Robust self-supervised multimodal image fusion method and device based on attention mechanism

Abstract

The present invention relates to the field of digital imaging technology, and in particular to a robust self-supervised multimodal image fusion method and device based on an attention mechanism. The method comprises: using a shooting device to collect images, obtaining visible light source images and infrared source images; performing focus fusion processing according to the visible light source images and the infrared source images to obtain a focus fusion map; constructing modules based on the attention mechanism and the Transformer network structure to obtain a joint learning module; based on the focus fusion map, inputting the visible light source image and the infrared source image into the joint learning module for image fusion to obtain a second fused image; inputting the second fused image into a preset semantic model for semantic addition to obtain a third fused image. The present invention is a self-supervised multimodal image fusion method with high efficiency and strong robustness based on an attention mechanism.

Description

Technical Field

[0001] The present invention relates to the field of digital imaging technology, and in particular to a robust self-supervised multimodal image fusion method and device based on an attention mechanism. Background Art

[0002] Images can reveal rich information under different shooting conditions and modes. In the medical field, multimodal magnetic resonance imaging (MRI) images are subdivided into four different categories: Flair, T1, T1ce, and T2, based on different imaging weighting techniques. Among them, the Flair technology uses a specific pre-pulse technology to suppress the signal of the surrounding fluid, thereby enhancing the signal in the tumor area; T1, T1ce, and T2 technologies focus on highlighting specific areas in brain tumor images, such as fat and edema areas. Through image fusion technology, researchers can combine information from multimodal images into a comprehensive image result. Multimodal image fusion tasks, including infrared and visible image fusion (IVIF) and medical image fusion (MIF), are key research directions in the field of multimodal image fusion.

[0003] In recent research on image fusion processing, strategies based on deep learning are gradually replacing traditional methods. Deep learning methods adaptively extract important features from the original image through the designed neural network model, simplifying the complexity of model adjustment. In order to achieve intuitive and effective fusion effects, a series of advanced structures, such as convolutional neural networks, Transformers, and encoder-decoder modules, have been introduced into the design of fusion networks. In addition, image fusion is often combined with subsequent analysis tasks such as detection and segmentation to comprehensively evaluate the effectiveness of the fused image.

[0004] Deep learning technology has become the mainstream method in image fusion research due to its high sensitivity to complex data feature extraction and powerful learning ability. This advanced technology enables image information obtained from different sources to be effectively integrated and optimized, thereby generating richer and more accurate fused images. In the medical field, the application of image fusion technology is particularly important, and it has become an important tool to assist doctors in diagnosing diseases. By fusing images from different imaging modes (such as CT, MRI, PET, etc.), medical image fusion can display various lesion information in a single image, which can not only provide doctors with a more comprehensive view of the disease, but also significantly improve the accuracy and efficiency of medical diagnosis. Researchers' continuous exploration and innovation of medical image fusion technology aims to further optimize the fusion algorithm and improve the quality of fused images in order to better meet the needs of clinical applications.

[0005] In the field of multimodal image fusion, IVIF and MIF are both core tasks, and researchers often develop a unified framework to handle these two tasks. For example, an innovative model based on multimodal image fusion is proposed, which is a conditional GAN ​​model with a dual discriminator mechanism, which is characterized by being able to process input images of any resolution. On the other hand, the "encoder-decoder" model that combines the feature extractors of Transformer and CNN aims to learn the features of the source images and apply appropriate strategies to generate fused images. Inter-modal registration plays a vital role in the image fusion process, which directly affects the quality of the fused image. Therefore, image registration is regarded as an important prerequisite task in the multimodal image fusion task. A mutually reinforcing network model is used to perform image registration and fusion simultaneously, which proves the effectiveness of this method.

[0006] There are still some unavoidable problems in the existing methods. In the feature extraction link, the fusion network learns the proposed context module in a cascaded manner to learn feature representations, which involves a transformation from one domain to another (e.g., from the spatial domain to the channel domain). However, the interdependence between different domains is rarely considered in this structure, which may lead to the loss of some key details and texture information in the fusion results. When training a general fusion framework, researchers often input source images from different fusion tasks into the network without distinction. In fact, there are significant differences between real-world scenes and medical scenes. The existing technology lacks effective strategies to enhance the adaptability of the network, which is not conducive to accurately capturing the key features in specific fusion tasks. Although there are specific datasets for semantic segmentation of advanced tasks (such as medical scenes), these datasets cannot be directly used for medical image fusion. Therefore, previous studies have failed to fully explore the downstream impact of image fusion in IVIF and MIF applications. This not only hinders the process of building a unified fusion framework, but also affects the performance of the corresponding downstream tasks.

[0007] In the existing technology, there is a lack of an efficient and robust self-supervised multimodal image fusion method based on the attention mechanism. Summary of the invention

[0008] In order to solve the technical problems of the loss of fused image details and insufficient robustness in the prior art, the embodiment of the present invention provides a robust self-supervised multimodal image fusion method and device based on an attention mechanism. The technical solution is as follows:

[0009] On the one hand, a robust self-supervised multimodal image fusion method based on an attention mechanism is provided, the method is implemented by a self-supervised multimodal image fusion device, and the method includes:

[0010] Use a shooting device to collect images and obtain visible light source images and infrared source images;

[0011] Performing focus fusion processing according to the visible light source image and the infrared source image to obtain a focus fusion image;

[0012] Modules are constructed based on the attention mechanism and Transformer network structure to obtain a joint learning module;

[0013] Based on the focus fusion map, inputting the visible light source image and the infrared source image into the joint learning module for image fusion to obtain a second fused image;

[0014] The second fused image is input into a preset semantic model for semantic addition to obtain a third fused image.

[0015] Optionally, performing focus fusion processing according to the visible light source image and the infrared source image to obtain a focus fusion image includes:

[0016] Performing pixel transformation on the visible light source image to obtain a visible light focus map;

[0017] Perform pixel changes on the infrared source image to obtain an infrared focus image;

[0018] The visible light focus image and the infrared focus image are fused to obtain a focus fusion image.

[0019] Wherein, the joint learning module includes a space domain learning module and a channel domain learning module;

[0020] The spatial domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Reshape layers and 1 softmax layer;

[0021] The channel domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Rotate layers and 1 softmax layer.

[0022] Optionally, based on the focus fusion map, inputting the visible light source image and the infrared source image into the joint learning module for image fusion to obtain a second fused image includes:

[0023] Based on a multi-domain attention mechanism, the visible light source image and the infrared source image are fused to obtain a first fused image;

[0024] Based on the focus fusion image, a self-supervised strategy is used to generate an image according to the first fused image to obtain a second fused image.

[0025] Optionally, fusing the visible light source image and the infrared source image to obtain a first fused image includes:

[0026] Performing feature extraction on the visible light source image to obtain a visible light feature map;

[0027] Extracting features from the infrared source image to obtain an infrared feature map;

[0028] Fusing the visible light feature map and the infrared feature map to obtain a fused feature map;

[0029] Perform multi-domain feature calculation according to the fused feature map to obtain spatial feature representation and channel feature representation;

[0030] Connecting the spatial feature representation and the channel feature representation to obtain a fused feature representation;

[0031] An image is generated according to the joint feature representation to obtain a first fused image.

[0032] Optionally, performing multi-domain feature calculation according to the fused feature map to obtain a spatial feature representation and a channel feature representation includes:

[0033] Perform spatial domain calculation according to the fused feature map to obtain spatial feature representation;

[0034] Channel domain calculation is performed according to the spatial feature representation to obtain a channel feature representation.

[0035] On the other hand, a robust self-supervised multimodal image fusion device based on an attention mechanism is provided, and the device is applied to a robust self-supervised multimodal image fusion method based on an attention mechanism, and the device includes:

[0036] A source image acquisition module is used to acquire images using a shooting device to obtain visible light source images and infrared source images;

[0037] A focus fusion image acquisition module, used for performing focus fusion processing according to the visible light source image and the infrared source image to obtain a focus fusion image;

[0038] The module construction module is used to construct modules based on the attention mechanism and the Transformer network structure to obtain the joint learning module;

[0039] A second fused image acquisition module is used to input the visible light source image and the infrared source image into the joint learning module for image fusion based on the focus fusion map to obtain a second fused image;

[0040] The third fused image acquisition module is used to input the second fused image into a preset semantic model for semantic addition to obtain a third fused image.

[0041] Optionally, the focus fusion image acquisition module is further used to:

[0042] Performing pixel transformation on the visible light source image to obtain a visible light focus map;

[0043] Perform pixel changes on the infrared source image to obtain an infrared focus image;

[0044] The visible light focus image and the infrared focus image are fused to obtain a focus fusion image.

[0045] Wherein, the joint learning module includes a space domain learning module and a channel domain learning module;

[0046] The spatial domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Reshape layers and 1 softmax layer;

[0047] The channel domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Rotate layers and 1 softmax layer.

[0048] Optionally, the second fused image acquisition module is further used to:

[0049] Based on a multi-domain attention mechanism, the visible light source image and the infrared source image are fused to obtain a first fused image;

[0050] Based on the focus fusion image, a self-supervised strategy is used to generate an image according to the first fused image to obtain a second fused image.

[0051] Optionally, the second fused image acquisition module is further used to:

[0052] Performing feature extraction on the visible light source image to obtain a visible light feature map;

[0053] Extracting features from the infrared source image to obtain an infrared feature map;

[0054] Fusing the visible light feature map and the infrared feature map to obtain a fused feature map;

[0055] Perform multi-domain feature calculation according to the fused feature map to obtain spatial feature representation and channel feature representation;

[0056] Connecting the spatial feature representation and the channel feature representation to obtain a fused feature representation;

[0057] An image is generated according to the joint feature representation to obtain a first fused image.

[0058] Optionally, the second fused image acquisition module is further used to:

[0059] Perform spatial domain calculation according to the fused feature map to obtain spatial feature representation;

[0060] Channel domain calculation is performed according to the spatial feature representation to obtain a channel feature representation.

[0061] On the other hand, a self-supervised multimodal image fusion device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned robust self-supervised multimodal image fusion methods based on the attention mechanism is implemented.

[0062] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned robust self-supervised multimodal image fusion methods based on the attention mechanism.

[0063] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0064] The present invention proposes a robust self-supervised multimodal image fusion method based on an attention mechanism, which adopts a multi-scale joint framework of a self-supervised mechanism. The framework is specifically aimed at general multimodal image fusion tasks and can perform image fusion more effectively. In the fusion process, an attention mechanism is introduced, which can capture self-attention features in parallel in two dimensions, space and channel, to ensure the accuracy of the fused image in visual presentation. By using original images of different modalities to create a visual focus map as a pseudo-label, the fusion network is self-supervised. This strategy effectively maintains the key details in the fused image and avoids interference from other information extraction processes. The present invention is a self-supervised multimodal image fusion method with high efficiency and strong robustness based on an attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0066] Figure 1It is a flow chart of a robust self-supervised multimodal image fusion method based on an attention mechanism provided by an embodiment of the present invention;

[0067] Figure 2 is a structural diagram of a joint learning module provided by an embodiment of the present invention;

[0068] Figure 3 It is a block diagram of a robust self-supervised multimodal image fusion device based on an attention mechanism provided by an embodiment of the present invention;

[0069] Figure 4 It is a structural schematic diagram of a self-supervised multimodal image fusion device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0071] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0072] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0073] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0074] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0075] The embodiment of the present invention provides a robust self-supervised multimodal image fusion method based on an attention mechanism, which can be implemented by a self-supervised multimodal image fusion device, which can be a terminal or a server. Figure 1 The flowchart of the robust self-supervised multimodal image fusion method based on the attention mechanism is shown in the figure. The processing flow of the method may include the following steps:

[0076] S1. Use a shooting device to collect images and obtain visible light source images and infrared source images.

[0077] In a feasible implementation, the self-supervised multimodal image fusion architecture proposed in the present invention can be adjusted in a targeted manner according to different types of tasks to obtain a robust fusion result with good visual effects. Based on the architecture proposed in the present invention, infrared image and visible light image fusion tasks can be realized, and multimodal medical image fusion tasks can also be well completed.

[0078] S2. Perform focus fusion processing according to the visible light source image and the infrared source image to obtain a focus fusion image.

[0079] Optionally, performing focus fusion processing according to the visible light source image and the infrared source image to obtain a focus fusion image includes:

[0080] Perform pixel transformation on the visible light source image to obtain a visible light focus map;

[0081] Perform pixel changes on the infrared source image to obtain an infrared focus image;

[0082] The visible light focus image and the infrared focus image are fused to obtain a focus fusion image.

[0083] In a feasible implementation, in order to better preserve the obvious texture features in the final fused image, the present invention considers incorporating a self-supervised learning strategy into the multimodal image fusion process. Selecting an appropriate self-supervised target is the key.

[0084] The present invention extracts the visual focus information from the source image and uses this information as an intermediate step in the fusion process. Such an intermediate result helps to highlight the image details more finely, and the visual focus map only represents the eye-catching elements in the image. However, the fusion output based on visual focus may be different from human visual perception. Therefore, these focus maps are only used as a means to capture key information.

[0085] The visible light source image input is , map it to grayscale for pixel transformation. The pixel intensity is Pixel point k, visual focus value The calculation formula is shown in the following formula (1):

[0086] (1);

[0087] in, express Any pixel in The pixel standard intensity of the source image pixel.

[0088] The visual focus value of each pixel is applied as a weight to the corresponding source image to generate the visual focus map. The process is shown in the following formula (2):

[0089] (2);

[0090] in, is the visual focus value matrix.

[0091] According to the above steps, when the infrared source image is used as the input image, the focus map of the infrared source image can be obtained. The visible light focus map is fused with the infrared background image to generate a focus fusion map. The generated focus fusion map is mapped back to the RGB channel.

[0092] S3. Module construction is performed based on the attention mechanism and Transformer network structure to obtain a joint learning module.

[0093] Among them, the joint learning module includes a space domain learning module and a channel domain learning module;

[0094] The spatial domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Reshape layers and 1 softmax layer;

[0095] The channel domain learning module consists of 4 Conv layers, 3 Dconv layers, 3 Rotate layers and 1 softmax layer.

[0096] In a feasible implementation, the specific architecture of the joint learning module is as follows: Figure 2 As shown in the figure. The joint learning module is a transformer structure, which extracts and learns the spatial domain and channel domain attention at the same time. When learning and extracting the spatial domain attention, it includes 4 convolutional layers (Conv), 3 deep convolutional layers (Dconv), 3 reshape layers (Reshape) and 1 activation layer (softmax). When learning and extracting the channel attention, it includes 4 convolutional layers (Conv), 3 deep convolutional layers (Dconv), 3 rotation layers (Rotate) and an activation layer (softmax).

[0097] S4. Based on the focus fusion image, the visible light source image and the infrared source image are input into the joint learning module for image fusion to obtain a second fused image.

[0098] Optionally, based on the focus fusion map, the visible light source image and the infrared source image are input into a joint learning module for image fusion to obtain a second fused image, including:

[0099] Based on the multi-domain attention mechanism, the visible light source image and the infrared source image are fused to obtain a first fused image;

[0100] Based on the focus fusion map, a self-supervised strategy is used to generate an image according to the first fused image to obtain a second fused image.

[0101] In one possible implementation, the self-attention mechanism only works within a single domain, sometimes ignoring feature representations from other domains. The limitation of using multiple attention modules in series is that it is difficult to fully capture the learning features across multiple domains at the same time, resulting in undesirable halo effects and artifacts in the fused image.

[0102] The joint learning module proposed in the present invention adopts a parallel multi-domain attention mechanism. The module can extract valuable features from the spatial domain while taking into account the characteristics of the channel domain.

[0103] Using the visual focus fusion map as a medium, the real fusion result is self-supervised. This self-supervised strategy can adaptively learn different salient features according to the input of different source images, thereby achieving the effect of highlighting the salient information in the fused image.

[0104] When using the visual focus fusion map to generate the fusion result of self-supervised learning, the L1-norm norm is used as the loss function in the self-supervised process for calculation, so that different significant information in different tasks is highlighted, the visual effect of the fused image is enhanced, and it is easier to observe and understand.

[0105] Optionally, fusing the visible light source image and the infrared source image to obtain a first fused image includes:

[0106] Extract features from the visible light source image to obtain a visible light feature map;

[0107] Extract features from the infrared source image to obtain an infrared feature map;

[0108] The visible light feature map and the infrared feature map are fused to obtain a fused feature map;

[0109] Multi-domain feature calculation is performed based on the fused feature map to obtain spatial feature representation and channel feature representation;

[0110] Connect the spatial feature representation and the channel feature representation to obtain a fused feature representation;

[0111] An image is generated according to the joint feature representation to obtain a first fused image.

[0112] In a feasible implementation, a convolutional neural network or other types of neural networks are used in the present invention to extract features from visible light source images and infrared source images. Useful feature representations can be extracted through the neural network.

[0113] The extracted visible light feature map and infrared feature map are fused to generate a richer fused feature map. Based on the self-attention mechanism of the Transformer model network structure, the relationship between the visible light feature map and the infrared feature map is captured and integrated into the generated fused feature map.

[0114] According to the fusion feature map obtained in the above steps, the feature representations of the spatial domain and the channel domain of the source image are extracted in parallel. The feature representations of the spatial domain and the channel domain are connected to obtain a fusion feature representation. According to the fusion feature representation, the first fusion image is obtained by decoding through a preset image generation network.

[0115] Optionally, multi-domain feature calculation is performed according to the fused feature map to obtain a spatial feature representation and a channel feature representation, including:

[0116] Perform spatial domain calculation based on the fused feature map to obtain spatial feature representation;

[0117] Channel domain calculation is performed according to the spatial feature representation to obtain the channel feature representation.

[0118] In a feasible implementation, based on the joint learning module proposed in the present invention, the input fusion feature F is sent to the normalization layer to obtain sub-features with similar distribution .

[0119] in, , Represent the height, width and number of channels of these features respectively. Linear projection is performed through convolutional layer (Conv) and deep convolutional layer (Dconv) to embed queues, key values ​​and real values, that is, , the process can be quantified as follows (3):

[0120] (3);

[0121] in, yes Linear projection of .

[0122] In order to realize the self-attention operation in the spatial domain, Reshape .pass The spatial self-attention map can be calculated , the calculation process is as follows (4):

[0123] (4);

[0124] in, generated in the spatial domain; is a learnable parameter in the spatial domain, used to balance The value of .

[0125] Will and Multiply them together to get the spatial feature representation , the calculation process is as follows (5):

[0126] (5);

[0127] In the above steps, the joint learning module adopts a window strategy of size 8 to promote global learning.

[0128] Will As the input of channel self-attention. Different from the spatial domain, it needs to be rotated to the channel dimension before matrix embedding. The calculation process is as follows (6):

[0129] (6);

[0130] in, Represents a rotation operation; is the rotated queue, key value and real value. Similar to the spatial domain, Generating channel feature representation The calculation process is as follows (7):

[0131] (7);

[0132] in, and The role of is similar to that of , which is a learnable parameter in the channel domain.

[0133] To reduce subsequent complexity, the model also Rotate to obtain fusion feature representation The parallel interaction structure proposed in the present invention can fully express information and avoid the limitation of a single domain.

[0134] S5. Input the second fused image into a preset semantic model for semantic addition to obtain a third fused image.

[0135] In a feasible implementation, the present invention adds semantics based on the second fused image through a preset semantic pre-training model of a convolutional neural network. Using a semantic training model to add semantics to an image is widely used in technical fields such as knowledge retrieval, intelligent driving, human-computer interaction, and medical imaging, which is a prior art and will not be described in detail here.

[0136] In a feasible implementation manner, the present invention generates images with good visual effects in the task of infrared and visible light fusion images.

[0137] Based on a robust self-supervised multimodal image fusion method based on attention mechanism proposed in the present invention, two visual comparison examples from a public dataset for image fusion tasks (Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection, M3FD) and a benchmark dataset for image fusion and segmentation research (Full-time Multi-modality Benchmark Dataset, FMB) were used, and two specific areas were enlarged for observation.

[0138] In the M3FD example, a dual-discriminator conditional generative adversarial network (DDcGAN) for multi-resolution image fusion and a target detection method (target-aware dual adversarial learning, TarDAL) for infrared and visible light image fusion tend to retain too much thermal imaging target information, which does not fully conform to the actual visual experience of the human eye.

[0139] In the expression of the outlines and details of distant buildings, a deep learning network (Coconut Network, CoCoNet) for multimodal image fusion appears to lack contrast, resulting in the loss of some key textures. In the FMB example, a deep learning architecture for infrared and visible image fusion (DenseFuse: A Fusion Approach to Infrared and Visible Images, DenseFuse) presents slightly blurred details, which will lead to the effect of the generated image in subsequent applications. The results of a deep learning method for image analysis (Inception-ResNet Network, IGNet) tend to overemphasize infrared modes, such as dark skies captured in daytime scenes. In contrast, the method proposed in the present invention can avoid the above-mentioned problems and can make adaptive adjustments according to the actual shooting environment. For example, in the FMB example, the processing of walls and cars in the example is closer to the visual perception of the human eye, avoiding the problem of excessive bias towards any single imaging mode.

[0140] The present invention generates images with good visual effects in multimodal medical image fusion tasks. In a study using a multimodal medical image dataset (Multi-grained Multi-curriculum Denoising Framework, M2DF), three different medical image fusion effects were generated by fusing flair images with t1, t1ce and t2 images. Affected by technical limitations, a conditional generative adversarial network (DualDiscriminator Conditional Generative Adversarial Network, DDcGAN) and CoCoNet for multi-resolution image fusion produced undesirable artifacts in the background, which had a negative impact on the visual effects. A deep learning method for medical image fusion (Multi-scale Adaptive Transformer, MATR) overemphasized the highlight effect in the generated image results, so that almost no t1 / t1ce / t2 information could be seen in the fused image. In the fusion results of a method for multimodal image fusion (Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion, CDDFuse), edge features appear too prominent, which does not fully meet the needs of actual medical imaging. A general image fusion framework based on cross-domain long-range learning and Swin Transformer (Cross-domain Long-range Learning for General Image Fusion via Swin Transformer, Swin Fusion) and a multimodal image registration and fusion method (Mutually Reinforcing Multi-Modal Image Registration and Fusion, MURF) do not appear to be prominent enough in the representation of details and textures (such as the edema area around the tumor). By introducing self-supervised learning, the method proposed in the present invention can effectively circumvent the above problems, and the fusion technology of the present invention can effectively retain the key information from the original input.

[0141] The present invention performs well in downstream tasks. In order to further evaluate the performance of the fusion effect, object detection tasks are performed based on the M3FD dataset. Two examples of daytime and nighttime scenes are included. In nighttime environments, methods such as DDcGAN, SwinFusion, CoCoNet, and a target detection method for infrared and visible light image fusion (Target-aware Dual Adversarial Learning, TarDAL) have the problem of failing to detect some objects, such as trucks and pedestrians, which may be due to the lack of sufficient semantic content in their fused images. CDDFuse, MURF, and DenseFuse not only miss some targets, but also have false detections, which reflects the inadequate performance of the fused image. The performance of object detection based on the method of the present invention is still the best. In daytime scenes, people in the distance are detected. Except for TarDAL and IGNet, which failed to identify all targets, the method proposed in the present invention shows that each target can be detected with a higher accuracy, which reflects that the fused image in the method contains richer high-level features. In addition, compared with the detection using infrared or visible light images alone, the technical effect of the present invention shows better performance. This proves that the method proposed in the present invention can achieve excellent results in both image fusion and downstream tasks.

[0142] The present invention proposes a robust self-supervised multimodal image fusion method based on an attention mechanism, which adopts a multi-scale joint framework of a self-supervised mechanism. The framework is specifically aimed at general multimodal image fusion tasks and can perform image fusion more effectively. In the fusion process, an attention mechanism is introduced, which can capture self-attention features in parallel in two dimensions, space and channel, to ensure the accuracy of the fused image in visual presentation. By using original images of different modalities to create a visual focus map as a pseudo-label, the fusion network is self-supervised. This strategy effectively maintains the key details in the fused image and avoids interference from other information extraction processes. The present invention is a self-supervised multimodal image fusion method with high efficiency and strong robustness based on an attention mechanism.

[0143] Figure 3 1 is a block diagram of a robust self-supervised multimodal image fusion device based on an attention mechanism according to an exemplary embodiment. The device is used in a robust self-supervised multimodal image fusion method based on an attention mechanism. Figure 3 The device includes a source image acquisition module 310, a focus fusion map acquisition module 320, a module construction module 330, and a training module 340. Among them:

[0144] The source image acquisition module 310 is used to acquire images using a shooting device to obtain visible light source images and infrared source images;

[0145] A focus fusion image acquisition module 320 is used to perform focus fusion processing according to the visible light source image and the infrared source image to obtain a focus fusion image;

[0146] A module construction module 330 is used to construct a module based on an attention mechanism and a Transformer network structure to obtain a joint learning module;

[0147] A second fused image acquisition module 340 is used to input the visible light source image and the infrared source image into the joint learning module for image fusion based on the focus fusion map to obtain a second fused image;

[0148] The third fused image acquisition module 350 is used to input the second fused image into a preset semantic model to perform semantic addition to obtain a third fused image.

[0149] Optionally, the focus fusion image acquisition module 320 is further used to:

[0150] Perform pixel transformation on the visible light source image to obtain a visible light focus map;

[0151] Perform pixel changes on the infrared source image to obtain an infrared focus image;

[0152] The visible light focus image and the infrared focus image are fused to obtain a focus fusion image.

[0153] Among them, the joint learning module includes a space domain learning module and a channel domain learning module;

[0154] The spatial domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Reshape layers and 1 softmax layer;

[0155] The channel domain learning module consists of 4 Conv layers, 3 Dconv layers, 3 Rotate layers and 1 softmax layer.

[0156] Optionally, the second fused image acquisition module 340 is further used to:

[0157] Based on the multi-domain attention mechanism, the visible light source image and the infrared source image are fused to obtain a first fused image;

[0158] Based on the focus fusion map, a self-supervised strategy is used to generate an image according to the first fused image to obtain a second fused image.

[0159] Optionally, the second fused image acquisition module 340 is further used to:

[0160] Extract features from the visible light source image to obtain a visible light feature map;

[0161] Extract features from the infrared source image to obtain an infrared feature map;

[0162] The visible light feature map and the infrared feature map are fused to obtain a fused feature map;

[0163] Multi-domain feature calculation is performed based on the fused feature map to obtain spatial feature representation and channel feature representation;

[0164] Connect the spatial feature representation and the channel feature representation to obtain a fused feature representation;

[0165] An image is generated according to the joint feature representation to obtain a first fused image.

[0166] Optionally, the second fused image acquisition module 340 is further used to:

[0167] Perform spatial domain calculation based on the fused feature map to obtain spatial feature representation;

[0168] Channel domain calculation is performed according to the spatial feature representation to obtain the channel feature representation.

[0169] The present invention proposes a robust self-supervised multimodal image fusion method based on an attention mechanism, which adopts a multi-scale joint framework of a self-supervised mechanism. The framework is specifically aimed at general multimodal image fusion tasks and can perform image fusion more effectively. In the fusion process, an attention mechanism is introduced, which can capture self-attention features in parallel in two dimensions, space and channel, to ensure the accuracy of the fused image in visual presentation. By using original images of different modalities to create a visual focus map as a pseudo-label, the fusion network is self-supervised. This strategy effectively maintains the key details in the fused image and avoids interference from other information extraction processes. The present invention is a self-supervised multimodal image fusion method with high efficiency and strong robustness based on an attention mechanism.

[0170] Figure 4 is a schematic diagram of the structure of a self-supervised multimodal image fusion device provided by an embodiment of the present invention, such as Figure 4 As shown, the self-supervised multimodal image fusion device may include the above Figure 3 The robust self-supervised multimodal image fusion device based on the attention mechanism is shown. Optionally, the self-supervised multimodal image fusion device 410 may include a first processor 2001.

[0171] Optionally, the self-supervised multimodal image fusion device 410 may further include a memory 2002 and a transceiver 2003 .

[0172] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0173] Combine the following Figure 4 The components of the self-supervised multimodal image fusion device 410 are specifically introduced:

[0174] The first processor 2001 is the control center of the self-supervised multimodal image fusion device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0175] Optionally, the first processor 2001 may perform various functions of the self-supervised multimodal image fusion device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0176] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0177] In a specific implementation, as an embodiment, the self-supervised multimodal image fusion device 410 may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0178] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0179] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0180] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0181] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0182] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and communicate with the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0183] It should be noted that Figure 4 The structure of the self-supervised multimodal image fusion device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0184] In addition, the technical effects of the self-supervised multimodal image fusion device 410 can refer to the technical effects of the robust self-supervised multimodal image fusion method based on the attention mechanism described in the above method embodiment, and will not be repeated here.

[0185] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0186] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0187] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0188] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0189] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0190] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0191] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0192] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0193] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0196] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0197] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A robust self-supervised multimodal image fusion method based on attention mechanism, characterized in that: The method comprises: Use a shooting device to collect images and obtain visible light source images and infrared source images; Performing focus fusion processing according to the visible light source image and the infrared source image to obtain a focus fusion image; The step of performing focus fusion processing on the visible light source image and the infrared source image to obtain a focus fusion image includes: Performing pixel transformation on the visible light source image to obtain a visible light focus map; Perform pixel changes on the infrared source image to obtain an infrared focus image; Performing image fusion on the visible light focus image and the infrared focus image to obtain a focus fusion image; Modules are constructed based on the attention mechanism and Transformer network structure to obtain a joint learning module; Wherein, the joint learning module includes a space domain learning module and a channel domain learning module; The spatial domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Reshape layers and 1 softmax layer; The channel domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Rotate layers and 1 softmax layer; Based on the focus fusion map, inputting the visible light source image and the infrared source image into the joint learning module for image fusion to obtain a second fused image; Wherein, based on the focus fusion map, inputting the visible light source image and the infrared source image into the joint learning module for image fusion to obtain a second fused image includes: Based on a multi-domain attention mechanism, the visible light source image and the infrared source image are fused to obtain a first fused image; Based on the focus fusion image, a self-supervised strategy is used to generate an image according to the first fused image to obtain a second fused image; The step of fusing the visible light source image and the infrared source image to obtain a first fused image includes: Performing feature extraction on the visible light source image to obtain a visible light feature map; Extracting features from the infrared source image to obtain an infrared feature map; Fusing the visible light feature map and the infrared feature map to obtain a fused feature map; Perform multi-domain feature calculation according to the fused feature map to obtain spatial feature representation and channel feature representation; Connecting the spatial feature representation and the channel feature representation to obtain a fused feature representation; Generate an image according to the fusion feature representation to obtain a first fusion image; The multi-domain feature calculation is performed according to the fused feature map to obtain a spatial feature representation and a channel feature representation, including: Perform spatial domain calculation according to the fused feature map to obtain spatial feature representation; Perform channel domain calculation according to the spatial feature representation to obtain a channel feature representation; The second fused image is input into a preset semantic model for semantic addition to obtain a third fused image.

2. A robust self-supervised multimodal image fusion device based on an attention mechanism, wherein the robust self-supervised multimodal image fusion device based on an attention mechanism is used to implement the robust self-supervised multimodal image fusion method based on an attention mechanism as claimed in claim 1, characterized in that: The device comprises: A source image acquisition module is used to acquire images using a shooting device to obtain visible light source images and infrared source images; A focus fusion image acquisition module, used for performing focus fusion processing according to the visible light source image and the infrared source image to obtain a focus fusion image; Wherein, the focus fusion image acquisition module is further used for: Performing pixel transformation on the visible light source image to obtain a visible light focus map; Perform pixel changes on the infrared source image to obtain an infrared focus image; Performing image fusion on the visible light focus image and the infrared focus image to obtain a focus fusion image; The module construction module is used to construct modules based on the attention mechanism and the Transformer network structure to obtain the joint learning module; Wherein, the joint learning module includes a space domain learning module and a channel domain learning module; The spatial domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Reshape layers and 1 softmax layer; The channel domain learning module includes 4 Conv layers, 3 Dconv layers, 3 Rotate layers and 1 softmax layer; A second fused image acquisition module is used to input the visible light source image and the infrared source image into the joint learning module for image fusion based on the focus fusion map to obtain a second fused image; Wherein, the second fused image acquisition module is further used for: Based on a multi-domain attention mechanism, the visible light source image and the infrared source image are fused to obtain a first fused image; Based on the focus fusion image, a self-supervised strategy is used to generate an image according to the first fused image to obtain a second fused image; Wherein, the second fused image acquisition module is further used for: Performing feature extraction on the visible light source image to obtain a visible light feature map; Extracting features from the infrared source image to obtain an infrared feature map; Fusing the visible light feature map and the infrared feature map to obtain a fused feature map; Perform multi-domain feature calculation according to the fused feature map to obtain spatial feature representation and channel feature representation; Connecting the spatial feature representation and the channel feature representation to obtain a fused feature representation; Generate an image according to the fusion feature representation to obtain a first fusion image; Wherein, the second fused image acquisition module is further used for: Perform spatial domain calculation according to the fused feature map to obtain spatial feature representation; Perform channel domain calculation according to the spatial feature representation to obtain a channel feature representation; The third fused image acquisition module is used to input the second fused image into a preset semantic model for semantic addition to obtain a third fused image.

3. A self-supervised multimodal image fusion device, characterized in that: The self-supervised multimodal image fusion device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to claim 1 is implemented.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to claim 1.

Images