Multi-domain image fusion method for learning specific domain and general domain feature representation

Abstract

The invention provides a multi-domain image fusion method for learning specific domain and general domain feature representation, which belongs to the technical field of image information processing and comprises three steps of a sharing principle network, an adaptive domain feature extraction strategy and a domain activation mechanism. The method comprises the steps of 1, sharing a principle network; and designing a shared network for cross-domain universal feature extraction based on a deep learning method. And 2, an adaptive domain feature extraction strategy. And feature extraction of different fields is carried out through a field-specific sub-network. And step 3, a domain activation mechanism. And by learning a weight distribution mechanism, contribution degrees of different features are dynamically adjusted, and the importance of specific features in the field is further enhanced, so that the fused image better meets the appearance requirements of the specific field. The generalization ability of the cross-domain application and the sensitivity of the specific domain application are improved by learning the feature representation of the specific domain and the universal domain.

Description

Technical Field

[0001] The present invention relates to the technical field of image information processing, and in particular to a method for multi-focus image fusion, specifically a multi-domain image fusion method that learns feature representations of specific domains and general domains. Background Art

[0002] The cameras in the photographic system are different from simple pinhole devices. They are equipped with a convex lens system that can focus on a specific distance at the moment of shooting, so they have a certain depth of field limitation. V. Aslantas et al. mentioned in the study "Depth from automatic defocusing" that in the pictures taken by the actual lens, only objects within a certain distance can appear clear (focused), while the rest of the objects at different distances will appear blurred (out of focus). Compared with blurred images, clear images can provide more useful information for human eyes and computer analysis. Therefore, in computer vision applications such as image analysis, target detection, recognition and monitoring, a comprehensive and clear image is crucial. To achieve this goal, the commonly used technology is multi-focus image fusion (Multi-Focus Image Fusion), which integrates multiple images taken under different focus settings in the same scene.

[0003] Traditional multi-focus image fusion mainly includes methods based on manually extracted features, spatial domain and transform domain. Bai et al. proposed a weighted kernel method based on image gradient in "Multi-focus image fusion through gradient-based decision map construction and mathematical morphology", which mainly focuses on how to construct a gradient-based decision map and perform image fusion through mathematical morphology; Z. Liu et al. proposed a multi-focus image fusion method based on image decomposition in "Anovel multi-focus image fusion approach based on image decomposition", and M. Abdipour et al. proposed a multi-focus image fusion technology designed for visual sensor networks using sharpness criteria in wavelet domain in "Multi-focus image fusion using sharpness criteria for visual sensor networks in wavelet domain". These methods focus on improving the image fusion effect by separating the cartoon content and texture content of the image and wavelet transform. In the implementation process of most current image fusion technologies, key elements such as feature extraction, focus area definition and fusion criteria need to be manually planned and set. This manual design process itself is considered a very challenging task, which not only requires a high degree of professional knowledge and experience accumulation, but also faces many uncertainties in practice.

[0004] Recently, deep neural networks (DNNs), especially deep CNNs, have been used in various computer vision tasks, including image classification, object detection, image denoising, and semantic segmentation. Some preliminary attempts to apply DNNs to image fusion have been made. Notably, these DNN-based methods are used to process long-range images, aiming to integrate the complementary spectral and spatial features of multispectral (MS) and panchromatic (PAN) images into one product. Huang et al. introduced an innovative improved version of sparse denoising autoencoder, namely Modified Sparse Denoising Autoencoder (MSDA), specifically for enhancing the fusion effect of PAN and MS images in their work "A New Pan-Sharpening Method with Deep Neural Networks". Liu et al. proposed a method for multi-focus image fusion in their work "Multi-focus image fusion with a deep convolutional neural network", in which the constructed network architecture essentially follows the design idea of ​​classification tasks. Specifically, by assigning a single binary label (i.e., 0 or 1) to the training image patch pairs to indicate whether they are focused or not, such a design framework may lead to inaccurate problems in distinguishing the boundaries between focused and unfocused areas.

[0005] From the application perspective, a general fusion framework should have both universality across multiple domains and sensitivity to single-domain applications. Several attempts have been made to address the problem of universality. For example, Li et al. developed an image fusion method based on spatial filter decomposition in their work A spatially constrained Gaussian mixture model for unsupervised image fusion, which fully exploits spatial consistency. Zhang et al. proposed a unified framework in their work A Unified Framework for Multi-Modality Medical Image Fusion via Learning Application-Adaptive Importance Measure to select important regions by learning application importance measures adapted to each domain. Although these methods perform well in some specific domains, they are based on hand-crafted representation methods, resulting in insufficient robustness in complex scenarios. Recently, a general image fusion framework (IFCNN) based on convolutional neural network (CNN) was also proposed in IFCNN: A General Image Fusion Framework Based on Convolutional Neural Network, which takes into account that images from different domains can share common features (such as edge details). However, these CNN-based frameworks lack domain-specific feature representations and ignore the intrinsic connections between applications in different domains, ultimately resulting in limited performance. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a multi-domain image fusion method for learning feature representations of specific domains and general domains. By learning feature representations of specific domains and general domains, the method aims to solve the problems of universality in multiple domains and sensitivity in specific domains, thereby achieving high generalization ability in cross-domain applications and showing high sensitivity in specific domain applications.

[0007] The technical solution of the present invention is:

[0008] A multi-domain image fusion method that learns domain-specific and domain-general feature representations consists of three steps:

[0009] Step 1: Shared principle network, a shared network for cross-domain general feature extraction is designed based on deep learning methods.

[0010] Step 2: Adaptive domain feature extraction, feature extraction of different domains through domain-specific sub-networks.

[0011] Step 3: Domain activation, through learning the weight distribution mechanism, dynamically adjust the contribution of different features, further enhance the importance of domain-specific features, and make the fused image more in line with the appearance requirements of the specific domain.

[0012] Furthermore,

[0013] A universal convolutional neural network is used to extract common features of images in different fields.

[0014] Based on the shared network, domain-specific sub-networks are added, which perform feature extraction according to the characteristics of specific fields.

[0015] A domain activation mechanism is introduced to dynamically adjust the weights of features during the fusion process, so that the final fused image is more in line with the appearance requirements of a specific domain.

[0016] The final stage optimizes the network parameters using a no-reference perceptual metric loss function based on edge details and contrast.

[0017] Going further,

[0018] The no-reference perceptual metric loss function evaluates the fusion effect by calculating the edge details and contrast of the fused image.

[0019] Edge detail measurement reflects the edge information of an image by calculating the gradient of the image.

[0020] Contrast measurement reflects the contrast information of an image by calculating the grayscale changes of the image.

[0021] Going further,

[0022] Edge detail measurement: Calculates the edge information of an image.

[0023] Contrast measurement: Calculates the contrast information of an image.

[0024] Loss function: Design a no-reference perceptual metric loss function based on edge details and contrast.

[0025] The beneficial effects of the present invention are

[0026] The experimental results show that the fusion image obtained by the proposed method outperforms the existing state-of-the-art methods on the MRIF and other three datasets. On the MRIF dataset, compared with SN: Structure Preservation Network, PSN: Patch Similarity Network, and MLCNN: Multi-Level Deeply Supervised Convolutional Neural Network, it can be found that the proposed method has better performance, as shown in the following table

[0027] Metric SN PSN MLCNN This method AG 6.1791 5.362 8.3058 8.5000 LIF 0.4903 0.4861 0.4765 0.4700 CEIF 0.3669 0.3704 0.3903 0.4000 MSD 0.1026 0.1018 0.1071 0.105 GLD 11.8573 12.6081 14.4316 15.00 BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the workflow of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0030] Due to the limitation of the number of focused images, common non-focused areas often appear in multi-focus images. This factor seriously reduces the fusion quality of multi-focus images. Low-level feature extraction can capture low-frequency content, while high-level feature extraction can effectively capture high-frequency details, further combining multi-level outputs so that the most visually distinct features can be extracted, fused and enhanced.

[0031] Image fusion using deep learning can further enhance image quality, especially improving the performance of common non-focus areas, showing strong technical advantages and application potential. With the continuous advancement of deep learning, end-to-end fusion methods are expected to become a standard tool for multi-focus image processing and drive the industry forward.

[0032] The technical solution of the present invention includes three main parts: a sharing principle network, an adaptive domain feature extraction strategy and a domain activation mechanism. The sharing principle network is used for universal feature extraction across domains to ensure that the framework has good generalization ability. The adaptive domain feature extraction strategy allows the framework to adjust the feature extraction method according to the needs of specific domains, thereby improving the sensitivity of specific domain applications. The domain activation mechanism further enhances the importance of domain-specific features, making the fused image more in line with the appearance requirements of specific domains. In addition, we propose a reference-free perceptual metric loss function based on edge details and contrast to optimize the learning process so that the fused image can show a more specific appearance.

[0033] In order to achieve cross-domain universal feature extraction and ensure that the framework has good generalization ability, a shared principle network is proposed. In the multi-domain image fusion task, although images from different fields are different in specific applications, they are similar in basic visual features. By sharing a common feature extraction network, the complexity of the model can be reduced, the generalization ability of the model can be improved, and the duplication of work and resource waste caused by designing feature extraction networks separately in each field can be avoided. Strong generalization ability: The shared network can learn common feature representations from a large number of images in different fields, improving the performance of the model in unseen fields. High resource utilization efficiency: By sharing the network, the number of model parameters is reduced, and the computational cost of training and inference is reduced. Strong scalability: When adding a new field, only the shared network needs to be fine-tuned or expanded, without redesigning the entire model. The core idea of ​​the shared principle network is to extract common features of images in different fields through a common convolutional neural network. Through multi-layer convolution operations, the convolutional neural network can gradually abstract low-level features (such as edges, textures) and high-level features (such as shapes, objects) of the image. These features have certain commonalities in different fields and can therefore be effectively extracted and utilized by the shared network. As shown in formula (1):

[0034]

[0035] Where In is the input image of different fields, Wi is the weight of the i-th layer network, and Fn (L) is the output of the Lth layer network.

[0036] In order to adjust the feature extraction method according to the needs of specific fields and improve the sensitivity of specific field applications, an adaptive domain feature extraction strategy is proposed. Specifically, on the basis of the shared network, domain-specific sub-networks are added, and these sub-networks extract features according to the characteristics of specific fields.

[0037] Although images from different fields have commonalities in basic visual features, each field also has its own unique features and application requirements. For example, medical images and infrared images have different requirements for edge details and contrast. By introducing domain-specific sub-networks, these unique features can be better captured and utilized, improving the fusion effect of specific fields. The adaptive domain feature extraction strategy captures the unique features of images from different fields by adding domain-specific sub-networks on the basis of a shared network. These sub-networks can be customized according to the characteristics of specific fields to better meet the application requirements of specific fields. Domain-specific sub-networks usually contain several layers of convolutional layers and activation functions to extract and enhance features in specific fields. The processing process is shown in formula (2).

[0038]

[0039] Where W specific Represents the weight matrix of the domain-specific convolution kernel. ReLu represents the activation function.

[0040] In order to further enhance the importance of domain-specific features and make the fused image more in line with the appearance requirements of a specific domain, a realm activation mechanism is proposed. In multi-domain image fusion, different features have different importance in different domains. By introducing the realm activation mechanism, the weights of features can be dynamically adjusted during the fusion process so that the final fused image is more in line with the appearance requirements of a specific domain. For example, in medical image fusion, edge details may be more important, while in infrared image fusion, contrast may be more critical. The realm activation mechanism dynamically adjusts the contribution of different features by learning a weight distribution mechanism. These weights can be adaptively adjusted according to the image content and domain characteristics, so that important features are highlighted and unimportant features are suppressed during the fusion process. The weight distribution is usually achieved through a small neural network or attention mechanism, which can dynamically generate a weight map based on the features of the input image.

[0041] The specific process is shown in formula (3).

[0042]

[0043] where w n is the weight of the nth input image, and ⊙ represents element-wise multiplication.

[0044] In the final stage, the network parameters are optimized using a non-reference perceptual metric loss function based on edge details and contrast. In multi-domain image fusion, edge details and contrast are important factors affecting the quality of the fused image. By introducing a non-reference perceptual metric loss function based on edge details and contrast, the model can be guided to better optimize these key features during the training process and improve the visual effect of the fused image. The non-reference perceptual metric loss function evaluates the fusion effect by calculating the edge details and contrast of the fused image. The edge detail measurement reflects the edge information of the image by calculating the gradient of the image, and the contrast measurement reflects the contrast information of the image by calculating the grayscale change of the image. These metrics can help the model better optimize the quality of the fused image during the training process, making it visually clearer and more natural. Edge detail measurement: calculate the edge information of the image. Contrast measurement: calculate the contrast information of the image. Loss function: design a non-reference perceptual metric loss function based on edge details and contrast. The specific process is shown in formulas (4)(5)(6).

[0045]

[0046] L α，β =MAE(F,α·Con(I1,I2,...,I N )+β·Edg(I1,I2,...,I N )) (6)

[0047] Where MAE stands for mean absolute difference, which represents the average difference between each pixel value between the two inputs. Edg(·) represents the edge rate, where ▽ represents the edge value of the input image I, and ||·||1 represents the L1 norm, which is the sum of all edge values. L α,β Represents the input feature F α and F β The absolute mean difference of the edge value matrix between .

[0048] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A multi-domain image fusion method for learning domain-specific and general domain feature representations, characterized in that: There are three steps involved: Step 1: Shared principle network: a shared network for cross-domain general feature extraction is designed based on deep learning methods; Step 2: Adaptive domain feature extraction, extracting features from different domains through domain-specific sub-networks; Step 3: Domain activation, through learning the weight distribution mechanism, dynamically adjust the contribution of different features, further enhance the importance of domain-specific features, and make the fused image more in line with the appearance requirements of the specific domain.

2. The method according to claim 1, characterized in that A universal convolutional neural network is used to extract common features of images in different fields.

3. The method according to claim 2, characterized in that Based on the shared network, domain-specific sub-networks are added, which perform feature extraction according to the characteristics of specific fields.

4. The method according to claim 3, characterized in that A domain activation mechanism is introduced to dynamically adjust the weights of features during the fusion process, so that the final fused image is more in line with the appearance requirements of a specific domain.

5. The method according to claim 4, characterized in that The final stage optimizes the network parameters using a no-reference perceptual metric loss function based on edge details and contrast.

6. The method according to claim 5, characterized in that The no-reference perceptual metric loss function evaluates the fusion effect by calculating the edge details and contrast of the fused image.

7. The method according to claim 6, characterized in that Edge detail measurement reflects the edge information of an image by calculating the gradient of the image; Contrast measurement reflects the contrast information of an image by calculating the grayscale changes of the image.

8. The method according to claim 7, characterized in that Edge detail measurement: calculate the edge information of the image; Contrast measurement: calculate the contrast information of the image; Loss function: Design a no-reference perceptual metric loss function based on edge details and contrast.

Images