A Multifocal Image Fusion Method Based on Attention Mechanism

Abstract

This invention proposes a multifocal image fusion method based on an attention mechanism. The method includes: first, constructing a general end-to-end multifocal image fusion network model by leveraging the properties of convolutional neural networks and the attention mechanism. The input is the multifocal image to be input, and the output is the fused all-focus image. Second, constructing a near-real-world multifocal image dataset and successfully training the proposed fusion model on this dataset. Third, inputting the image to be fused into the trained fusion model to obtain the final fused image. This invention overcomes the shortcomings of other traditional or deep learning methods that neglect important information in the source image and possesses high efficiency. Experimental results show that this method can effectively fuse various multifocal images.

Description

Technical Field

[0001] This invention relates to the field of image processing and fusion, and more specifically, to a method for multifocal image fusion using a two-stream structure model, so as to preserve the sharpness and detail of multiple focal regions in the fused image. Background Technology

[0002] With the rapid advancement of digital imaging technology, image fusion has become a hot research area in computer vision. Multifocal image fusion aims to merge multiple images from different focal points into a single, clear image, which is extremely useful in many applications, such as medical imaging, remote sensing, and photography.

[0003] Currently, there are various methods for multifocal image fusion. Traditional methods are mainly based on the spatial domain and transform domain. Spatial domain methods are typically based on pixel-level operations, directly operating on the spatial domain of the input image to obtain the fused image. Transform domain methods, on the other hand, involve transforming the image to another domain (such as the wavelet domain, frequency domain, etc.), performing the fusion in that domain, and then transforming it back to the spatial domain.

[0004] While these methods work well in certain scenarios, they often fall short in complex and dynamic scenes. Furthermore, many traditional methods can lead to fusion artifacts, especially when there are significant differences between the source images. Transform domain-based methods often fail to fuse images by generating decision maps, resulting in color differences in the output image compared to the focused regions in the source images. Spatial domain-based methods use block-based strategies, making it difficult to classify the boundaries between focused and out-of-focus areas, leading to loss of detail at the fusion boundaries and a lack of certain image details.

[0005] In recent years, deep learning has been introduced into multifocal image fusion. These deep learning-based methods typically use neural network models, such as convolutional neural networks (CNNs) or generative adversarial networks (GANs), for image fusion. These methods can automatically learn useful features of images and provide more accurate and natural fusion results.

[0006] However, despite the immense potential of deep learning for image fusion, several challenges remain. For example, many deep learning methods require large amounts of labeled data for training, which is difficult to obtain in multifocal image fusion scenarios. Many deep learning-based methods lose important feature information during convolution, negatively impacting the fusion results. Furthermore, many past methods require post-processing techniques after generating the decision map, heavily relying on the refined decision map and struggling to achieve high-quality fused images. Another drawback is model complexity, preventing end-to-end fusion. Additionally, the high computational cost of deep learning methods may limit their use in real-time applications. While existing image fusion methods achieve satisfactory overall performance, improvements are still needed. New methods are required to address the limitations of existing approaches and provide higher quality, faster, and more robust fusion results. Summary of the Invention

[0007] To more effectively process multifocal images and generate high-quality fused images, this invention proposes a deep learning-based multifocal image fusion method. This overcomes the limitations of previous multifocal image fusion methods, which could lead to blurring or distortion in the fused image.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for image fusion based on attention mechanism and multifocal focus includes the following key steps:

[0010] Step 1: Construct a novel multifocal image fusion convolutional neural network model based on an attention mechanism. An appropriate attention mechanism is designed according to the characteristics of the source images and the convolution process. A local energy operator is introduced to enhance the feature fusion capability. The input to this network is the partially focused images to be fused. The output is the fused image.

[0011] Step two involves constructing a high-quality, large-scale dataset, including 40,000 real images and their corresponding binary mask images. The proposed fusion model is then trained on this dataset.

[0012] Step 3: Input the multifocal image into the trained fusion model to obtain the final fused image.

[0013] Furthermore, in the fusion model described in step one, the specific description of the model is as follows:

[0014] First, in the feature extraction unit, feature extraction blocks are used to systematically collect multi-scale deep features corresponding to each source image. Second, an SFLE module based on the local energy operator is used to compute the activity levels of two deep features from the feature extraction unit, which are then fed into a series of convolutional operations, including a residual module and the feature extraction blocks, to further extract deep features for accurate decision classification of each pixel, where the decision classification is the probability of each pixel being focused in each source image. The output of the feature fusion unit is the initial decision map. In the feature enhancement unit, a guided filter is applied to smooth the boundaries of the initial decision map and obtain the final decision map.

[0015] Furthermore, in the construction of the multifocal image fusion model described in step one, the construction process of the model is as follows:

[0016] First, the near-focus and far-focus images are input into the Feature Extraction Unit (FEU). This unit uses a Siamese-based network to extract depth features from the input near-focus / far-focus images. This architecture consists of two identical sub-networks that share the same weight set and are used to extract depth features from the input images. Each branch in the model contains four attention-based feature extraction blocks. The main function of these blocks is to perform channel-level feature map calibration, dynamically adjusting the weights between different channels to emphasize important channels and suppress unimportant ones. Simultaneously, it prioritizes extracting information-rich salient regions from the feature maps.

[0017] Next, the output of the feature extraction unit is fed into the Feature Fusion Unit (FFU). In this unit, we use an SFLE module to compute the activity levels of two deep features, then input them into a series of convolutional operations, including residual blocks and feature extraction blocks, to further extract deep features, thereby enabling accurate decision classification for each pixel. The output of this unit is the initial decision map.

[0018] Subsequently, a guided filter is applied in the Feature Strengthening Unit (FSU) to smooth the boundaries of the initial decision map and obtain the final decision map. The purpose of this unit is to further enhance and refine the features of the initial decision map, making it more accurate and vivid.

[0019] Finally, the final decision map and source images are cascaded and fused to produce the final fused output. This Image Fusion Unit (IFU) is responsible for combining the final decision map and source images to generate a clear, high-quality fused image.

[0020] In summary, this network model achieves a complete process from input near-focus and far-focus images to the final fused image through four main components—feature extraction, feature fusion, feature enhancement, and image fusion. This design ensures that the model can effectively capture and utilize key information from the image at each stage, thereby producing high-quality fusion results.

[0021] Furthermore, in step two, a large-scale dataset for model training needs to be constructed. First, masks and multifocal images are generated through object detection algorithms. This process generates a large number of multifocal image pairs in the training dataset, providing rich data for model training.

[0022] The construction process is as shown in Algorithm 1:

[0023]

[0024] This process ensures that the training dataset contains a large number of multifocal image pairs, thus providing rich data for model training.

[0025] Next, these multifocal image pairs underwent further processing. First, image enhancement techniques (including random rotation, random flipping, and random color distortion) were used to increase the diversity of the dataset. These enhancement techniques not only improve the model's generalization ability but also help the model better cope with various situations in real-world applications.

[0026] Furthermore, to ensure the model can effectively handle images of different sizes and resolutions, multiple versions of each image were generated at different sizes. These versions were randomly cropped to a fixed size and used as input to the model. This step ensures the model is exposed to images of various sizes during training, thereby improving its robustness in real-world applications.

[0027] Finally, we divided all multifocal image pairs and their corresponding masks into training and validation sets. The training set was used for model training, while the validation set was used to evaluate model performance and perform hyperparameter tuning. To ensure that model performance was not affected by the dataset split, we used stratified sampling to ensure that the image distributions in the training and validation sets were similar.

[0028] Through the above steps, a large-scale, diverse, and high-quality multifocal image dataset was successfully constructed, providing a solid foundation for model training.

[0029] The loss function for network training is:

[0030] L = L Dice +λLdetail ,

[0031]

[0032] Where N represents the total number of pixels in the mask, p i and g i represents the value at position i in the prediction decision graph and the ground-truth, respectively.

[0033] L detail =1-SSIM(O,I),

[0034]

[0035] in and σ represents the average intensity of image O and I, respectively; O and σ I σ represents the standard deviation of intensity in images O and I. OI It is the covariance between images O and I. Specifically, the SSIM loss calculates the structural difference between images O and I.

[0036] Furthermore, in step three, the multifocal image is input into the trained model to obtain the final fused image.

[0037] The main advantages of this method are: 1. The two-stream structure model can be trained end-to-end, directly obtaining the decision map and fused image without any post-processing. 2. In the feature extraction stage of the network, an attention-based feature extraction module is designed. During feature extraction, attention is paid to the positional information of the image to extract useful features, obtain a high-quality fused image with clearer details, and avoid losing important information during convolution. 3. A feature fusion module based on the Local Energy Operator (LE) and Spatial Frequency (SF) is proposed. The calculation of LE combines the depth feature map and the source image. This helps to better preserve the information of the source image in the fused image. 4. A series of comprehensive experiments were conducted to evaluate the effectiveness of our fusion model. The results of both qualitative and quantitative evaluations show that our fusion model performs excellently, surpassing existing state-of-the-art multifocal image fusion algorithms. Attached Figure Description

[0038] Figure 1 This is a flowchart of the multifocal image fusion method based on the attention mechanism of the present invention;

[0039] Figure 2 This invention proposes a multi-focus image fusion model.

[0040] Figure 3 This is a diagram illustrating the training data required for model training.

[0041] Figure 4 and Figure 5 This is the fusion result of the test images and its difference map. Detailed Implementation

[0042] The present invention will now be described in detail with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be noted that those skilled in the art can make modifications to the invention without departing from its core principles, and these modifications all fall within the scope of protection of the present invention.

[0043] like Figure 1 As shown, this invention provides a multifocal image fusion method based on a convolutional neural network, comprising the following steps:

[0044] Step 1: Construct a novel multifocal image fusion convolutional neural network model based on an attention mechanism. An appropriate attention mechanism is designed according to the characteristics of the source images and the convolution process. A local energy operator is introduced to enhance the feature fusion capability. The input to this network is the partially focused images to be fused. The output is the fused image.

[0045] First, the near-focus and far-focus images are input into the Feature Extraction Unit (FEU). This unit uses a Siamese-based network to extract depth features from the input near-focus / far-focus images. This architecture consists of two identical sub-networks that share the same weight set and are used to extract depth features from the input images. Each branch in the model contains four attention-based feature extraction blocks. The main function of these blocks is to perform channel-level feature map calibration, dynamically adjusting the weights between different channels to emphasize important channels and suppress unimportant ones. Simultaneously, it prioritizes extracting information-rich salient regions from the feature maps.

[0046] Next, the output of the feature extraction unit is fed into the Feature Fusion Unit (FFU). In this unit, we use an SFLE module to compute the activity levels of two deep features, then input them into a series of convolutional operations, including residual blocks and feature extraction blocks, to further extract deep features, thereby enabling accurate decision classification for each pixel. The output of this unit is the initial decision map.

[0047] Subsequently, a guided filter is applied in the Feature Strengthening Unit (FSU) to smooth the boundaries of the initial decision map and obtain the final decision map. The purpose of this unit is to further enhance and refine the features of the initial decision map, making it more accurate and vivid.

[0048] Finally, the final decision map and source images are cascaded and fused to produce the final fused output. This Image Fusion Unit (IFU) is responsible for combining the final decision map and source images to generate a clear, high-quality fused image.

[0049] Step two involves constructing a high-quality, large-scale dataset, including 40,000 real images and their corresponding binary mask images. The proposed fusion model is then trained on this dataset. The construction process is shown in Algorithm 1.

[0050] Step 3: Input the multifocal image into the trained fusion model to obtain the final fused image.

[0051] In this embodiment, Figure 2 This invention proposes a multi-scale fusion model based on convolutional neural networks, whose input is the image to be fused and whose output is the fused image. Figure 3 Examples of training datasets required for model training: (a) is the fused image (original image), (b) is the binary mask, (c) is the near-focus image, and (d) is the far-focus image.

[0052] Figure 4 and Figure 5 The result of image fusion is for testing. Figure 4 For the fusion result, Figure 5 This is a graph showing the difference between the fused result and the near-focus image.

[0053] exist Figure 4 In the magnified area, it's clear that this algorithm completely preserves the golf ball at the edge of the defocused image. Other magnification methods don't quite achieve this. Given that the magnified area lies on the boundary between near and far focus, the ability to retain the golf ball reflects the method's ability to preserve important information in the source image. Clearly, this method outperforms other algorithms in preserving source image details and blending boundaries.

[0054] Figure 5 The differences between different algorithms are presented in the graphs. In subgraphs (a), (c), (e), and (g), it is evident that large areas of information are significantly lost. In subgraphs (b), (d), (h), and (i), details are lost at the focus and defocus boundaries or in other areas. The far-focus area marked in (i) shows obvious artifacts in the grass details of the fused image. Table 1 shows... Figure 4The evaluation results of the various algorithms were presented on six different datasets, and eight metrics were selected for comprehensive evaluation. The best results for each metric are highlighted in bold, demonstrating that the proposed method outperforms other algorithms on most metrics. Table 2 shows... Figure 4 The table compares the average time cost of each algorithm tested on the Lytro dataset. As can be seen from the table, the proposed method has higher fusion efficiency.

[0055] Table 1: Evaluation Results of Image Fusion by Each Algorithm

[0056]

[0057]

[0058]

[0059] Table 2: Average running time of each algorithm

[0060]

Claims

1. A multifocal image fusion method based on an attention mechanism, the key steps of which include: Step 1: Construct a new multifocal image fusion convolutional neural network model based on an attention mechanism. Design an appropriate attention mechanism according to the characteristics of the source image and the convolution process. At the same time, introduce a local energy operator to enhance the feature fusion capability. The input of this network is a partially focused image to be fused, and the output is the fused image. Step 2: Construct a high-quality, large-scale dataset, including 40,000 real images and their corresponding binary mask images, and train the proposed fusion model on this dataset. Step 3: Input the multifocal image into the trained fusion model to obtain the final fused image; In step one, the construction process of the multifocal image fusion convolutional neural network model is as follows: First, the near-focus and far-focus images are input into the feature extraction unit. In this unit, a network based on the Siamese architecture is used to extract depth features from the input near-focus / far-focus images. This architecture includes two identical sub-networks that share the same weight set and are used to extract depth features from the input images. Each branch in the model contains four feature extraction blocks based on the attention mechanism. The main function of these blocks is to perform channel-level feature map calibration, which emphasizes important channels and suppresses unimportant channels by dynamically adjusting the weights between different channels. At the same time, it also prioritizes the extraction of information-rich salient regions in the feature maps. Next, the output of the feature extraction unit is fed into the feature fusion unit. In this unit, we use an SFLE module to calculate the activity levels of two deep features and then input them into a series of convolutional operations, including residual blocks and feature extraction blocks, to further extract deep features, thereby making accurate decision classifications for each pixel. The output of this unit is the initial decision map. Subsequently, a guided filter is applied in the feature enhancement unit to smooth the boundaries of the initial decision map and obtain the final decision map. The purpose of this unit is to further enhance and refine the features of the initial decision map, making it more accurate and vivid. Finally, the cascaded final decision map and source images are fused together to produce the final fused output. This image fusion unit is responsible for combining the final decision map and source images to generate a clear, high-quality fused image. This network model achieves a complete process from input near-focus and far-focus images to the final fused image through these four main components: feature extraction, feature fusion, feature enhancement, and image fusion. This ensures that the model can effectively capture and utilize key information of the image at each stage, producing high-quality fusion results.

2. The multifocal image fusion method based on an attention mechanism according to claim 1, characterized in that, The fused convolutional neural network model described in step one is specifically described as follows: First, in the feature extraction unit, a feature extraction block is used to systematically collect multi-scale deep features corresponding to each source image. Second, an SFLE module based on local energy operators is used to calculate the activity levels of two deep features from the feature extraction unit, and then input them into a series of convolution operations, including a residual module and a feature extraction block, to further extract deep features for accurate decision classification of each pixel, where the decision classification is the probability of each pixel being focused in each source image. The output of the feature fusion unit is the initial decision map; in the feature enhancement unit, a guided filter is applied to smooth the boundaries of the initial decision map and obtain the final decision map.

3. The multifocal image fusion method based on an attention mechanism according to claim 1, characterized in that, In step two, a large-scale dataset needs to be built for model training. The construction process is as follows: First, masks and multifocal images are generated using object detection algorithms. This process involves training a large number of multifocal image pairs in the training dataset, providing rich data for model training. Next, these multifocal image pairs were further processed. First, enhancement techniques including random rotation, random flipping, and random color distortion were used to increase the diversity of the dataset. These enhancement techniques can not only improve the generalization ability of the model, but also help the model better cope with various situations in real-world applications. In addition, to ensure that the model can effectively handle images of different sizes and resolutions, multiple versions of each image were generated in different sizes. These versions of the image were randomly cropped to a fixed size and used as input to the model. Finally, all multifocal image pairs and their corresponding masks are divided into training and validation sets. The training set is used for model training, while the validation set is used to evaluate model performance and perform hyperparameter tuning. By using stratified sampling, it is ensured that the image distributions in the training and validation sets are similar.

4. A multifocal image fusion method based on an attention mechanism according to claim 1 or 3, characterized in that, In step three, the multifocal image is input into the trained model to obtain the final fused image.

Images