Multi-focus image fusion method and system based on multi-scale image fusion network

Abstract

The invention provides a multi-focus image fusion method and system based on a multi-scale image fusion network, and relates to the technical field of image fusion. First, a first focus image and a second focus image are acquired. And constructing and training a multi-scale image fusion network to obtain a trained multi-scale image fusion network. The multi-scale image fusion network comprises a feature extraction unit used for extracting multi-scale features of the first focus image and the second focus image, a feature integration unit used for integrating the multi-scale features, and a feature reconstruction unit used for generating a final fusion image. And finally, inputting the first focusing image and the second focusing image into a trained multi-scale image fusion network, and performing multi-focusing image fusion to obtain a final fusion image. The multi-scale image fusion network constructed by the invention is used for performing multi-focus image fusion, so that the requirement of an actual application scene on a multi-focus fusion image can be met, and a high-quality full-focus image is generated.

Description

Technical Field

[0001] This invention belongs to the technical field of image fusion, and more specifically, relates to a multi-focus image fusion method and system based on a multi-scale image fusion network. Background Technology

[0002] Due to the depth-of-field limitations of optical lenses, only the focused areas in an image can appear sharp, making it difficult to obtain a fully sharp image of the entire scene. Multi-focus image fusion (MFIF) is an effective method to address this problem. It extracts complementary information from two or more locally focused images of the same scene to generate a fully sharp fused image. Currently, image fusion has been widely applied in fields such as medicine, military, agriculture, and security.

[0003] Existing multi-focus image fusion methods are mainly classified into three categories: transform domain methods, spatial domain methods, and deep learning methods. Transform domain methods decompose the focused image into high-frequency and low-frequency components, applying different fusion rules to each component. However, this method ignores the spatial consistency issues that may arise during the processing of the source focused image, leading to distortion of the fused focused image and loss of some spatial information. Spatial domain methods can be further divided into pixel-based, block-based, and region-based methods. Pixel-based spatial domain methods generate pixel-by-pixel weight maps by evaluating the focus of pixels in the source focused image, preserving information for each pixel, but requiring significant computation. Block-based spatial domain methods are suitable for situations with large differences in local image features, but may produce block artifacts. Region-based spatial domain methods utilize segmentation techniques to divide the source focused image into regions, but require sophisticated segmentation techniques. Spatial domain methods are more consistent with the human visual system (HVS), but activity measurement and fusion rules require manual design.

[0004] Deep learning methods have gradually become the mainstream technology in the field of multi-focus image fusion due to their powerful feature representation capabilities. While supervised learning-based image fusion models can identify the in-focus areas of an image, their network structures are too simple and require post-processing to construct the decision map. The MFF-SSIM (Multi-Focus Fusion Structural Similarity Index) method, a quality assessment method for multi-focus image fusion, can effectively reduce defocus diffusion effects, but it is limited to generating decision maps, and the fused image requires post-processing. Furthermore, it does not focus on local structural features, resulting in insensitivity to key information such as edge information, noise, and color. The end-to-end network U2Fusion can solve the fusion problem of different types of images, but due to differences in image feature extraction and image fusion across different domains, U2Fusion struggles to achieve optimal performance in multi-focus image fusion. The unsupervised end-to-end network MFF-GAN (Multi-Focus Fusion-Generative Adversarial Network) can automatically classify in-focus and defocused regions by designing adaptive decision blocks. However, this network has insufficient ability to extract key features, leading to color aberration issues. The GACN architecture improves fusion performance by generating decision maps and fused images simultaneously through end-to-end training. However, the feature information it extracts and fuses is of a single scale, making it unable to accurately capture complex edges and texture details in images.

[0005] In summary, while deep learning methods have improved the quality of focused image fusion to some extent compared to traditional fusion algorithms, they still have certain limitations. Most existing models extract features at a single scale, making it difficult to cover detailed information at different levels of the image. Some methods require additional post-processing steps to optimize the results, which not only increases computational time but also easily leads to the loss of key information during processing. Furthermore, existing models suffer from insufficient precision in extracting and representing key features such as image edges, textures, and colors, resulting in distorted final fused images with missing details (such as edges and textures), making it difficult to meet the demands of complex real-world applications for multi-focused fusion images. Summary of the Invention

[0006] To address the problem that existing multi-focus image fusion methods suffer from limited feature extraction scale and difficulty in highlighting key information, this invention provides a multi-focus image fusion method and system based on a multi-scale image fusion network. This method meets the needs of complex real-world application scenarios for multi-focus fused images and generates high-quality all-focus images.

[0007] To achieve the above-mentioned technical effects, the technical solution proposed by this invention is as follows: S1: Acquire the first focused image and the second focused image; S2: Construct a multi-scale image fusion network; the multi-scale image fusion network includes: a feature extraction unit for extracting multi-scale features of the first focused image and the second focused image, a feature integration unit for integrating multi-scale features, and a feature reconstruction unit for generating the final fused image; S3: Train the multi-scale image fusion network to obtain a trained multi-scale image fusion network; S4: Input the first and second focused images into the trained multi-scale image fusion network to perform multi-focus image fusion and obtain the final fused image.

[0008] Furthermore, the feature extraction unit includes: a first feature extraction branch and a second feature extraction branch; both the first and second feature extraction branches are provided with N cascaded feature extraction modules; each feature extraction module includes: a convolutional layer and a convolutional block attention layer (CBAM) connected in sequence, inputting the first focused image to the first feature extraction module of the first feature extraction branch, and inputting the second focused image to the first feature extraction module of the second feature extraction branch; The output of the first feature extraction module is used as the input of the second feature extraction module. The output of the first feature extraction module and the output of the i-th feature extraction module are used together as the input of the (i+1)-th feature extraction module. i is greater than or equal to 2, i is a positive integer, and i+1 is less than or equal to N. When i is greater than 2, the output of the (i-1)-th feature extraction module is also used as the input of the (i+1)-th feature extraction module. The output of the j-th feature extraction module of the first feature extraction branch and the output of the j-th feature extraction module of the second feature extraction branch are respectively subjected to spatial frequency SF focusing measurement to obtain the first focusing measurement value and the second focusing measurement value. The first focusing measurement value is subtracted from the second focusing measurement value to obtain the pixel-by-pixel decision map. The pixel-by-pixel decision map is input to the feature integration unit, where j is greater than or equal to 1 and j is a positive integer. The feature integration unit includes: a Concat module and a multi-scale selection kernel module MSK connected in sequence; The feature reconstruction unit includes M cascaded feature reconstruction modules; each feature reconstruction module includes a convolutional layer and an efficient channel attention layer (ECA) connected in sequence.

[0009] Furthermore, in the convolutional block attention layer CBAM: set up Let F represent the feature map of the input first or second focused image. First, global average pooling and global max pooling operations are performed on the feature map F to obtain the channel feature maps after global average pooling and global max pooling, respectively. Then, the channel feature maps after global average pooling and global max pooling are processed using a multilayer perceptron, followed by element-wise addition and sigmoid activation to obtain the channel attention weights. The expression is:

[0010]

[0011] In the formula, This represents a multilayer perceptron. This indicates the average pooling operation. This represents the global max pooling operation. This represents the channel feature map after global average pooling. This represents the channel feature map after global max pooling. This represents the sigmoid function. Represents the weights of the multilayer perceptron. Represents the weights of the multilayer perceptron. This represents the channel attention weight value; Channel attention weights and feature maps F Perform element-wise multiplication to obtain the channel attention feature map. The expression is:

[0012] In the formula, Represents the channel attention feature map. This indicates element-wise multiplication; Channel attention feature map Global average pooling and global max pooling are performed separately to generate spatial feature maps after global average pooling and global max pooling, respectively. These two spatial feature maps are then concatenated, followed by convolution and sigmoid activation to obtain the spatial attention weights, expressed as:

[0013] In the formula, This represents the spatial attention weight value. This represents a filter of size 7×7. This represents the spatial feature map after global average pooling. Represents the spatial feature map after global max pooling; The spatial attention weights and channel attention feature maps are multiplied element-wise to obtain the final output feature map, expressed as:

[0014] In the formula, This represents the feature map of the final output.

[0015] Furthermore, the process of measuring the spatial frequency SF focusing is as follows: Select a local block centered on each pixel in the first or second focused image; Calculate the frequency RF in the row direction and the frequency CF in the column direction of the local block; based on the frequency RF in the row direction and the frequency CF in the column direction, calculate the spatial frequency SF of the center pixel of the local block as a focus metric, and obtain the focus metric value; The expression for calculating the frequency RF in the row direction is:

[0016] In the formula, The x-coordinate is m The vertical axis is n The frequency RF of the pixel in the row direction, This indicates that the focused image is located on the horizontal axis. m+a The vertical axis is n+b The pixel at that location, This represents the pixel in the focused image located at x-coordinate m+a and y-coordinate n+b-1, where r represents the kernel radius. The expression for calculating the frequency CF in the column direction is:

[0017] In the formula, The frequency CF represents the column direction; The expression for calculating the focus metric is:

[0018] In the formula, This indicates the focus metric.

[0019] Furthermore, in the Multi-Scale Selection Kernel (MSK) module: multi-scale features are extracted through several deep convolutional layers of different sizes; Multi-scale features are added and fused together. Global information is obtained through the first global average pooling layer to obtain a global feature vector. The global feature vector is then passed through the first fully connected layer to reduce the dimensionality of the global information. The feature dimension is then restored through the second fully connected layer to obtain the weights of the multi-scale features. The weights of the multi-scale features are then passed through an activation function to generate normalized weights of the multi-scale features. The weights of the multi-normalized scale features are multiplied element-wise with the multi-scale features to obtain the weighted multi-scale features; the weighted multi-scale features are then summed to obtain the fused multi-scale features.

[0020] Furthermore, in the feature reconstruction unit: Generate an initial decision map, which represents the probability that the first focused image has higher sharpness than the second focused image at each pixel location; Gaussian filtering is applied to the initial decision map, and threshold calculation is used to select pixels within a set threshold as boundary regions. The initial decision graph is smoothed using a guided filter to obtain a smoothed decision graph; By using the boundary region as the threshold region, the smoothed decision map is fused with the initial decision map to obtain the final decision map; The final decision image and the source image are weighted and averaged, and then multi-focus image fusion is performed to obtain the final fused image.

[0021] Furthermore, in the efficient channel attention layer (ECA): the feature map is input into the first global average pooling layer for global average pooling operation, and aggregated features are output; the aggregated features are input into the first convolutional layer for capturing local cross-channel information, and initial channel weights are output; the initial channel weights are input into the first activation function layer for nonlinear compression operation, and channel weights are obtained; the input feature map and channel weights are multiplied element-wise to obtain the enhanced feature map.

[0022] Furthermore, during the training process of the multi-scale image fusion network: Obtain the original image set for training, process the original images using Gaussian filtering to generate near-focus images and far-focus images; construct near-focus image-far-focus image data pairs based on the near-focus image and far-focus image data pairs, and construct the training set based on the near-focus image-far-focus image data pairs; A gradient-aware loss function is constructed, and the training set is input into the constructed multi-scale image fusion network. The Adam optimizer is used to train the multi-scale image fusion network based on the gradient-aware loss function. During the training process, the parameters of the multi-scale image fusion network are updated through the backpropagation algorithm until the gradient-aware loss function converges, thus obtaining the trained multi-scale image fusion network.

[0023] Furthermore, the expression for the gradient-aware loss function is:

[0024] In the formula, This represents the gradient-aware loss function. This indicates Dice's loss. This indicates the loss of edge information. Indicates the weighting parameter; The expression for the Dice loss is:

[0025] In the formula, This indicates the total number of pixels in the focused image. This indicates the probability that the first focused image is sharper than the second focused image at each pixel. This represents the value of the baseline decision map at the i-th pixel. The expression for the edge information loss is:

[0026] In the formula, This represents the edge information retention value of pixel i in the first focused image. This represents the edge information retention value of pixel i in the second focused image. This represents the edge intensity of pixel i in the first focused image. This represents the edge intensity of pixel i in the second focused image. This represents the amount of information at the edge.

[0027] The present invention also provides a multi-focus image fusion system based on a multi-scale image fusion network, comprising: The data acquisition module is used for multi-focus image fusion methods based on multi-scale image fusion networks; A network construction module is used to construct a multi-scale image fusion network; the multi-scale image fusion network includes: a feature extraction unit for extracting multi-scale features of a first focused image and a second focused image, a feature integration unit for integrating multi-scale features, and a feature reconstruction unit for generating a final fused image; The training module is used to train the multi-scale image fusion network to obtain a trained multi-scale image fusion network. The fusion module is used to input the first focused image and the second focused image into the trained multi-scale image fusion network to perform multi-focus image fusion and obtain the final fused image.

[0028] Compared with existing technologies, the beneficial effects of this method are: This invention provides a multi-focus image fusion method and system based on a multi-scale image fusion network. The method involves constructing and training a multi-scale image fusion network to obtain a trained network. The multi-scale image fusion network includes a feature extraction unit, a feature integration unit, and a feature reconstruction unit. The feature extraction unit extracts multi-scale features from the first and second focused images, capturing key details from different levels of focused images. The feature integration unit integrates multi-scale features, addressing the problem of insufficient representation by single-scale features. The feature reconstruction unit generates the final fused image. The multi-scale image fusion network constructed in this invention can meet the needs of practical applications for multi-focus fused images, generating high-quality full-focus images. Attached Figure Description

[0029] Figure 1 A flowchart illustrating the multi-focus image fusion method based on a multi-scale image fusion network proposed in this embodiment of the invention; Figure 2 This diagram illustrates the structure of the multi-scale image fusion network proposed in this embodiment of the invention. Figure 3 This diagram illustrates the structure of the Convolutional Block Attention Layer (CBAM) proposed in this embodiment of the invention. Figure 4 This diagram illustrates the structure of the channel attention unit proposed in this embodiment of the invention. Figure 5 This diagram illustrates the structure of the spatial attention unit proposed in this embodiment of the invention. Figure 6 This diagram illustrates the structure of the multi-scale selection kernel module (MSK) proposed in this embodiment of the invention. Figure 7 This diagram illustrates the structure of the efficient channel attention layer ECA proposed in this embodiment of the invention. Figure 8 An example diagram showing the training images of the training set proposed in the embodiments of the present invention; Figure 9 This image shows a comparison of the final fused image obtained by the multi-focus image fusion method based on a multi-scale image fusion network proposed in this invention and existing methods. Figure 10 This diagram illustrates the structure of the multi-focus image fusion system based on a multi-scale image fusion network proposed in this embodiment of the invention. Detailed Implementation

[0030] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions; It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Example 1 This embodiment proposes a multi-focus image fusion method based on a multi-scale image fusion network, such as... Figure 1 The flowchart shown illustrates the method, and the method proposed in this embodiment generally includes the following steps: S1: Acquire the first focused image and the second focused image; S2: Construct a multi-scale image fusion network; the multi-scale image fusion network includes: a feature extraction unit for extracting multi-scale features of the first focused image and the second focused image, a feature integration unit for integrating multi-scale features, and a feature reconstruction unit for generating the final fused image; S3: Train the multi-scale image fusion network to obtain a trained multi-scale image fusion network; S4: Input the first and second focused images into the trained multi-scale image fusion network to perform multi-focus image fusion and obtain the final fused image.

[0033] In this embodiment, as Figure 2 The diagram shows the structure of a multi-scale image fusion network. The feature extraction unit in the multi-scale image fusion network includes a first feature extraction branch and a second feature extraction branch. Both the first and second feature extraction branches have N cascaded feature extraction modules. Each feature extraction module includes a convolutional layer and a convolutional block attention layer (CBAM) connected in sequence. The first focused image is input to the first feature extraction module of the first feature extraction branch, and the second focused image is input to the first feature extraction module of the second feature extraction branch. The output of the first feature extraction module is used as the input of the second feature extraction module. The output of the first feature extraction module and the output of the i-th feature extraction module are used together as the input of the (i+1)-th feature extraction module. i is greater than or equal to 2, i is a positive integer, and i+1 is less than or equal to N. When i is greater than 2, the output of the (i-1)-th feature extraction module is also used as the input of the (i+1)-th feature extraction module. The output of the j-th feature extraction module of the first feature extraction branch and the output of the j-th feature extraction module of the second feature extraction branch are respectively subjected to spatial frequency SF focusing measurement to obtain the first focusing measurement value and the second focusing measurement value. The first focusing measurement value is subtracted from the second focusing measurement value to obtain the pixel-by-pixel decision map. The pixel-by-pixel decision map is input to the feature integration unit, where j is greater than or equal to 1 and j is a positive integer. The feature integration unit includes: a Concat module and a multi-scale selection kernel module MSK connected in sequence; The feature reconstruction unit includes M cascaded feature reconstruction modules; each feature reconstruction module includes a convolutional layer and an efficient channel attention layer (ECA) connected in sequence.

[0034] In this embodiment, the feature extraction unit is a twin-shared structure. N cascaded feature extraction modules employ a densely connected structure, which enhances feature propagation and reduces the number of parameters. Simultaneously, to accurately locate details in the first and second focused images and to preserve as much information as possible from both images, no pooling layer is used.

[0035] For example, see Figure 2 The first and second feature extraction branches each have five cascaded feature extraction modules. The output of the first feature extraction module serves as the input to the second, third, fourth, and fifth feature extraction modules. The output of the second feature extraction module serves as the input to the third, fourth, and fifth feature extraction modules. The output of the third feature extraction module serves as the input to the fourth and fifth feature extraction modules. The output of the fourth feature extraction module serves as the input to the fifth feature extraction module.

[0036] The outputs of the first feature extraction module of the first feature extraction branch and the first feature extraction module of the second feature extraction branch are subjected to spatial frequency SF focusing measurement to obtain a first focusing measurement value and a second focusing measurement value. The first focusing measurement value is subtracted from the second focusing measurement value to obtain a first pixel-wise decision map, which is then input into the feature integration unit. The outputs of the second feature extraction module of the first feature extraction branch and the second feature extraction module of the second feature extraction branch are subjected to spatial frequency SF focusing measurement to obtain a first focusing measurement value and a second focusing measurement value. The first focusing measurement value is subtracted from the second focusing measurement value to obtain a second pixel-wise decision map, which is then input into the feature integration unit. The outputs of the third feature extraction module of the first feature extraction branch and the third feature extraction module of the second feature extraction branch are subjected to spatial frequency SF focusing measurement to obtain a first focusing measurement value and a second focusing measurement value. The first focusing measurement value is subtracted from the second focusing measurement value to obtain a third pixel-wise decision map, which is then input into the feature integration unit. The outputs of the fourth feature extraction module in the first feature extraction branch and the fourth feature extraction module in the second feature extraction branch are subjected to spatial frequency (SF) focusing measurements to obtain a first focusing measurement value and a second focusing measurement value. The first focusing measurement value is subtracted from the second focusing measurement value to obtain the fourth pixel-wise decision map, which is then input into the feature integration unit. Similarly, the outputs of the fifth feature extraction module in the first feature extraction branch and the fifth feature extraction module in the second feature extraction branch are subjected to spatial frequency (SF) focusing measurements to obtain a first focusing measurement value and a second focusing measurement value. The first focusing measurement value is subtracted from the second focusing measurement value to obtain the fifth pixel-wise decision map, which is then input into the feature integration unit.

[0037] In this embodiment, as Figure 3 The diagram shows the structure of the Convolutional Block Attention Module (CBAM). CBAM extracts more details and key features. The Multi-scale Selective Kernel (MSK) uses multiple convolutional kernels to extract features at different scales, expanding the receptive field and ensuring more complete feature information. The Effective Channel Attention (ECA) layer adaptively adjusts the weights of each channel, enhancing the expressive power of the feature map. The multi-scale image fusion network simultaneously extracts deep and key features, effectively integrating and reconstructing multi-scale features; it also generates decision maps and fusion results, optimizing the visual effect of the image.

[0038] To preserve more detailed information from the first or second focused image, a Convolutional Block Attention (CBAM) layer is implemented. This layer extracts attention features simultaneously in both channel and spatial dimensions, thereby enhancing the contour and texture details of the fused image. This allows for more accurate and comprehensive extraction of features from either the first or second focused image. The CBAM layer consists of channel attention units and spatial attention units connected in series. The channel attention units enhance important features within a channel while suppressing unimportant features.

[0039] In this embodiment, in the convolutional block attention layer CBAM: like Figure 4 The diagram shown illustrates the structure of a channel attention unit. Within this channel attention unit, let... Let F represent the feature map of the input first or second focused image. First, global average pooling and global max pooling operations are performed on the feature map F to obtain the channel feature maps after global average pooling and global max pooling, respectively. Then, the channel feature maps after global average pooling and global max pooling are processed using a multilayer perceptron, followed by element-wise addition and sigmoid activation to obtain the channel attention weights. The expression is:

[0040]

[0041] In the formula, This represents a multilayer perceptron. This indicates the average pooling operation. This represents the global max pooling operation. This represents the channel feature map after global average pooling. This represents the channel feature map after global max pooling. This represents the sigmoid function. Represents the weights of the multilayer perceptron. Represents the weights of the multilayer perceptron. This represents the channel attention weight value; Channel attention weights and feature maps F Perform element-wise multiplication to obtain the channel attention feature map. The expression is:

[0042] In the formula, Represents the channel attention feature map. This indicates element-wise multiplication; like Figure 5The diagram shown illustrates the structure of a spatial attention unit. Within this unit, channel attention feature maps are... Global average pooling and global max pooling are performed separately to generate spatial feature maps after global average pooling and global max pooling, respectively. These two spatial feature maps are then concatenated, followed by convolution and sigmoid activation to obtain the spatial attention weights, expressed as:

[0043] In the formula, This represents the spatial attention weight value. This represents a filter of size 7×7. This represents the spatial feature map after global average pooling. Represents the spatial feature map after global max pooling; The spatial attention weights and channel attention feature maps are multiplied element-wise to obtain the final output feature map, expressed as:

[0044] In the formula, This represents the feature map of the final output.

[0045] In this embodiment, the process of spatial frequency (SF) focusing measurement is as follows: Select a local block centered on each pixel in the first or second focused image; Calculate the frequency RF in the row direction and the frequency CF in the column direction of the local block; based on the frequency RF in the row direction and the frequency CF in the column direction, calculate the spatial frequency SF of the center pixel of the local block as a focus metric, and obtain the focus metric value; The expression for calculating the frequency RF in the row direction is:

[0046] In the formula, The x-coordinate is m The vertical axis is n The frequency RF of the pixel in the row direction; This indicates that the focused image is located on the horizontal axis. m+a The vertical axis is n+b The pixel at that location, This represents the pixel in the focused image located at x-coordinate m+a and y-coordinate n+b-1, where r represents the kernel radius. The expression for calculating the frequency CF in the column direction is:

[0047] In the formula, The frequency CF represents the column direction; The expression for calculating the focus metric is:

[0048] In the formula, This indicates the focus metric.

[0049] To effectively integrate multi-scale feature information, a multi-scale selection kernel module (MSK) is set up. For example... Figure 6 The diagram shown illustrates the structure of the Multi-Scale Selection Kernel (MSK) module. In the MSK module, multi-scale features are extracted through several deep convolutional layers of different sizes. Multi-scale features are added and fused together. Global information is obtained through the first global average pooling layer (GAP) to obtain a global feature vector. The global feature vector is then passed through the first fully connected layer (FC) to reduce the dimensionality of the global information. The feature dimension is then restored through the second fully connected layer to obtain the weights of the multi-scale features. The weights of the multi-scale features are then passed through an activation function to generate normalized weights of the multi-scale features. The weights of the multi-normalized scale features are multiplied element-wise with the multi-scale features to obtain the weighted multi-scale features; the weighted multi-scale features are then summed to obtain the fused multi-scale features.

[0050] Existing conventional Selective Kernel (SK) modules use 3×3 and 5×5 convolutional kernels to extract and integrate multi-scale information. However, using only two scales of convolutional kernels is insufficient for comprehensive feature extraction and lacks the ability to handle complex features, potentially leading to loss of detail in the integrated feature information. Therefore, the multi-scale Selective Kernel (MSK) module in this embodiment, based on the SK module, reconstructs multi-scale depth convolutions of different sizes. By capturing information at more scales, it learns more global and local information in the focused image, resulting in more complete and accurate integrated feature information.

[0051] For example, in the multi-scale kernel selection module MSK, convolutional layers with depths of 1×1, 3×3, 5×5, and 7×7 are set respectively.

[0052] In this embodiment, in the feature reconstruction unit: Generate an initial decision map that is close to binarization, wherein the initial decision map represents the probability value that the first focused image has higher sharpness than the second focused image at each pixel position; Gaussian filtering is applied to the initial decision map, and threshold calculation is used to select pixels within a set threshold as boundary regions. The initial decision graph is smoothed using a guided filter to obtain a smoothed decision graph; By using the boundary region as the threshold region, the smoothed decision map is fused with the initial decision map to obtain the final decision map; The final decision image and the source image are weighted and averaged, and then multi-focus image fusion is performed to obtain the final fused image.

[0053] For example, the range of the threshold is set to .

[0054] In this embodiment, as Figure 7 The diagram shows the structure of the efficient channel attention layer (ECA). In the efficient channel attention layer (ECA): the feature map is input into the first global average pooling layer (GAP) for global average pooling operation, and the output is aggregated features; the aggregated features are input into the first convolutional layer (Conv1d) for capturing local cross-channel information, and the output is initial channel weights; the initial channel weights are input into the first activation function layer (Sigmoid) for non-linear compression operation to obtain channel weights; the input feature map and channel weights are multiplied element-wise to obtain the enhanced feature map.

[0055] The Efficient Channel Attention (ECA) layer enhances the expressive power of feature maps. Employing a non-dimensionality-reduction local cross-channel interaction mechanism, it strengthens feature representation, improves network performance, and significantly reduces model complexity. The non-dimensionality-reduction design avoids feature information loss, focusing more on key features and effectively preserving important details in both the first and second focused images, thereby improving the expressive power of the feature maps.

[0056] Example 2 This embodiment provides a detailed explanation of the process of training the multi-scale image fusion network in the multi-focus image fusion method based on the multi-scale image fusion network proposed in Embodiment 1.

[0057] During the training of multi-scale image fusion networks: Obtain the original image set for training, process the original images using Gaussian filtering to generate near-focus images and far-focus images; construct near-focus image-far-focus image data pairs based on the near-focus image and far-focus image data pairs, and construct the training set based on the near-focus image-far-focus image data pairs; A gradient-aware loss function is constructed, and the training set is input into the constructed multi-scale image fusion network. The Adam optimizer is used to train the multi-scale image fusion network based on the gradient-aware loss function. During the training process, the parameters of the multi-scale image fusion network are updated through the backpropagation algorithm until the gradient-aware loss function converges, thus obtaining the trained multi-scale image fusion network.

[0058] For example, during training, the learning rate is set to 0.0004, the batch size is set to 8, and the epochs are set to 50.

[0059] In this embodiment, 2913 original images are selected from VOC2012 and resized to 256×256 to form the original image set. During the Gaussian filtering process, the background is blurred to form a near-focus image, and the foreground is blurred to form a far-focus image. A defocus diffusion effect model is used to improve the realism of the generated near-focus and far-focus images. Figure 8 The image shown is an example of the training images in the training set. The first row of images represents the example images in the original image set, the second row represents the near-focus images, and the third row represents the far-focus images.

[0060] In this embodiment, the expression for the gradient-aware loss function is:

[0061] In the formula, This represents the gradient-aware loss function. This indicates Dice's loss. This indicates the loss of edge information. Indicates the weighting parameter; The expression for the Dice loss is:

[0062] In the formula, This indicates the total number of pixels in the focused image. This indicates the probability that the first focused image is sharper than the second focused image at each pixel. This represents the value of the baseline decision map at the i-th pixel. The expression for the edge information loss is:

[0063] In the formula, This represents the edge information retention value of pixel i in the first focused image. This represents the edge information retention value of pixel i in the second focused image. This represents the edge intensity of pixel i in the first focused image. This represents the edge intensity of pixel i in the second focused image. This represents the amount of information at the edge.

[0064] In this embodiment, the training of the multi-scale image fusion network is further tested and evaluated using the Lytro multifocal dataset and the MFI-WHU Wuhan University multifocal image dataset. Fifty pairs of common multifocal images from the Lytro and MFI-WHU datasets are selected as test images, with 20 pairs taken from the Lytro dataset and 30 pairs from the MFI-WHU dataset.

[0065] To verify the effectiveness of the trained multi-scale image fusion network, the multi-focus image fusion method based on the multi-scale image fusion network proposed in Example 1, along with the guided filter fusion algorithm (GFF), cross-scale representation fusion algorithm (CSR), convolutional neural network-based fusion algorithm (CNN), multi-focus image fusion quality evaluation method (Multi-Focus Fusion Structural Similarity Index, MFF-SSIM), unsupervised end-to-end network (Multi-Focus Fusion-Generative Adversarial Network, MFF-GAN), graph attention convolutional network-based fusion algorithm (Graph Attention Convolutional Network-based Fusion, GACN), unsupervised end-to-end image fusion network (Unified and Universal Image Fusion Network, U2Fusion), zero-mean filter-based multi-focus fusion algorithm (ZMFF), and fusion algorithm based on diffusion model (Fusion based on Diffusion) are compared. This comparison focuses on ten typical algorithms, including Models (FusionDiff) and Multi-Focus Fusion based on Fourier Transform (MFFT).

[0066] like Figure 9The accompanying image shows a comparison of the final fused images obtained by the multi-focus image fusion method based on a multi-scale image fusion network and existing methods. The differences in the fusion results of different multi-focus image fusion methods are highlighted with small red rectangles and magnified locally, placed in the upper right corner of the image. The selected detail is the boundary between the photographer's index finger joint and the globe background. From the joint edges, it can be seen that in comparison images (c), (d), (e), and (k), there is a large amount of noise at the joint edges. In comparison images (g) and (i), there are noise and color distortion problems at the edges. In comparison image (i), the texture details at the joint are lost. In comparison images (h), (f), and (l), defocus diffusion effects appear at the joint edges, and the boundaries become blurred. In comparison image (j), the boundaries at the joint edges are blurred, and jagged ripples appear. In comparison image (m), the multi-focus image fusion method proposed in Example 1 preserves rich detail information at the boundaries without noise or artifacts.

[0067] In this embodiment, to comprehensively evaluate multi-focus image fusion, objective metrics are used to quantitatively analyze the fusion results of different fusion methods. These objective metrics include: information-theoretic metrics (cross-entropy CE, feature mutual information FMI_dct, mutual information...). ), Image feature-based metrics ( Gradient preservation ), a measure based on image structural similarity ( Human perception-based metrics (contrast-based fusion performance metrics) ), a metric based on source and generated images (pixel-based fusion quality weighted sum) Noise-aware fusion quality metrics ), of which the indicators CE and The smaller the value, the better the quality of the final fused image; for the other indicators, the larger the value, the better the quality of the final fused image.

[0068] Table 1 shows the average objective metrics for multi-focus image fusion using 20 images selected from the Lytro multifocal dataset and various multifocal image fusion methods. In each metric, bold values ​​indicate the best result, and underlined values ​​indicate the second best result. The multifocal image fusion method proposed in this invention achieves optimal values ​​in most objective metric evaluations. Specifically, CE and... Two objective metrics showed improvements of 1.2% and 2.2% respectively compared to the second-ranked comparison algorithm.

[0069] Table 1

[0070] Table 2 shows the average objective metrics for multi-focus image fusion using 30 images selected from the MFI-WHU Wuhan University multi-focus image dataset and various multi-focus image fusion methods. In each metric, bold values ​​indicate the best result, and underlined values ​​indicate the second best result. The multi-focus image fusion method proposed in Example 1 achieves optimal values ​​in most objective metrics, with CE and... The two metrics improved by 1.1% and 5.5% respectively compared to the second-ranked comparison algorithm.

[0071] Table 2

[0072] A comprehensive comparison of the multi-focus image fusion method based on a multi-scale image fusion network proposed in Example 1 and existing methods on the Lytro and MFI-WHU datasets reveals that the multi-focus image fusion method based on a multi-scale image fusion network achieves optimal results on the vast majority of metrics, including CE and The multi-focus image fusion method based on multi-scale image fusion networks shows significant improvements in performance metrics, demonstrating remarkable effectiveness in preserving information integrity and reducing noise in the fused image. The final fused image yields richer detail information, excels in preserving edge information, and minimizes the generation of undesirable results such as noise and artifacts, resulting in superior visual effects compared to other algorithms.

[0073] Example 3 This invention also includes a multi-focus image fusion system based on a multi-scale image fusion network, the system composition of which is as follows: Figure 10 As shown, it includes: The data acquisition module is used for multi-focus image fusion methods based on multi-scale image fusion networks; A network construction module is used to construct a multi-scale image fusion network; the multi-scale image fusion network includes: a feature extraction unit for extracting multi-scale features of a first focused image and a second focused image, a feature integration unit for integrating multi-scale features, and a feature reconstruction unit for generating a final fused image; The training module is used to train the multi-scale image fusion network to obtain a trained multi-scale image fusion network. The fusion module is used to input the first focused image and the second focused image into the trained multi-scale image fusion network to perform multi-focus image fusion and obtain the final fused image.

[0074] The embodiments described are merely examples to clearly illustrate the present invention and are not intended to limit the implementation of the invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all possible implementations. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A multi-focus image fusion method based on a multi-scale image fusion network, characterized in that, Includes the following steps: Acquire the first focused image and the second focused image; Constructing a multi-scale image fusion network; The multi-scale image fusion network includes: a feature extraction unit for extracting multi-scale features from the first focused image and the second focused image, a feature integration unit for integrating multi-scale features, and a feature reconstruction unit for generating the final fused image. The multi-scale image fusion network is trained to obtain a trained multi-scale image fusion network; The first and second focused images are input into a trained multi-scale image fusion network to perform multi-focus image fusion and obtain the final fused image.

2. The multi-focus image fusion method based on a multi-scale image fusion network according to claim 1, characterized in that, The feature extraction unit includes: a first feature extraction branch and a second feature extraction branch; both the first and second feature extraction branches are provided with N cascaded feature extraction modules; each feature extraction module includes: a convolutional layer and a convolutional block attention layer (CBAM) connected in sequence, inputting a first focused image to the first feature extraction module of the first feature extraction branch, and inputting a second focused image to the first feature extraction module of the second feature extraction branch; The output of the first feature extraction module is used as the input of the second feature extraction module. The output of the first feature extraction module and the output of the i-th feature extraction module are used together as the input of the (i+1)-th feature extraction module. i is greater than or equal to 2, i is a positive integer, and i+1 is less than or equal to N. When i is greater than 2, the output of the (i-1)-th feature extraction module is also used as the input of the (i+1)-th feature extraction module. The output of the j-th feature extraction module of the first feature extraction branch and the output of the j-th feature extraction module of the second feature extraction branch are respectively subjected to spatial frequency SF focusing measurement to obtain the first focusing measurement value and the second focusing measurement value. The first focusing measurement value is subtracted from the second focusing measurement value to obtain the pixel-by-pixel decision map. The pixel-by-pixel decision map is input to the feature integration unit, where j is greater than or equal to 1 and j is a positive integer. The feature integration unit includes: a Concat module and a multi-scale selection kernel module MSK connected in sequence; The feature reconstruction unit includes M cascaded feature reconstruction modules; each feature reconstruction module includes a convolutional layer and an efficient channel attention layer (ECA) connected in sequence.

3. The multi-focus image fusion method based on a multi-scale image fusion network according to claim 2, characterized in that, In the convolutional block attention layer CBAM: set up Let F represent the feature map of the input first or second focused image. First, global average pooling and global max pooling operations are performed on the feature map F to obtain the channel feature maps after global average pooling and global max pooling, respectively. Then, the channel feature maps after global average pooling and global max pooling are processed using a multilayer perceptron, followed by element-wise addition and sigmoid activation to obtain the channel attention weights. The expression is: In the formula, This represents a multilayer perceptron. This indicates the average pooling operation. This represents the global max pooling operation. This represents the channel feature map after global average pooling. This represents the channel feature map after global max pooling. This represents the sigmoid function. Represents the weights of the multilayer perceptron. Represents the weights of the multilayer perceptron. This represents the channel attention weight value; Channel attention weights and feature maps F Perform element-wise multiplication to obtain the channel attention feature map. The expression is: In the formula, Represents the channel attention feature map. This indicates element-wise multiplication; Channel attention feature map Global average pooling and global max pooling are performed separately to generate spatial feature maps after global average pooling and global max pooling, respectively. These two spatial feature maps are then concatenated, followed by convolution and sigmoid activation to obtain the spatial attention weights, expressed as: In the formula, This represents the spatial attention weight value. This represents a filter of size 7×7. This represents the spatial feature map after global average pooling. Represents the spatial feature map after global max pooling; The spatial attention weights and channel attention feature maps are multiplied element-wise to obtain the final output feature map, expressed as: In the formula, This represents the feature map of the final output.

4. The multi-focus image fusion method based on a multi-scale image fusion network according to claim 2, characterized in that, The process of measuring the spatial frequency SF focusing is as follows: Select a local block centered on each pixel in the first or second focused image; Calculate the frequency RF in the row direction and the frequency CF in the column direction of the local block; based on the frequency RF in the row direction and the frequency CF in the column direction, calculate the spatial frequency SF of the center pixel of the local block as a focus metric, and obtain the focus metric value; The expression for calculating the frequency RF in the row direction is: In the formula, The x-coordinate is m The vertical axis is n The frequency RF of the pixel in the row direction, This indicates that the focused image is located on the horizontal axis. m+a The vertical axis is n+b The pixel at that location, This represents the pixel in the focused image at x-coordinate m+a and y-coordinate n+b-1, where r represents the kernel radius. The expression for calculating the frequency CF in the column direction is: In the formula, The frequency CF represents the column direction; The expression for calculating the focus metric is: In the formula, This indicates the focus metric.

5. A multi-focus image fusion method based on a multi-scale image fusion network according to claim 2, characterized in that, In the Multi-Scale Selection Kernel (MSK) module: multi-scale features are extracted through several deep convolutional layers of different sizes; Multi-scale features are added and fused together. Global information is obtained through the first global average pooling layer to obtain a global feature vector. The global feature vector is then passed through the first fully connected layer to reduce the dimensionality of the global information. The feature dimension is then restored through the second fully connected layer to obtain the weights of the multi-scale features. The weights of the multi-scale features are then passed through an activation function to generate normalized weights of the multi-scale features. The weights of the multi-normalized scale features are multiplied element-wise with the multi-scale features to obtain the weighted multi-scale features; the weighted multi-scale features are then summed to obtain the fused multi-scale features.

6. The multi-focus image fusion method based on a multi-scale image fusion network according to claim 2, characterized in that, In the feature reconstruction unit: Generate an initial decision map, which represents the probability that the first focused image has higher sharpness than the second focused image at each pixel location; Gaussian filtering is applied to the initial decision map, and threshold calculation is used to select pixels within a set threshold as boundary regions. The initial decision graph is smoothed using a guided filter to obtain a smoothed decision graph; By using the boundary region as the threshold region, the smoothed decision map is fused with the initial decision map to obtain the final decision map; The final decision image and the source image are weighted and averaged, and then multi-focus image fusion is performed to obtain the final fused image.

7. A multi-focus image fusion method based on a multi-scale image fusion network according to claim 2, characterized in that, In the efficient channel attention layer (ECA): the feature map is input into the first global average pooling layer for global average pooling operation, and the aggregated feature is output; the aggregated feature is input into the first convolutional layer for capturing local cross-channel information, and the initial channel weights are output; the initial channel weights are input into the first activation function layer for nonlinear compression operation to obtain channel weights; the input feature map and channel weights are multiplied element-wise to obtain the enhanced feature map.

8. The multi-focus image fusion method based on a multi-scale image fusion network according to claim 1, characterized in that, During the training of the multi-scale image fusion network: Obtain the original image set for training, process the original images using Gaussian filtering, and generate near-focus and far-focus images; Based on near-focus and far-focus images, construct near-focus image-far-focus image data pairs, and construct a training set based on the near-focus image-far-focus image data pairs; A gradient-aware loss function is constructed, and the training set is input into the constructed multi-scale image fusion network. The Adam optimizer is used to train the multi-scale image fusion network based on the gradient-aware loss function. During the training process, the parameters of the multi-scale image fusion network are updated through the backpropagation algorithm until the gradient-aware loss function converges, thus obtaining the trained multi-scale image fusion network.

9. A multi-focus image fusion method based on a multi-scale image fusion network according to claim 8, characterized in that, The expression for the gradient-aware loss function is: In the formula, This represents the gradient-aware loss function. This indicates Dice's loss. This indicates the loss of edge information. Indicates the weighting parameter; The expression for the Dice loss is: In the formula, This indicates the total number of pixels in the focused image. This indicates the probability that the first focused image is sharper than the second focused image at each pixel. This represents the value of the baseline decision map at the i-th pixel. The expression for the edge information loss is: In the formula, This represents the edge information retention value of pixel i in the first focused image. This represents the edge information retention value of pixel i in the second focused image. This represents the edge intensity of pixel i in the first focused image. This represents the edge intensity of pixel i in the second focused image. This represents the amount of information at the edge.

10. A multi-focus image fusion system based on a multi-scale image fusion network, used to implement the multi-focus image fusion method based on a multi-scale image fusion network as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used for multi-focus image fusion methods based on multi-scale image fusion networks; The network building module is used to construct multi-scale image fusion networks; The multi-scale image fusion network includes: a feature extraction unit for extracting multi-scale features from the first focused image and the second focused image, a feature integration unit for integrating multi-scale features, and a feature reconstruction unit for generating the final fused image. The training module is used to train the multi-scale image fusion network to obtain a trained multi-scale image fusion network. The fusion module is used to input the first focused image and the second focused image into the trained multi-scale image fusion network to perform multi-focus image fusion and obtain the final fused image.

Images