A multi-focus image fusion method based on deep learning

Abstract

The application discloses a kind of multi-focus image fusion methods based on deep learning, the method first converts training sample set and test sample set from RGB color space to YCbCr space, retain the Y channel image in it as training dataset and test dataset.Then according to Laplace operator, the edge information of multi-focus image pair is obtained, and the fusion image label is obtained according to the maximum selection strategy.Again, the multi-focus image pair is input into the generator fusion network to obtain the fusion image.In addition, according to the designed texture enhancement module, the high-frequency texture information of the source image is extracted, and the corresponding loss function is designed to optimize the fusion image.Finally, the fusion image label and the fusion image are input into the discriminator to identify true and false, and the generator is optimized according to the identification result.The application solves the problem of texture and edge information loss in the multi-focus image fusion process, and the fused image can be used for further image processing.

Description

Technical Field

[0001] This invention is a fusion method for generating a clear image using multiple images acquired with different focus settings. The aim is to create a fused image with a fully clear visual experience, while selectively retaining the detailed information contained in the multiple source images. The fused image can then be used for subsequent image processing tasks. Background Technology

[0002] Due to the characteristic of lenses being thicker in the middle and thinner at the edges, images captured by such lenses exhibit distortion at points far from the focal point, resulting in blurry images. Driven by the need for sharper images, multifocal image fusion has become an important topic in image processing. Due to the limitations of optical lenses, only objects near the focal point and within the depth of field can obtain a fully focused and sharp image; objects far from the focal point and outside the depth of field only produce blurry images. Currently, multifocal image fusion algorithms have been proposed to address this problem. These algorithms capture the sharp appearance of multiple source images with different focal points within the same scene and synthesize a fully focused image. The fused image can be applied to many fields such as photogrammetric visualization, object tracking, medical diagnosis, and remote sensing monitoring.

[0003] To date, image fusion can be divided into two categories: traditional fusion methods and deep learning-based fusion methods. Traditional image fusion algorithms focus on processing the transform domain and spatial domain of images. Transform domain-based algorithms first transform the image to different feature domains, then perform weighted fusion in these feature domains, and finally use the fused features to generate a fused image. Even images from different modalities share similar properties in their feature domains, making these algorithms suitable for fusion between multimodal images, such as infrared-visible light image fusion and CT-MR image fusion. Spatial domain-based algorithms first divide the input image into multiple small blocks or regions, then measure the saliency of each block, and finally fuse the most salient region into a new image. Spatial domain-based algorithms are also suitable for images of the same modality, such as multifocal image fusion. However, traditional image fusion algorithms inevitably suffer from poor versatility, low efficiency, and blurred edges. Deep learning algorithms, with their powerful feature representation capabilities, have been widely applied in the field of image fusion. Researchers can complete image fusion tasks by designing appropriate network structures and corresponding loss functions. Deep learning-based methods first use convolutional neural networks to automatically learn features in each image patch, learning based on the labels of segmented focused and defocused regions. The network parameters are then continuously optimized according to a designed loss function. Researchers have improved the quality and efficiency of image fusion by adjusting the network structure of deep networks to address different application scenarios, such as pixel-level fusion CNNs, encoder-decoder fusion networks, residual fusion networks, and end-to-end multifocal image fusion algorithms. While these methods achieve good fusion results, they still suffer from drawbacks such as high computational cost and network complexity.

[0004] Typical algorithms still have shortcomings in the following three aspects: (1) As the number of network layers increases, the gradient expansion or contraction effect accumulates, which can easily cause the model to fail to converge and affect the final fusion effect; (2) Due to the lack of perfect image reconstruction capability in the feature fusion process, the fused image contains redundant information; (3) Deep learning models need to calculate a large number of parameters and loss functions, resulting in high complexity and high computation time cost of image fusion algorithms. In order to solve the above problems, this patent proposes a multi-focus image fusion algorithm based on texture enhancement. The algorithm is based on generative adversarial network. First, it uses the texture enhancement module to extract the high-frequency information of the source image and designs a reasonable loss function; second, it uses a generator with a dual-channel mechanism to extract the features of the image to be fused; then, it performs concat fusion of the obtained features; finally, it inputs the generated fused image and the real image label into the discriminator for adversarial learning. Summary of the Invention

[0005] This invention proposes an improved algorithm for deep learning-based multifocal image fusion technology, which can fuse source images with different focal points into an image with a fully focused region and high texture features. The fused image can be used for image processing tasks such as image segmentation and object recognition.

[0006] The method of the present invention specifically includes the following steps:

[0007] Step 1: Convert the training and test sample images from RGB space to YCbCr space, and retain the Y channel images as the training and test datasets.

[0008] Step 2: Using the Laplace operator ( Laplacian The edge information of each image in the training dataset is obtained, and the fused image label I is obtained according to the maximum selection strategy. r The specific formula is as follows:

[0009] I r =Maximum(Laplacian(I1)+Laplacian(I2))

[0010] I1 and I2 are multifocal image pairs in the training dataset.

[0011] Step 3: Construct a generative adversarial network model, which includes a generator network and a discriminator network;

[0012] Step 4: Construct the texture enhancement module (ITEB) and input the multifocal image pairs I1 and I2 corresponding to the training dataset into the texture enhancement module to extract high-frequency information.

[0013] The image deep features are extracted through a texture enhancement module to obtain high-frequency texture information of the image.

[0014] The specific steps include: (1) constructing a texture enhancement model; (2) using convolution operations to obtain shallow feature information of the input image; (3) adding a channel attention mechanism to rescale the extracted feature channels and assign different weights to each channel; (4) adding a ReLU function to make the output of some network neurons zero, reducing the interdependence of parameters and alleviating overfitting; (5) adding residual connections to connect the extracted shallow information as input to the output of the next extraction stage; (6) continuously superimposing the above 5 steps to obtain the depth feature information of the source image.

[0015] Step 5: Incorporate texture and edge information, along with image structure, and design a reasonable loss function. The loss function can be divided into generator loss function and discriminator loss function, where generator loss includes content loss. Adversarial loss and SSIM loss Content loss is used to extract and reconstruct information, adversarial loss is used to enhance texture details, and SSIM loss is used to constrain the generator to produce images with a structure consistent with the real image. The generator loss function can be expressed as:

[0016]

[0017] The hyperparameters α and β are used to balance the three parameters to the same level, with α and β set to 100 and 0.01 respectively. The generator's content loss... It is the mean square error of pixels in the fused image and the input image, constraining the focused regions of the fused image and the source image to have the same intensity distribution and texture details, and is defined as:

[0018]

[0019] Where G(z) is the fused image, z represents the pixel distribution of the fused image, X1 and X2 are the input source images, i and j represent the pixel values ​​in the i-th row and j-th column of the gradient map or source image, and W and H are the width and length of the image, respectively. and This is the gradient map of the source image after texture enhancement, and its size is the same as the fused image. The generator's adversarial loss can further enhance the texture details of the fused image, and it is defined as:

[0020]

[0021] Where N represents the number of fused images during training, and a is the probability label that the generator expects the discriminator to identify in the fused image, which is set to 1 here. This indicates the operation of generating an image gradient map using the Laplacian operator. This adversarial game results in a more refined texture in the fused image.

[0022] Because mean squared error is very sensitive to large errors, simply using mean squared error loss may result in an overly smooth image without considering the overall structure of the image. To ensure that the fused image maintains the same structure as the source image, [the following is omitted as it is not directly related to the previous sentence]. Added to the loss function, it is defined as:

[0023]

[0024] Among them I f I1 and I2 represent the source images, and the SSIM function is used to calculate the brightness, contrast, and structural differences between the fused image and the source images. The calculation formula is as follows:

[0025]

[0026] Where μ and σ represent the mean and standard deviation of the entire image pixel matrix, respectively, and c1 and c2 are two minimal constants to prevent the denominator from being zero. The larger the SSIM value, the higher the structural similarity between the two.

[0027] The loss function of the discriminator enables it to accurately distinguish between real and fake data. The input to the discriminator is the fused image I generated by the generator. real_fused And the fused image I obtained based on the maximum selection principle and reconstruction fake_fused The loss function of the discriminator is specifically expressed as:

[0028]

[0029] Where N is the number of fused images during training, AVE is the pixel average function, b represents the probability that the discriminator should be able to identify real data (set to b=1), and c represents the probability that the discriminator should be able to identify fake data (set to c=0). Through this constraint, the discriminator can continuously improve its ability to distinguish between real and fake data, thus guiding the generator to produce fused images with strong textures.

[0030] Step 6: Input the multifocal image pairs I1 and I2 corresponding to the training dataset into the generator network to obtain the feature sets F1 = (x1, x2, ... x2) of I1 and I2. n F2 = (x1, x2, ... x) n Then, the generator's fused image I is obtained through concat fusion and reconstruction. f The specific formula is as follows:

[0031] I f =reconstruct(concat(F1,F2))

[0032] Step 7: Merge the generator's image I f With fused image label I r The input to the discriminator of the generative adversarial network is used for discrimination. Adversarial rules are established between the generator and the discriminator to optimize the fused image. Specific optimization steps include:

[0033] 7-1: The discriminator identifies whether the fused image is a real image;

[0034] 7-2: If not, then minimize the difference between the fused image and the fused image label through the loss function, and feed the identification result back to the generator. Based on the identification result, adjust the generator's fusion rules and optimize the fusion result.

[0035] 7-3: If so, the generator's fused image is the optimal fused image.

[0036] Step 8: Perform color space conversion on the merged grayscale image to obtain the final merged image.

[0037] The beneficial effects of this invention are:

[0038] The deep learning-based multifocal image fusion method disclosed in this invention employs a texture enhancement module to extract high-frequency texture information from the source image and designs a loss function that promotes rapid network convergence to adaptively adjust the generator's fusion rules, effectively reducing the loss of important information during feature extraction. Furthermore, adversarial rules are established between the generator and discriminator, and a loss function is designed to optimize the fusion result. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a deep learning-based multifocal image fusion method provided by the present invention.

[0040] Figure 2 This is a schematic diagram of the generator network structure in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the discriminator network in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the texture enhancement module in an embodiment of the present invention; Detailed Implementation

[0043] The invention will be further described below with reference to the accompanying drawings, clearly and meticulously illustrating the purpose, technical solution, and key points of the invention. Please refer to the appendix for the overall process of the invention. Figure 1 The specific steps are as follows:

[0044] Step 1: Convert the training and test sample sets from RGB space to YCbCr (Y: luminance component, Cb: blue chrominance component, Cr: red chrominance component) space, and retain the Y channel images as the training dataset Train_set and the test dataset Test_set.

[0045] Step 2: Use the Laplacian operator to obtain the edge information of multifocal image pairs in the training set, and obtain the fused image label I according to the maximum selection strategy. r .

[0046] Step 3: Input the multifocal images in the training set into the texture enhancement module to obtain the high-frequency texture information of the images, and design a loss function to optimize the generator.

[0047] Step 4: Input the multifocal image pairs from the training set into the generator network to obtain the fused image I. f For a detailed description of the generator network structure, please refer to the appendix. Figure 2 .

[0048] Step 5: Merge image label I r and fused image I f The input image is fed into the discriminator to obtain probability labels, and the discriminator determines whether the input image is a real image based on these labels. The specific discrimination process is as follows: If the discriminator determines that the input image is not a real image, it uses the loss function described in step 3 to reduce the difference between the input image and the real image, and feeds the discrimination result back to the generator. The generator adaptively adjusts the fusion rules based on the feedback result to optimize the fusion result. If the discriminator determines that the input image is a real image, then the fused image is the optimal result. For a detailed description of the discriminator's structure, please refer to the appendix. Figure 3 .

[0049] Furthermore, in step 3, the texture enhancement module calculation can be divided into three parts: image feature extraction, reconstruction, and design of the loss function.

[0050] 3-1. Deep feature extraction of images to obtain high-frequency texture information of images.

[0051] 3-1-1. Obtaining image edge information. First, use the Laplacian operator ( Laplacian Extracting edge information from the image's focal region:

[0052] image p =Laplacian(image)

[0053] Where image is the source image in the training dataset. p It is the edge information map after transformation by the Laplace operator.

[0054] 3-1-2. Extracting deep features from the image. First, use a 3×3 convolution kernel to extract deep features from the image. p The single-channel feature set Origin is obtained by converting it into 32 channels and then using a 3×3 convolution kernel to convert the 32 channels into 128 channels. The above convolution process is called the conv operation.

[0055] 3-1-3. Attention Weight Allocation. After channel extraction, the feature map corresponding to each channel is compressed into a real number, represented by the set {x1, x2, ... x}. 128 Then, weights are assigned to each feature channel using parameter W. W is learned to explicitly model the correlation between feature channels. The assigned weights are then applied to each of the original feature channels to obtain the channel attention set, Channel_set, which is represented as {w1, w2, ... w...}. 128}, combined with deep learning methods, can learn the importance of different channels.

[0056] 3-1-4. Residual Skip Connection. After assigning different weights to each feature channel, the feature set origin is added to the channel attention set Channel_Set to obtain the output of each layer of the texture enhancement module:

[0057] ITEB = Origin + Channel_set

[0058] ITB represents the result obtained from each layer of the texture enhancement module.

[0059] 3-1-5. Extract deeper feature maps from the image. Iterate through each layer of the texture enhancement block (ITEB) process to obtain the extraction result F, expressed by the following formula:

[0060] F = IITEB(ITEB(…ITEB(I p )))

[0061] 3-2. Reconstruct the image to restore it to its original size. Use a 1×1 convolutional kernel to restore the 128-channel feature map into a single-channel image. The specific formula is as follows:

[0062] F ITEB =constructor(F)

[0063] Where F ITEB Output of the texture enhancement module.

[0064] 3-3. Loss Function Design. Incorporating evaluation information from the texture enhancement module, a loss function is designed to promote rapid network convergence. The specific formula for the loss function is as follows:

[0065]

[0066] Where W and H are the width and height of the image, respectively, and l1 and l2 are multifocal image pairs in the training dataset, I f It is the fused image output by the generator. and This is the gradient map of the source image after enhancement by the texture enhancement module; its size is the same as the fused image. For detailed structure of the texture enhancement module, please refer to the appendix. Figure 4 .

[0067] Furthermore, in step 4, the generator model consists of two channels, and its calculation is divided into two parts: image feature extraction and image restoration.

[0068] 4-1. Extract features from the image.

[0069] 4-1-1. First layer feature extraction. First, a 1×1 convolution kernel is used to convert the source image from a single channel to 16 channels of features, and its output is represented as layer1.

[0070] 4-1-2. Second layer feature extraction. A 3×3 convolution kernel is used to convert the 16 channels of the first layer into 32 channels of features, which is represented as layer2.

[0071] 4-1-3. Fully Connected Layer. The outputs of the first and second layers are connected together as the input to the third layer. This can be represented as:

[0072] layer2_3=concat(layer1,layer2)

[0073] layer2_3 represents the output of the fully connected layer, which has 48 feature channels.

[0074] 4-1-4. Feature extraction in the third layer. The output of the fully connected layer is converted from 48 channels to 16 channels using a 3×3 convolutional kernel, which is represented as layer3.

[0075] 4-1-5. Fully Connected Layer. The outputs of the first three layers are connected together as the input of the fourth layer, which can be represented as:

[0076] layer3_4=concat(layer1,layer2,layer3)

[0077] layer3_4 is the output of the fully connected layer, which has 64 feature channels.

[0078] 4-1-6. Fourth layer for feature extraction. A 3×3 convolutional kernel is used to convert the output of the fully connected layer from 64 channels to 16 channels, which is represented as layer4.

[0079] 4-1-7. Fully Connected Layer. The outputs of the first four layers are concatenated and used as the input to the fusion layer, which is represented as:

[0080] layer4_5=concat(layer1,layer2,layer3,layer4)

[0081] layer4_5 represents the output of the fully connected layer, which has 128 feature channels.

[0082] 4-2. Image Restoration. A 1×1 convolution kernel is used to transform the image from 128 channels to a single channel, denoted as layer5. The hyperbolic tangent function tanh is then applied to the final single-channel image. The tanh function is specifically expressed as:

[0083]

[0084] The generator's output is represented as:

[0085] I f =tanh(layer5)

[0086] Furthermore, in step 5, the discriminator consists of four convolutional layers and one linear layer, and its specific steps are as follows:

[0087] 5-1. Extract the image features used for identification.

[0088] 5-1-1. First, use a 3×3 convolution kernel to convert the input image from a single channel to 16 channels;

[0089] 5-1-2. Use a 3×3 convolution kernel to convert the input features from 16 channels to 32 channels;

[0090] 5-1-3. Use a 3×3 convolution kernel to convert the input features from 32 channels to 64 channels;

[0091] 5-1-4. Use a 3×3 convolution kernel to convert the input features from 64 channels to 128 channels;

[0092] 5-1-5. Use a 3×3 convolution kernel to convert the input features from 128 channels to 256 channels, and obtain the height H of the output features;

[0093] 5-2. Linear layer, to obtain the probability labels of the input image.

[0094] 5-2-1. First, the dimensions of the output feature channels are readjusted to 32, with each channel having a size of H×H×256. The output can be represented as reshape_var.

[0095] 5-2-2. Set the normalization matrix recover, which is represented as:

[0096] recover = [H × H × 256, 1]

[0097] 5-2-3. Finally, multiply the readjusted feature channels by the normalization matrix to obtain the probability distribution labels:

[0098] Probability=reshape_var×recover

[0099] Probability represents the output of the discriminator.

[0100] Experimental Results Analysis

[0101] To more intuitively demonstrate the advantages of this invention, the algorithm used in this invention is compared with five state-of-the-art fusion algorithms, including BF, GRW, Quadtree, CNN, and MFF-GAN. Among them, BF, GRW, and Quadtree are traditional fusion algorithms, while CNN and MFF-GAN are based on deep learning methods.

[0102] To objectively evaluate the image fusion algorithm used in this invention, six popular statistical metrics were selected as objective indicators for measuring the fusion results. These include the average gradient (Q). AG Information entropy (Q) EN ), spatial frequency (Q) SF Gradient-based fusion performance (Q) AB|F ), visual fidelity (Q) VIF ), the sum of differential correlations (Q) SCD The objective comparison results are shown in the table below:

[0103] Comparison results of this invention with other algorithms

[0104]

[0105] Explanation of indicators in the table:

[0106] Q AG Q is the average value of the image gradient, used to measure the sharpness of the fused image. AG The larger the value, the clearer the image. Q AG The definition is as follows:

[0107]

[0108] Where M and N are the height and width of the image, I f It is a fused image, where i and j represent the i-th row and j-th column of the image.

[0109] Q EN Used to measure the information content of an image. Q EN The larger the value, the more information the fused image contains. Q EN The definition is as follows:

[0110]

[0111] Where p i It is the normalized probability corresponding to the grayscale value.

[0112] Q SF Used to measure the structural texture of the fused image. Q SF The larger the value, the richer the edges and textures of the image. SF The definition is as follows:

[0113]

[0114] Where RF and CF represent row frequency and column frequency, respectively, and their definitions are as follows:

[0115]

[0116]

[0117] Q AB|F Used to measure the degree of edge information preserved from the source image to the fused image. Q AB|F The larger the value, the more edge information is preserved. AB|F The definition is as follows:

[0118]

[0119] in and These are the edge intensity and direction values ​​at position (i,j) in the image, respectively, w A and w B These are the weight values ​​of the fused image corresponding to the two source images.

[0120] Q VIF It is used to measure the information fidelity of fused images, similar to the human visual system. First, Q... VIF The source and fused images are filtered and divided into different blocks. Each block is then evaluated for any distorted visual information. The visual fidelity of each block is then calculated. Finally, the overall visual fidelity is calculated.

[0121] Q SCD It is used to measure the correlation between source image information and fused image information, and can evaluate the pseudo-information contained in the fused image. Q SCD The larger the value, the better the fusion performance and the less false information it contains. SCD The definition is as follows:

[0122] Q SCD =r(I1,I f )+r(I2,I f )

[0123] Where I1 and I2 are the source images, I f It is a fused image, and r(·) is the correlation coefficient function, defined as follows:

[0124] .

Claims

1. A multifocal image fusion method based on deep learning, characterized in that... Includes the following steps: Step 1: Convert the training and test sample sets from RGB color space to YCbCr color space, retaining the Y channel images as the training and test datasets; Step 2: Use the Laplacian operator to obtain edge information of multifocal image pairs in the training set. And obtain the fused image labels according to the maximum selection strategy. ; Step 3: Construct a deep convolutional neural network model and a texture enhancement module; the deep convolutional neural network adopts a generative adversarial network architecture, including a generator and a discriminator; the generator uses a dual-channel approach to extract features, and the input is multifocal image pairs from the training dataset. and The output is a fused image. The input to the discriminator is the fused image. and merge image tags The output is a probability label; adversarial rules are established between the generator and the discriminator to optimize the fused image; deep features of the image are extracted through the texture enhancement module to obtain high-frequency texture information of the image; The specific steps include: (1) constructing a texture enhancement model; (2) using convolution operations to obtain shallow feature information of the input image; (3) adding a channel attention mechanism to rescale the extracted feature channels and assign different weights to each channel; (4) adding a ReLU function to make the output of some network neurons zero, reducing the interdependence of parameters and alleviating overfitting; (5) adding residual connections to connect the extracted shallow information as input to the output of the next extraction stage; (6) continuously superimposing the above 5 steps to obtain the depth feature information of the source image. Step 4: Reconstruct the image to restore it to its original size; and use the texture enhancement module described in Step 3 to obtain the high-frequency texture information of the test sample, and design the corresponding loss function based on the high-frequency texture information; The specific formula for the loss function is as follows: in and These are the width and height of the image, respectively. f1 and f2 are the gradient maps of the source image after being enhanced by the texture enhancement module, and their size is the same as that of the fused image. Step 5: Train the deep convolutional neural network model described in Step 3 using the training dataset to obtain the trained deep convolutional neural network model; Step 6: Based on the trained deep convolutional neural network model, obtain the deep feature set of the training dataset, and perform concat fusion on the deep feature set to obtain the fused grayscale image. Step 7: Perform color space conversion on the merged grayscale image to obtain the final merged image.

2. The deep learning-based multifocal image fusion method according to claim 1, characterized in that: The reconstructed image restores the image to its original size; specifically: A 1×1 convolution kernel is used to restore a 128-channel feature map into a single-channel image.

3. The deep learning-based multifocal image fusion method according to claim 1, characterized in that: fused images and merge image tags The input is fed into the discriminator for discrimination and differentiation. Adversarial rules are established between the discriminator and the generator to optimize the fused image. The steps include: (1) If the discriminator discriminates the fused image If the image is fake, then the fused image is minimized using the loss function described in step 4. and merge image tags The difference between them is identified, and the identification result is fed back to the generator, and the generator's fusion rules are adaptively adjusted to optimize the fused image; (2) If the discriminator identifies the fused image If it is a real image, then merge the images. The optimal fused image.

4. The deep learning-based multifocal image fusion method according to claim 1, characterized in that: Step 3 involves extracting deep features from the image using a texture enhancement module to obtain high-frequency texture information, including: 3-1. Extract deep features from the image; First, use a 3×3 convolution kernel to... The single-channel feature set is converted to 32 channels, and then a 3×3 convolution kernel is used to convert the 32 channels to 128 channels, resulting in the feature set Origin. 3-2. Attention allocation weights; After channel extraction, the feature map corresponding to each channel is compressed into a real number, and its set is represented as follows: Then, weights are assigned to each feature channel using parameter W. W is learned to explicitly model the correlation between feature channels. The assigned weights are then applied to each of the original feature channels to obtain the channel attention set, Channel_set, which is represented as follows: In conjunction with deep learning methods, the importance of different channels is learned; 3-3. Residual jump connection; After assigning different weights to each feature channel, the feature set origin is added to the channel attention set Channel_set to obtain the output of each layer of the texture enhancement module: in It is the result obtained from each layer of the texture enhancement module; 3-4. Extract deeper feature maps from the image; The extraction result F is obtained by iterating through each layer of the texture enhancement block (ITEB).

Images