A method for fusing multi-focus image stacks and a computer-readable storage medium

Abstract

This invention discloses a fusion method for multi-focus image stacks and a computer-readable storage medium. The method includes acquiring high-dimensional features of different layers of images in the input multi-focus image stack and generating high-dimensional feature maps; calculating the pixel-level average spatial frequency gradient matrix of all high-dimensional feature maps for each layer; calculating the high-dimensional spatial frequency gradient response of pixels with the same coordinates in each image of the input multi-focus image stack and generating a decision matrix; and generating the final fused image based on the decision matrix. This invention introduces gradient loss and spatial frequency loss into the loss function, constraining the overall spatial frequency features of the image during encoder training, thus making the encoder output more accurate. The encoder of this invention can be used to efficiently and conveniently complete multi-focus fusion tasks of image stacks with different numbers, resolutions, and categories. Furthermore, the final fusion only requires generating a single decision matrix, greatly improving fusion efficiency and simplifying the fusion process.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for fusing multi-focus image stacks and a computer-readable storage medium. Background Technology

[0002] Due to the physical limitations of optical lenses, only areas within the depth of field can be clearly imaged. Multifocal image fusion technology can fuse multiple images with different focal parts into a complete and clear image. Therefore, image fusion, as an effective technique, has broad application prospects in various fields. In biomedical microscopy, image fusion technology can integrate Z-axis information from cells in different imaging layers, which is crucial for biomedical observation. In optical engineering, it can be used for post-processing of imaging results from light field cameras, underwater microscopes, etc. In industry, it can be used for structural measurement of nonwoven fabrics and to improve the quality of printed circuit board microscopic images.

[0003] Traditional multifocal image fusion methods can be divided into two categories: transform domain methods and spatial domain methods. Transform domain methods achieve good fusion results by transforming the original image to another domain (such as the frequency domain), such as pyramid-based methods, wavelet-based methods, non-subsampled contour transform methods, and sparse representation methods. Transform domain-based image fusion methods are prone to altering image contrast, leading to color differences, and their processing is relatively slow. Spatial domain methods focus on pixel-level operations on the original image, are simple, intuitive, and fast. These methods mainly fall into three categories: block-based methods, region-based methods, and pixel-based methods. Spatial domain-based image fusion methods are more susceptible to noise interference, leading to misfusion, and they largely rely on manually designed image activity level measurements and fusion strategies. Furthermore, these two traditional multifocal fusion methods heavily depend on manually designed image activity level measurements and fusion strategies, and require extensive image processing experience. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for fusing multi-focus image stacks and a computer-readable storage medium.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for fusing multi-focus image stacks, comprising:

[0006] Step 1: Obtain the high-dimensional features of different layers of the input multi-focus image stack I and generate high-dimensional feature maps. Calculate the pixel-level average spatial frequency gradient matrix of all high-dimensional feature maps for each layer, and calculate the pixel-level average spatial frequency gradient matrix φ of the image at layer i. i Expressed as:

[0007]

[0008] in, Let c be the high-dimensional feature matrix of the i-th layer image.

[0009]

[0010] Where m is the row width of the image at layer i, n is the column width of the image at layer i, and SF i Let (x, y) be the pixel-level spatial frequency of the image with layer i, and (x, y) be the coordinate index, where x ranges from [0, m-1] and y ranges from [0, n-1].

[0011] Step 2: Calculate the high-dimensional spatial frequency gradient response of pixels with the same coordinates in each image of the input multi-focus image stack I, and generate a decision matrix. The index of each element in the decision matrix corresponds to the layer index position in the input multi-focus image stack I. The decision matrix is ​​represented as:

[0012]

[0013] Where, Φ ini For the decision matrix,

[0014]

[0015] in, Φ is the decision matrix ini The element with coordinate (x, y) is the element at coordinate (x, y), max{.} is the maximum value function, N is the number of layers in the multi-focus image stack, and the value of i is in the range of [0, N-1].

[0016] Step 3: Generate the final fused image based on the decision matrix, as follows:

[0017]

[0018] Among them, O fusion For the generated fused image, I(i) (x，y) The pixel value at coordinates (x, y) in the image of layer i of the input multifocus image stack I.

[0019] Furthermore, the high-dimensional features are obtained based on an encoder trained through unsupervised learning. The encoder is trained by introducing a decoder and a loss function during the training phase. The decoder is used to reconstruct the image stack O based on the high-dimensional features output by the encoder.

[0020] Furthermore, the loss function is expressed as:

[0021]

[0022] Among them, L i Let I be the loss function for the input multifocus image stack I and the reconstructed image stack O output by the decoder, where the image at layer i is located. Let I be the mean square error of the image with layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder. The structural similarity loss is calculated between the input multifocus image stack I and the reconstructed image stack O output by the decoder for the image at layer i. The gradient loss is calculated for the input multifocus image stack I and the reconstructed image stack O output by the decoder, where the image at layer i is the gradient loss. Let λ be the spatial frequency loss of the image with layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder. λ, η, α, and β are all weighting coefficients, and the sum of λ, η, α, and β is 1.

[0023] Furthermore, the mean square error of the image at layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder is... The calculation method is as follows:

[0024]

[0025] Among them, O(i) (x，y) The pixel value at coordinates (x, y) of the image in layer i of the reconstructed image stack O output by the decoder.

[0026] Furthermore, the structural similarity loss between the input multifocus image stack I and the image at layer i in the reconstructed image stack O output by the decoder... The calculation method is as follows:

[0027]

[0028] Wherein, SSIM(O(i), I(i)) is the structural similarity index between the input multifocus image stack I and the image with layer i in the reconstructed image stack O output by the decoder, including the brightness, contrast and structural information of the two images;

[0029]

[0030] Where, μ I (i) represents the average pixel value of the image with layer i in the input multifocus image stack I, μ O(i) μ is the average pixel value of the image with layer i in the reconstructed image stack O output by the decoder. O(i)I(i) C1 and C2 are the standard deviations of the pixels in the input multifocus image stack I and the reconstructed image stack O output by the decoder, where layer i is the image.

[0031] Furthermore, the gradient loss of the image at layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder... The calculation method is as follows:

[0032]

[0033] Among them, G I(i) G is a combination of gradient information in the horizontal and vertical directions of the high-dimensional feature map of the image with layer i in the input multifocus image stack I. O(i) This is a combination of gradient information in the horizontal and vertical directions of the high-dimensional feature map of the image with layer i in the reconstructed image stack O output by the decoder.

[0034]

[0035] Where K = O, I, The gradient information in the horizontal direction of the high-dimensional feature map of the image with layer i in the multifocus image stack. This refers to the gradient information in the vertical direction of the high-dimensional feature map of the image with layer i in the multifocus image stack.

[0036]

[0037]

[0038] Where H(x, y) is the feature response at coordinates (x, y) of the high-dimensional feature map.

[0039] Furthermore, the spatial frequency loss of the image at layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder... The calculation method is as follows:

[0040]

[0041] SF K(i) The spatial frequency of the image with layer i.

[0042]

[0043] Among them, RF K(i) For the row frequency of the high-dimensional feature map of the image with layer i in the multifocus image stack, CF K(i) The column frequencies of the high-dimensional feature map of the image with layer i in the multifocus image stack.

[0044]

[0045]

[0046] Furthermore, the pixel-level spatial frequency of the image of layer i at coordinates (x, y) for:

[0047]

[0048] Where r is the calculation radius, Let be the pixel-level row frequency of the image at coordinates (x, y) for layer i. The pixel-level column frequency of the image at coordinates (x, y) for layer i;

[0049]

[0050]

[0051] The values ​​of a and b are both in the range of -r to r.

[0052] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0053] Beneficial effects: 1. This invention introduces gradient loss and spatial frequency loss into the loss function, which constrains the overall spatial frequency characteristics of the image during encoder training, making the encoder output more accurate;

[0054] 2. The encoder of the present invention can be used to efficiently and conveniently complete multi-focal-length fusion tasks of image stacks with different numbers, resolutions and categories. In addition, the final fusion only needs to generate a decision matrix, which greatly improves the fusion efficiency and simplifies the fusion process. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of a multi-focus image stack fusion method;

[0056] Figure 2 This is a schematic diagram of the encoder and decoder in an embodiment of the present invention;

[0057] Figure 3 This is a diagram illustrating the changes in pixel sharpness. Detailed Implementation

[0058] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0059] like Figure 1 As shown, this embodiment of the invention provides a fusion method for a multi-focus image stack, including:

[0060] Step 1: Obtain the high-dimensional features of different layers of the input multi-focus image stack I and generate high-dimensional feature maps. Calculate the pixel-level average spatial frequency gradient matrix of all high-dimensional feature maps for each layer, and calculate the pixel-level average spatial frequency gradient matrix φ of the image at layer i. i Expressed as:

[0061]

[0062] in, Let c be the high-dimensional feature matrix of the i-th layer image.

[0063]

[0064] Where m is the row width of the image at layer i, n is the column width of the image at layer i, and SF i Let (x, y) be the pixel-level spatial frequency of the image with layer i, and (x, y) be the coordinate index, where x ranges from [0, m-1] and y ranges from [0, n-1].

[0065] The above layer represents the pixel-level spatial frequency of image i at coordinates (x, y). for:

[0066]

[0067] Where r is the calculation radius, Let be the pixel-level row frequency of the image at coordinates (x, y) for layer i. The pixel-level column frequency of the image at coordinates (x, y) for layer i;

[0068]

[0069]

[0070] Where H(x, y) is the feature response at coordinate (x, y) of the high-dimensional feature map, and the values ​​of a and b are both in the range of -r to r.

[0071] Step 2: Calculate the high-dimensional spatial frequency gradient response of pixels with the same coordinates in each image of the input multi-focus image stack I, and generate a decision matrix. The index of each element in this decision matrix corresponds to the layer index position in the input multi-focus image stack I. After calculating the pixel-level spatial frequency gradient of different layers, in order to fuse pixels at different positions in the image stack, it is desirable to obtain the sharpest pixel from images along different Z-axis. If the Z-axis images are acquired continuously, the variation in pixel sharpness in the images may be as follows: Figure 3As shown. This invention combines neural networks and traditional image algorithms to find relatively good or optimal choices for each pixel in an image stack, and uses these selected pixels to generate a decision matrix. The decision matrix can be expressed as:

[0072]

[0073] Where, Φ ini For the decision matrix,

[0074]

[0075] in, Φ is the decision matrix ini The element with coordinates (x, y) is defined by max{.}, which is the maximum value function. N is the number of layers in the multifocus image stack, and the value of i above ranges from [0, N-1].

[0076] Step 3: Generate the final fused image based on the decision matrix, as follows:

[0077]

[0078] Among them, O fusion For the generated fused image, I(i)(x, y) is the pixel value at coordinates (x, y) in layer i of the input multi-focus image stack I. Correspondingly, This refers to the layers of the input multifocal image stack I. The pixel value at coordinates (0,0) in the image.

[0079] The aforementioned high-dimensional features are acquired based on an encoder trained using unsupervised learning. During the training phase, the encoder is trained by introducing a decoder and a loss function. The decoder is used to reconstruct the image stack O based on the high-dimensional features output by the encoder. To calculate the spatial frequency gradient of these high-dimensional features, the key is to establish a network that can support high-resolution image processing while preventing information loss during network propagation. Specifically, as follows... Figure 2As shown, to preserve the original spatial features of the image to the greatest extent, this invention does not use any downsampling layers. During deep encoding, due to the dense connections between layers, the deep network can still utilize image features from lower layers, which improves computational efficiency and overcomes problems such as gradient vanishing. To increase the weights of frequency-related channels, a coordinated attention module is used in each convolutional layer, as it can work in conjunction with the spatial-frequency loss function to extract more useful image features for subsequent computations. The decoder consists of several convolutional layers with decreasing dimensions to reconstruct the image. During network training, each image in the image stack is fed into the encoder one by one, and its high-dimensional features are calculated. Then, the decoder decodes each image, reconstructing the original image from the high-dimensional feature map. After training, in the fusion stage, the image stack to be fused is simply fed into the trained encoder to obtain the encoding result, which is then used for subsequent computations.

[0080] Specifically, the loss function described above is used to drive the encoder's output to approximate the input image. The loss function eventually approaches 0, indicating that the input and output are approximately the same. The loss function is expressed as:

[0081]

[0082] Among them, L i Let I be the loss function for the input multifocus image stack I and the reconstructed image stack O output by the decoder, where the image at layer i is located. Let I be the mean square error of the image with layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder. The structural similarity loss is calculated between the input multifocus image stack I and the reconstructed image stack O output by the decoder for the image at layer i. The gradient loss is calculated for the input multifocus image stack I and the reconstructed image stack O output by the decoder, where the image at layer i is the gradient loss. Let be the spatial frequency loss of the image at layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder. λ, η, α, and β are weighting coefficients, and the sum of λ, η, α, and β is 1. The mean square error is... and structural similarity loss The combination of these loss functions is a common feature in image reconstruction; however, it has several limitations: 1. It does not specifically consider the spatial frequency information of the entire image. 2. The encoder output may be unreliable due to image noise. To accurately encode spatial frequencies and their gradients, this invention adds gradient loss to the loss function. and spatial frequency loss

[0083] Specifically, the mean square error of the image at layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder is... The calculation method is as follows:

[0084]

[0085] Among them, O(i) (x，y) The pixel value at coordinates (x, y) of the image layer i in the reconstructed image stack O output by the decoder.

[0086] The structural similarity loss between the input multifocus image stack I and the reconstructed image stack O output by the decoder for the image with layer i is calculated. The calculation method is as follows:

[0087]

[0088] Wherein, SSIM(O(i), I(i)) is the structural similarity index between the input multifocus image stack I and the image with layer i in the reconstructed image stack O output by the decoder, including the brightness, contrast and structural information of the two images;

[0089]

[0090] Where, μ I(i) μ is the average pixel value of the image with layer i in the input multifocus image stack I. O(i) σ is the average pixel value of the image with layer i in the reconstructed image stack O output by the decoder. O(i)I(i) C1 and C2 are the standard deviations of the pixels in the input multifocus image stack I and the reconstructed image stack O output by the decoder, where layer i is the image.

[0091] The gradient loss of the image at layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder. The calculation method is as follows:

[0092]

[0093] Among them, C I(i) G is a combination of gradient information in the horizontal and vertical directions of the high-dimensional feature map of the image with layer i in the input multifocus image stack I. O(i) This is a combination of gradient information in the horizontal and vertical directions of the high-dimensional feature map of the image with layer i in the reconstructed image stack O output by the decoder.

[0094]

[0095] Where K = O, I, The gradient information in the horizontal direction of the high-dimensional feature map of the image with layer i in the multifocus image stack. This refers to the gradient information in the vertical direction of the high-dimensional feature map of the image with layer i in the multifocus image stack.

[0096]

[0097]

[0098] Spatial frequency loss of the image at layer i in the input multifocus image stack I and the reconstructed image stack O output by the decoder. The calculation method is as follows:

[0099]

[0100] SF K(i) The spatial frequency of the image with layer i.

[0101]

[0102] Among them, RF K(i) For the row frequency of the high-dimensional feature map of the image with layer i in the multifocus image stack, CF K(i) The column frequencies of the high-dimensional feature map of the image with layer i in the multifocus image stack.

[0103]

[0104]

[0105] Based on the above embodiments, those skilled in the art can easily understand that the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0106] The above description is merely a preferred embodiment of the present invention. It should be noted that for those skilled in the art, other parts not specifically described are existing technology or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A fusion method for multi-focus image stacks, characterized in that, include: Step 1: Obtain the input multifocal image stack The high-dimensional features of images from different layers are analyzed and high-dimensional feature maps are generated. The pixel-level average spatial frequency gradient matrix of all high-dimensional feature maps for each layer is calculated sequentially. The layers are... The pixel-level average spatial frequency gradient matrix of the image Represented as: ； in, For the first The c-th high-dimensional feature matrix of the layer image, ； Where m is the layer The row width of the image, where n is the layer width. The column width of the image, For layer as The pixel-level spatial frequency of the image. For coordinate index, The value range of is [0, m-1]. The value range is [0, n-1]; Step 2: Calculate the input multifocal image stack The high-dimensional spatial frequency gradient responses of pixels with the same coordinates in each image are used to generate a decision matrix. The index of each element in the decision matrix corresponds to the input multi-focus image stack. The layer index position in the matrix, the decision matrix is ​​represented as: ； in, For the decision matrix, ； in, Decision matrix The upper coordinate is elements, The function is used to find the maximum value, where N is the number of layers in the multifocal image stack. The value range is [0, N-1]; Step 3: Generate the final fused image based on the decision matrix, as follows: ； in, For the generated fused image, For input multifocal image stacks The layer is The coordinates in the image are Pixel values; The layer is The image in coordinates pixel-level spatial frequency for: ； Where r is the calculation radius, For layer as The image in coordinates Pixel-level line frequency at the location For layer as The image in coordinates Pixel-level column frequency at the location; ； ； Where the values ​​of a and b both range from -r to r, The coordinates of the high-dimensional feature map are The characteristic response at that location.

2. The fusion method for a multi-focus image stack according to claim 1, characterized in that, The high-dimensional features are obtained based on an encoder trained through unsupervised learning. During the training phase, the encoder is trained by introducing a decoder and a loss function. The decoder is used to reconstruct the image stack based on the high-dimensional features output by the encoder. .

3. The fusion method for a multi-focus image stack according to claim 2, characterized in that, The loss function is expressed as: ； in, For the input multifocal image stack Reconstructed image stack with decoder output The middle layer is The loss function for the image, For the input multifocal image stack Reconstructed image stack with decoder output The middle layer is The mean square error of the image. For the input multifocal image stack Reconstructed image stack with decoder output The middle layer is Image structural similarity loss, For the input multifocal image stack Reconstructed image stack with decoder output The middle layer is Gradient loss of the image, For the input multifocal image stack Reconstructed image stack with decoder output The middle layer is Spatial frequency loss of the image , , , All are weighting coefficients, and , , , The sum of is 1.

4. The fusion method for a multi-focus image stack according to claim 3, characterized in that, The input multifocus image stack Reconstructed image stack with decoder output The middle layer is Mean square error of the image The calculation method is as follows: ； in, The reconstructed image stack output by the decoder The middle layer is The image coordinates are The pixel value.

5. The fusion method for a multi-focus image stack according to claim 4, characterized in that, The input multifocus image stack Reconstructed image stack with decoder output The middle layer is Image structural similarity loss The calculation method is as follows: ； in, For the input multifocal image stack Reconstructed image stack with decoder output The middle layer is The structural similarity index of the images includes the brightness, contrast and structural information of the two images; ； in, For the input multifocal image stack The middle layer is The average pixel value of the image. The reconstructed image stack output by the decoder The middle layer is The average pixel value of the image. For the input multifocal image stack Reconstructed image stack with decoder output The middle layer is The standard deviation of the pixels in the image, where C1 and C2 are constants.

6. The fusion method for a multi-focus image stack according to claim 3, characterized in that, The input multifocus image stack Reconstructed image stack with decoder output The middle layer is Gradient loss of the image The calculation method is as follows: ； in, For the input multifocal image stack The middle layer is The high-dimensional feature map of the image combines gradient information in the horizontal and vertical directions. The reconstructed image stack output by the decoder The middle layer is The high-dimensional feature map of the image combines gradient information in the horizontal and vertical directions. ； Where K = O, I, For the multi-focus image stack, the layers are The high-dimensional feature map of the image contains gradient information in the horizontal direction. For the multi-focus image stack, the layers are The gradient information in the vertical direction of the high-dimensional feature map of the image. ； 。 7. The fusion method for a multi-focus image stack according to claim 6, characterized in that, The input multifocus image stack Reconstructed image stack with decoder output The middle layer is Spatial frequency loss of images The calculation method is as follows: ； For layer as The spatial frequency of the image, ； in, For the multi-focus image stack, the layers are The row frequency of the high-dimensional feature map of the image. For the multi-focus image stack, the layers are Column frequencies of the high-dimensional feature map of the image. ； 。 8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images