Multi-focus microscopic image sequence fusion method based on diffusion model

The multi-focus microscopic image sequence fusion method using a diffusion model addresses noise interference and detail loss by enhancing feature extraction and reconstruction, achieving improved fusion quality and robustness.

CN119067866APending Publication Date: 2024-12-03SHANXI UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411107190.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing multi-focus image fusion methods, including transform domain, spatial domain, and deep learning methods, face limitations in capturing detailed information and are susceptible to noise interference, especially in microscopic image sequences, and struggle with complex scene fusion tasks.

Method used

A multi-focus microscopic image sequence fusion method based on a diffusion model that involves adjusting focal planes to capture a sequence of images, adding Gaussian noise, and using a series of sequence and diffusion operations to enhance feature extraction and reconstruction, ultimately producing a fully focused fused image.

Benefits of technology

The method effectively retains detailed information and enhances noise robustness, suitable for few-sample learning scenarios, overcoming limitations of traditional methods and improving fusion quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BDA0004991159910000025
    Figure BDA0004991159910000025
  • Figure BDA0004991159910000029
    Figure BDA0004991159910000029
  • Figure BDA0004991159910000036
    Figure BDA0004991159910000036
Patent Text Reader

Abstract

The invention discloses a multi-focus microscopic image sequence fusion method based on a diffusion model, and the method comprises the steps: firstly initializing a noise image which has the same tensor as a source image and obeys Gaussian distribution at a T moment, then splicing the initialized image and a multi-focus image sequence, inputting the spliced image and the multi-focus image sequence into a feature extraction network, and predicting the noise of the previous step; and calculating a noise picture at the T-1 moment according to the predicted noise and the initialized picture. And by parity of reasoning, a fused full-focus image can be obtained in T time steps. According to the method, the problems of noise interference, detail information loss and the like in an existing multi-focus image fusion method are solved, and meanwhile, an image fusion task of few-sample learning can be effectively realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the image fusion technology in the field of computer vision, and specifically relates to a multi-focus microscopic image sequence fusion method based on a diffusion model. Background Art

[0002] Due to the small scale of microscopic objects, in order to clearly observe the detailed information of microscopic objects, a large aperture lens is required for imaging, which will cause the depth of field of a single imaging process to become smaller, resulting in only a partial area of ​​a single microscopic image being focused. In order to observe all the clear details of the microscopic objects, it is necessary to effectively synthesize microscopic image sequences with different focusing levels. Multi-focus image fusion technology can effectively solve the above problems. In addition, this technology is also widely used in medical image processing, remote sensing images, transportation and other fields.

[0003] Multi-focus image fusion methods are mainly divided into three categories: transform domain, spatial domain, and deep learning. The transform domain method first uses the time-frequency transform tool to map the multi-focus image sequence to the frequency domain, then performs high- and low-frequency decomposition and fusion operations on the image in the frequency domain, and finally obtains the final fusion result through inverse transform. Common transform domain methods include wavelet transform fusion, discrete cosine transform fusion, etc., but these methods can only capture high-frequency detail information in a specific direction, which easily leads to the loss of detail information in the fusion result; the spatial domain method first calculates the clarity of each single pixel in the multi-focus image sequence, and then selects the clearest pixel value in the multi-focus image sequence as the final fusion image based on the clarity evaluation result. Although this type of method has high computational efficiency, it is prone to artifacts and distortion in the process of dealing with complex scene fusion problems. In summary, the above two types of methods basically use pre-set focus evaluation and fusion criteria for image fusion. Due to the limitations of focus level design and the subjectivity of fusion rule judgment, these methods cannot meet the needs of different application scenarios and image content, resulting in unsatisfactory fusion effects. Therefore, mainstream image fusion methods mainly focus on deep learning methods. Currently, deep learning methods mainly include two categories: decision graph-based methods and end-to-end deep network models. Among them, the decision graph-based method first uses a deep network model to extract the focus features of the image, then builds a decision graph based on the extracted focus features, and finally selects the best pixel value of each pixel in the multi-focus image for fusion according to the information in the decision graph. Due to the decision of the pixel in the spatial domain, this type of method will have a pseudo-Gibbs effect; the end-to-end deep network model-based method uses a deep network model to extract image focus features, and directly maps the multi-focus image sequence to a fused image through an encoder-decoder structure. This type of method mainly relies on the powerful feature representation capabilities of network models such as convolutional neural networks or Transformer, but because this type of method introduces more gain modules (such as attention mechanisms, residual blocks, multi-scale feature extraction, etc.) to improve performance, as the complexity of the network increases, the performance improvement of the model gradually tends to saturation, and the performance improvement space becomes more and more difficult.

[0004] Through the above research status analysis, we believe that the current multi-focus microscopic image sequence fusion faces the following problems: the applicability of traditional time domain and frequency domain multi-focus image fusion methods is limited by the focus evaluation method and the setting of fusion rules; and deep learning image fusion methods currently only fuse two images. If it is extended to the task of multi-microscopic image fusion, it will lead to an increase in model complexity, and the existing model cannot effectively capture the detail information of microscopic images with different focus levels during the fusion process. Therefore, this patent designs a multi-focus microscopic image sequence fusion method based on a diffusion model, which can not only retain more detail information of microscopic images, but also has strong noise robustness. Summary of the invention

[0005] In order to overcome the shortcomings of the existing solutions, the present invention aims to provide a multi-focus microscopic image sequence fusion method based on a diffusion model, comprising the following steps:

[0006] Step 1: Change the focal plane by adjusting the focal length of the microscope to obtain a multi-focus microscopic image sequence covering the entire depth of field of the scene Where N represents the total number of image sequences, n represents the position of the image sequence under a certain focal plane, and its value range is 1≤n≤N;

[0007] Step 2: In the forward noise addition process, given the fully focused image X0 of the scene, add Gaussian noise using equation (1):

[0008]

[0009] Among them, X t Yes X t-1 The noisy version of β t Represents a fixed sequence of values ​​used to control the noise intensity t represents the number of diffusion steps, and its value range is 1≤t≤T, T is the maximum number of diffusion steps, ε is a Gaussian distribution with a mean of 0 and a variance of 1, let α t =1-β t , Formula (1) can be used to obtain formula (2);

[0010]

[0011] Step 3: Initialize the noise image X at time T using formula (3): T ,

[0012] X T =randn(shape(X1)) (3)

[0013] Where X1 represents the multi-focus image sequence In the first picture, shape() is a function to obtain the image shape, and randn() is a function to generate a random array that obeys Gaussian distribution;

[0014] Step 4: According to formula (4), the multi-focus image sequence in step 1 is And the noise image X in step 3 T After concatenation in the channel dimension, sequence operations are performed to obtain the initial feature set Where N' represents the total number of feature sets, n' represents the number of feature sets, and its value range is 1≤n'≤N'.

[0015]

[0016] Among them, Concat() is a concatenation operation, and Seq() is a sequence operation;

[0017] Step 5: The initial feature set obtained in step 4 According to formula (5), the sequence operation is performed, and then the time embedding vector is added and the sequence operation is performed again. Finally, the input feature set Repeat the above operation after residual connection to obtain the feature set

[0018]

[0019] Circulat() is a two-loop operation, shortcut() is a channel matching operation, and Te() is a time embedding operation.

[0020] Step 6: According to formula (6), the feature set obtained in step 5 Perform downsampling operation to obtain feature set

[0021]

[0022] Where Downsample() is the downsampling operation;

[0023] Step 7: The feature set obtained in step 6 Recursively repeat steps 5 and 6 to further obtain the feature set Then the feature set According to formula (7), the feature set is obtained

[0024]

[0025] Circulat() is a two-cycle operation;

[0026] Step 8: According to formula (8), the feature set obtained in step 7 Perform operations to obtain feature sets

[0027]

[0028] Step 9: The feature set obtained in step 8 According to the operation of formula (9), the feature set is obtained

[0029]

[0030] Then according to formula (10), the feature set and Concatenate to get the initial feature set of the decoder

[0031]

[0032] Among them, Concat() is a concatenation operation;

[0033] Step 10: Initial feature set of the encoder obtained in step 9 According to formula (11), we can get the feature set

[0034]

[0035] Step 11: The feature set obtained in step 10 A feature set of the same size as the corresponding one in the encoder path Splice along the channel dimension and repeat the operation of formula (11) to obtain the feature set Then, according to formula (12), we get the feature set

[0036]

[0037] Among them, Upsample() is the upsampling operation;

[0038] Step 12: The feature set obtained in step 11 Recursively repeat steps 10 to 11 twice to obtain the final upsampled feature set. Then, after group normalization and activation function activation according to formula (13), the noise ε at time T is obtained through convolution operation. T ;

[0039]

[0040] Step 13: During the reverse diffusion process, the posterior diffusion conditional probability P(X t-1 |X t ,X0),

[0041]

[0042] Where ∝ means proportional to, exp is the exponential function, as well as are all known quantities in step 2, X t is the noise image at time t, and then according to formula (15) we get X t-1 ,

[0043]

[0044] The noise ε obtained by step 12 TAnd step 3 to get the noise image X T According to formula (16), we can get X T-1 ;

[0045]

[0046] Step 14: X obtained in step 13 T-1 Repeat steps 4 to 13 for T-1 times to obtain a fully focused fused image of the microscopic image.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] (1) The present invention proposes a multi-focus microscopic image sequence fusion method based on a diffusion model, which overcomes the problems of existing multi-focus image fusion methods that are susceptible to noise interference and loss of detail information;

[0049] (2) The present invention proposes a multi-focus microscopic image sequence fusion method based on a diffusion model, which can realize few-sample learning and is particularly suitable for multi-focus image fusion tasks where samples are difficult to collect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a multi-focus microscopic image sequence fusion method based on a diffusion model of the present invention;

[0051] Figure 2 A network schematic diagram of a multi-focus microscopic image sequence fusion method based on a diffusion model of the present invention;

[0052] Figure 3 A sequence of 10 multi-focus cell images with different focal planes collected in step 1 in Example 1 of the present invention;

[0053] Figure 4 It is the fused image generated in step 14 in Example 1 of the present invention. DETAILED DESCRIPTION

[0054] Example 1

[0055] like Figure 1 , Figure 2 As shown, a multi-focus microscopic image sequence fusion method based on a diffusion model includes the following steps:

[0056] Step 1: Change the focal plane by adjusting the focal length of the microscope to obtain a multi-focus microscopic image sequence covering the entire depth of field of the scene , where n represents the position of the image sequence under a certain focal plane, and its value range is 1≤n≤N, and N=10 means that the total number of image sequences is 10, such as Figure 3 As shown;

[0057] Step 2: In the forward noise addition process, given the fully focused image X0 of the scene, add Gaussian noise using equation (1):

[0058]

[0059] Among them, X t Yes X t-1 The noisy version of β t Represents a fixed sequence of values ​​used to control the noise intensity t represents the number of diffusion steps, and its value range is 1≤t≤T, T is the maximum number of diffusion steps, ε is a Gaussian distribution with a mean of 0 and a variance of 1, let α t =1-β t , Formula (1) can be used to obtain formula (2);

[0060]

[0061] Step 3: Initialize the noise image X at time T = 2000 using formula (3) T ,

[0062] X T =randn(shape(X1)) (3)

[0063] Where X1 represents the multi-focus image sequence In the first picture, shape() is a function to obtain the image shape, and randn() is a function to generate a random array that obeys Gaussian distribution;

[0064] Step 4: According to formula (4), the multi-focus image sequence in step 1 is And the noise image X in step 3 T After concatenation in the channel dimension, sequence operations are performed to obtain the initial feature set Where N' represents the total number of feature sets, n' represents the number of feature sets, and its value range is 1≤n'≤N'.

[0065]

[0066] Among them, Concat() is a concatenation operation, and Seq() is a sequence operation;

[0067] Step 5: The initial feature set obtained in step 4 According to formula (5), the sequence operation is performed, and then the time embedding vector is added and the sequence operation is performed again. Finally, the input feature set Repeat the above operation after residual connection to obtain the feature set

[0068]

[0069] Circulat() is a two-loop operation, shortcut() is a channel matching operation, and Te() is a time embedding operation.

[0070] Step 6: According to formula (6), the feature set obtained in step 5 Perform downsampling operation to obtain feature set

[0071]

[0072] Where Downsample() is the downsampling operation;

[0073] Step 7: The feature set obtained in step 6 Recursively repeat steps 5 and 6 to further obtain the feature set Then the feature set According to formula (7), the feature set is obtained

[0074]

[0075] Circulat() is a two-cycle operation;

[0076] Step 8: According to formula (8), the feature set obtained in step 7 Perform operations to obtain feature sets

[0077]

[0078] Step 9: The feature set obtained in step 8 According to the operation of formula (9), the feature set is obtained

[0079]

[0080] Then according to formula (10), the feature set and Concatenate to get the initial feature set of the decoder

[0081]

[0082] Among them, Concat() is a concatenation operation;

[0083] Step 10: Initial feature set of the encoder obtained in step 9 According to formula (11), we can get the feature set

[0084]

[0085] Step 11: The feature set obtained in step 10 A feature set of the same size as the corresponding one in the encoder path Splice along the channel dimension and repeat the operation of formula (11) to obtain the feature set Then, according to formula (12), we get the feature set

[0086]

[0087] Among them, Upsample() is the upsampling operation;

[0088] Step 12: The feature set obtained in step 11 Recursively repeat steps 10 to 11 twice to obtain the final upsampled feature set. Then, after group normalization and activation function activation according to formula (13), the noise ε at time T is obtained through convolution operation. T ;

[0089]

[0090] Step 13: During the reverse diffusion process, the posterior diffusion conditional probability P(X t-1 |X t ,X0),

[0091]

[0092] Where ∝ means proportional to, exp is the exponential function, as well as are all known quantities in step 2, X t is the noise image at time t, and then according to formula (15) we get X t-1 ,

[0093]

[0094] The noise ε obtained by step 12 T And step 3 to get the noise image X T According to formula (16), we can get X T-1 ;

[0095]

[0096] Step 14: X obtained in step 13 T-1 Repeat steps 4 to 13 T-1 times to get Figure 4 Full-focus fusion image of the microscopic image.

Claims

1. A multi-focus microscopic image sequence fusion method based on a diffusion model, characterized by: The following steps are involved: Step 1: Change the focal plane by adjusting the focal length of the microscope to obtain a multi-focus microscopic image sequence covering the entire depth of field of the scene Where N represents the total number of image sequences, n represents the position of the image sequence under a certain focal plane, and its value range is 1≤n≤N; Step 2: In the forward noise addition process, given the fully focused image X0 of the scene, add Gaussian noise using equation (1): Among them, X t Yes X t-1 The noisy version of β t Represents a fixed sequence of values ​​used to control the noise intensity t represents the number of diffusion steps, and its value range is 1≤t≤T, T is the maximum number of diffusion steps, ε is a Gaussian distribution with a mean of 0 and a variance of 1, let α t =1-β t , Formula (1) can be used to obtain formula (2); Step 3: Initialize the noise image X at time T using formula (3): T , X T =randn(shape(X1)) (3) Where X1 represents the multi-focus image sequence In the first picture, shape() is a function to obtain the image shape, and randn() is a function to generate a random array that obeys Gaussian distribution; Step 4: According to formula (4), the multi-focus image sequence in step 1 is And the noise image X in step 3 T After concatenation in the channel dimension, sequence operations are performed to obtain the initial feature set Where N' represents the total number of feature sets, n' represents the number of feature sets, and its value range is 1≤n'≤N'. Among them, Concat() is a concatenation operation, and Seq() is a sequence operation; Step 5: The initial feature set obtained in step 4 According to formula (5), the sequence operation is performed, and then the time embedding vector is added and the sequence operation is performed again. Finally, the input feature set Repeat the above operation after residual connection to obtain the feature set Circulat() is a two-cycle operation, shortcut() is a channel matching operation, and Te() is a time embedding operation; Step 6: According to formula (6), the feature set obtained in step 5 Perform downsampling operation to obtain feature set Where Downsample() is the downsampling operation; Step 7: The feature set obtained in step 6 Recursively repeat steps 5 and 6 to further obtain the feature set Then the feature set According to formula (7), the feature set is obtained Circulat() is a two-cycle operation; Step 8: According to formula (8), the feature set obtained in step 7 Perform operations to obtain feature sets Step 9: The feature set obtained in step 8 According to the operation of formula (9), the feature set is obtained Then according to formula (10), the feature set and Concatenate to get the initial feature set of the decoder Among them, Concat() is a concatenation operation; Step 10: Initial feature set of the encoder obtained in step 9 According to formula (11), we can get the feature set Step 11: The feature set obtained in step 10 A feature set of the same size as the corresponding one in the encoder path Splice along the channel dimension and repeat the operation of formula (11) to obtain the feature set Then, according to formula (12), we get the feature set Among them, Upsample() is the upsampling operation; Step 12: The feature set obtained in step 11 Recursively repeat steps 10 to 11 twice to obtain the final upsampled feature set. Then, after group normalization and activation function activation according to formula (13), the noise ε at time T is obtained through convolution operation. T ; Step 13: During the reverse diffusion process, the posterior diffusion conditional probability P(X t-1 |X t ,X0), Where ∝ means proportional to, exp is the exponential function, α t , as well as are all known quantities in step 2, X t is the noise image at time t, and then according to formula (15) we get X t-1 , The noise ε obtained by step 12 T And step 3 to get the noise image X T According to formula (16), we can get X T-1 ; Step 14: X obtained in step 13 T-1 Repeat steps 4 to 13 for T-1 times to obtain a fully focused fused image of the microscopic image.