Fluorescent image processing model training method and fluorescent image processing method

Abstract

The application provides a fluorescence image processing model training method and a fluorescence image processing method, and comprises the following steps: building an unsupervised neural network model, wherein the unsupervised neural network model comprises an image conversion module and an image recovery module; obtaining a three-dimensional fluorescence image of a sample to make a training data set; setting a training parameter; training the unsupervised neural network model with the minimum loss function as the target to obtain a fluorescence image processing model. The fluorescence image axial resolution improvement method based on unsupervised deep learning disclosed in the application does not need gold standard training data and does not need to perform complex physical modeling on the imaging process, greatly improves the model performance and applicability, and can effectively recover three-dimensional data of various fluorescence imaging.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for training a fluorescence image processing model and a fluorescence image processing method. Background Technology

[0002] Most optical microscopes exhibit anisotropic 3D resolution, with their axial resolution typically being 2-3 times lower than their lateral resolution. This anisotropic optical resolution leads to axial artifacts during 3D reconstruction, affecting the accurate analysis and measurement of the object's 3D structure. Early methods to improve the axial resolution of 3D imaging involved hardware modifications, such as multi-view light sheet microscopes and confocal 4Pi microscopes. However, these methods rely on meticulously calibrated optical paths and complex, expensive components, limiting their applicability.

[0003] With the development of computer vision and deep learning, some methods have attempted to improve axial resolution using neural networks. Since the lateral and axial data of 3D images are inherently mismatched, this poses a challenge to training supervised networks. Therefore, they employ synthetic data methods, physically modeling the imaging process to degrade the lateral data to a level close to the axial data, then training the neural network using the synthetic paired data, and finally applying the network to real axial data. The drawback of this method is that the model's performance is highly dependent on the modeling performance, and the modeling process is merely a simulation of the real scene, inevitably deviating from reality. Therefore, the performance of the trained model in real-world applications is limited. Summary of the Invention

[0004] To overcome the above-mentioned technical shortcomings, this invention proposes a fluorescence image processing model training method and a fluorescence image processing method. This method does not require gold standard training data or complex physical modeling of the imaging process, which greatly improves the model performance and applicability and can effectively recover the three-dimensional data of various fluorescence imaging.

[0005] To achieve the above objectives, this invention discloses a method for training a fluorescence image processing model, specifically comprising the following steps:

[0006] An unsupervised neural network model is constructed, which includes an image conversion module and an image restoration module. The image conversion module is used to learn the real degradation process of an image in an unpaired manner, and the image restoration module is used to learn the resolution enhancement process of an image.

[0007] Obtain three-dimensional fluorescence images of the samples to create an unpaired training dataset;

[0008] By setting training parameters and aiming to minimize the loss function, an unsupervised neural network model is trained to obtain a fluorescence image processing model.

[0009] In the above technical solution, since the horizontal and axial data of three-dimensional fluorescence images are naturally non-corresponding, an unsupervised network model is used to train on the unpaired training dataset. The image conversion module can learn the real degradation process from horizontal data to axial data in an unpaired manner, and then the image restoration module learns the resolution enhancement process of the image. Therefore, the unsupervised neural network model can directly learn the distribution characteristics of the real data without modeling and estimating the imaging process, avoiding the problem of model performance degradation caused by modeling errors. The obtained fluorescence image processing model can be applied to the processing of fluorescence images that cannot form a paired training set.

[0010] This invention also discloses a fluorescence image processing method, comprising:

[0011] Acquire the three-dimensional fluorescence image to be processed;

[0012] The three-dimensional fluorescence image to be processed is sliced ​​and then processed using the aforementioned fluorescence image processing model to obtain a three-dimensional fluorescence image with improved axial resolution.

[0013] This method is based on the aforementioned fluorescence image processing model. It can improve the axial resolution of three-dimensional fluorescence image data at the algorithm level, overcome the limitations of hardware conditions, and its performance and applicability are superior to existing neural network methods. Attached Figure Description

[0014] Figure 1 This is a flowchart of a fluorescence image processing model training method.

[0015] Figure 2 This is a flowchart of the training process for an unsupervised neural network model.

[0016] Figure 3 A flowchart of a fluorescence image processing method

[0017] Figure 4 These are comparison images showing the effect of applying the processing method of this application to improve the axial resolution of three-dimensional fluorescence images. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] The unsupervised deep learning-based fluorescence image axial resolution enhancement method of the present invention does not require paired training data or complex physical modeling of the imaging process, which greatly improves the model performance and applicability and can effectively recover the three-dimensional data of various fluorescence imaging.

[0020] like Figure 1 As shown, this invention proposes a method for training a fluorescence image processing model, comprising the following steps:

[0021] S101, Build an unsupervised neural network model, which includes an image conversion module and an image restoration module. The image conversion module is used to learn the real degradation process of an image in an unpaired manner, and the image restoration module is used to learn the resolution enhancement process of an image.

[0022] S102, Obtain the three-dimensional fluorescence image of the sample and create an unpaired training dataset;

[0023] S103, set the training parameters, use an unpaired training dataset, train an unsupervised neural network model with the goal of minimizing the loss function, and obtain a fluorescence image processing model.

[0024] Specifically, the image conversion module in S101 is an image conversion module based on generative adversarial networks, which contains two structurally identical generative networks G. A G B And two discriminant networks D with the same structure A D B Generative Network G A G B The structure can be a residual neural network with residual connections or a U-shaped network containing downsampling and upsampling paths; the discriminant network D A D B All of them are convolutional neural networks composed of multiple convolutional layers connected in series.

[0025] The image restoration module in S101 is an image restoration module H based on a convolutional neural network. Its structure can be a residual neural network with residual connections or a U-shaped network containing downsampling paths and upsampling paths.

[0026] Specifically, the creation of the training dataset for S102 includes:

[0027] The acquired three-dimensional fluorescence image is sliced ​​along the horizontal and axial directions to obtain its horizontal and axial sections. The horizontal section is the horizontal high-resolution data X.

[0028] The horizontal high-resolution data X is downsampled along the column direction of the image and then upsampled to obtain the horizontal low-resolution data Y. The axial section is upsampled along the column direction of the image to obtain the axial low-resolution data Z.

[0029] The horizontal high-resolution data X, the horizontal low-resolution data Y, and the axial low-resolution data Z are split to form a series of unpaired image blocks of the same size.

[0030] The image block pixels in the horizontal high-resolution data X and the horizontal low-resolution data Y correspond one-to-one, while the image block pixels in the horizontal high-resolution data X and the axial low-resolution data Z do not correspond, and the image block pixels in the horizontal low-resolution data Y and the axial low-resolution data Z do not correspond.

[0031] The three-dimensional fluorescence images acquired in S102 were obtained using a fluorescence microscope. Because the objective lens aperture angle cannot reach 90 degrees in this type of microscopic optical imaging—a natural physical limitation—this limitation results in the axial spatial frequency received by the objective lens being much lower than the lateral spatial frequency. Consequently, the axial image resolution is worse than the lateral image resolution. During the three-dimensional acquisition process, the axial and lateral image content of the same object are completely non-corresponding. This means that although the training dataset created in S102 can simultaneously acquire high- and low-resolution images, they are unpaired. Therefore, conventional supervised neural networks cannot be used for training; instead, the method described in this application is required to train an unsupervised neural network model.

[0032] Specifically, the transverse and axial sections are obtained directly from the acquired 3D fluorescence image. Since the transverse and axial sampling rates are generally unequal, all axial sections need to be interpolated and upsampled along the column direction to ensure their pixel resolution matches the transverse high-resolution data X, thus obtaining the axial low-resolution data Z. The transverse low-resolution data Y is obtained by downsampling and then upsampling the transverse high-resolution data X along the column direction of the image at a certain magnification, ensuring that the pixel resolution of the transverse low-resolution data Y matches that of the transverse high-resolution data X. Assuming the microscope's transverse sampling rate is *a* and the axial sampling rate is *b*, then its upsampling magnification is *b / a*. The axial low-resolution data Z is obtained by upsampling the axial sections of the directly acquired 3D fluorescence image along the column direction of the image at a magnification of *b / a*. The transverse low-resolution data Y is obtained by downsampling and then upsampling the transverse high-resolution data X along the column direction of the image at a certain magnification, with both downsampling and upsampling magnifications being *b / a*. S2 also includes splitting the data in X, Y, and Z to form a series of 64×64 image blocks.

[0033] In some embodiments, the present invention utilizes a fluorescence microscopy optical section tomography system to acquire three-dimensional fluorescence images of mouse livers at a sampling rate of 0.32 × 0.32 × 2 μm. A three-dimensional data block of 1600 × 1600 × 256 voxels (corresponding to an actual size of 512 × 512 × 512 μm) is selected to create a training dataset. This data block is divided into horizontal and axial parts. The horizontal data consists of 256 images of 1600 × 1600 pixels, corresponding to a resolution of 0.32 × 0.32 μm, constituting the horizontal high-resolution data X. All images of X are downsampled along the column direction at a downsampling factor of 2 / 0.32, resulting in 256 images of 1600 × 256 pixels. Subsequently, these images are upsampled by interpolation along the column direction by a factor of 2 / 0.32, resulting in 256 low-resolution images of 1600 × 1600 pixels, constituting the horizontal low-resolution data Y. The axial data consists of 1600 images of 1600×256 pixels each, corresponding to a resolution of 0.32×2μm. The axial images are interpolated and upsampled along the column direction at a ratio of 2 / 0.32, resulting in 1600 images of 1600×1600 pixels each, forming the axial low-resolution data Z. Finally, the image data in X, Y, and Z are split into a series of 64×64 image blocks. The image blocks in the X and Y datasets have a one-to-one pixel correspondence, while the image blocks in X and Z, and Y and Z, do not have a pixel correspondence.

[0034] like Figure 2 As shown, during training in S103, in each iteration optimization cycle, the image conversion module is optimized first, followed by the image restoration module, and the image conversion module is trained in a cyclically consistent manner. That is, the training of the unsupervised neural network model in S103 involves the alternating optimization of the image conversion module and the image restoration module. The image conversion module consists of a cyclically consistent generative adversarial network and is trained in a cyclically consistent manner, meaning that the lateral low-resolution data Y is processed by the generator G. A and G B Afterwards, it needs to be consistent with itself, and the axial low-resolution data Z is processed by G. B and G A Then it needs to be consistent with itself. Among them, G A Responsible for processing image patches y in the lateral low-resolution data Y i Image patch z converted to axial low-resolution data Z i And G B Responsible for processing image patches z in the axial low-resolution data Z i Image patch y converted to lateral low-resolution data Y i Discriminator D A Responsible for distinguishing real image blocks z i and the image patch G generated by the network A (yi Discriminator D B Responsible for distinguishing real image blocks y i and the image patch G generated by the network B (z i Since this application uses an unpaired training dataset for learning, the image conversion module is used to learn the real degradation process from lateral low-resolution data Y to axial low-resolution data Z in an unpaired manner.

[0035] When implementing cycle-consistent training, the goal is to minimize the loss function. The loss function L in S3 includes the cycle-consistent loss L. cycle Generate adversarial loss L gan Content loss L con and feedback loss L feed ,Right now:

[0036] L = L cycle +λL gan +ρL con +σL feed (1)

[0037] Where λ, ρ, and σ represent the generative adversarial loss L, respectively. gan Content loss L con and feedback loss L feed The weights are λ∈(0,1], ρ∈(0,10] and σ∈(0,1).

[0038] Cyclic Consistent Loss L cycle Defined as:

[0039]

[0040] Where y i , z i These represent image blocks in the horizontal low-resolution data Y and the axial low-resolution data Z, respectively, with N being the total number of image blocks. The purpose of this is to ensure structural consistency during the image conversion process.

[0041] To prevent the network from learning the identity mapping, generative adversarial loss is introduced to force the generator G to learn the identity mapping. A and G B Generate image patches from the corresponding image data. Generate adversarial loss L. gan From the generation loss L gen and combat losses L dis composition:

[0042] L gan =L gen +L dis (3)

[0043] L genL represents the generation loss. dis Represents the adversarial loss, where the generation loss L gen Defined as:

[0044]

[0045] Combat loss L dis Defined as:

[0046]

[0047] The image restoration module consists of a convolutional neural network H based on residual connections, which is responsible for restoring the axially low-resolution image patch z. i Restored to horizontal high-resolution image patch x i This involves learning the mapping from low-resolution axial data Z to high-resolution lateral data X. Because x i and z i Also, the pixels don't correspond, making direct training impossible. And x... i and y i It is pixel-based, therefore this application utilizes the generative network G trained by the image transformation module. A y i Image converted to axial low-resolution data Z At this time x i and It's pixel-based; specifically, this can be achieved by minimizing the content loss L. con Training is performed. Content loss is defined as:

[0048]

[0049] Where x i This represents an image patch within the horizontal high-resolution data X. This represents the generator network G after the weights are frozen. A .

[0050] The training performance of the image restoration module can provide feedback to the image conversion module; therefore, a feedback loss is introduced to improve the performance of the generator network G. A To better learn the mapping from lateral low-resolution data Y to axial low-resolution data Z. Feedback loss L feed Defined as:

[0051]

[0052] More specifically, the total loss function is a combination of these four loss functions with certain weights:

[0053] L = L cycle +λL gan +ρL con +σL feed(1)

[0054] In some embodiments, λ = 0.1, ρ = 1, and σ = 0.1 can be selected. The specific training parameters set in S103 can be: the optimizer that minimizes the loss function is the Adam optimization method, the initial learning rate is 0.0001, the learning rate is halved every 15 iterations, and a total of 60 training rounds are conducted to finally obtain the trained unsupervised neural network model.

[0055] like Figure 3 As shown, the present invention also proposes a fluorescence image processing method, comprising:

[0056] S201, Obtain the three-dimensional fluorescence image to be processed;

[0057] S202, the three-dimensional fluorescence image to be processed is sliced, and then processed using the aforementioned fluorescence image processing model to obtain a three-dimensional fluorescence image with improved axial resolution.

[0058] Specifically, the three-dimensional fluorescence image to be processed is sliced, including:

[0059] The three-dimensional fluorescence image to be processed is re-sliced ​​along the x and y directions to obtain axial sections in the xz and yz directions. Then, the axial sections in the xz and yz directions are sequentially input into the image restoration module H in the fluorescence image processing model to obtain xz and yz section images with improved axial resolution. Finally, the section images in these two directions are re-sliced ​​back to the original xy directions and averaged to obtain a three-dimensional fluorescence image with improved axial resolution.

[0060] In some embodiments, an 800×1000×100 voxel data block (containing 100 horizontal images of 800×1000 pixels each) is arbitrarily selected from a 3D image of a mouse liver. The voxel resolution is 0.32×0.32×2μm, and the actual size of the 3D fluorescence image data block is 256×320×200μm. This 3D fluorescence image data block is resliced ​​along the x and y directions to obtain axial section data blocks in the xz and yz directions. The axial section data block in the xz direction contains 1000 low-resolution axial images of 800×100 pixels each, and the axial section data block in the yz direction contains 800 low-resolution axial images of 1000×100 pixels each. Each axial image in the xz and yz 3D fluorescence image data blocks is upsampled by a factor of 2 / 0.32 along the column direction, resulting in 1000 low-resolution axial images of size 800×625 pixels and 800 low-resolution axial images of size 1000×625 pixels, respectively. These low-resolution axial images are sequentially input into the image restoration module H trained in step S3, resulting in 1000 high-resolution axial images of size 800×625 pixels in the xz direction and 800 high-resolution axial images of size 1000×625 pixels in the yz direction, respectively. The data blocks in these two directions are re-sliced ​​and restored to the xy direction, resulting in two data blocks of size 800×1000×625 voxels, with the voxel resolution of the data blocks increased to 0.32×0.32×0.32μm. The arithmetic mean of these two data blocks is used to obtain the final 3D fluorescence image data with improved axial resolution.

[0061] Figure 4 The image shows axial images before (left) and after (right) restoration. Visually, it's clear that the liver axial image before restoration is very blurry, and the vascular structures are difficult to distinguish. The axial image after restoration shows a significant improvement in resolution; the previously blurry artifacts are removed, high-frequency image details are restored, and the vascular structures become clear and sharp. Using this method to improve the axial resolution of three-dimensional fluorescence image data can greatly help us perform fine morphological analysis of three-dimensional fluorescence images.

[0062] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for training a fluorescence image processing model, characterized in that, Includes the following steps: An unsupervised neural network model is constructed, comprising an image transformation module and an image restoration module. The image transformation module is used to learn the true degradation process of an image unpaired, and the image restoration module is used to learn the resolution enhancement process of an image. The image transformation module is a generative adversarial network-based image transformation module, containing two structurally identical generative networks G. A G B And two discriminant networks D with the same structure A D B G A Responsible for processing image patches y in the lateral low-resolution data Y i Image patch z converted to axial low-resolution data Z i ;and G B Responsible for processing image patches z in the axial low-resolution data Z i Image patch y converted to lateral low-resolution data Y i Discriminator D A Responsible for distinguishing real image blocks z i and the image patch G generated by the network A (y i Discriminator D B Responsible for distinguishing real image blocks y i and the image patch G generated by the network B (z i The generator network is a residual neural network with residual connections or a U-shaped network containing downsampling and upsampling paths; the discriminator network is a convolutional neural network composed of multiple convolutional layers connected in series; the image restoration module is an image restoration module H based on a convolutional neural network, with a structure of a residual neural network with residual connections or a U-shaped network containing downsampling and upsampling paths. The image restoration module is responsible for restoring image patches z in the axial low-resolution data Z. i Recover the image patch x from the horizontal high-resolution data X i , where image patch x i and image block y i The pixel correspondence; the training effect of the image restoration module provides feedback to the image conversion module, therefore a feedback loss is introduced to allow the generator network G to... A G B To better learn the mapping between lateral low-resolution data Y and axial low-resolution data Z; To acquire three-dimensional fluorescence images of samples and create an unpaired training dataset, the following steps are taken: slicing the acquired three-dimensional fluorescence images along the horizontal and axial directions to obtain horizontal and axial sections, wherein the horizontal section is the horizontal high-resolution data X; downsampling and then upsampling the horizontal high-resolution data X along the column direction of the image to obtain horizontal low-resolution data Y; and upsampling the axial section along the column direction of the image to obtain axial low-resolution data Z. The horizontal high-resolution data X, the horizontal low-resolution data Y, and the axial low-resolution data Z are split to form a series of unpaired image blocks of the same size; By setting training parameters and using an unpaired training dataset, the unsupervised neural network model is trained with the goal of minimizing the loss function to obtain a fluorescence image processing model.

2. The training method according to claim 1, characterized in that, The image block pixels in the horizontal high-resolution data X and the horizontal low-resolution data Y correspond one-to-one. The image block pixels in the horizontal high-resolution data X and the axial low-resolution data Z do not correspond, and the image block pixels in the horizontal low-resolution data Y and the axial low-resolution data Z do not correspond.

3. The training method according to claim 2, characterized in that, During training, in each iteration optimization cycle, the image conversion module is optimized first, and then the image restoration module is optimized, and the image conversion module is trained in a consistent cyclic manner.

4. The training method according to claim 3, characterized in that, The loss function L includes the cycle-consistent loss L. cycle Generate adversarial loss L gan Content loss L con and feedback loss L feed ,Right now: Where λ, ρ, and σ represent the generative adversarial loss L, respectively. gan Content loss L con and feedback loss L feed The weights are λ∈(0,1], ρ∈(0,10] and σ∈(0,1).

5. A fluorescence image processing method, characterized in that, include: Acquire the three-dimensional fluorescence image to be processed; The three-dimensional fluorescence image to be processed is sliced, and then processed using the fluorescence image processing model as described in any one of claims 1-4 to obtain a three-dimensional fluorescence image with improved axial resolution.

6. The fluorescence image processing method according to claim 5, characterized in that, Slicing the three-dimensional fluorescence image to be processed includes: The three-dimensional fluorescence image to be processed is re-sliced ​​along the x and y directions to obtain axial sections in the xz and yz directions.

7. The fluorescence image processing method according to claim 6, characterized in that, A fluorescence image processing model is used to obtain a three-dimensional fluorescence image with improved axial resolution, including: The axial cross-sections in the xz and yz directions are sequentially input into the image restoration module of the fluorescence image processing model to obtain xz and yz cross-section images with improved axial resolution. Then, the cross-section images in these two directions are re-sliced ​​back to the original xy directions and averaged to finally obtain a three-dimensional fluorescence image with improved axial resolution.

Images