Method for obtaining a multiphoton imaging virtual simulation model and method for generating a three-dimensional image

Abstract

The embodiment of the present application provides a kind of acquisition method of multi-photon imaging virtual simulation model and three-dimensional image generation method, it is related to multi-photon imaging technical field.The acquisition method of multi-photon imaging virtual simulation model includes: constructing the training set of multiple sample data including target tissue, each sample data includes: the original dyeing image of target tissue matched with the real three-dimensional image obtained by multi-photon imaging;Multi-photon imaging virtual simulation model based on GAN network and self-attention mechanism is constructed, the input of the multi-photon imaging virtual simulation model is the dyeing image of target tissue, and the output is the virtual three-dimensional image of target tissue;The multiple sample data are used to train the multi-photon imaging virtual simulation model, and the final multi-photon imaging virtual simulation model is obtained.The present application realizes the conversion from dyeing image to virtual three-dimensional image, and conversion precision is higher.

Description

Technical Field

[0001] This invention relates to the field of multiphoton imaging technology, specifically to a method for obtaining a virtual simulation model of multiphoton imaging and a method for generating three-dimensional images. Background Technology

[0002] Radiation-induced lung injury (RALE) is a major dose-limiting toxicity associated with thoracic radiotherapy, with a grade II or higher incidence rate as high as 20%-40%. Radiation-induced pulmonary fibrosis (RACF), as the end-stage manifestation of lung injury, can lead to irreversible destruction of lung parenchymal structure, severely impacting patient efficacy and quality of life. Currently, the visualization and quantitative assessment of RACF in the early stages of pulmonary fibrosis remains significantly limited. Traditional methods, such as immunohistochemical detection of α-SMA (alpha-smooth muscle actin), Masson staining, and hydroxyproline staining, while widely used for assessing collagen deposition in pulmonary fibrosis, suffer from insufficient specificity for collagen detection, low resolution in collagen structure resolution, and high detection thresholds for changes in collagen increment, making them only suitable for late-stage fibrosis. There is an urgent need for a high-resolution, label-free evaluation tool capable of accurately quantifying the three-dimensional morphology and spatial arrangement of collagen in the early stages of injury.

[0003] Multiphoton microscopy (MPM), as an advanced optical imaging technology, possesses unique advantages such as subcellular spatial resolution, label-free detection, and deep tissue penetration. Based on the unique non-centrosymmetric crystal structure of collagen molecules, they can generate specific second harmonics (SHGs) under two-photon excitation, producing SHG signals. Cellular components such as lipids and nucleic acids, lacking this structural characteristic, do not generate interfering signals, thus enabling visualization of collagen distribution. Notably, MPM can not only accurately quantify collagen content but also analyze the morphological characteristics and spatial arrangement patterns of collagen fibers through three-dimensional reconstruction technology.

[0004] However, multiphoton imaging (MPM) technology is limited by the high cost of lasers, complex optical systems, and precision scanning equipment, preventing its widespread application. Summary of the Invention

[0005] The purpose of this invention is to provide a method for obtaining a virtual simulation model of multiphoton imaging and a method for generating three-dimensional images. The obtained virtual simulation model of multiphoton imaging combines the image virtual simulation capability of the CycleGAN model and the image global information extraction capability of the Transformer self-attention neural network, realizing the conversion from H&E staining images to virtual MPM three-dimensional images with high conversion accuracy. This breaks through the cost barrier of MPM technology and has significant application value and social benefits.

[0006] To achieve the above objectives, the present invention provides a method for obtaining a virtual simulation model for multiphoton imaging, comprising: A training set is constructed that includes multiple sample data of the target tissue, each of the sample data including: the original stained image of the target tissue after pairing and the real three-dimensional image obtained by multiphoton imaging; A virtual simulation model for multiphoton imaging based on GAN network and self-attention mechanism is constructed. The input of the virtual simulation model is a stained image of the target tissue, and the output is a virtual three-dimensional image of the target tissue. The multiphoton imaging virtual simulation model is trained using the multiple sample data to obtain the final multiphoton imaging virtual simulation model.

[0007] The present invention also provides a method for generating three-dimensional images through multiphoton imaging, comprising: Acquire a stained image of the tissue to be tested; input the stained image of the tissue to be tested into a multiphoton imaging virtual simulation model to obtain a virtual three-dimensional image of the tissue to be tested based on multiphoton imaging. The multiphoton imaging virtual simulation model is obtained based on the above-described method for acquiring the multiphoton imaging virtual simulation model.

[0008] The present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for acquiring a virtual simulation model of multiphoton imaging, and / or the above-described method for generating a three-dimensional image of multiphoton imaging.

[0009] In one embodiment, the multiphoton imaging virtual simulation model includes at least: a first generator based on a self-attention mechanism, a second generator based on a self-attention mechanism, a first discriminator, and a second discriminator; The first generator takes the original stained image as input and outputs a virtual 3D image, while the second generator takes the real 3D image as input and outputs a virtual stained image. The first discriminator takes the virtual 3D image and the real 3D image as input and outputs the similarity between the virtual 3D image and the real 3D image. The input to the second discriminator is the original stained image and the virtual stained image, and the output is the similarity between the original stained image and the virtual stained image.

[0010] In one embodiment, the multiphoton imaging virtual simulation model further includes: a third generator based on a self-attention mechanism and a fourth generator based on a self-attention mechanism; The third generator takes the virtual 3D image as input and outputs the restored stained image. The fourth generator takes the virtual stained image as input and outputs a reconstructed 3D image. In one embodiment, the loss function used when training the multiphoton imaging virtual simulation model is the sum of the following three terms: The content loss function term is: the similarity measure between the virtual 3D image and the real 3D image, and the sum of the similarity measures between the virtual stained image and the original stained image; The discriminant loss function term is the sum of the similarity output by the first discriminator and the similarity output by the second discriminator. The cycle-consistent loss function term is the sum of the similarity measure between the real 3D image and the reconstructed 3D image, and the similarity measure between the original stained image and the reconstructed stained image.

[0011] In one embodiment, each sample data is obtained in the following way: Acquire several paired complete stained images and complete three-dimensional images corresponding to the target tissue, and segment the complete stained images into multiple stained image blocks of a specified size, and segment the complete three-dimensional images into multiple three-dimensional image blocks of a specified size; The paired stained image blocks are rigidly registered with the three-dimensional image blocks; The pixel values ​​of the rigidly registered stained image block and the three-dimensional image block are normalized to serve as the original stained image of the target tissue and the real three-dimensional image obtained by multiphoton imaging.

[0012] In one embodiment, the first generator, the second generator, the third generator, and the fourth generator each include a Transformer encoding module and a Transformer decoding module. The Transformer encoding module includes three stacked encoders, each of which includes a feedforward neural network layer and a multi-head attention layer; The Transformer decoding module includes three stacked decoders, each of which includes a feedforward neural network layer and a multi-head attention layer.

[0013] In one embodiment, the first discriminator and the second discriminator comprise: a two-dimensional convolutional layer and a fully connected layer.

[0014] In one embodiment, the staining image of the target tissue is a hematoxylin and eosin staining image. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the method for obtaining a virtual simulation model of multiphoton imaging according to the first embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the multiphoton imaging virtual simulation model according to the first embodiment of the present invention; Figure 3 This is a schematic diagram of the training process of the multiphoton imaging virtual simulation model according to the first embodiment of the present invention; Figure 4 This is a flowchart illustrating the three-dimensional image generation method for multiphoton imaging according to the second embodiment of the present invention. Figure 5 This is a schematic diagram comparing the simulation results of a multiphoton imaging three-dimensional image generation method according to a third embodiment of the present invention using a multiphoton imaging virtual simulation model, a conventional CycleGAN model, and a CNN convolutional network model. Detailed Implementation

[0016] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings to provide a clearer understanding of the purpose, features, and advantages of the present invention. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of the present invention, but are merely illustrative of the essential spirit of the technical solution of the present invention.

[0017] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known apparatuses, structures, and techniques associated with this application may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0018] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.

[0019] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0020] The singular forms “a” and “the” used in this specification and the appended claims include plural references unless otherwise expressly stated herein. It should be noted that the term “or” is generally used to include the meaning of “or / and” unless otherwise expressly stated herein.

[0021] In the following description, in order to clearly demonstrate the structure and working method of the present invention, a number of directional terms will be used. However, terms such as "front", "back", "left", "right", "outer", "inner", "outer", "inner", "up", and "down" should be understood as convenient terms and not as limiting terms.

[0022] Since hematoxylin-eosin (H&E) staining images and MPM real 3D images have a high degree of consistency at the cellular and tissue structure level, and H&E staining images are inexpensive and readily available, the virtual conversion of H&E staining images to MPM real 3D images based on virtual generation technology is expected to break through the cost barrier of MPM technology-generated images and has significant application value and social benefits.

[0023] Generative Adversarial Networks (GANs) have become a fundamental deep learning framework in the field of cross-modal medical image translation. In histopathology, GAN models can be used to translate images with different staining processes, improving diagnostic efficacy for conditions such as cavernous hemangioma and cholangiocarcinoma. Studies by numerous scholars both domestically and internationally on the translation of H&E-stained images with other histopathological images have also demonstrated the superior performance of GAN networks in cross-modal generation of pathological images, suggesting the potential of GAN models to convert H&E-stained images into realistic 3D images of virtual MPMs. However, traditional GAN ​​models use convolutional neural networks (CNNs) as generators. Limited by the size of the convolutional kernel, although CNNs have strong local image feature extraction capabilities, they lack a global understanding of the image and do not fully utilize the contextual information between images. Increasing the kernel size also suffers from low efficiency and decreased accuracy. In contrast, Transformer networks are not limited by the kernel size of CNNs and can learn long-distance pixel dependencies in images, capturing full-text feature information. Therefore, how to retain the image virtual simulation capability of GAN models while making full use of the correlation information of long-distance pixels in the image to improve the accuracy of MPM real 3D image virtual simulation is an important applied research problem.

[0024] The first embodiment of the present invention relates to a method for obtaining a multiphoton imaging virtual simulation model, which is used to obtain a multiphoton imaging virtual simulation model that can generate a virtual three-dimensional image of a target tissue. For example, the target tissue is pulmonary fibrosis tissue. The multiphoton imaging virtual simulation model can input a stained image of pulmonary fibrosis tissue (for example, a hematoxylin and eosin stained image, i.e., a H&E stained image).

[0025] The specific process of obtaining the multiphoton imaging virtual simulation model in this embodiment is as follows: Figure 1 As shown.

[0026] Step 101: Construct a training set comprising multiple sample data of the target tissue. Each sample data includes: a paired original stained image of the target tissue and a real three-dimensional image obtained by multiphoton imaging. The original stained image and the real three-dimensional image within each sample data correspond to the same target tissue.

[0027] For example, each of the sample data is obtained in the following way: Acquire several paired complete stained images and complete three-dimensional images corresponding to the target tissue, and segment the complete stained images into multiple stained image blocks of a specified size, and segment the complete three-dimensional images into multiple three-dimensional image blocks of a specified size; The paired stained image blocks are rigidly registered with the three-dimensional image blocks; The pixel values ​​of the rigidly registered stained image block and the three-dimensional image block are normalized to serve as the original stained image of the target tissue and the real three-dimensional image obtained by multiphoton imaging.

[0028] For example, taking pulmonary fibrosis tissue as the target tissue, complete H&E staining images and complete MPM three-dimensional images of 100 paired pulmonary fibrosis tissues were obtained. For each paired pulmonary fibrosis tissue, complete H&E staining images and complete MPM 3D images were used: the complete H&E staining images were divided into 300 1024×1024 pixel staining image blocks, and the complete MPM 3D images were divided into 300 1024×1024 pixel 3D image blocks. Based on the above, the complete H&E staining images and complete MPM three-dimensional images of 100 paired pulmonary fibrosis tissues can be segmented to obtain 300 paired 1024×1024 pixel staining image blocks and 300 paired 1024×1024 pixel three-dimensional image blocks respectively; that is, 30,000 paired staining image blocks and three-dimensional image blocks can be obtained.

[0029] Subsequently, rigid image registration was performed on all paired stained image blocks and the three-dimensional image block. The image registration method adopted was a feature point-based rigid registration method. The feature points were selected based on the Harris corner detection algorithm to extract anatomical landmarks on the stained image block and the three-dimensional image block as registration references. After rigid registration was completed, the pixel values ​​of the registered stained image block and the three-dimensional image block were normalized. The pixel value distribution range of the registered stained image block and the three-dimensional image block after normalization was [-1, 1].

[0030] The normalized and registered stained image blocks and the three-dimensional image blocks together form a sample data. In the sample data, the normalized and registered stained image blocks are the original stained images, and the three-dimensional image blocks are the real three-dimensional images obtained by multiphoton imaging.

[0031] Based on the above process, sample data from 30,000 target organizations can be obtained.

[0032] The sample data of 30,000 target organizations is then divided into training set, validation set and test set, for example, the sample data of 30,000 target organizations is divided into training set, validation set and test set in a ratio of 8:1:1.

[0033] After obtaining the training set, the training set can be expanded. The expansion methods include: flipping the colored image blocks and the three-dimensional image blocks in at least a portion of the sample data in the training set horizontally, and combining the flipped colored image blocks and the three-dimensional image blocks in each sample data as new sample data and adding them to the training set; or flipping the colored image blocks and the three-dimensional image blocks in at least a portion of the sample data in the training set vertically, and combining the flipped colored image blocks and the three-dimensional image blocks in each sample data as new sample data and adding them to the training set.

[0034] Furthermore, the sample data obtained by left-right flipping and up-down flipping can be flipped again as described above to obtain new sample data to be added to the training set.

[0035] Thus, a richer training set can be obtained, for example, the training set includes: 24,000 sample data (including: paired normalized registered stained image patches and the three-dimensional image patch), and new sample data obtained based on the above left-right and up-down flipping.

[0036] It should be noted that, in some embodiments, complete H&E staining images and MPM real 3D images can also be used directly to form sample data for subsequent model training, which is also feasible.

[0037] Step 102: Construct a multiphoton imaging virtual simulation model based on GAN network and self-attention mechanism. The input of the multiphoton imaging virtual simulation model is the stained image of the target tissue, and the output is a virtual three-dimensional image of the target tissue.

[0038] Step 103: Use the multiple sample data to train the multiphoton imaging virtual simulation model to obtain the final multiphoton imaging virtual simulation model.

[0039] For example, the multiphoton imaging virtual simulation model includes at least: a first generator based on a self-attention mechanism, a second generator based on a self-attention mechanism, a first discriminator, and a second discriminator; Furthermore, the multiphoton imaging virtual simulation model also includes: a third generator based on a self-attention mechanism and a fourth generator based on a self-attention mechanism; The first generator takes the original stained image as input and outputs a virtual 3D image. The second generator takes the real 3D image as input and outputs a virtual stained image. The third generator takes the virtual 3D image as input and outputs the restored stained image. The fourth generator takes the virtual stained image as input and outputs a reconstructed 3D image. The first discriminator takes the virtual 3D image and the real 3D image as input and outputs the similarity between the virtual 3D image and the real 3D image. The input to the second discriminator is the original stained image and the virtual stained image, and the output is the similarity between the original stained image and the virtual stained image.

[0040] When training the multiphoton imaging virtual simulation model, the loss function used is the sum of the following three terms: The content loss function term is: the similarity measure between the virtual 3D image and the real 3D image, and the sum of the similarity measures between the virtual stained image and the original stained image; The discriminant loss function term is the sum of the similarity output by the first discriminator and the similarity output by the second discriminator. The cycle-consistent loss function term is the sum of the similarity measure between the real 3D image and the reconstructed 3D image, and the similarity measure between the original stained image and the reconstructed stained image.

[0041] Please refer to Figure 2 As can be seen, the paired original H&E staining images and original MPM 3D images in the sample data are used as inputs into the constructed multiphoton imaging virtual simulation model. Specifically: The constructed multiphoton imaging virtual simulation model is the SACT-GAN model based on the recurrent CycleGAN network and the Transformer self-attention mechanism. It uses the Transformer network to replace the convolutional network of the CycleGAN generator to obtain the multiphoton imaging virtual simulation model.

[0042] The multiphoton image virtual simulation model includes a first generator G1, a second generator G2, a third generator G3, a fourth generator G4, and two discriminators. The connection relationship between the generators and discriminators is as follows: Figure 2 As shown.

[0043] The inputs to the first generator G1 and the second generator G2 are the paired H&E staining image patches and MPM 3D image patches in the sample data. The output of the first generator G1 is a virtual MPM 3D image patch; the output of the second generator G2 is a virtual H&E staining image patch.

[0044] At this time, the virtual MPM 3D image patch output by the first generator G1 is input to the third generator G3, and the virtual H&E staining image patch output by the second generator G2 is input to the fourth generator G4; the output of the third generator G3 is the restored H&E staining image patch, and the output of the fourth generator G4 is the restored MPM 3D image patch.

[0045] The first generator G1, the second generator G2, the third generator G3, and the fourth generator G4 have the same structure and are all used to transform the input image. Specifically, each generator (G1 to G4) includes a Transformer encoding module and a Transformer decoding module. The Transformer encoding module includes three stacked encoders, each of which includes a feedforward neural network layer and a multi-head attention layer. The Transformer decoding module includes three stacked decoders, each of which includes a feedforward neural network layer and a multi-head attention layer.

[0046] The two discriminators are the first discriminator D1 and the second discriminator D2, both of which are used to determine whether the input image is a real image or a virtual image.

[0047] The first discriminator D1 takes as input the original MPM 3D image patch corresponding to the same sample data and the virtual MPM 3D image patch output by the first generator G1. It outputs an image discrimination result that characterizes the similarity between the original MPM 3D image patch and the virtual MPM 3D image patch. For example, it outputs a similarity value R, which is between 0 and 1. The closer R is to 0, the more likely it is to be a virtual image. The closer R is to 1, the more likely it is to be a real image.

[0048] The input to the second discriminator D2 is the original H&E stained image patch corresponding to the same sample data and the virtual H&E stained image patch output by the second generator G2. The output is an image discrimination result that characterizes the similarity between the original H&E stained image patch and the virtual H&E stained image patch. For example, it outputs a similarity value R, which is between 0 and 1. The closer R is to 0, the more likely it is to be a virtual image. The closer R is to 1, the more likely it is to be a real image.

[0049] The first discriminator D1 and the second discriminator D2 have the same structure. Each discriminator (the first discriminator D1 and the second discriminator D2) includes three identical two-dimensional convolutional layers and one fully connected layer. The convolutional kernel size of the two-dimensional convolutional layer is 3×3, the stride is 1, and the padding value is 1. The activation function used is the linear rectified function. The input and output of each two-dimensional convolutional layer use a skip connection. After executing the fully connected layer once, the output image discrimination result R is generated. The value of the image discrimination result R is between 0 and 1. R = 1 indicates that the output is a completely real image, and R = 0 indicates that the output is a completely virtual image.

[0050] For example, please refer to Figure 2 , Figure 3 The process of training the constructed SACT-GAN model (i.e., multiphoton image virtual simulation model) using sample data from the training set is as follows: The paired H&E staining image patches and MPM 3D image patches contained in the sample data in the training set are input into the SACT-GAN model respectively; that is, the H&E staining image patches in the sample data are input into the first generator G1, and the MPM 3D image patches are input into the second generator G2.

[0051] The first generator G1 converts the original H&E stained image patch into a virtual MPM 3D image patch. The second generator G2 converts the MPM 3D image patch into a virtual H&E stained image patch. The third generator G3 restores the virtual MPM 3D image patch to a restored H&E stained image patch. The fourth generator G4 restores the virtual H&E stained image patch to a restored MPM 3D image patch.

[0052] The virtual MPM 3D image patch output by the first generator G1 is input to the first discriminator D1, and the virtual H&E staining image patch output by the second generator G2 is input to the second discriminator D2.

[0053] The virtual MPM 3D image patch and the real MPM 3D image patch corresponding to the same sample data are input into the first discriminator D1. The first discriminator D1 outputs a similarity value R1 (0~1) that represents the degree of similarity between the virtual MPM 3D image patch and the real MPM 3D image patch. The closer R1 is to 0, the more likely it is a virtual image. The closer R1 is to 1, the more likely it is a real image.

[0054] The virtual H&E staining image patch corresponding to the same sample data and the original H&E staining image patch are input into the second discriminator D2. The second discriminator D2 outputs a similarity value R2 (0~1) that represents the degree of similarity between the virtual H&E staining image patch and the original H&E staining image patch. The closer R2 is to 0, the more likely it is a virtual image. The closer R2 is to 1, the more likely it is a real image.

[0055] The loss function for model training is constructed as the sum of the content loss function term, the discriminative loss function term, and the cycle consistency loss function term, as follows: Calculate the similarity measure S1 between the virtual MPM 3D image patch and the real MPM 3D image patch corresponding to the same sample data, including mean squared error, structural similarity coefficient, and peak signal-to-noise ratio. At the same time, calculate the similarity measure S2 between the virtual H&E staining image patch and the original H&E staining image patch corresponding to the same sample data, including mean squared error, structural similarity coefficient, and peak signal-to-noise ratio. Then calculate the sum of similarity measures S1 and S2 as the content loss function term.

[0056] The sum of the similarity values ​​R1 output by the first discriminator D1 and R2 output by the second discriminator D2 is used as the discrimination loss function term. Alternatively, the output results of the first discriminator D1 and the second discriminator D2 are compared with the true results (0 for virtual images and 1 for real images), and used as the discrimination loss function term.

[0057] Calculate the similarity measure S3 between the reconstructed MPM 3D image patch and the real MPM 3D image patch corresponding to the same sample data, including mean squared error, structural similarity coefficient, and peak signal-to-noise ratio; simultaneously calculate the similarity measure S4 between the reconstructed H&E staining image patch and the original H&E staining image patch corresponding to the same sample data, including mean squared error, structural similarity coefficient, and peak signal-to-noise ratio; then calculate the sum of similarity measures S3 and S4 as the cycle-consistent loss function term.

[0058] The total loss function obtained by adding the three loss function terms is the loss function for model training. Then, backpropagation of the SACT-GAN model is performed to update the parameters of each layer of the model.

[0059] The SACT-GAN model is repeatedly trained until the number of training rounds reaches the preset number. The value of the loss function is then checked to see if it has converged. If it has, the current SACT-GAN model is used as the final multiphoton imaging virtual simulation model; otherwise, the hyperparameters of the SACT-GAN model are modified and the model is retrained.

[0060] The H&E staining image patch from the test set sample data is input into the trained multiphoton imaging virtual simulation model. The multiphoton imaging virtual simulation model automatically outputs a virtual MPM 3D image patch. Then, the similarity between the real MPM 3D image patch paired with the H&E staining image patch in the sample data and the virtual MPM 3D image patch generated by the multiphoton imaging virtual simulation model can be compared to evaluate the similarity between the virtual MPM 3D image patch generated by the multiphoton imaging virtual simulation model and the real MPM 3D image patch, thereby assessing the accuracy of the model.

[0061] The hyperparameters of the multiphoton imaging virtual simulation model are adjusted using sample data from the validation set. Image patches of H&E staining images and MPM 3D images contained in the sample data of the validation set are input into the model for training. The obtained virtual MPM 3D image patches are compared with real MPM image patches to obtain the target loss function results, and then the hyperparameters of the multiphoton imaging virtual simulation model are adjusted. The target loss function includes mean squared error, structural similarity coefficient, and peak signal-to-noise ratio.

[0062] The second embodiment of the present invention relates to a three-dimensional image generation method for multiphoton imaging, which is used to generate a corresponding virtual MPM three-dimensional image based on a stained image of a tissue to be tested; the tissue to be tested can be any biological tissue, such as pulmonary fibrosis tissue, and the stained image is, for example, an H&E stained image.

[0063] The specific process of the three-dimensional image generation method for multiphoton imaging in this embodiment is as follows: Figure 4 As shown.

[0064] Step 201: Obtain the stained image of the tissue to be tested.

[0065] Step 202: Input the stained image of the tissue to be tested into the multiphoton imaging virtual simulation model to obtain a virtual three-dimensional image of the tissue to be tested based on multiphoton imaging. The multiphoton imaging virtual simulation model is obtained based on the acquisition method of the multiphoton imaging virtual simulation model in the first embodiment.

[0066] Specifically, after directly training or directly acquiring the multiphoton imaging virtual simulation model trained by the method for acquiring the multiphoton imaging virtual simulation model based on the first embodiment, the acquired stained image of the tissue to be tested is divided into multiple stained image blocks of a specified size. Then, each stained image block is input into the multiphoton imaging virtual simulation model, thereby generating multiple virtual three-dimensional image blocks. All the virtual three-dimensional image blocks are combined according to their corresponding positions to obtain a complete virtual three-dimensional image.

[0067] For example, such as Figure 5 As shown, the tissue to be tested is pulmonary fibrosis tissue. H&E staining images of pulmonary fibrosis tissue were acquired. After the H&E staining images of pulmonary fibrosis tissue were input into the multiphoton imaging virtual simulation model, a virtual MPM image was obtained. The H&E staining images were converted into virtual second harmonic SHG images. The virtual SHG images can simulate the high-contrast imaging characteristics of SHG on structures such as collagen fibers.

[0068] Similarly, H&E staining images of pulmonary fibrosis tissue can be input into existing conventional CycleGAN and conventional CNN convolutional models to obtain corresponding virtual 3D images. Subsequently, the virtual MPM images generated by the multiphoton imaging virtual simulation model of this embodiment, as well as the virtual 3D images obtained by the conventional CycleGAN and conventional CNN convolutional models, are compared with virtual SHG images. The results show that the mean squared error (MSE) of the multiphoton imaging virtual simulation model of this embodiment is 0.002, the structural similarity coefficient (SSIM) reaches 0.84, and the peak signal-to-noise ratio (PSNR) is 33.2 dB, which is significantly better than the conventional CycleGAN model (MSE: 0.004, SSIM: 0.76, PSNR: 26.3 dB) and the conventional CNN model (MSE: 0.005, SSIM: 0.65, PSNR: 25.8 dB).

[0069] In summary, compared with existing technologies, the multiphoton imaging virtual simulation model of this invention combines the image virtual simulation capability of the CycleGAN model with the image global information extraction capability of the Transformer self-attention neural network, realizing the conversion from H&E staining images to virtual MPM three-dimensional images with high conversion accuracy. This breaks through the cost barrier of MPM technology and has significant application value and social benefits.

[0070] Since the first embodiment corresponds to this embodiment, this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and the technical effects that can be achieved in the first embodiment can also be achieved in this embodiment. To reduce repetition, they will not be repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0071] The third embodiment of the present invention relates to an electronic device, such as a mobile phone, tablet computer, desktop computer, cloud server, etc.; the electronic device includes: at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the method for acquiring a virtual simulation model of multiphoton imaging as in the first embodiment, and / or the method for generating a three-dimensional image of multiphoton imaging as in the second embodiment.

[0072] Since the first and second embodiments correspond to this embodiment, this embodiment can be implemented in conjunction with the first and second embodiments. The relevant technical details mentioned in the first and second embodiments remain valid in this embodiment, and the technical effects achievable in the first and second embodiments can also be realized in this embodiment. To reduce repetition, they will not be repeated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the first and second embodiments.

[0073] The preferred embodiments of the present invention have been described in detail above, but it should be understood that, if necessary, aspects of the embodiments can be modified to utilize aspects, features, and concepts from various patents, applications, and publications to provide other embodiments.

[0074] In light of the detailed description above, these and other changes can be made to the embodiments. Generally, the terminology used in the claims should not be considered limited to the specific embodiments disclosed in the specification and claims, but should be understood to include all possible embodiments together with the full scope of equivalents enjoyed by these claims.

Claims

1. A method for obtaining a virtual simulation model of multiphoton imaging, characterized in that, include: A training set is constructed that includes multiple sample data of the target tissue, each of the sample data including: the original stained image of the target tissue after pairing and the real three-dimensional image obtained by multiphoton imaging; A virtual simulation model for multiphoton imaging based on GAN network and self-attention mechanism is constructed. The input of the virtual simulation model is a stained image of the target tissue, and the output is a virtual three-dimensional image of the target tissue. The multiphoton imaging virtual simulation model is trained using the multiple sample data to obtain the final multiphoton imaging virtual simulation model.

2. The method for obtaining a virtual simulation model for multiphoton imaging according to claim 1, characterized in that, The multiphoton imaging virtual simulation model includes at least: a first generator based on a self-attention mechanism, a second generator based on a self-attention mechanism, a first discriminator, and a second discriminator; The first generator takes the original stained image as input and outputs a virtual 3D image, while the second generator takes the real 3D image as input and outputs a virtual stained image. The first discriminator takes the virtual 3D image and the real 3D image as input and outputs the similarity between the virtual 3D image and the real 3D image. The input to the second discriminator is the original stained image and the virtual stained image, and the output is the similarity between the original stained image and the virtual stained image.

3. The method for obtaining a virtual simulation model for multiphoton imaging according to claim 2, characterized in that, The multiphoton imaging virtual simulation model also includes: a third generator based on a self-attention mechanism and a fourth generator based on a self-attention mechanism; The third generator takes the virtual 3D image as input and outputs the restored stained image. The fourth generator takes the virtual stained image as input and outputs a reconstructed 3D image.

4. The method for obtaining a virtual simulation model for multiphoton imaging according to claim 3, characterized in that, When training the multiphoton imaging virtual simulation model, the loss function used is the sum of the following three terms: The content loss function term is: the similarity measure between the virtual 3D image and the real 3D image, and the sum of the similarity measures between the virtual stained image and the original stained image; The discriminant loss function term is the sum of the similarity output by the first discriminator and the similarity output by the second discriminator. The cycle-consistent loss function term is the sum of the similarity measure between the real 3D image and the reconstructed 3D image, and the similarity measure between the original stained image and the reconstructed stained image.

5. The method for obtaining a virtual simulation model for multiphoton imaging according to claim 1, characterized in that, The method for obtaining each sample data is as follows: Acquire several paired complete stained images and complete three-dimensional images corresponding to the target tissue, and segment the complete stained images into multiple stained image blocks of a specified size, and segment the complete three-dimensional images into multiple three-dimensional image blocks of a specified size; The paired stained image blocks are rigidly registered with the three-dimensional image blocks; The pixel values ​​of the rigidly registered stained image block and the three-dimensional image block are normalized to serve as the original stained image of the target tissue and the real three-dimensional image obtained by multiphoton imaging.

6. The method for obtaining a virtual simulation model for multiphoton imaging according to claim 3, characterized in that, The first generator, the second generator, the third generator, and the fourth generator each include a Transformer encoding module and a Transformer decoding module. The Transformer encoding module includes three stacked encoders, each of which includes a feedforward neural network layer and a multi-head attention layer; The Transformer decoding module includes three stacked decoders, each of which includes a feedforward neural network layer and a multi-head attention layer.

7. The method for obtaining a virtual simulation model for multiphoton imaging according to claim 2, characterized in that, The first discriminator and the second discriminator each include a two-dimensional convolutional layer and a fully connected layer.

8. The method for obtaining a virtual simulation model for multiphoton imaging according to claim 1, characterized in that, The staining images of the target tissue are hematoxylin and eosin staining images.

9. A method for generating three-dimensional images using multiphoton imaging, characterized in that, include: Acquire stained images of the tissue to be tested; The stained image of the tissue to be tested is input into a multiphoton imaging virtual simulation model to obtain a virtual three-dimensional image of the tissue to be tested based on multiphoton imaging. The multiphoton imaging virtual simulation model is obtained based on the method for obtaining the multiphoton imaging virtual simulation model according to any one of claims 1 to 8.

10. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method for acquiring a virtual simulation model of multiphoton imaging as described in any one of claims 1 to 8, and / or the method for generating a three-dimensional image of multiphoton imaging as described in claim 9.

Images