Microscope imaging modal migration method, system and storage medium

Abstract

The present invention discloses a microscope imaging modal migration method, system, and storage medium. By utilizing a generative network for model training, the corneal endothelial microscopic image data of domain A and the confocal microscopic image data of domain B are mutually converted to learn the images of the other domain. The generated domain B synthetic image data and domain A synthetic image data are used as inputs to continuously cycle through the training of the generated image data, and the continuously updated domain B synthetic image data and domain A synthetic image data are continuously discriminated. The generated image data can gradually be obtained to be closer and closer to the real image data in the original domain. The training process mainly improves the performance of the generative network by reducing the value of the acquired loss function. Therefore, this method alleviates the limitations of non-contact corneal endothelial cell microscopy imaging technology, better evaluates the clinical parameters of corneal endothelial cells, and provides more reliable evaluation assistance.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a microscope imaging modality migration method, system and storage medium. Background Art

[0002] Corneal endothelial cells are a densely packed, non-renewable layer of cells located on the posterior surface of the cornea, mostly in a hexagonal shape. Besides aging, external damage, corneal diseases, and other factors can lead to corneal endothelial cell death and a decrease in cell density. The cornea requires a sufficient number of endothelial cells to maintain normal function, with a critical density typically at 400-500 cells per square millimeter. Following corneal endothelial cell death, neighboring cells migrate and expand to occupy the area of ​​dead corneal endothelial cells, resulting in unpredictable cell elongation, thinning, and area increase.

[0003] Currently, the main methods for acquiring corneal endothelial cell images are divided into contact corneal endothelial cell microscopes and non-contact microscopes. In vivo confocal microscopy is a contact microscope with clear imaging. Since the equipment needs to be in direct contact with the cornea through the gel during measurement, it is not patient-friendly and difficult to operate. It requires adequate preparation and disinfection before and after the examination, and may cause eye damage. The use of a non-contact corneal endothelial cell microscope can reduce the risk of infection and potential eye damage in the examinee. Since the field of view of the non-contact corneal endothelial microscope is relatively narrow and it cannot avoid the patient's nystagmus, it is prone to low imaging quality factors such as image blur, defocus, uneven lighting, and low contrast, which increases the challenge of downstream tasks of corneal endothelial cell images such as segmentation tasks and parameter estimation tasks. Summary of the Invention

[0004] The present invention aims to address at least one of the technical problems existing in the prior art. To this end, the present invention proposes a microscope imaging modality migration method that addresses the current problem of acquiring corneal endothelial cell images using a contact microscope, which is unfriendly to patients, while using a non-contact microscope results in blurred images.

[0005] The present invention also provides a microscope imaging modality migration system and a computer-readable storage medium.

[0006] The microscope imaging modality transfer method according to the first aspect of the present invention comprises the following steps:

[0007] Preparing a domain A image dataset and a domain B image dataset and constructing a generative network, wherein the domain A is a corneal endothelial imaging domain, the domain B is a confocal microscopy imaging domain, and the domain A image dataset and the domain B image dataset are paired or unpaired;

[0008] Inputting the domain A image dataset and the domain B image dataset into the generative network for model training;

[0009] Training the generative network using a reduced loss function to improve performance;

[0010] The domain A image to be processed is input into the trained generative network for processing to realize an ophthalmic image processing system for domain B modality migration.

[0011] The microscope imaging modal migration method according to the embodiment of the present invention has at least the following beneficial effects:

[0012] By using a generative network for model training, the corneal endothelial image data of domain A and the confocal microscope image data of domain B are converted to each other to learn the images of the other domain, and the generated domain B synthetic image data and domain A synthetic image data are further used as input to continuously cycle the training of the generated image data, and continue to discriminate the continuously updated domain B synthetic image data and domain A synthetic image data, so that the generated image data can gradually become closer to the real image data in the original domain, and the training process mainly improves the performance of the generative network by reducing the value of the acquired loss function. Therefore, by using a generative network in the embodiment method of the present invention, bidirectional modal migration between corneal endothelial imaging and confocal microscope imaging is achieved. Modal migration is conducive to alleviating the limitations of non-contact corneal endothelial cell microscope imaging technology and improving image quality under certain equipment cost constraints. An effective image quality enhancement method is conducive to obtaining more accurate corneal endothelial cell segmentation results, which is conducive to better evaluating clinical parameters related to corneal endothelial cells and providing more reliable evaluation assistance for clinical workers.

[0013] According to some embodiments of the present invention, the generation network includes a first generator, a second generator, a first discriminator, and a second discriminator;

[0014] The step of inputting the domain A image dataset and the domain B image dataset into the generative network for model training comprises the following steps:

[0015] Using the first generator, converting domain A original image data into domain B composite image data, and using the second generator, converting domain B original image data into domain A composite image data, wherein the domain A image dataset includes a plurality of domain A original image data, and the domain B image dataset includes a plurality of domain B original image data;

[0016] transforming the domain A composite image data using the first generator to update the domain B composite image data, and transforming the domain B composite image data using the second generator to update the domain A composite image data;

[0017] The domain B synthetic image data is processed using the first discriminator to obtain a discrimination result, and the domain A synthetic image data is processed using the second discriminator to obtain a discrimination result.

[0018] According to some embodiments of the present invention, the loss function includes a first frequency domain information loss function and a second frequency domain information loss function;

[0019] The method of training the generative network by reducing the loss function to improve performance includes the following steps:

[0020] reducing the first frequency domain information loss function to train the first generator so that the first generator maintains cell boundary pattern consistency;

[0021] The second generator is trained by reducing the second frequency domain information loss function so that the second generator maintains the consistency of the cell boundary pattern.

[0022] According to some embodiments of the present invention, the first generator includes a convolutional layer encoder, an aggregated residual transformer, and a frequency domain convolutional layer decoder, and the first frequency domain information loss function is constrained by the following mathematical model:

[0023]

[0024] in, represents the first frequency domain information loss function, represents the frequency domain image data of domain A, Represents domain B frequency domain image data;

[0025] The first frequency domain information loss function is obtained by the following steps:

[0026] Acquire domain B frequency domain image data, where the domain B frequency domain image data is obtained by sequentially processing the domain A original image data through the convolutional layer encoder, the aggregated residual converter, and the frequency domain convolutional layer decoder;

[0027] Acquire domain A frequency domain image data, wherein the domain A frequency domain image data is obtained by discrete Fourier transforming the domain A original image data;

[0028] A mean square error calculation is performed on the domain B frequency domain image data and the domain A frequency domain image data to obtain the first frequency domain information loss function.

[0029] According to some embodiments of the present invention, the loss function further includes a first discriminator performance function, a second discriminator performance function, and a cycle loss function;

[0030] The method of training the generative network by reducing the loss function to improve performance further includes the following steps:

[0031] reducing the first discriminator performance function to train the first discriminator, so as to improve the ability of the first discriminator to distinguish generated data from original data;

[0032] reducing the performance function of the second discriminator to train the second discriminator, so as to improve the ability of the second discriminator to distinguish between generated data and original data;

[0033] The cycle loss function is reduced to ensure cycle stability during data generation.

[0034] According to some embodiments of the present invention, the first discriminator performance function and the second discriminator performance function are respectively constrained by the following mathematical models:

[0035]

[0036]

[0037] in, represents the first discriminator performance function, represents the second discriminator performance function, a is the image data in domain A, is the image data in domain B generated by the second generator, b is the image data in domain B, The image data in domain A is generated by the first generator, a~p data (a) and b~p data (b) represents samples obtained from data domain A and data domain B respectively, and Denote the expected cycle generation loss calculated using the L1 norm, D b Denotes the first discriminator, D a represents the second discriminator;

[0038] The cycle loss function is constrained by the following mathematical model:

[0039]

[0040] in, Represents the cycle loss function, G represents the first generator, and F represents the second generator.

[0041] According to some embodiments of the present invention, the first generator includes a convolutional layer encoder, an aggregated residual transformer, and an image convolutional layer decoder;

[0042] The method of converting the domain A original image data into domain B synthetic image data by using the first generator includes the following steps:

[0043] Processing the domain A original image data through the convolutional layer encoder to obtain encoded image data;

[0044] Processing the encoded image data through the aggregated residual converter to obtain converted image data;

[0045] The converted image data is processed by the image convolution layer decoder to obtain the domain B synthetic image data.

[0046] According to some embodiments of the present invention, the first discriminator includes a perception module, a plurality of first convolutional layers, and a fully connected layer;

[0047] The step of processing the domain B synthetic image data using the first discriminator to obtain a discrimination result includes the following steps:

[0048] Using the perception module to extract multiple scale features from the domain B synthetic image data, and obtaining first feature extraction data after upsampling;

[0049] Performing a convolution operation on the first feature extraction data through a plurality of the first convolutional layers in sequence to obtain second feature extraction data;

[0050] The second feature extraction data is processed using the fully connected layer to obtain the discrimination result.

[0051] A microscope imaging modality migration system according to a second aspect of the present invention includes:

[0052] a data preparation and model building unit, configured to prepare a domain A image dataset and a domain B image dataset and construct a generative network, wherein the domain A is a corneal endothelial imaging domain, the domain B is a confocal microscopy imaging domain, and the domain A image dataset and the domain B image dataset are paired or unpaired;

[0053] a model training unit, configured to input the domain A image dataset and the domain B image dataset into the generation network for model training;

[0054] A training optimization unit, configured to train the generative network to improve performance by reducing a loss function;

[0055] The modality transfer unit is used to input the domain A image to be processed into the trained generation network for processing, so as to realize the ophthalmic image processing system of domain B modality transfer.

[0056] The microscope imaging modal migration system according to the embodiment of the present invention has at least the following beneficial effects:

[0057] By using a generative network for model training, the corneal endothelial image data of domain A and the confocal microscope image data of domain B are converted to each other to learn the images of the other domain, and the generated domain B synthetic image data and domain A synthetic image data are further used as input to continuously cycle the training of the generated image data, and continue to discriminate the continuously updated domain B synthetic image data and domain A synthetic image data, so that the generated image data can gradually become closer to the real image data in the original domain, and the training process mainly improves the performance of the generative network by reducing the value of the acquired loss function. Therefore, by using a generative network in the embodiment system of the present invention, bidirectional modal migration between corneal endothelial imaging and confocal microscope imaging is achieved. Modal migration is conducive to alleviating the limitations of non-contact corneal endothelial cell microscope imaging technology and improving image quality under certain equipment cost constraints. An effective image quality enhancement method is conducive to obtaining more accurate corneal endothelial cell segmentation results, which is conducive to better evaluating clinical parameters related to corneal endothelial cells and providing more reliable evaluation assistance for clinical workers.

[0058] According to the computer-readable storage medium of the third aspect embodiment of the present invention, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the microscope imaging modality migration method as described in the first aspect embodiment of the present invention.

[0059] It can be understood that the beneficial effects of the third aspect compared with the relevant technology are the same as the beneficial effects of the first aspect compared with the relevant technology. Please refer to the relevant description in the first aspect and no further details will be given here.

[0060] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0062] Figure 1 is a flow chart of a microscope imaging modality migration method according to an embodiment of the present invention;

[0063] Figure 2is a schematic diagram of a generation network according to an embodiment of the present invention;

[0064] Figure 3 is a schematic diagram of a first generator according to an embodiment of the present invention;

[0065] Figure 4 FIG. 1 is a schematic diagram of a first discriminator according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0067] In the description of the present invention, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0068] In the description of the present invention, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0069] In the description of the present invention, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0070] The technical solutions of the present invention will be described clearly and completely below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of the present invention, not all embodiments.

[0071] See also Figure 1 FIG. 1 is a flow chart of a microscope imaging modality migration method provided by an embodiment of the present invention, and the method includes the following steps:

[0072] Prepare domain A image dataset and domain B image dataset and construct a generation network, domain A is a corneal endothelial imaging domain, domain B is a confocal microscope imaging domain, and the domain A image dataset and domain B image dataset are paired or unpaired;

[0073] Input the domain A image dataset and the domain B image dataset into the generative network for model training;

[0074] Use the reduced loss function to train the generative network to improve performance;

[0075] The domain A image to be processed is input into the trained generative network for processing to realize the ophthalmic image processing system of domain B modality transfer.

[0076] Specifically, if Figure 1 As shown, it can be understood that the image data sets of domain A (corneal endothelial imaging) and domain B (confocal microscopy imaging) are first prepared, and a generative network is constructed. Based on the characteristics of the generative network, the domain A image data set and the domain B image data set do not need to be precisely paired, that is, the domain A image data set and the domain B image data set are paired or unpaired. The generative network is then trained using the domain A image data set and the domain B image data set. During the training process, the generative network can be trained by reducing the loss function to improve performance, and finally a trained generative network is obtained. The trained network can generate confocal modal images from the input corneal endothelial image through modal migration, and vice versa. By utilizing the modal migration method of an embodiment of the present invention, the cost and difficulty of acquiring confocal microscopy images can be reduced.

[0077] In this embodiment, by using a generative network for model training, the corneal endothelial image data of domain A and the confocal microscope image data of domain B are converted to each other to learn the images of the other domain, and further the generated domain B synthetic image data and domain A synthetic image data are used as input to continuously cycle the training of the generated image data, and continue to discriminate the continuously updated domain B synthetic image data and domain A synthetic image data, so that the generated image data can gradually be obtained. The real image data in the original domain is closer and closer, and the training process mainly improves the performance of the generative network by reducing the value of the obtained loss function. Therefore, by using a generative network in the embodiment method of the present invention, bidirectional modal migration between corneal endothelial imaging and confocal microscope imaging is achieved. Modal migration is conducive to alleviating the limitations of non-contact corneal endothelial cell microscope imaging technology and improving image quality under certain equipment cost constraints. An effective image quality enhancement method is conducive to obtaining more accurate corneal endothelial cell segmentation results, which is conducive to better evaluating clinical parameters related to corneal endothelial cells and providing more reliable evaluation assistance for clinical workers.

[0078] In some embodiments, as Figure 2 As shown, the generation network includes a first generator, a second generator, a first discriminator, and a second discriminator;

[0079] The domain A image dataset and the domain B image dataset are input into the generative network for model training, which includes the following steps:

[0080] Using a first generator, converting domain A original image data into domain B composite image data, and using a second generator, converting domain B original image data into domain A composite image data, wherein the domain A image dataset includes a plurality of domain A original image data, and the domain B image dataset includes a plurality of domain B original image data;

[0081] Transforming the domain A composite image data using a first generator to update the domain B composite image data, and transforming the domain B composite image data using a second generator to update the domain A composite image data;

[0082] The domain B synthetic image data is processed using the first discriminator to obtain a discrimination result, and the domain A synthetic image data is processed using the second discriminator to obtain a discrimination result.

[0083] Specifically, refer to Figure 2 It will be appreciated that in some embodiments, the generative network is a recurrent generative adversarial network, consisting of two generator-discriminator pairs. First, a first generator and a second generator are used to initialize and learn confocal microscopy image data for domain B and corneal endothelial microscopy image data for domain A, respectively, to generate synthetic image data for domain B and synthetic image data for domain A. Furthermore, the synthetic image data for domain B and domain A are continuously used for cyclic generative updates. That is, the first generator and the second generator are continuously trained and learned, ultimately enabling the first generator to generate image data from domain B with high similarity, and similarly enabling the second generator to generate image data from domain A with high similarity. During the training process, the first and second discriminators are used to determine whether the synthetic image data for domain B and the synthetic image data for domain A each generated exhibit a high degree of similarity. Simultaneously, a loss function is obtained and subjected to function reduction, thereby improving the performance of the generative network during training.

[0084] In some embodiments, the loss function includes a first frequency domain information loss function and a second frequency domain information loss function;

[0085] Using the reduced loss function to train the generative network to improve performance includes the following steps:

[0086] Lowering the first frequency domain information loss function to train the first generator so that the first generator maintains the consistency of the cell boundary pattern;

[0087] The second frequency domain information loss function is reduced to train the second generator so that the second generator maintains the consistency of the cell boundary pattern.

[0088] Specifically, it can be understood that based on the corneal endothelial cell image, its frequency domain information can effectively extract the cell structure contour information. Therefore, in the first generator and the second generator designed in the embodiment of the present invention, the frequency domain information contrast constraint is used to enhance the pattern consistency. Specifically, the generated frequency domain spatial image is compared with the original frequency domain spatial image to train the first generator and the second generator so that they maintain the consistency of the cell boundary pattern.

[0089] In some embodiments, as Figure 3 As shown, the first generator includes a convolutional layer encoder, an aggregated residual converter, and a frequency domain convolutional layer decoder. The first frequency domain information loss function is constrained by the following mathematical model:

[0090]

[0091] in, represents the first frequency domain information loss function, represents the frequency domain image data of domain A, Represents domain B frequency domain image data;

[0092] The first frequency domain information loss function is obtained by the following steps:

[0093] Obtain domain B frequency domain image data, which is obtained by processing domain A original image data through a convolutional layer encoder, an aggregated residual converter, and a frequency domain convolutional layer decoder in sequence;

[0094] Acquire frequency domain image data of domain A, where the frequency domain image data of domain A is obtained by discrete Fourier transforming the original image data of domain A;

[0095] A mean square error is calculated for the frequency domain image data of domain B and the frequency domain image data of domain A to obtain a first frequency domain information loss function.

[0096] Specifically, refer to Figure 3 , the embodiment of the present invention only takes the first generator as an example. It can be understood that the second generator also includes a frequency domain convolution layer decoder, so the second frequency domain information loss function can be obtained in the same way. Furthermore, after the original image data of domain A is processed by the convolution layer encoder and the aggregated residual converter in sequence, it will be processed by the frequency domain convolution layer decoder to extract the frequency domain image data of the generated domain B synthetic image data, that is, the domain B frequency domain image data, and the original frequency domain spatial image, that is, the domain A frequency domain image data, is obtained by performing discrete Fourier transform or fast Fourier transform on the original image data of domain A. By calculating the mean square error of the domain B frequency domain image data and the domain A frequency domain image data, the first frequency domain information loss function can be obtained. Therefore, the first generator can be trained by reducing the mean square error value to better maintain the consistency of the cell boundary pattern.

[0097] Furthermore, the above mathematical model only takes the first frequency domain information loss function as an example, and the second frequency domain information loss function can be obtained in the same way. The embodiment of the present invention considers the frequency domain information loss as a loss in the training process, and therefore constructs a frequency domain information loss function. It can be understood that the two frequency domain information loss functions are obtained by two generators respectively. The frequency domain information loss function specifically calculates the difference between the frequency domain result of the original image input to the generator and the frequency domain image generated by the generator. Since the frequency domain information can represent the cell structure and geometric distribution characteristics of the corneal endothelial cell image, maintaining the frequency domain information can better constrain the consistency of the cell contour in the generation process. By reducing the values ​​of the two frequency domain information loss functions, the purpose of constraining the generated image cell structure and making it as consistent as possible with the source domain input image cell structure can be achieved.

[0098] In some embodiments, the loss function further includes a first discriminator performance function, a second discriminator performance function, and a cycle loss function;

[0099] Training the generative network to improve performance by reducing the loss function also includes the following steps:

[0100] Reducing the performance function of the first discriminator to train the first discriminator, so as to improve the ability of the first discriminator to distinguish between generated data and original data;

[0101] Reducing the performance function of the second discriminator to train the second discriminator to improve the ability of the second discriminator to distinguish between generated data and original data;

[0102] Reduce the cycle loss function to ensure cycle stability during data generation.

[0103] Specifically, it is understood that during the training process, embodiments of the present invention improve the performance of the generator and discriminator by reducing the loss function. As the loss function decreases, the generator is able to generate data that is more similar to the target domain data, and the discriminator is able to improve its ability to distinguish generated data from the original data. The main loss functions should also include the first discriminator performance function, the second discriminator performance function, and the loop loss function.

[0104] In some embodiments, the first discriminator performance function and the second discriminator performance function are respectively constrained by the following mathematical models:

[0105]

[0106]

[0107] in, represents the first discriminator performance function, represents the second discriminator performance function, a is the image data in domain A, The image data in domain B is generated by the second generator, b is the image data in domain B, The image data in domain A is generated by the first generator, a~p data (a) and b~p data (b) represents samples obtained from data domain A and data domain B respectively, and Denote the expected cycle generation loss calculated using the L1 norm, D b represents the first discriminator, D a represents the second discriminator;

[0108] Specifically, the lower the values ​​of the first and second discriminator performance functions, the better the first and second discriminators are at distinguishing the difference between the original image and the generated image. By improving the discriminator performance, the similarity between the generator's generated data and the target domain data is indirectly improved.

[0109] The cycle loss function is constrained by the following mathematical model:

[0110]

[0111] in, represents the cycle loss function, G represents the first generator, and F represents the second generator.

[0112] Furthermore, to ensure cyclic stability during data generation, a cyclic loss function is used during training. This cyclic process first generates data from the original data into a fictitious target domain. Another generator is then used to generate fictitious data belonging to the target domain into data in the same domain as the original data. The cyclic loss function calculates the difference between the original data and the fictitious data in the same domain after the cyclic generation process. Lowering the cyclic loss function allows the data to transfer style during the generation process while preserving the original structure and content as much as possible.

[0113] In some embodiments, as Figure 3 As shown, the first generator includes a convolutional layer encoder, an aggregated residual converter, and an image convolutional layer decoder;

[0114] Using a first generator, converting domain A original image data into domain B synthetic image data includes the following steps:

[0115] Processing the original image data of domain A through a convolutional layer encoder to obtain encoded image data;

[0116] Processing the encoded image data through an aggregated residual transformer to obtain transformed image data;

[0117] The converted image data is processed by the image convolution layer decoder to obtain domain B synthetic image data.

[0118] Specifically, refer to Figure 3 Taking the first generator as an example, the original image data of domain A is first encoded by the convolutional layer encoder, then further processed by the aggregated residual transformer to complete the deep extraction of convolutional features, and finally decoded by the image convolution layer decoder corresponding to the convolutional layer encoder to generate the synthesized image data of domain B. It can be understood that the structure of the second generator in this embodiment of the present invention is the same or similar to that of the first generator, and the main difference lies in the different image data processing domains.

[0119] It should be noted that the aggregated residual transformer, namely the Aggregated Residual Transformations Networks (ResNeXt), inherits the two major features of ResNet's strategy of repeating layers and skip connections, and introduces a novel topological structure, namely the Aggregated Residual Transformations module (ART), which performs convolutional feature extraction in the form of groups; each small module in ResNeXt is stacked in the form of group convolution, and the group convolution in the aggregated residual block can supervise the network to learn more distinctive representation information through multi-way parallel group convolution operations; and it also plays a network regularization effect, and the extracted convolution operations are more sparse, which greatly reduces the complexity of each sub-network, thereby avoiding the risk of model overfitting.

[0120] In some embodiments, as Figure 4 As shown, the first discriminator includes a perception module, multiple first convolutional layers, and a fully connected layer;

[0121] Processing the domain B synthetic image data using the first discriminator to obtain a discrimination result includes the following steps:

[0122] Using the perception module to extract multiple scale features from the domain B synthetic image data, and obtaining first feature extraction data after upsampling;

[0123] Performing a convolution operation on the first feature extraction data through multiple first convolutional layers in sequence to obtain second feature extraction data;

[0124] The second feature extraction data is processed using a fully connected layer to obtain a discrimination result.

[0125] Specifically, refer to Figure 4Taking the first discriminator as an example, the input domain B synthetic image data is processed. First, the perception module extracts features at three scales: x1, x1 / 2, and x1 / 4. The extracted features are upsampled and concatenated before being input to the subsequent part of the first discriminator. The subsequent part of the first discriminator consists of three convolution operations and a fully connected convolution layer with a softmax function. The final output is the probability that the domain B synthetic image data comes from the original domain B dataset rather than the generated data, that is, the quantized discriminant result.

[0126] In some embodiments, as Figure 4 As shown in the figure, the perception module includes an average pooling layer, a second convolutional layer, a batch normalization layer, and a ReLU function connected in sequence.

[0127] In addition, an embodiment of the present invention further provides a microscope imaging modality transfer system, comprising a data preparation and model building unit, a model training unit, a training optimization unit, and a modality transfer unit. The data preparation and model building unit is used to prepare a domain A image dataset and a domain B image dataset and construct a generative network, where domain A is a corneal endothelial imaging domain and domain B is a confocal microscope imaging domain, and the domain A image dataset and domain B image dataset are paired or unpaired; the model training unit is used to input the domain A image dataset and the domain B image dataset into the generative network for model training; the training optimization unit is used to train the generative network by reducing the loss function to improve performance; and the modality transfer unit is used to input the domain A image to be processed into the trained generative network for processing, thereby realizing an ophthalmic image processing system for domain B modality transfer.

[0128] Specifically, refer to Figure 1 It can be understood that the microscope imaging modal migration system of the embodiment of the present application is used to implement the microscope imaging modal migration method. The microscope imaging modal migration system of the embodiment of the present application corresponds to the aforementioned microscope imaging modal migration method. For the specific processing process, please refer to the aforementioned microscope imaging modal migration method, which will not be repeated here.

[0129] In this embodiment, by using a generative network for model training, the corneal endothelial image data of domain A and the confocal microscope image data of domain B are converted to each other to learn the images of the other domain, and the generated domain B synthetic image data and domain A synthetic image data are further used as input to continuously cycle the training of the generated image data, and continue to discriminate the continuously updated domain B synthetic image data and domain A synthetic image data, so that the generated image data can gradually become closer to the real image data in the original domain, and the training process mainly improves the performance of the generative network by reducing the value of the obtained loss function. Therefore, by using a generative network in the system of the embodiment of the present invention, bidirectional modal migration between corneal endothelial imaging and confocal microscope imaging is achieved. Modal migration is conducive to alleviating the limitations of non-contact corneal endothelial cell microscope imaging technology and improving image quality under certain equipment cost constraints. Effective image quality enhancement methods are conducive to obtaining more accurate corneal endothelial cell segmentation results, which is conducive to better evaluation of clinical parameters related to corneal endothelial cells and providing more reliable evaluation assistance for clinical workers.

[0130] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by one or more control processors, which can enable the one or more control processors to execute a microscope imaging modality migration method in the above method embodiment, for example, to execute the above described Figure 1 The method in .

[0131] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0132] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the scope of the present invention.

Claims

1. A microscope imaging modal migration method, characterized in that: The following steps are involved: Preparing a domain A image dataset and a domain B image dataset and constructing a generative network, wherein the domain A is a corneal endothelial imaging domain, the domain B is a confocal microscopy imaging domain, and the domain A image dataset and the domain B image dataset are paired or unpaired; Inputting the domain A image dataset and the domain B image dataset into the generative network for model training; Training the generative network using a reduced loss function to improve performance; Inputting the domain A image to be processed into the trained generative network for processing, so as to realize the ophthalmic image processing system of domain B modality transfer; The generative network includes a first generator, a second generator, a first discriminator, and a second discriminator; and inputting the domain A image dataset and the domain B image dataset into the generative network for model training includes the following steps: Using the first generator, converting domain A original image data into domain B composite image data, and using the second generator, converting domain B original image data into domain A composite image data, wherein the domain A image dataset includes a plurality of domain A original image data, and the domain B image dataset includes a plurality of domain B original image data; transforming the domain A composite image data using the first generator to update the domain B composite image data, and transforming the domain B composite image data using the second generator to update the domain A composite image data; Processing the domain B synthetic image data using the first discriminator to obtain a discrimination result, and processing the domain A synthetic image data using the second discriminator to obtain a discrimination result; The loss function includes a first frequency domain information loss function, the first generator includes a convolutional layer encoder, an aggregated residual converter, and a frequency domain convolutional layer decoder; the first frequency domain information loss function is obtained by the following steps: Acquire domain B frequency domain image data, where the domain B frequency domain image data is obtained by sequentially processing the domain A original image data through the convolutional layer encoder, the aggregated residual converter, and the frequency domain convolutional layer decoder; Acquire domain A frequency domain image data, wherein the domain A frequency domain image data is obtained by discrete Fourier transforming the domain A original image data; performing mean square error calculation on the domain B frequency domain image data and the domain A frequency domain image data to obtain the first frequency domain information loss function; The first discriminator includes a perception module, a plurality of first convolutional layers, and a fully connected layer; and the processing of the domain B synthetic image data by the first discriminator to obtain a discrimination result includes the following steps: Using the perception module to extract multiple scale features from the domain B synthetic image data, and obtaining first feature extraction data after upsampling; Performing a convolution operation on the first feature extraction data through a plurality of the first convolutional layers in sequence to obtain second feature extraction data; The second feature extraction data is processed using the fully connected layer to obtain the discrimination result.

2. The microscope imaging modal migration method according to claim 1, characterized in that: The loss function includes a first frequency domain information loss function and a second frequency domain information loss function; The method of training the generative network by reducing the loss function to improve performance includes the following steps: reducing the first frequency domain information loss function to train the first generator so that the first generator maintains cell boundary pattern consistency; The second generator is trained by reducing the second frequency domain information loss function so that the second generator maintains the consistency of the cell boundary pattern.

3. The microscope imaging modal migration method according to claim 2, characterized in that: The first generator includes a convolutional layer encoder, an aggregated residual converter, and a frequency domain convolutional layer decoder, and the first frequency domain information loss function is constrained by the following mathematical model: ， in, represents the first frequency domain information loss function, represents the frequency domain image data of domain A, Represents domain B frequency domain image data.

4. The microscope imaging modal migration method according to claim 1, characterized in that: The loss function also includes a first discriminator performance function, a second discriminator performance function, and a cycle loss function; The method of training the generative network by reducing the loss function to improve performance further includes the following steps: reducing the first discriminator performance function to train the first discriminator, so as to improve the ability of the first discriminator to distinguish generated data from original data; reducing the performance function of the second discriminator to train the second discriminator, so as to improve the ability of the second discriminator to distinguish between generated data and original data; The cycle loss function is reduced to ensure cycle stability during data generation.

5. The microscope imaging modal migration method according to claim 4, characterized in that: The first discriminator performance function and the second discriminator performance function are respectively constrained by the following mathematical models: in, represents the first discriminator performance function, represents the second discriminator performance function, is the image data in domain A, The image data in domain B is generated by the second generator, is the image data in domain B, The image data in domain A is generated by the first generator, and Represent samples obtained from data domain A and data domain B respectively, and They represent the expected cycle generation loss calculated using the L1 norm, represents the first discriminator, represents the second discriminator; The cycle loss function is constrained by the following mathematical model: in, represents the cycle loss function, represents the first generator, Represents the second generator.

6. The microscope imaging modal migration method according to claim 1, characterized in that: The first generator includes a convolutional layer encoder, an aggregated residual converter, and an image convolutional layer decoder; The method of converting the domain A original image data into domain B synthetic image data by using the first generator includes the following steps: Processing the domain A original image data through the convolutional layer encoder to obtain encoded image data; Processing the encoded image data through the aggregated residual converter to obtain converted image data; The converted image data is processed by the image convolution layer decoder to obtain the domain B synthetic image data.

7. A microscope imaging modal migration system, characterized in that: The microscope imaging modal migration method according to any one of claims 1 to 6 comprises: a data preparation and model building unit, configured to prepare a domain A image dataset and a domain B image dataset and construct a generative network, wherein the domain A is a corneal endothelial imaging domain, the domain B is a confocal microscopy imaging domain, and the domain A image dataset and the domain B image dataset are paired or unpaired; a model training unit, configured to input the domain A image dataset and the domain B image dataset into the generation network for model training; A training optimization unit, configured to train the generative network to improve performance by reducing a loss function; The modality transfer unit is used to input the domain A image to be processed into the trained generation network for processing, so as to realize the ophthalmic image processing system of domain B modality transfer.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the microscope imaging modality migration method according to any one of claims 1 to 6.

Images