A method and system for generating postoperative contour images of ophthalmic OCT based on domain adaptation
Through the domain adaptation-based ophthalmic OCT postoperative contour map generation method, the generator, discriminator and domain classifier models are used to solve the problem of difficulty in generating postoperative OCT images in the prior art, and efficient and accurate contour maps are achieved Generate, supports doctors to evaluate postoperative recovery effects and expand the scale of training data.
Patent Information
- Application Number
- CN202210487626.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-06
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-05-06
AI Technical Summary
When using deep learning technology to generate postoperative images of ophthalmic OCT, the prior art has difficulty in image generation and poor effect, and it is difficult to accurately evaluate the surgical treatment effect.
Using the domain adaptation-based ophthalmic OCT postoperative contour map generation method, by constructing a model including generator, discriminator and domain classifier, the convolutional neural network is used to extract features and generate postoperative contour maps, combined with Multi-objective loss function is trained to generate accurate postoperative contour maps.
Reduces the technical difficulty of generating high-resolution postoperative OCT images, improves the accuracy and efficiency of generating contour maps, reduces noise, enables doctors to evaluate patients' postoperative recovery effects, and expands the scale of training data, improving deep learning models. effect.
Smart Images

Figure CN115147273B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of retinal image simulation generation, and in particular to a method and system for generating an ophthalmic OCT postoperative contour image based on domain adaptation. Background Art
[0002] OCT examination in ophthalmology is the abbreviation of optical coherence tomography. It is a tomography technology that uses the interference phenomenon of light to form an image. Combined with computer imaging technology, it can observe the cross-sectional image of the ultra-microscopic tissue structure in the eye. It is a non-invasive and non-contact examination with high resolution and fast imaging, and is called CT in ophthalmology. OCT has important value in clinical research on retinal diseases, macular diseases, optic nerve diseases, glaucoma, etc. It can provide valuable basis for the diagnosis and differential diagnosis of retinal diseases, especially macular diseases, observe the morphological changes of subtle lesions, directly measure tissues, and quantitatively analyze fundus lesions. It can find some extremely tiny fundus lesions and distinguish between bloody and serous detachments; it can provide valuable information for the selection of surgical indications for epiretinal membranes, macular holes, submacular choroidal neovascular membranes, vitreomacular traction syndrome, etc., and can also be used as one of the bases for evaluating the success of surgical treatment; it can track the changes of certain lesions and the treatment effects.
[0003] OCT is effective in diagnosing eye diseases, but the cost is relatively high. Some patients with eye diseases need follow-up after surgery, such as persistent subretinal fluid (SRF) after rhegmatogenous retinal detachment (RRD). Patients need to undergo relevant clinical examinations before surgery, 1 month after surgery, and 3 months after surgery. Multiple OCT scans are required during this period, which will impose a considerable financial burden on patients. In order to solve this problem, some studies have pointed out that deep learning technology can be used to directly generate postoperative OCT images based on preoperative OCT images. However, due to the high resolution of OCT images and limited paired annotation data, image generation is difficult and the effect is poor. It is easy to have problems such as ghosting and blurring, making it difficult to accurately evaluate the effect of surgical treatment.
[0004] The prior art discloses a feasible and effective method for generating retinal OCT disease images. The generated OCT images can be used to expand the training data set of the OCT disease image classification algorithm. The method is based on the conditional generative adversarial network cGAN, and the network structure consists of a generator and a discriminator. By combining the generative adversarial loss function cGAN loss with a new structural similarity loss function SSIM loss, normal retinal OCT images are converted into OCT disease images that can be used to supplement the classification model training set. This patent automatically generates retinal OCT images of various lesions based on existing OCT images by training a generative adversarial network. Due to the high resolution of OCT images and limited paired annotation data, image generation is difficult and the effect is poor. Summary of the invention
[0005] The purpose of the present invention is to provide a simple, efficient and highly accurate method and system for generating postoperative contour images of ophthalmological OCT based on domain adaptation.
[0006] In order to achieve the above object, the present invention provides a method for generating an ophthalmic OCT postoperative contour map based on domain adaptation, comprising the following steps:
[0007] S1. Collect preoperative OCT images A and postoperative OCT images B of different fundus diseases as a data set;
[0008] S2, divide the dataset into different fields according to the types of fundus diseases;
[0009] S3, according to the preoperative OCT image A, obtain the preoperative OCT contour map AL; according to the postoperative OCT image B corresponding to the preoperative OCT image A, obtain the postoperative real contour map real_BL to form a new data set;
[0010] S4. Construct an ophthalmic OCT postoperative contour image generation model based on domain adaptation. The ophthalmic OCT postoperative contour image generation model based on domain adaptation includes a generator G1, a discriminator D1, and a domain classifier C. The data set obtained in S3 is sent to the ophthalmic OCT postoperative contour image generation model based on domain adaptation for training, including:
[0011] S4.1 extracts features from the preoperative OCT contour map AL through a convolutional neural network and converts it into a tensor representation; then it is input into the generator G1 to obtain the predicted postoperative generated contour map fake_BL;
[0012] S4.2, input the preoperative OCT contour image AL, the postoperative real contour image real_BL and the postoperative generated contour image fake_BL into the discriminator D1, calculate the discriminator loss loss_D1 according to the discrimination results and update the parameters of the discriminator D1;
[0013] S4.3, input the preoperative OCT contour image AL, the postoperative real OCT contour image real_BL and the postoperative generated contour image fake_BL into the domain classifier C, calculate the domain classifier loss loss_C according to the discrimination result of the domain and update the parameters of the domain classifier C;
[0014] S4.4. Calculate the generator loss loss_G1 and update the parameters of the generator G1, where the discrimination results of the discriminator D1 and the domain classifier C participate in the generator loss loss_G;
[0015] S4.5, repeating steps S4.2, S4.3 and S4.4 until the training is completed, and obtaining a trained domain adaptation-based ophthalmic OCT postoperative contour image generation model;
[0016] S5. Obtain a preoperative OCT contour map corresponding to a newly captured preoperative OCT image, input the preoperative OCT contour map corresponding to the newly captured preoperative OCT image into the generator G1 of the trained domain adaptation-based ophthalmic OCT postoperative contour map generation model, and output the corresponding postoperative generated OCT contour map.
[0017] As a preferred solution, in step S4.2, the preoperative OCT contour image AL and the postoperative generated contour image fake_BL are concatenated to obtain negative_img_D1, and then negative_img_D1 is sent to the discriminator D1 to obtain the tensor pred_fake_D1, and the discriminator D1 labels it with 0, that is, the discriminator D1 believes that this image is fake, and calculates the GAN loss loss_D1_fake of pred_fake_D1;
[0018] The preoperative OCT contour image AL and the postoperative real OCT contour image real_BL are concatenated to obtain positive_img_D1, and then positive_img_D1 is sent to the discriminator D1 to obtain pred_real_D1. The discriminator D1 labels it with 1, that is, the discriminator D1 believes that this image is real, and calculates the GAN loss loss_D1_real of pred_real_D1;
[0019] Add the GAN losses of loss_D1_fake and loss_D1_real to get the discriminator loss loss_D1.
[0020] As a preferred solution, in step S4.3, the preoperative OCT contour image AL and the postoperative generated contour image fake_BL are concatenated to obtain negative_img_C, and then negative_img_C is sent to the domain classifier C to obtain the tensor pred_fake_C. The label marked by the domain classifier C is not the label corresponding to the field in which it is located, that is, the domain classifier C does not identify the correct field of this image, and the GAN loss loss_C_fake of pred_fake_C is calculated;
[0021] The preoperative OCT contour image AL and the postoperative real OCT contour image real_BL are concatenated to obtain positive_img_C, and then positive_img_C is sent to the domain classifier C to obtain pred_real_C. The domain classifier C is labeled with the corresponding label of its domain, that is, the domain classifier C determines the domain of this image and calculates the GAN loss loss_C_real of pred_real_C;
[0022] Add the GAN losses of loss_C_fake and loss_C_real to get the domain classifier loss loss_C.
[0023] As a preferred solution, in step S4.4, it includes:
[0024] S4.4.1. Get the GAN loss loss_D1_real of pred_real_D1 in step S4.2.
[0025] S4.4.2. Get the GAN loss loss_C_fake of pred_fake_C in step 4.3;
[0026] S4.4.3. Calculate the pixel-by-pixel loss L1 of pred_fake_D1 and pred_real_D1 in step S4.2;
[0027] S4.4.3, calculate the pixel-by-pixel loss L2 of pred_fake_C and pred_real_C in step S4.3;
[0028] S4.4.5. Calculate the VGG feature loss of the preoperative OCT image A and the postoperative generated contour map fake_BL, denoted as loss_G_VGG;
[0029] S4.4.6. Add loss_D1_real, loss_C_fake, L1, L2 and loss_G_VGG to get the generator loss loss_G.
[0030] As a preferred solution, both the discriminator D1 and the domain classifier C are multi-scale discriminators.
[0031] As a preferred solution, the generator G1 includes a global generator network G1.1 and a local enhancement network G1.2. The input and output resolutions of the global generator network G1.1 are consistent, and the output size of the local enhancement network G1.2 is 4 times the input size. The local enhancement network G1.2 is used to expand the resolution of the image generated by the global generator network G1.1. During training, the preoperative OCT contour map AL is input into the global generator network G1.1, and the image output by the global generator network G1 is input into the local enhancement network G1.2.
[0032] As a preferred solution, the network structure of the global generator network G1.1 is U-Net, which is responsible for encoding and decoding image features.
[0033] As a preferred solution, in step S2, the inner surface of the nerve fiber layer, the inner surface of the inner nuclear layer, the inner surface of the outer nuclear layer, the ellipsoidal band and the outer surface of the RPE in the preoperative OCT image A and the postoperative OCT image B are outlined to obtain the preoperative OCT contour image AL and the postoperative real contour image real_BL.
[0034] As a preferred solution, in step S2, the preoperative OCT image A is input into the preoperative OCT contour map generation model, and the preoperative OCT contour map AL is output; the preoperative contour map generation model is a generative adversarial network; the preoperative contour map generation model includes a generator G2 and a discriminator D2.
[0035] The present invention also provides a system for generating an ophthalmic OCT postoperative contour map based on domain adaptation, comprising:
[0036] A data acquisition module, used to acquire preoperative OCT images and postoperative OCT images of different fundus diseases to form a data set;
[0037] The classification module is used to classify the dataset into different fields according to the types of fundus diseases;
[0038] The contour image acquisition module is used to obtain a preoperative OCT contour image AL according to the preoperative OCT image A; and obtain a postoperative real contour image real_BL according to the postoperative OCT image B corresponding to the preoperative OCT image A;
[0039] A model building module is used to build an ophthalmic OCT postoperative contour map generation model based on domain adaptation. The ophthalmic OCT postoperative contour map generation model based on domain adaptation includes a generator G1, a discriminator D1 and a domain classifier C. The generator G1 is used to generate a postoperative generated contour map fake_BL according to a preoperative OCT image A. The discriminator D1 is used to distinguish between the postoperative generated contour map fake_BL and the postoperative real OCT contour map real_BL. The domain classifier C is used to distinguish between the domains of the postoperative generated contour map fake_BL and the postoperative real OCT contour map real_BL.
[0040] A loss function construction module is used to construct a multi-objective loss function; the multi-objective loss function includes the loss function of the generator G1, the loss function of the discriminator D1 and the loss function of the domain classifier C;
[0041] A training module, used for taking the samples consisting of the preoperative OCT contour map AL and the postoperative real OCT contour map real_BL as the input of the ophthalmic OCT postoperative contour map generation model based on domain adaptation, iteratively training the ophthalmic OCT postoperative contour map generation model based on domain adaptation based on a multi-objective loss function, and obtaining a trained ophthalmic OCT postoperative contour map generation model based on domain adaptation;
[0042] The application module is used to input the preoperative OCT contour map corresponding to the newly captured preoperative OCT image into the generator G1 of the trained domain adaptation-based ophthalmic OCT postoperative contour map generation model to obtain the corresponding postoperative generated OCT contour map.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] The present invention obtains a preoperative OCT contour map corresponding to a preoperative OCT image, and generates a postoperative OCT contour map according to the preoperative OCT contour map through an ophthalmic OCT postoperative contour map generation model based on domain adaptation. Compared with directly generating a postoperative OCT image with high resolution and complex texture information, the present invention has significantly reduced technical difficulty, is simple and efficient, and has extremely low noise in the generated contour map, so that the generated contour map has high accuracy. The postoperative OCT contour map can be used to measure some key pathological indicators without the assistance of postoperative OCT images. Doctors can also evaluate the postoperative recovery effect of patients to a certain extent, so the present invention is highly practical. In addition, the intrinsic connection between preoperative OCT images of different fundus diseases and their postoperative OCT contour maps has certain similarities. The present invention uses transfer learning technology to integrate data from multiple sources in the source field and the target field, effectively expanding the scale of training data and improving the training effect of the deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1It is a flowchart of a method for generating an ophthalmic OCT postoperative contour map based on domain adaptation according to an embodiment of the present invention.
[0046] Figure 2 It is a framework diagram of a model for generating postoperative contour images of ophthalmic OCT based on domain adaptation in an embodiment of the present invention.
[0047] Figure 3 4 is a framework diagram of the discriminator D1 of the ophthalmic OCT postoperative contour image generation model based on domain adaptation in an embodiment of the present invention.
[0048] Figure 4 4 is a framework diagram of a domain classifier C of a domain adaptation-based ophthalmic OCT postoperative contour image generation model according to an embodiment of the present invention.
[0049] Figure 5 This is a framework diagram of the generator G1 of the domain adaptation-based ophthalmic OCT postoperative contour image generation model in an embodiment of the present invention.
[0050] Figure 6 This is a framework diagram of the global generator network G1.1 of the domain adaptation-based ophthalmic OCT postoperative contour image generation model in an embodiment of the present invention.
[0051] Figure 7 It is a principle block diagram of a system for generating an ophthalmic OCT postoperative contour image based on domain adaptation according to an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0053] Embodiment 1
[0054] like Figures 1 to 6 As shown, a method for generating an ophthalmic OCT postoperative contour map based on domain adaptation in a preferred embodiment of the present invention is characterized by comprising the following steps:
[0055] S1. Collect preoperative OCT images A and postoperative OCT images B of different fundus diseases as a data set;
[0056] S2, divide the dataset into different fields according to the types of fundus diseases;
[0057] S3, according to the preoperative OCT image A, obtain the preoperative OCT contour map AL; according to the postoperative OCT image B corresponding to the preoperative OCT image A, obtain the postoperative real contour map real_BL to form a new data set;
[0058] S4. Construct an ophthalmic OCT postoperative contour image generation model based on domain adaptation. The ophthalmic OCT postoperative contour image generation model based on domain adaptation includes a generator G1, a discriminator D1, and a domain classifier C. The data set obtained in S3 is sent to the ophthalmic OCT postoperative contour image generation model based on domain adaptation for training, including:
[0059] S4.1 extracts features from the preoperative OCT contour map AL through a convolutional neural network and converts it into a tensor (multidimensional vector) representation; then it is input into the generator G1 to obtain the predicted postoperative generated contour map fake_BL;
[0060] S4.2, input the preoperative OCT contour image AL, the postoperative real contour image real_BL and the postoperative generated contour image fake_BL into the discriminator D1, calculate the discriminator loss loss_D1 according to the discrimination results and update the parameters of the discriminator D1;
[0061] S4.3, input the preoperative OCT contour image AL, the postoperative real OCT contour image real_BL and the postoperative generated contour image fake_BL into the domain classifier C, calculate the domain classifier loss loss_C according to the discrimination result of the domain and update the parameters of the domain classifier C;
[0062] S4.4. Calculate the generator loss loss_G1 and update the parameters of the generator G1, where the discrimination results of the discriminator D1 and the domain classifier C participate in the generator loss loss_G;
[0063] S4.5, repeating steps S4.2, S4.3 and S4.4 until the training is completed, and a trained domain adaptation-based ophthalmic OCT postoperative contour map generation model is obtained; the training of this embodiment is stopped until the validation set index tends to be optimal to prevent overfitting;
[0064] S5. Obtain a preoperative OCT contour map corresponding to a newly captured preoperative OCT image, input the preoperative OCT contour map corresponding to the newly captured preoperative OCT image into the generator G1 of the trained domain adaptation-based ophthalmic OCT postoperative contour map generation model, and output the corresponding postoperative generated OCT contour map.
[0065] This embodiment obtains the preoperative OCT contour map corresponding to the preoperative OCT image, and generates the postoperative OCT contour map based on the domain adaptation ophthalmic OCT postoperative contour map generation model according to the preoperative OCT contour map. Compared with directly generating high-resolution postoperative OCT images with complex texture information, the technical difficulty is significantly reduced, simple and efficient, and the noise of the generated contour map is extremely small, so that the generated contour map has high accuracy. The postoperative OCT contour map can be used to measure some key pathological indicators without the assistance of postoperative OCT images. Doctors can also evaluate the postoperative recovery effect of patients to a certain extent, which is highly practical. In addition, the inherent connection between the preoperative OCT images of different fundus diseases and their postoperative OCT contour maps has certain similarities. This embodiment uses transfer learning technology to integrate data from multiple sources in the source domain and the target domain, effectively expands the scale of training data, and improves the training effect of the deep learning model. This embodiment effectively integrates deep learning and transfer learning technologies, makes full use of preoperative OCT images and their features, predicts and generates postoperative OCT contour maps of patients, and then calculates relevant pathological indicators, effectively assisting doctors in evaluating the postoperative recovery effect of patients. Through transfer learning, the model proposed in the present invention takes multimodal data into consideration, and the training data comes from multiple fields rather than single modality data. By comprehensively considering multiple factors, the generated postoperative OCT contour map is more accurate and has richer feature details.
[0066] The contour map of this embodiment is obtained by outlining the inner surface of the nerve fiber layer, the inner surface of the inner nuclear layer, the inner surface of the outer nuclear layer, the ellipsoidal band, and the outer surface of the RPE.
[0067] In an embodiment, the ophthalmic OCT postoperative contour map generation model based on domain adaptation is a generative adversarial network. In step S2, the preoperative OCT contour map AL is obtained by an ophthalmologist annotating the contours of the preoperative OCT image A. The preoperative OCT contour map AL is paired with its corresponding postoperative real OCT contour map real_BL and used as a labeled sample to train the network to automatically generate the postoperative contour map fake_BL from the preoperative OCT contour map AL. The ophthalmic OCT postoperative contour map generation model based on domain adaptation mainly consists of three parts: a generator G1, a discriminator D1, and a domain classifier C. The training data includes a source domain and a target domain. The so-called domain refers to a data set corresponding to the disease type of the OCT image. In order to make the generated OCT contour map have accurate target features, the data set is divided into a source domain and a target domain. The input preoperative OCT contour map AL and the postoperative real OCT contour map real_BL pairing data may come from the source domain or the target domain. During the adversarial training process, the model first extracts the image features of the paired data of the preoperative OCT contour image AL and the postoperative real OCT contour image real_BL, and then sends them to the discriminator D1 and the domain classifier C respectively, calculates the loss and iterates the network parameters of the discriminator D1 and the domain classifier C, so that the discriminator D1 can distinguish the postoperative real OCT contour image real_BL and the postoperative generated contour image fake_BL, while the domain classifier C cannot identify the source domain of the preoperative OCT contour image AL. Finally, the network parameters of the discriminator D1 and the domain classifier C are fixed, and the generator G1 generates an accurate postoperative generated contour image fake_BL.
[0068] Optionally, the discriminator D1 and the domain classifier C in this embodiment are both multi-scale discriminators.
[0069] Specifically, in step S2, an ophthalmologist outlines the inner surface of the nerve fiber layer, the inner surface of the inner nuclear layer, the inner surface of the outer nuclear layer, the ellipsoid band and the outer surface of the RPE in the preoperative OCT image A and the postoperative OCT image B to obtain the preoperative OCT contour map AL and the postoperative real contour map real_BL.
[0070] Updating the discriminator D1 can reduce the loss value calculated by its loss function, that is, the discriminator D1 can accurately distinguish the real postoperative contour image real_BL and the postoperative generated contour image fake_BL annotated by an ophthalmologist. However, the discriminator D1 after adversarial training has difficulty distinguishing the real postoperative contour image real_BL and the postoperative generated contour image fake_BL, which means that the postoperative generated contour image fake_BL has the image features of the real postoperative contour image real_BL, and can be regarded as the generated image is accurate. In step S4.2 of this embodiment, the preoperative OCT contour image AL and the postoperative generated contour image fake_BL are spliced together to obtain negative_img_D1, and then negative_img_D1 is sent to the discriminator D1 to obtain the tensor pred_fake_D1, and the discriminator D1 labels it with 0, that is, the discriminator D1 believes that this image is fake, and calculates the GAN loss loss_D1_fake of pred_fake_D1; the preoperative OCT contour image AL and the postoperative real OCT are spliced together. The contour image real_BL is spliced together to get positive_img_D1, and then positive_img_D1 is sent to the discriminator D1 to get pred_real_D1. The discriminator D1 labels it with 1, that is, the discriminator D1 believes that this image is real, and calculates the GAN loss loss_D1_real of pred_real_D1; the GAN losses of loss_D1_fake and loss_D1_real are added to get the discriminator loss loss_D1.
[0071] In addition, the domain classifier C is updated to reduce the loss value calculated by its loss function, that is, the domain classifier C can accurately distinguish the source of the image domain and discriminate the domain of the postoperative generated contour map fake_BL generated by the generator G1. When the domain classifier C after training and optimization cannot identify the domain of the postoperative generated contour map fake_BL, it means that the postoperative generated contour map fake_BL generated by the generator G1 has learned the common features of the postoperative real contour map real_BL in each target domain, that is, the target features to be generated. In step S4.3 of this embodiment, the preoperative OCT contour image AL and the postoperative generated contour image fake_BL are spliced together to obtain negative_img_C, and then negative_img_C is sent to the domain classifier C to obtain the tensor pred_fake_C. The label marked by the domain classifier C is not the label corresponding to the field to which it belongs, that is, the domain classifier C does not determine the correct field to which this image belongs, and the GAN loss loss_C_fake of pred_fake_C is calculated; the preoperative OCT contour image AL and the real postoperative OCT contour image real_BL are spliced together to obtain positive_img_C, and then positive_img_C is sent to the domain classifier C to obtain pred_real_C. The domain classifier C is marked with the label corresponding to the field to which it belongs, that is, the domain classifier C determines the field to which this image belongs, and the GAN loss loss_C_real of pred_real_C is calculated; the GAN losses of loss_C_fake and loss_C_real are added to obtain the domain classifier loss loss_C.
[0072] In addition, the generator G1 is updated to reduce the loss value calculated by its loss function, so that the generated image is realistic enough and has the main features of the target generated image. The generator G1 includes a global generator network G1.1 and a local enhancement network G1.2. The input and output resolutions of the global generator network G1.1 are consistent (such as 1024*512), and the output size (2048*1024) of the local enhancement network G1.2 is 4 times the input size (1024*512), that is, the height and width of the output image of the local enhancement network G1.2 are both twice the input image; the local enhancement network G1.2 is used to expand the resolution of the image generated by the global generator network G1.1. During training, the preoperative OCT contour map AL is input into the global generator network G1.1, and the image output by the global generator network G1 is input into the local enhancement network G1.2. The network structure of the global generator network G1.1 is U-Net, which is responsible for encoding and decoding image features; the local enhancement network G1.2 enables the model to generate images with higher resolution.
[0073] In step S4.4 of this embodiment, the following steps are included:
[0074] S4.4.1. Get the GAN loss loss_D1_real of pred_real_D1 in step S4.2.
[0075] S4.4.2. Get the GAN loss loss_C_fake of pred_fake_C in step 4.3;
[0076] S4.4.3. Calculate the pixel-by-pixel loss L1 of pred_fake_D1 and pred_real_D1 in step S4.2;
[0077] S4.4.3, calculate the pixel-by-pixel loss L2 of pred_fake_C and pred_real_C in step S4.3;
[0078] S4.4.5. Calculate the VGG feature loss of the preoperative OCT image A and the postoperative generated contour map fake_BL, denoted as loss_G_VGG;
[0079] S4.4.6. Add loss_D1_real, loss_C_fake, L1, L2 and loss_G_VGG to get the generator loss loss_G.
[0080] The loss function of generator G1 consists of three parts: GAN loss, pixel-by-pixel loss, and VGG loss, which is conducive to model parameter optimization. Optionally, the GAN loss can be calculated using the MSE function, BCE function, etc.
[0081] Therefore, this embodiment first collects the patient's preoperative OCT image A and paired postoperative OCT image B. The collected images contain a variety of fundus diseases, and the images are required to be clear and have high resolution, which are used to train the deep transfer learning model-the ophthalmic OCT postoperative contour map generation model based on domain adaptation, and the model is trained using a data set processed by medical professionals, so that the optimized model weight parameters can bring better image generation effects; this embodiment uses three data sets consisting of OCT images of diabetic retinopathy, RVO and AMD to train the model; then the collected OCT image data sets are classified and labeled with domain labels, and then the OCT contour map is annotated and preprocessed, and 512*512 image data is retained; then the ophthalmic OCT postoperative contour map generation model based on domain adaptation is trained, and the postoperative real contour map real_BL is extracted by the convolutional neural network and sent to the discriminator D1 and the domain classifier C for training; the preoperative OCT contour map AL is respectively input into the generator G1, the discriminator D1 and the domain classifier C for training, and the iteration is stopped before the validation set index is overfitted and the respective model weights are retained. And the loss function is calculated, and the weight parameter value of the model is adjusted according to the calculated loss function value to minimize the function value, thereby obtaining a smaller loss function result and an optimized model weight, thereby iterating the network parameters of the generator G1, the discriminator D1 and the domain classifier C, so that the generator G1 can accurately generate the postoperative OCT contour map. This embodiment integrates deep learning and transfer learning, and by expanding the scale of training data, it can more accurately extract the features of OCT images of various eye diseases, improve the generation results of postoperative OCT contour maps, and effectively enhance the model prediction ability.
[0082] Embodiment 2
[0083] The difference between this embodiment and the first embodiment is that, based on the first embodiment, this embodiment also includes a preoperative OCT contour image generation model
[0084] In step S2, the preoperative OCT image A is input into the preoperative contour map generation model, and the preoperative OCT contour map AL is output; the preoperative contour map generation model is a generative adversarial network; the preoperative contour map generation model includes a generator G2 and a discriminator D2. The preoperative contour map generation model is trained based on the domain adaptation of the ophthalmic OCT postoperative contour map generation model according to the preoperative OCT contour map AL generated by the preoperative OCT image A, which can reduce the workload of annotation.
[0085] In step S5, the newly captured preoperative OCT image is input into the trained preoperative contour image generation model, and the corresponding preoperative OCT contour image is output. The preoperative OCT contour image output by the preoperative contour image generation model is then input into the generator G1 of the trained domain adaptation-based ophthalmic OCT postoperative contour image generation model, and the corresponding postoperative generated OCT contour image is output.
[0086] Therefore, the method of this embodiment includes two stages. In the first stage, a preoperative contour map generation model is trained so that the preoperative contour map generation model can generate an accurate preoperative OCT contour map; in the second stage, an ophthalmic OCT postoperative contour map generation model based on domain adaptation is trained so that the ophthalmic OCT postoperative contour map generation model based on domain adaptation can generate an accurate postoperative OCT contour map based on the preoperative OCT contour map.
[0087] Optionally, the preoperative contour map generation model of this embodiment may adopt models such as classic pix2pix, pix2pixHD and SPADE to achieve the conversion of preoperative OCT images to preoperative OCT contour maps.
[0088] The other steps of this embodiment are the same as those of the first embodiment and will not be described again here.
[0089] Embodiment 3
[0090] This embodiment proposes a system for generating an ophthalmic OCT postoperative contour map based on domain adaptation according to the first or second embodiment, including:
[0091] A data acquisition module, used to acquire preoperative OCT images and postoperative OCT images of different fundus diseases to form a data set;
[0092] The classification module is used to classify the dataset into different fields according to the types of fundus diseases;
[0093] The contour image acquisition module is used to obtain a preoperative OCT contour image AL according to the preoperative OCT image A; and obtain a postoperative real contour image real_BL according to the postoperative OCT image B corresponding to the preoperative OCT image A;
[0094] A model building module is used to build an ophthalmic OCT postoperative contour map generation model based on domain adaptation. The ophthalmic OCT postoperative contour map generation model based on domain adaptation includes a generator G1, a discriminator D1 and a domain classifier C. The generator G1 is used to generate a postoperative generated contour map fake_BL according to a preoperative OCT image A. The discriminator D1 is used to distinguish between the postoperative generated contour map fake_BL and the postoperative real OCT contour map real_BL. The domain classifier C is used to distinguish between the domains of the postoperative generated contour map fake_BL and the postoperative real OCT contour map real_BL.
[0095] A loss function construction module is used to construct a multi-objective loss function; the multi-objective loss function includes the loss function of the generator G1, the loss function of the discriminator D1 and the loss function of the domain classifier C;
[0096] A training module, used for taking the samples consisting of the preoperative OCT contour map AL and the postoperative real OCT contour map real_BL as the input of the ophthalmic OCT postoperative contour map generation model based on domain adaptation, iteratively training the ophthalmic OCT postoperative contour map generation model based on domain adaptation based on a multi-objective loss function, and obtaining a trained ophthalmic OCT postoperative contour map generation model based on domain adaptation;
[0097] The application module is used to input the preoperative OCT contour map corresponding to the newly captured preoperative OCT image into the generator G1 of the trained domain adaptation-based ophthalmic OCT postoperative contour map generation model to obtain the corresponding postoperative generated OCT contour map.
[0098] In summary, the embodiment of the present invention provides a method for generating an ophthalmic OCT postoperative contour map based on domain adaptation, which obtains a preoperative OCT contour map corresponding to a preoperative OCT image, and generates a postoperative OCT contour map based on a domain adaptation ophthalmic OCT postoperative contour map generation model according to the preoperative OCT contour map. Compared with directly generating a postoperative OCT image with high resolution and complex texture information, the technical difficulty is significantly reduced, simple and efficient, and the noise of the generated contour map is extremely small, so that the accuracy of the generated contour map is high. Using the postoperative OCT contour map, some key pathological indicators can be measured, and the postoperative OCT image does not need to be assisted, and the doctor can also evaluate the patient's postoperative recovery effect to a certain extent, which is highly practical; and the preoperative OCT images of different fundus diseases and their postoperative OCT contour maps have certain similarities in the intrinsic connection. This embodiment uses transfer learning technology to integrate data from multiple sources in the source domain and the target domain, effectively expands the scale of training data, and improves the training effect of the deep learning model. This embodiment effectively integrates deep learning and transfer learning technologies, makes full use of preoperative OCT images and their features, predicts and generates the patient's postoperative OCT contour map, and then calculates related pathological indicators, effectively assisting doctors in evaluating the patient's postoperative recovery effect. Through transfer learning, the model proposed in the present invention takes multimodal data into consideration, and the training data comes from multiple fields rather than single modality data. By comprehensively considering multiple factors, the generated postoperative OCT contour map is more accurate and has richer feature details.
[0099] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.
Claims
1. A method for generating postoperative contour images of ophthalmic OCT based on domain adaptation, characterized in that: The steps include: S1. Collect preoperative OCT images A and postoperative OCT images B of different fundus diseases as a data set; S2, divide the dataset into different fields according to the types of fundus diseases; S3, according to the preoperative OCT image A, obtain the preoperative OCT contour map AL; according to the postoperative OCT image B corresponding to the preoperative OCT image A, obtain the postoperative real contour map real_BL to form a new data set; S4. Construct an ophthalmic OCT postoperative contour image generation model based on domain adaptation. The ophthalmic OCT postoperative contour image generation model based on domain adaptation includes a generator G1, a discriminator D1, and a domain classifier C. The data set obtained in S3 is sent to the ophthalmic OCT postoperative contour image generation model based on domain adaptation for training, including: S4.1 extracts features from the preoperative OCT contour map AL through a convolutional neural network and converts it into a tensor representation; then it is input into the generator G1 to obtain the predicted postoperative generated contour map fake_BL; S4.2, input the preoperative OCT contour image AL, the postoperative real contour image real_BL and the postoperative generated contour image fake_BL into the discriminator D1, calculate the discriminator loss loss_D1 according to the discrimination results and update the parameters of the discriminator D1; S4.3, input the preoperative OCT contour image AL, the postoperative real OCT contour image real_BL and the postoperative generated contour image fake_BL into the domain classifier C, calculate the domain classifier loss loss_C according to the discrimination result of the domain and update the parameters of the domain classifier C; S4.
4. Calculate the generator loss loss_G1 and update the parameters of the generator G1, where the discrimination results of the discriminator D1 and the domain classifier C participate in the generator loss loss_G; S4.5, repeating steps S4.2, S4.3 and S4.4 until the training is completed, and obtaining a trained domain adaptation-based ophthalmic OCT postoperative contour image generation model; S5. Obtain a preoperative OCT contour map corresponding to a newly captured preoperative OCT image, input the preoperative OCT contour map corresponding to the newly captured preoperative OCT image into the generator G1 of the trained domain adaptation-based ophthalmic OCT postoperative contour map generation model, and output the corresponding postoperative generated OCT contour map.
2. The method for generating postoperative contour images of ophthalmic OCT based on domain adaptation according to claim 1, characterized in that: In step S4.2, the preoperative OCT contour image AL and the postoperative generated contour image fake_BL are concatenated to obtain negative_img_D1, and then negative_img_D1 is sent to the discriminator D1 to obtain the tensor pred_fake_D1. The discriminator D1 labels it with 0, that is, the discriminator D1 believes that this image is fake, and calculates the GAN loss loss_D1_fake of pred_fake_D1; The preoperative OCT contour image AL and the postoperative real OCT contour image real_BL are concatenated to obtain positive_img_D1, and then positive_img_D1 is sent to the discriminator D1 to obtain pred_real_D1. The discriminator D1 labels it with 1, that is, the discriminator D1 believes that this image is real, and calculates the GAN loss loss_D1_real of pred_real_D1; Add the GAN losses of loss_D1_fake and loss_D1_real to get the discriminator loss loss_D1.
3. The method for generating postoperative contour images of ophthalmic OCT based on domain adaptation according to claim 1, characterized in that: In step S4.3, the preoperative OCT contour image AL and the postoperative generated contour image fake_BL are concatenated to obtain negative_img_C, and then negative_img_C is sent to the domain classifier C to obtain the tensor pred_fake_C. The label added by the domain classifier C is not the label corresponding to the domain it belongs to, that is, the domain classifier C does not identify the correct domain of this image, and the GAN loss loss_C_fake of pred_fake_C is calculated; The preoperative OCT contour image AL and the postoperative real OCT contour image real_BL are concatenated to obtain positive_img_C, and then positive_img_C is sent to the domain classifier C to obtain pred_real_C. The domain classifier C is labeled with the corresponding label of its domain, that is, the domain classifier C determines the domain of this image and calculates the GAN loss loss_C_real of pred_real_C; Add the GAN losses of loss_C_fake and loss_C_real to get the domain classifier loss loss_C.
4. The method for generating postoperative contour images of ophthalmic OCT based on domain adaptation according to claim 1, characterized in that: In step S4.4, it includes: S4.4.
1. Get the GAN loss loss_D1_real of pred_real_D1 in step S4.
2. S4.4.
2. Get the GAN loss loss_C_fake of pred_fake_C in step 4.3; S4.4.
3. Calculate the pixel-by-pixel loss L1 of pred_fake_D1 and pred_real_D1 in step S4.2; S4.4.3, calculate the pixel-by-pixel loss L2 of pred_fake_C and pred_real_C in step S4.3; S4.4.
5. Calculate the VGG feature loss of the preoperative OCT image A and the postoperative generated contour map fake_BL, denoted as loss_G_VGG; S4.4.
6. Add loss_D1_real, loss_C_fake, L1, L2 and loss_G_VGG to get the generator loss loss_G.
5. The method for generating postoperative contour images of ophthalmic OCT based on domain adaptation according to claim 1, characterized in that: Both the discriminator D1 and the domain classifier C are multi-scale discriminators.
6. The method for generating postoperative contour images of ophthalmic OCT based on domain adaptation according to claim 1, characterized in that: Generator G1 includes a global generator network G1.1 and a local enhancement network G1.
2. The input and output resolutions of the global generator network G1.1 are consistent, and the output size of the local enhancement network G1.2 is 4 times the input size. The local enhancement network G1.2 is used to expand the resolution of the image generated by the global generator network G1.
1. During training, the preoperative OCT contour map AL is input into the global generator network G1.1, and the image output by the global generator network G1 is input into the local enhancement network G1.
2.
7. The method for generating postoperative contour images of ophthalmic OCT based on domain adaptation according to claim 6, characterized in that: The network structure of the global generator network G1.1 is U-Net, which is responsible for encoding and decoding image features.
8. The method for generating postoperative contour images of ophthalmic OCT based on domain adaptation according to claim 1, characterized in that: In step S2, the inner surface of the nerve fiber layer, the inner surface of the inner nuclear layer, the inner surface of the outer nuclear layer, the ellipsoidal band and the outer surface of the RPE in the preoperative OCT image A and the postoperative OCT image B are outlined to obtain the preoperative OCT contour image AL and the postoperative real contour image real_BL.
9. The method for generating postoperative contour images of ophthalmic OCT based on domain adaptation according to claim 1, characterized in that: In step S2, the preoperative OCT image A is input into the preoperative OCT contour map generation model, and the preoperative OCT contour map AL is output; the preoperative contour map generation model is a generative adversarial network; the preoperative contour map generation model includes a generator G2 and a discriminator D2.
10. A system for generating postoperative contour images of ophthalmic OCT based on domain adaptation, characterized in that: include: A data acquisition module, used to acquire preoperative OCT images and postoperative OCT images of different fundus diseases to form a data set; The classification module is used to classify the dataset into different fields according to the types of fundus diseases; The contour image acquisition module is used to obtain a preoperative OCT contour image AL according to the preoperative OCT image A; and obtain a postoperative real contour image real_BL according to the postoperative OCT image B corresponding to the preoperative OCT image A; A model building module is used to build an ophthalmic OCT postoperative contour map generation model based on domain adaptation. The ophthalmic OCT postoperative contour map generation model based on domain adaptation includes a generator G1, a discriminator D1 and a domain classifier C. The generator G1 is used to generate a postoperative generated contour map fake_BL according to a preoperative OCT image A. The discriminator D1 is used to distinguish between the postoperative generated contour map fake_BL and the postoperative real OCT contour map real_BL. The domain classifier C is used to distinguish between the domains of the postoperative generated contour map fake_BL and the postoperative real OCT contour map real_BL. A loss function construction module is used to construct a multi-objective loss function; the multi-objective loss function includes the loss function of the generator G1, the loss function of the discriminator D1 and the loss function of the domain classifier C; A training module, used for taking the samples consisting of the preoperative OCT contour map AL and the postoperative real OCT contour map real_BL as the input of the ophthalmic OCT postoperative contour map generation model based on domain adaptation, iteratively training the ophthalmic OCT postoperative contour map generation model based on domain adaptation based on a multi-objective loss function, and obtaining a trained ophthalmic OCT postoperative contour map generation model based on domain adaptation; The application module is used to input the preoperative OCT contour map corresponding to the newly captured preoperative OCT image into the generator G1 of the trained domain adaptation-based ophthalmic OCT postoperative contour map generation model to obtain the corresponding postoperative generated OCT contour map.
Citation Information
Patent Citations
Three-dimensional retina OCT image vascular detection and registration method, equipment and application
CN108257126A
Optical Apparatus
US20200146545A1
Image classification method for improvement of auxiliary classifier GAN
WO2020172838A1
