Retinal image segmentation using semi-supervised learning

Abstract

A system and method for performing automated retinal segmentation. Initial imaging data associated with a target domain is received. The initial imaging data captures the retina. Image inputs are formed in a machine learning model using the initial imaging data. A segmentation output is generated via the machine learning model, graphically locating a set of retinal elements relative to the initial imaging data. The machine learning model is trained using a loss function that combines supervised learning loss and controlled learning loss. The machine learning model is trained using a training dataset containing labeled imaging data associated with a set of source domains, where the set of source domains is different from the target domain.

Claims

1. Receiving initial imaging data associated with a target domain, wherein the initial imaging data captures the retina, Using the aforementioned initial imaging data, the image input for the machine learning model is formed, This includes generating a segmentation output that graphically localizes a set of retinal elements relative to the initial imaging data via the machine learning model, The aforementioned machine learning model is trained using a loss function that combines supervised learning loss and controlled learning loss. The machine learning model is trained using a training dataset that includes labeled imaging data associated with a set of source domains, wherein the set of source domains is different from the target domain.

2. The method according to claim 1, wherein the training dataset further comprises unlabeled imaging data associated with the target domain.

3. The method according to claim 1 or 2, wherein the training dataset further comprises unlabeled imaging data associated with at least one source domain from the set of source domains.

4. The method according to any one of claims 1 to 3, wherein the target domain includes imaging data obtained from an imaging device different from the imaging data associated with the set of source domains.

5. The method according to any one of claims 1 to 4, wherein the target domain includes imaging data that captures a retinal disease or retinal symptom different from the imaging data associated with the set of source domains.

6. The method according to any one of claims 1 to 5, wherein the initial imaging data includes an OCT volume containing a plurality of OCTB scans.

7. The method according to any one of claims 1 to 6, wherein the machine learning model is a collaborative learning model comprising a segmentation backbone, an encoder, and a contrast projection module.

8. The method according to claim 7, wherein the segmentation backbone includes a UNet architecture.

9. The method according to claim 7 or 8, wherein the encoder comprises a UNet encoder.

10. The method according to any one of claims 7 to 9, wherein the contrast projection module performs channel-by-channel aggregation and learns from pairs of images constructed using at least one of an expansion-based pairing strategy, a slice-based pairing strategy, or a combined pairing strategy that incorporates both the expansion-based pairing strategy and the slice-based pairing strategy.

11. The method according to claim 10, wherein the training dataset comprises an OCT volume comprising a plurality of OCTB scans, the expansion-based pairing strategy comprises constructing positive pairs, the constructing positive pairs comprising selecting one OCTB scan from the plurality of OCTB scans as the first image of the positive pair, and expanding the OCTB scan to form the second image of the positive pair, the expansion of the OCTB scan comprising at least one of horizontal flipping, horizontal translation, vertical translation, zoom in, zoom out, or chromatic distortion.

12. The method according to claim 10 or 11, wherein the training dataset comprises an OCT volume comprising a plurality of OCTB scans, the slice-based pairing strategy comprises constructing positive pairs, the constructing positive pairs comprising selecting a first OCTB scan as a first image from the plurality of OCTB scans, and selecting a second OCTB scan as a second image from the plurality of OCTB scans, the second OCTB scan being within a selected distance from the first OCTB scan.

13. The method according to any one of claims 10 to 12, wherein the combination pairing strategy comprises constructing a pair of images using the slice-based pairing strategy and expanding at least one of the images in the pair of images using the expansion-based pairing strategy.

14. Using the initial imaging data to form the image input means The method according to any one of claims 1 to 13, comprising performing at least one of the following operations: normalization, scaling, resizing, horizontal flipping, vertical flipping, trimming, rotation, or noise filtering.

15. The method according to any one of claims 1 to 14, wherein one of the retinal elements in the set of retinal elements comprises at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), retinal fluid pocket, or fracture.

16. The method according to any one of claims 1 to 15, wherein one of the set of retinal elements is associated with a retinal layer selected from the group consisting of the internal limiting membrane (ILM) layer, the external limiting membrane (ELM) layer, the outer plexiform layer-Henle fiber layer (OPL-HFL), the retinal pigment epithelium (RPE) layer, the RPE detachment layer, the Bruch's membrane (BM) layer, and the ellipsoidal region (EZ).

17. The method according to any one of claims 1 to 16, wherein the segmentation output includes a segmentation map that graphically locates at least one of the following: a color indicator, a shape indicator, a pattern indicator, a shading indicator, a line, a curve, a marker, a label, a tag, or text.

18. A method for training a machine learning model to perform automatic segmentation, Forming a training dataset that includes labeled imaging data associated with a set of source domains, This includes training the machine learning model to perform the automatic segmentation using the training dataset and a loss function that combines supervised learning loss and controlled learning loss, The trained machine learning model can process imaging data associated with a target domain to generate segmentation output with a desired level of performance. The aforementioned target domain differs from the aforementioned set of source domains, A method for excluding any labeled imaging data associated with the target domain from the training dataset.

19. The method according to claim 18, wherein the training dataset further comprises unlabeled imaging data associated with the target domain.

20. The method according to claim 18 or 19, wherein the training dataset further comprises unlabeled imaging data associated with at least one source domain from the set of source domains.

21. The method according to any one of claims 18 to 20, wherein the target domain includes imaging data obtained from an imaging device different from the imaging data associated with the set of source domains.

22. The method according to any one of claims 18 to 21, wherein the target domain includes imaging data that captures a retinal disease or retinal symptom different from the imaging data associated with the set of source domains.

23. The method according to any one of claims 18 to 22, wherein the machine learning model is a collaborative learning model comprising a segmentation backbone, an encoder, and a contrast projection module.

24. The method according to claim 23, wherein the segmentation backbone includes a UNet architecture, the encoder includes a UNet encoder, and the contrast projection module performs channel-by-channel aggregation.

25. The training dataset includes an OCT volume containing multiple OCTB scans, and training the machine learning model involves: The method according to any one of claims 18 to 24, comprising constructing a plurality of pairs using the training dataset for use in calculating the controlled learning loss using at least one of an expansion-based pairing strategy, a slice-based pairing strategy, or a combination pairing strategy that incorporates both the expansion-based pairing strategy and the slice-based pairing strategy.

26. The method according to claim 25, wherein the expansion-based pairing strategy comprises constructing a positive pair, comprising selecting an OCTB scan from a plurality of OCTB scans as the first image of the positive pair, and expanding the OCTB scan to form the second image of the positive pair, wherein expanding the OCTB scan comprises at least one of horizontal flipping, horizontal translation, vertical translation, zoom in, zoom out, or chromatic distortion.

27. The method according to claim 25 or 26, wherein the slice-based pairing strategy comprises constructing a positive pair, wherein constructing a positive pair comprises selecting a first OCTB scan as a first image from the plurality of OCTB scans and selecting a second OCTB scan as a second image from the plurality of OCTB scans, the second OCTB scan being within a selected distance from the first OCTB scan.

28. The method according to any one of claims 25 to 27, wherein the combination pairing strategy comprises constructing a pair of images using the slice-based pairing strategy and expanding at least one of the images in the pair of images using the expansion-based pairing strategy.

29. One or more data processors, A non-temporary computer-readable storage medium containing instructions, wherein when the instructions are executed by one or more data processors, the one or more data processors... Initial imaging data associated with the target domain is received, and the initial imaging data captures the retina, The initial imaging data is used to form the image input for the machine learning model. Through the machine learning model, a segmentation output is generated that graphically identifies a set of retinal elements in relation to the initial imaging data. The aforementioned machine learning model is trained using a loss function that combines supervised learning loss and controlled learning loss. A system comprising a non-temporary computer-readable storage medium, wherein the machine learning model is trained using a training dataset that includes labeled imaging data associated with a set of source domains, and the set of source domains is different from the target domain.

30. The system according to claim 29, wherein the training dataset further comprises unlabeled imaging data associated with the target domain.

31. The system according to claim 29 or 30, wherein the training dataset further comprises unlabeled imaging data associated with at least one source domain from the set of source domains.

32. The system according to any one of claims 29 to 31, wherein the target domain includes imaging data acquired from an imaging device different from the imaging data associated with the set of source domains.

33. The system according to any one of claims 29 to 32, wherein the target domain includes imaging data that captures a retinal disease or retinal symptom different from the imaging data associated with the set of source domains.

34. The system according to any one of claims 29 to 33, wherein the initial imaging data includes an OCT volume containing a plurality of OCTB scans.

35. The system according to any one of claims 29 to 34, wherein the machine learning model is a collaborative learning model comprising a segmentation backbone, an encoder, and a contrast projection module.

36. The system according to claim 35, wherein the segmentation backbone includes a UNet architecture.

37. The system according to claim 35 or 36, wherein the encoder comprises a UNet encoder.

38. The system according to any one of claims 35 to 37, wherein the contrast projection module performs channel-by-channel aggregation and learns from pairs of images constructed using at least one of an expansion-based pairing strategy, a slice-based pairing strategy, or a combined pairing strategy that incorporates both the expansion-based pairing strategy and the slice-based pairing strategy.

39. The system according to claim 38, wherein the training dataset comprises an OCT volume comprising a plurality of OCTB scans, the expansion-based pairing strategy comprises constructing positive pairs, the constructing positive pairs comprising selecting an OCTB scan from the plurality of OCTB scans as the first image of the positive pair, and expanding the OCTB scan to form the second image of the positive pair, the expansion of the OCTB scan comprising at least one of horizontal flipping, horizontal translation, vertical translation, zoom in, zoom out, or chromatic distortion.

40. The system according to claim 38 or 39, wherein the training dataset comprises an OCT volume comprising a plurality of OCTB scans, and the slice-based pairing strategy comprises constructing positive pairs, wherein constructing positive pairs comprises selecting a first OCTB scan as a first image from the plurality of OCTB scans and selecting a second OCTB scan as a second image from the plurality of OCTB scans, the second OCTB scan being within a selected distance from the first OCTB scan.

41. The system according to any one of claims 38 to 40, wherein the combination pairing strategy comprises constructing pairs of images using the slice-based pairing strategy and expanding at least one of the images in the pair of images using the expansion-based pairing strategy.

42. Using the initial imaging data to form the image input means The system according to any one of claims 29 to 41, comprising performing at least one of the following operations: normalization, scaling, resizing, horizontal flipping, vertical flipping, trimming, rotation, or noise filtering.

43. The system according to any one of claims 29 to 42, wherein one of the retinal elements in the set of retinal elements includes at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), retinal fluid pocket, or fracture.

44. The system according to any one of claims 29 to 43, wherein one of the retinal elements in the set of retinal elements is associated with a retinal layer selected from the group consisting of the internal limiting membrane (ILM) layer, the external limiting membrane (ELM) layer, the outer plexiform layer-Henle fiber layer (OPL-HFL), the retinal pigment epithelium (RPE) layer, the RPE detachment layer, the Bruch's membrane (BM) layer, and the ellipsoidal region (EZ).

45. The system according to any one of claims 29 to 44, wherein the segmentation output includes a segmentation map that graphically locates at least one retinal element from the set of retinal elements, comprising at least one of a color indicator, a shape indicator, a pattern indicator, a shading indicator, a line, a curve, a marker, a label, a tag, or text.

46. A system for training machine learning models to perform automatic segmentation, One or more data processors, A non-temporary computer-readable storage medium containing instructions, wherein when the instructions are executed by one or more data processors, the one or more data processors... A training dataset is formed that includes labeled imaging data associated with a set of source domains. The machine learning model is trained to perform the automatic segmentation using the training dataset and a loss function that combines supervised learning loss and controlled learning loss. The trained machine learning model can process imaging data associated with the target domain to generate segmentation output with a desired level of performance. The target domain differs from the set of source domains, A system comprising a non-temporary computer-readable storage medium in which the training dataset excludes any labeled imaging data associated with the target domain.

47. The system according to claim 46, wherein the training dataset further comprises unlabeled imaging data associated with the target domain.

48. The system according to claim 46 or 47, wherein the training dataset further comprises unlabeled imaging data associated with at least one source domain from the set of source domains.

49. The system according to any one of claims 46 to 48, wherein the target domain includes imaging data acquired from an imaging device different from the imaging data associated with the set of source domains.

50. The system according to any one of claims 46 to 48, wherein the target domain includes imaging data that captures a retinal disease or retinal symptom different from the imaging data associated with the set of source domains.

51. The system according to any one of claims 46 to 50, wherein the machine learning model is a collaborative learning model comprising a segmentation backbone, an encoder, and a contrast projection module.

52. The system according to claim 51, wherein the segmentation backbone includes a UNet architecture, the encoder includes a UNet encoder, and the comparative projection module performs channel-by-channel aggregation.

53. The training dataset includes an OCT volume containing multiple OCTB scans, and the machine learning model is trained using this dataset. The system according to any one of claims 46 to 52, comprising constructing a plurality of pairs using the training dataset for use in calculating the contrast learning loss using at least one of an expansion-based pairing strategy, a slice-based pairing strategy, or a combination pairing strategy that incorporates both the expansion-based pairing strategy and the slice-based pairing strategy.

54. The system according to claim 53, wherein the expansion-based pairing strategy includes constructing a positive pair, the constructing of a positive pair includes selecting an OCTB scan from a plurality of OCTB scans as the first image of the positive pair, and expanding the OCTB scan to form the second image of the positive pair, the expanding of the OCTB scan includes at least one of horizontal flipping, horizontal translation, vertical translation, zoom in, zoom out, or chromatic distortion.

55. The system according to claim 53 or 54, wherein the slice-based pairing strategy includes constructing a positive pair, the construction of a positive pair includes selecting a first OCTB scan as a first image from the plurality of OCTB scans, and selecting a second OCTB scan as a second image from the plurality of OCTB scans, the second OCTB scan being within a selected distance from the first OCTB scan.

56. The system according to any one of claims 53 to 55, wherein the combination pairing strategy comprises constructing pairs of images using the slice-based pairing strategy and expanding at least one of the images in the pair of images using the expansion-based pairing strategy.

57. One or more data processors, A system comprising: a non-temporary computer-readable storage medium containing instructions, wherein when the instructions are executed on one or more data processors, the non-temporary computer-readable storage medium causes one or more data processors to perform one or more of the methods disclosed in claims 1 to 28.

58. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform some or all of the methods disclosed in claims 1 to 28.

Images