Systems and methods for detecting retinoblastoma

Abstract

Presented herein are systems and methods of detecting retinoblastoma using images of eyes. The systems and methods include identifying a first image of at least one first eye of a first subject prior to administration of treatment for retinoblastoma, the at least one eye having a structure of interest corresponding to an abnormality, applying the first image to a machine learning (ML) model established using a training dataset including examples, each of the examples identifying a respective second image of at least one second eye of a second subject and a respective second classification identifying one of presence or absence of retinoblastoma in the at least second eye, determining from applying the ML model, a first classification identifying one of presence or absence of retinoblastoma in the at least one first eye, and storing using one or more data structures, an association between the first subject and the classification.

Description

SYSTEMS AND METHODS FOR DETECTING RETINOBLASTOMACROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 605,148, filed December 1, 2023, the entire disclosure of which is hereby incorporated herein by reference.BACKGROUND

[0002] Retinoblastoma is a potentially life-threatening cancer that primarily affects the retina of young children, particularly the light-sensitive tissue at the back of the eye. One of the signs of retinoblastoma may be a white or cloudy appearance in the pupil, known as leukocoria.SUMMARY

[0003] An aspect of the present disclosure is directed to systems, methods, and computer-readable media for detecting retinoblastoma using images of eyes. One or more processors may identify a first image of at least one first eye of a first subject prior to administration of treatment for retinoblastoma, the at least one eye having a structure of interest corresponding to an abnormality. The one or more processors may determine the first image to a machine learning (ML) model. The ML model may be established using a training dataset including a plurality of examples. Each of the plurality of examples may identify: (i) a respective second image of at least one second eye of a second subject and (ii) a respective second classification identifying one of presence or absence of retinoblastoma in the at least second eye. The one or more processors may determine, from applying the ML model, a first classification identifying one of presence or absence of retinoblastoma in the at least one first eye of the first subject. The one or more processors may store, using one or more data structures, an association between the first subject and the classification.

[0004] In some embodiments, the one or more processors may identify clinical data of the first subject. The clinical data may include at least one of (i) an age, (ii) genetic data, (iii) family data, (iv) a sex, or (v) a trait of the first subject. The one or more processors may apply the clinical data to the ML model to determine the first classification.

[0005] In some embodiments, the one or more processors may determine, from applying the ML model, a likelihood of the presence of retinoblastoma in the at least one first eye. The one or more processors may determine the first classification identifying the presence of retinoblastoma in the at least one first eye of the first subject, responsive to the likelihood satisfying a threshold.

[0006] In some embodiments, the one or more processors may generate an output identifying the first subject as to be administered with for a treatment for the at least one first eye of the first subject, in response to a determination of the presence of retinoblastoma. In some embodiments, the one or more processors may generate an output identifying the first subject as not to be administered with a treatment for the at least one first eye of the first subject, in response to a determination of absence of retinoblastoma.

[0007] In some embodiments, the one or more processors may compare a likelihood of the presence of retinoblastoma in the at least one first eye with a threshold. The threshold may be determined during the establishment of the ML model. In some embodiments, the one or more processors may provide an output based on the association between the first subject and the first classification.

[0008] In some embodiments, the first image may include a fundus image of a back region of the at least one first eye acquired via a fundus camera from an interior or an exterior of the at least one first eye. In some embodiments, the first subject may be a human child under the age of five years, and the retinoblastoma may include at least one of (i) unilateral retinoblastoma, (ii) bilateral retinoblastoma, or (iii) a trilateral retinoblastoma.

[0009] In some embodiments, the one or more processors may apply the first image to a plurality of ML models. In some embodiments, determining the first classification includes determining, based on applying the first image to the plurality of ML models, a plurality of likelihoods corresponding to the plurality of ML models. Each of the plurality of likelihoods may identify the presence of retinoblastoma in at least one first eye of the first subject. In some embodiments, the one or more processors may select at least the ML model of the plurality of ML models based on the plurality of likelihoods.

[0010] Another aspect of the present disclosure is directed to systems, methods, and computer-readable media for training models to detect retinoblastoma using images of eyes. One or more processors may identify a training dataset including a plurality of examples. Each of the plurality of examples may include (i) a respective image of at least one eye of a subject prior to administration of treatment for retinoblastoma and (ii) a respective first classification identifying one of presence or absence of retinoblastoma in the at least one eye. For each example of the plurality of examples of the training dataset, the one or more processors may apply the respective image to a machine learning (ML) model including a plurality of weights to determine a second classification identifying one of presence or absence of retinoblastoma in the at least one eye. The one or more processors may compare the respective first classification and the second classification. The one or more processors may determine a loss metric based on comparing the respective first classification and the second classification. The one or more processors may update at least one weight of the ML model in accordance with the loss metric.

[0011] In some embodiments, each example of the plurality of examples may include clinical data, including at least one of (i) an age, (ii) genetic data, (iii) family data, (iv) a sex, or (v) a trait of the subject. The one or more processors may apply the clinical data of each example of the plurality of examples to the ML model to determine the second classification.

[0012] In some embodiments, the one or more processors may determine a confidence value for the second classification to identify the presence of retinoblastoma in the at least one first eye for each example of the plurality of examples. In some embodiments, the one or more processors may determine the loss metric as a function of the first classification and a confidence value for the second classification.

[0013] In some embodiments, the one or more processors may determine a threshold against which to compare confidence values based on the loss metric of the ML model. In some embodiments, the image in each example may include a fundus image of a back region of the at least one first eye acquired via a fundus camera from an interior or an exterior of the at least one first eye. In some embodiments, the subject may be a humanchild under the age of five years. The retinoblastoma includes at least one of (i) unilateral retinoblastoma, (ii) bilateral retinoblastoma, or (iii) a trilateral retinoblastoma.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The foregoing and other objects, aspects, features, and advantages of the disclosure will become more apparent and better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0015] FIG. 1 depicts a block diagram of a system for detecting retinoblastoma using images of eyes, in accordance with an illustrative embodiment.

[0016] FIG. 2 depicts a block diagram of a process for training retinoblastoma using images of eyes in the system for detecting retinoblastoma, in accordance with an illustrative embodiment.

[0017] FIG. 3 depicts a block diagram of a process for determining likelihood of retinoblastoma using images of eyes in the system for detecting retinoblastoma, in accordance with an illustrative embodiment.

[0018] FIG. 4 depicts a flow diagram of a method of detecting retinoblastoma using images of eyes, in accordance with an illustrative embodiment.

[0019] FIG. 5 depicts a flow diagram of a method of training models to detect retinoblastoma using images of eyes, in accordance with an illustrative embodiment.

[0020] FIG. 6 depicts example plots of model training history, in accordance with an illustrative embodiment.

[0021] FIG. 7 depicts an example of a confusion matrix for ResNet-101 with normal, in accordance with an illustrative embodiment.

[0022] FIG. 8 depicts an example of benign lesions simulating retinoblastoma (Coats disease), in accordance with an illustrative embodiment.

[0023] FIG. 9 depicts an example of pseudo-retinoblastoma incorrectly identified as retinoblastoma by the algorithm (left) but correctly identified as a benign retrolenticularlesion by the model on additional imaging (right), in accordance with an illustrative embodiment.

[0024] FIG. 10 depicts a block diagram of a server system and a client computer system in accordance with an illustrative embodiment.DETAILED DESCRIPTION

[0025] Following below are more detailed descriptions of various concepts related to, and embodiments of, systems and methods for detecting retinoblastoma from digital images. It should be appreciated that various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways, as the disclosed concepts are not limited to any particular manner of implementation. Examples of specific implementations and applications are provided primarily for illustrative purposes.

[0026] Section A describes systems and methods for detecting retinoblastoma from digital images.

[0027] Section B describes a network environment and computing environment which may be useful for practicing various computing related embodiments described herein.A. Systems and Methods for Detecting Retinoblastoma from Digital Images

[0028] Retinoblastoma is a deadly cancer of one or both eyes in children. Making an accurate diagnosis remains challenging, as errors in diagnosis can lead to the removal of an eye thought to have cancer that instead has a non-cancerous abnormality that is similar in appearance to cancer. One of the characteristics of retinoblastoma may include leukocoria, or “white eye”, in the patient’s pupil. Prompt, successful treatment can save vision, an eye, or life.

[0029] Ophthalmologists diagnose retinoblastoma early by examining the inside of the eye (e.g., using an ophthalmoscope and retinal imaging). However, the diagnostic techniques are limited to imaging or direct inspection. Furthermore, a biopsy may potentially spread the cancer (e.g., through the needle tract). Moreover, despite the use ofvarious imaging techniques, error rates remain high, particularly those of false positives, largely because of a lack of standard clinical data and / or reliable techniques. Many ophthalmologists have to rely on diagnoses of a subjective standard, leading to the high error rates. In addition, cancerous abnormalities and non-cancerous abnormalities (e.g., exudative retinitis) in eyes could look very similar in various modes of imaging modalities, which adds difficulty to accurately diagnosing retinoblastoma. The subjective nature of the diagnosis techniques currently in use and incorrect diagnoses (e.g., false positives) have led to many instances where children’s eyes are removed, even when no cancer is actually present.[0030 The techniques disclosed herein are to detect retinoblastoma (e.g., unilateral retinoblastoma, bilateral retinoblastoma, or trilateral retinoblastoma) using images of eyes based on artificial intelligence (Al), thereby reducing errors in diagnosing retinoblastoma and thus avoiding removing eyes due to false positives. For example, the techniques allow for a reliable detection of retinoblastoma by analyzing an eye image having a structure of interest corresponding to an abnormality to determine a likelihood of the presence of retinoblastoma. The detection of retinoblastoma may avoid having to rely on invasive techniques (e.g., biopsy needle) that would otherwise physically damage the eye and potentially spread the cancer.

[0031] Furthermore, the techniques disclosed herein can determine a likelihood of the presence of retinoblastoma based on eye images and / or structures therein and classify the eye based on the likelihood. Because ophthalmologists continue to make errors in diagnosis, the techniques could greatly assist ophthalmologists in accurately diagnosing retinoblastoma. Once the system notifies the detection of the retinoblastoma, the ophthalmologist can use this information to determine whether to treat the subject’s eyes for retinoblastoma. From a human-computer interaction (HCI) perspective, by having a computing device provide predictions of the retinoblastoma, the quality of HCI between the user and the computing device and the overall utility of the computing device may be greatly improved.

[0032] Referring now to FIG. 1, depicted is a block diagram of a system 100 for detecting retinoblastoma using images of eyes. In overview, the system 100 may include atleast one image processing system 105, at least one imaging device 110, and at least one display 115, among others, communicatively coupled with one another via at least one network 120. The image processing system 105 may include at least one data acquirer 125, at least one model trainer 130, at least one model applier 135, at least one eye evaluator 140, at least one detection model 145, and at least one database 150, among others. Each of the components in the system 100 as detailed herein may be implemented using hardware (e.g., one or more processors coupled with memory) or a combination of hardware and software as detailed herein in Section C.

[0033] In further detail, the image processing system 105 may (sometimes herein generally referred to as a computing system or a server) be any computing device, including one or more processors coupled with memory and software and capable of performing the various processes and tasks described herein. The image processing system 105 may be in communication with the imaging device 110 and display 115, and other devices, via the network 120. The image processing system 105 may be situated, located, or otherwise associated with at least one server group. The server group may correspond to a data center, a branch office, or a site at which one or more servers corresponding to the image processing system 105 is situated.

[0034] Within the image processing system 105, the data acquirer 125 may retrieve, identify, or receive images (e.g., eye images) from the imaging device 110 to be processed at the image processing system 105. The model applier 135 may apply the images to the detection model 145. The detection model 145 may have been initialized, trained, and established to determine a likelihood of the presence of retinoblastoma from the images using training data (e.g., in accordance with supervised learning techniques). The eye evaluator 140 may determine various characteristics associated with the retinoblastoma identified from the images. The eye evaluator 140 may generate information on the identified characteristics to provide for presentation on the display 115.

[0035] The detection model 145 may be any type of machine learning algorithm or model to detect retinoblastoma, such as a clustering algorithm (e.g., ^-means clustering), an artificial neural network (e.g., an encoder with a convolutional neural network architecture or a residual neural network (ResNet)), a support vector machine (SVM), a decision tree, aBayesian model, a regression model, or a transformer model, among others. In general, the detection model 145 may have at least one input and at least one output. The output and the input may be related via a set of weights. The input may be at least one image (e.g., a fundus image). The output may include at least a determination of a classification identifying one of the presence or absence of retinoblastoma from the application of the detection model 145 onto the input image in accordance with the set of weights. In some embodiments, a set of multiple detection models 145 may be maintained on the image processing system 105. Each detection model 145 may have a different architecture.

[0036] In addition, the set of weights of the detection model 145 may define corresponding parameters to be applied to the input image to generate the output image. In some embodiments, the set of weights may be arranged in one or more transform layers. Each layer may specify a combination or a sequence of application of the parameters to the input and resultant. The layers may be arranged in accordance with the machine learning algorithm or model for the detection model 145. In general, the ML model can have an input corresponding to an image (e.g., an eye image) and an output including a determination of a classification identifying one of the presence or absence of retinoblastoma. The ML model can include a set of weights (e.g., layers of tensors) that define a relationship between the input and the output. The ML model may have been initialized, trained, and established using a training dataset in accordance with learning techniques (e.g., supervised or semi-supervised). The training dataset can include or identify a set of examples. Each example can include a respective image of an eye, as well as an annotation defining a structure of interest and / or a feature associated with the presence or absence of retinoblastoma within the eye within the image. Upon training, the ML model can be used to recognize or detect the structure of interest and / or the feature within a respective input image.

[0037] The imaging device 110 (sometimes herein generally referred to as an imaging device or an image acquirer) may be any device to acquire images of eyes from subjects (e.g., patients, potential patients, etc.). The subjects may be under the guidance of a clinician or hospital staff while scanned by the imaging device 110. In some embodiments, the subjects can use their own device (e.g., camera) to capture their eye image. In some embodiments, the imaging device 110 may be a retinal fundus camera toacquire fundus images of the back part of the eye. In some embodiments, the imaging device 110 may be a camera to capture an image (e.g., a fundus image) of the eye from the exterior or an interior of the eye.

[0038] The display 115 may be communicatively coupled with the image processing system 105 or any other computing device, including one or more processors coupled with memory and software and capable of performing the various processes and tasks described herein. The display 115 may display, render, or otherwise present any information provided by the image processing system 105 or the images of subjects acquired via the imaging device 110. The information may be used by a clinician examining a subject to diagnose retinoblastoma.

[0039] Referring now to FIG. 2, depicted is a block diagram of a process 200 for training retinoblastoma using images of eyes in the system 100 for detecting retinoblastoma. More specifically, the process 200 may include or correspond to operations performed in the system 100 for acquiring an image 205 of an eye 220 from a subject 225 and training the detection model 145. The subject 225 may be a human at risk of retinoblastoma or suffering from the same. In some embodiments, the subject 225 may be a human child under the age of five to seven years. The image 205 may be an optical image (e.g., a regular camera image) of an exterior of an eye 220 of the subject 225 prior to administration of treatment for retinoblastoma. In some embodiments, the image 205 may be a fundus image of a back region of the eye 220 via a fundus camera from an exterior or an interior of the eye.

[0040] Under the process 200, the model trainer 130 can train the detection model 145 based on the training dataset 155. In some embodiments, the model trainer 130 can initialize, train, or establish multiple detection models 145. The training dataset 155 may identify or include a set of examples. Each example may identify or include the image 205 and the annotation 215 (e.g., a label) for the image 205. In some embodiments, the images 205 may be obtained through the imaging device 110. The imaging device 110 may output, produce, or otherwise generate at least one image 205 (e.g., an eye image). The imaging device 110 may scan, obtain, or otherwise acquire the image 205 of the eye 220 of the subject 225. In some embodiments, the imaging device 110 may generate a set of images205. Each image 205 in the set may be of a respective eye 220 from a respective subject 225. Each image 205 in the set may be acquired at a respective time instance. Upon acquisition, the imaging device 110 may send, transmit, or otherwise provide the image 205 to the image processing system 105. In some embodiments, the image processing system 105 may store and maintain the image 205 on the database 150.

[0041] In some embodiments, the data acquirer 125 executing on the image processing system 105 may retrieve, receive, or otherwise identify the image 205 of the eye 220 from the subject 225. In some embodiments, the data acquirer 125 may access the database 150 to retrieve the image 205. In some embodiments, the data acquirer 125 may identify a set of images 205 acquired over a corresponding set of time instances. Each image 205 in the set may be of a respective eye 220 obtained from a respective subject 225 at a particular time instance. In some embodiments, the data acquirer 125 may identify the set of images 205 acquired at a single time instance. In some embodiments, the data acquirer 125 may determine or identify the time instance corresponding to an acquisition of the image 205 from metadata associated with the image 205. For example, the metadata may identify or include a timestamp at which the image 205 is acquired. The image 205 may be in the form of an image file (e.g., with a BMP, TIFF, LJPEG, or PNG, among others).

[0042] In the training dataset 155, each example may include an annotation 215 identifying the classification identifying one of the presence or absence of retinoblastoma in the eye 220 of the corresponding image 205. In some embodiments, the training dataset 155 may include annotation 215 to label the image 205 as retinoblastoma or non-retinoblastoma. For example, the image 205 may include an image of an eye labeled as retinoblastoma when the eye has been reliably diagnosed. For example, the images 205 may include an image of an eye labeled as retinoblastoma when the eye has been removed because of retinoblastoma. In some embodiments, the annotation 215 may identify the image 205 as retinoblastoma or pseudo-retinoblastoma. In some embodiments, the training dataset 155 may include clinical data 210. The clinical data 210 may be or include information on the subject 225. For example, the clinical data 210 may include an age (e.g., defined in terms of weeks, months, or years), genetic data (e.g., presence of genes or other biomarkers correlated with retinoblastoma), family data (e.g., occurrence of retinoblastoma or other cancer in family), asex (e.g., male or female), or a trait of the subject 225, (e.g., location, country, or origin), among others. In some embodiments, the training dataset 155 used for training may differ depending on the detection model 145, when there are multiple detection models 145 to be trained. For example, one detection model 145 may be trained on the training dataset 155 with annotations 215 for retinoblastoma versus non-retinoblastoma, whereas another detection model 145 may be trained with annotations 215 for retinoblastoma versus pseudoretinoblastoma.

[0043] The model applier 135 executing on the image processing system 105 may feed or apply the image 205 (and / or the clinical data 210, etc.) to the detection model 145. The detection model 145 may include a set of weights arranged in accordance with the model architecture to process the input image 205. In feeding, in some embodiments, the model applier 135 may process the image 205 in accordance with the set of weights defined by the detection model 145. From processing in accordance with the detection model 145, the model applier 135 may produce, output, or otherwise generate a processed image.When the set of images 205 is identified, the model applier 135 may traverse through the set and apply each image 205 to the detection model 145 to generate a corresponding processed image. By applying the detection model 145 to the image 205 (and / or the clinical data 210) in the training dataset 155, the model applier 135 can feed the image 205 and / or the clinical data 210 to the detection model 145. Upon feeding, the model applier 135 can process the input using the set of weights of the detection model 145 to generate a likelihood 230. The likelihood 230 can identify a probability that the eye 315 of the subject 320 has (or does not have) retinoblastoma.

[0044] Using the likelihood 230, the eye evaluator 140 can determine can classify the eye 220 as retinoblastoma or non-retinoblastoma based on a comparison between the determined classification 235 and the classification stored in the training dataset 155 (e.g., the annotation 215). In some embodiments, prior to the comparison, the detection model 145 can determine a likelihood 230 (e.g., a confidence value) of the corresponding eye 220 of the image 205 to have retinoblastoma, and the eye evaluator 140 can classify the eye 220 as retinoblastoma or non-retinoblastoma based on the likelihood. When the likelihood 230 is greater than or equal to a threshold, the eye evaluator 140 can determine the classification 235 identifying the presence of retinoblastoma in the eye 315 of the subject 225.Conversely, when the likelihood 230 is less than a threshold, the eye evaluator 140 can determine the classification 235 identifying the absence of retinoblastoma in the eye 315 of the subject 225.

[0045] The model trainer 130 can calculate or determine a loss metric 240. The loss metric 240 may be calculated, generated, or otherwise determined based on a comparison between the classification 235 and the classification stored in the training dataset 155 (e.g., the annotation 215). For example, the comparison may be between the classification 235 as outputted by the detection model 145 and the annotation 215 in the training dataset 155. The loss metric 240 may indicate a deviation between the output from the detection model 145 and the expected output as identified in the annotation 215. In some embodiments, the loss metric 240 may be calculated in accordance with a root mean squared error, a relative root mean squared error, and a weighted cross-entropy, among others.

[0046] In some embodiments, the loss metric 240 may be calculated in accordance with any number of loss functions, such as a norm loss (e.g., LI or L2), mean squared error (MSE), a quadratic loss, a cross-entropy loss, and a Huber loss, among others. Using the loss metric 240, the model trainer 130 can modify, set, or otherwise update the detection model 145 (e.g., the weights). The updating of the detection model 145 may be in accordance with an optimization function (or an objective function) for the classification 235. The optimization function may define one or more rates or parameters at which the weights of the classification 235 are to be updated. The updating of the weights may be repeated until convergence. The model trainer 130 can update the detection model 145 in accordance with the loss metric 240. For example, the model trainer 130 can update at least one weight of the detection model 145 based on the loss metric 240. For example, the weight of the detection model 145 may be modified or updated using the loss metric 240 in accordance with an objective function (e.g., stochastic gradient descent (SGD)). The detection model 145 may be iteratively updated until convergence to complete the training.

[0047] In some embodiments, the model trainer 130 can determine the threshold against which to compare the likelihood 230 (e.g., a confidence value) based on the loss metric 240 of the detection model 145. In some embodiments, the model trainer 130 can determine the threshold based on an accuracy rate of the detection model 145. For example,the model trainer 130 can calculate the threshold as a function of the accuracy rate to minimize true negatives in outputting the classification 235. In some embodiments, the model trainer 130 can set the threshold to a pre-defined value (e.g., 80% to 90%). The model trainer 130 can repeat the initialization, training, and establishment over the set of detection models 145.

[0048] Referring now to FIG. 3, depicted is a block diagram of a process 300 for determining the likelihood of retinoblastoma using images of eyes in the system 100 for detecting retinoblastoma. More specifically, the process 300 may include or correspond to operations performed in the system 100 for applying an image 305 to the detection model 145 and determining a classification identifying one of the presence or absence of retinoblastoma in an eye 315 of a subject 320. The detection model 145 of FIG. 3 may be a model established using the training dataset 155 as discussed with respect to FIG. 2. The subject 320 may be a human at risk of retinoblastoma or suffering from the same. In some embodiments, the subject 320 may be a human child under the age of five to seven years. The image 305 may be an optical image (e.g., a regular camera image) of an exterior of an eye 315 of the subject 320 prior to administration of treatment for retinoblastoma. In some embodiments, the image 305 may be a fundus image of a back region of the eye 315 via a fundus camera (e.g., from an exterior or an interior of the eye 315).

[0049] Under the process 300, the data acquirer 125 can retrieve, identify, or otherwise receive the image 305 of the eye 315 of the subject 320, for example, through the imaging device 110. The eye 315 may include a structure of interest corresponding to an abnormality. For example, the abnormality may be associated with the presence or absence of retinoblastoma, as identified by a clinician examining the eye 315 of the subject 320. In some embodiments, a human (e.g., the subject 320) can upload the image 305 to the system 100. In some embodiments, the data acquirer 125 can receive the clinical data 310 of the subject 320. For example, the clinical data 310 may include an age, genetic data, family data, a sex, or a trait of the subject 320. The clinical data 310 may be entered via a graphical user interface (e.g., a form prompting the subject 320 to fill in age, family-related data, sex, or any other traits) on a computing device or may be retrieved from a database.

[0050] In some embodiments, the data acquirer 125 can identify or determine whether the image 305 is of an eye (e.g., as apparent in a fundus image). The determination can be performed using a computer vision algorithm (e.g., circle detection, intensity-based thresholding, feature extractor, or vessel segmentation) or machine learning models (e.g., deep learning neural network-based models, classifiers, or support vector machines). In some embodiments, the determination can be performed based on visual characteristics of the image 305. If the image 305 is determined to not contain any eyes, the data acquirer 125 can refrain from further processing the image 305. The data acquirer 125 can return, send, or otherwise provide a notification that the image 305 does not contain the eye to the imaging device 110. The notification may also indicate to reacquire the image 305 to include the eye 315 of the subject 320. If the image 305 is determined to contain an eye (e.g., the eye 315), the data acquirer 125 can continue processing the image 305.

[0051] With the receipt, the model applier 135 can apply the image 305 and / or the clinical data 310 to the detection model 145. To apply, the model applier 135 can feed the image 305 and / or the clinical data 310 to the detection model 145. Upon feeding, the model applier 135 can process the input using the set of weights of the detection model 145 to generate a likelihood 330. The likelihood 330 can identify a probability that the eye 315 of the subject 320 has (or does not have) retinoblastoma. In some embodiments, the likelihood 330 can identify or indicate a probability that the eye 315 has (or does not have) pseudoretinoblastoma. In some embodiments, the model applier 135 can apply the image 305 to a set of detection models 145. The model applier 135 can generate a set of likelihoods 330 from the set of detection model 145. Each likelihood 330 may indicate or identify a probability that the eye 315 of the subject 320 has (or does not have) retinoblastoma. Since the detection models 145 may have different architectures or may have been trained using different data, the likelihoods 330 may differ from one another.

[0052] Using the likelihood 330, the eye evaluator 140 can generate or determine a classification 335 identifying one of presence or absence of retinoblastoma in the eye 315 of the subject 320. In some embodiments, the detection model 145 can determine a likelihood 330 of the presence of retinoblastoma in the eye 315, and then determine the classification 335 based on the likelihood 330. In some embodiments, the detection model 145 can determine the classification 335 based on the likelihood 330 and a predetermined threshold(e.g., determined during establishment of the detection model 145). When the likelihood 330 is greater than or equal to a threshold, the eye evaluator 140 can determine the classification 335 identifying the presence of retinoblastoma in the eye 315 of the subject 320. For example, the detection model 145 can classify the eye 315 as retinoblastoma when the likelihood 330 (e.g., 95%) is higher than the predetermined threshold (e.g., 90%). Conversely, when the likelihood 330 is less than a threshold, the eye evaluator 140 can determine the classification 335, identifying the absence of retinoblastoma in the eye 315 of the subject 320. For example, the detection model 145 can classify the eye 315 as nonretinoblastoma when the likelihood 330 (e.g., 65%) is lower than the predetermined threshold (e.g., 70%).[00531 In some embodiments, when there are multiple detection models 145 used, the eye evaluator 140 can identify or select at least one of the set of detection models 145 based on the set of likelihoods 330. For instance, the eye evaluator 140 can select the detection model 145 with the highest likelihood 330 to use for analyzing the eye 315 of the subject 320. In some embodiments, the eye evaluator 140 can calculate, determine, or otherwise generate a combined likelihood using the set of likelihoods 330 generated from the set of detection model 145. The combined likelihood may correspond to a weighted sum, a mean, or a median of the set of likelihoods 330. The combined likelihood 330 may be used to determine the classification 335 as detailed herein.

[0054] In some embodiments, the eye evaluator 140 can output information 340 (e.g., display through the display 115) based on an association between the subject 320 and the classification 335. For example, the information 340 may include whether the eye 315 of the subject 320 has retinoblastoma or not, whether the abnormality of the eye 315 is cancerous or not, among others. In some embodiments, the information 340 may include the accuracy of the classification 335 (e.g., based on the detection model 145 used). In some embodiments, the information 340 may include a correlation between the classification 335 and the clinical data 310. In some embodiments, the eye evaluator 140 can store, using one or more data structures (e.g., linked list, tree, table, matrix, or array), the association between the subject 320 and the classification 335. The eye evaluator 140 can store and maintain the association between the subject 320 and the classification 335, along with any information of the subject 320, the database 150. In some embodiments, theeye evaluator 140 can encrypt the association and the information stored on the database 150.

[0055] In some embodiments, the eye evaluator 140 can create, produce, or otherwise generate output information 340 based on the identified presence or absence of retinoblastoma. In some embodiments, the eye evaluator 140 can generate an output identifying the subject 320 to be administered with a treatment for the eye 315 of the subject 320, when the classification 335 indicates the presence of retinoblastoma. For example, the eye evaluator 140 can generate an output identifying the subject 320 as to be administered with such a treatment as enucleation, chemotherapy, radiation therapy, laser therapy, cryotherapy, or immunotherapy, among others, for the eye 315 of the subject 320. Conversely, the eye evaluator 140 can generate an output identifying the subject 320 as not to be administered with a treatment for the eye 315 of the subject 320 when the classification 335 indicates the absence of retinoblastoma. With the generation, the eye evaluator 140 can send, transmit, or otherwise provide the output information 340 for presentation on the display 115.

[0056] With receipt, the display 115 can present, render, or otherwise display the output information 340. The information presented in the display 115 may indicate whether the eye 315 of the subject 320 has retinoblastoma or not, whether the abnormality of the eye 315 is cancerous or not, among others. In some embodiments, the information may indicate whether the subject 320 is identified as to be administered with a treatment for retinoblastoma. When the classification 335 indicates the presence of retinoblastoma, the information may indicate that the subject 320 is identified as to be administered with a therapy, such as enucleation, chemotherapy, radiation therapy, laser therapy, cryotherapy, or immunotherapy, among others. For instance, using the presented information, a clinician examining the subject 320 may administer the anti -retinoblastoma therapy, such as surgical or laser removal of the cancer within the affected eye 315. On the other hand, when the classification 335 indicates the absence of retinoblastoma, the information may indicate that the subject 320 is identified as not to be administered with a therapy. For instance, using the presented information, a clinician examining the subject 320 may withhold the antiretinoblastoma therapy or may conduct further examination of the eye 315.

[0057] Referring now to FIG. 4, depicted is a flow diagram of a method 400 of detecting retinoblastoma using images of eyes. The method 400 may be performed by or implemented using the system 100 described herein in conjunction with FIGs. 1-4 or the system 1000 detailed herein in Section C. Under the method 400, a computing system (e.g., the image processing system 105) may identify an image (e.g., the image 305) of an eye (e.g., the eye 315) from a subject (e.g., the subject 320) (405). The computing system may apply the image to a detection model (e.g., the detection model 145) including a plurality of weights to determine a classification (e.g., the classification 335) identifying one of presence or absence of retinoblastoma in the eye (410). The computing system may determine a likelihood (e.g., the likelihood 330) of the presence of retinoblastoma in the eye (415). The computing system may determine whether the likelihood satisfies a predetermined threshold (420). In response to a determination that the likelihood satisfies the predetermined threshold (“Y” in FIG. 4), the computing system may classify the eye as retinoblastoma (425). In response to a determination that the likelihood does not satisfy the predetermined threshold (“N” in FIG. 4), the computing system may classify the eye as nonretinoblastoma (430). The computing system may output information (e.g., the information 340) on the classification (435).

[0058] Referring now to FIG. 5, depicted is a flow diagram of a method 500 of training models to detect retinoblastoma using images of eyes. The method 500 may be performed by or implemented using the system 100 described herein in conjunction with FIGs. 1-4 or the system 1000 detailed herein in Section C. Under the method 500, a computing system (e.g., the image processing system 105) may identify an image (e.g., the image 205) of an eye (e.g., the eye 220) from a subject (e.g., the subject 225) (505) from the training dataset (e.g., the training dataset 155). The computing system may apply the image to a detection model (e.g., the detection model 145) (510). The computing system may determine a likelihood (e.g., the likelihood 230) of the presence of retinoblastoma in the eye (515). The computing system may determine whether the likelihood satisfies a predetermined threshold (520). In response to a determination that the likelihood satisfies the predetermined threshold (“Y” in FIG. 5), the computing system may classify the eye as retinoblastoma (525). In response to a determination that the likelihood does not satisfy the predetermined threshold (“N” in FIG. 5), the computing system may classify the eye as non-retinoblastoma (530). The computing system may compare the classification (e.g., the classification 235) with an annotation (e.g., the annotation 215) (535). The computing system may determine a loss metric (e.g., the loss metric 240) based on the comparison (540). The computing system may update the detection model (545).

[0059] Disclosed herein are techniques for detecting retinoblastoma using images of eyes based on artificial intelligence (Al) and / or machine learning (ML). The techniques include classifying eyes as retinoblastoma or non-retinoblastoma based on the images of eyes. In some embodiments, the techniques include classifying based on a likelihood of the presence of retinoblastoma. For example, the techniques may include determining the likelihood based on eye images and / or structures (e.g., microstructure), which cannot be reliably performed by a human. These non-invasive diagnostic techniques would greatly assist ophthalmologists in diagnosing retinoblastoma with reduced error rates. Thus, the techniques disclosed herein provide a solution to reliably diagnose retinoblastoma, thereby allowing for an early treatment of the eyes without having to remove eyes. The techniques disclosed herein can be used to diagnose various types of retinoblastoma. With the diagnosis of retinoblastoma, therapies can be administered to the eye of the subject to remove or treat the retinoblastoma.B. Detecting Retinoblastoma from Digital Images

[0060] Retinoblastoma is diagnosed and treated without biopsy based solely on appearance (with the indirect ophthalmoscope and imaging). More than 20 benign ophthalmic disorders resemble retinoblastoma, and errors in diagnosis continue to be made worldwide. A better non-invasive method for distinguishing retinoblastoma from pseudoretinoblastoma is needed.

[0061] RetCam imaging of retinoblastoma and pseudo-retinoblastoma from the largest retinoblastoma center in the U.S. was used for this study. Several neural networks (ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, and a Vision Image Transformer, or VIT) were used, using 80% of images for training, 10% for validation, and 10% for testing.

[0062] Two thousand eight hundred eighty -two RetCam images from patients with retinoblastoma at diagnosis, 1,970 images from pseudo-retinoblastoma at diagnosis, and 804 normal pediatric fundus images were included. The highest sensitivity (98.6%) was obtained with a ResNet-101 model, as were the highest accuracy and Fl scores of 97.3% and 97.7%. The highest specificity (97.0%) and precision (97.0%) were attained with a ResNet-152 model.

[0063] The machine learning algorithm successfully distinguished retinoblastoma from retinoblastoma with high specificity and sensitivity, and if implemented worldwide, will prevent hundreds of eyes from incorrectly being surgically removed yearly.

[0064] Intraocular tumors and retinoblastoma in particular are routinely diagnosed and treated without biopsy because biopsy and intraocular surgery on eyes with retinoblastoma (unsuspected) can result in extraocular extension and death. Because of that, retinoblastoma is routinely diagnosed with the indirect ophthalmoscope and confirmed with retinal imaging (usually the RetCam system). Ancillary imaging is often used but notoriously non-specific. CT scans are no longer used (because of the ionizing radiation), and while ultrasound and MRI are sometimes helpful, they are fraught with false positives and negatives.

[0065] Many intraocular lesions can look like retinoblastoma, and the differential diagnosis includes more than 20 separate conditions. Human errors are common. Despite improvements in ultrasound, MR, and retinal imaging, errors continue to occur. The incidence of errors worldwide is not known, but findings suggest an error rate of 10% in some countries.

[0066] The use of machine learning in medicine has expanded. In ophthalmology it has been investigated extensively for retinal diseases, including diabetes, retinal detachment, macular degeneration, macula holes, central retinal vein occlusion, retinitis pigmentosa, retinopathy of prematurity, reticular pseudodrusen, utilizing fundus photography, and OCT and fluorescein angiography. It has also been used for glaucoma, cataracts, oculoplastics, keratoconus, refractive surgery, and strabismus surgery.[0067 J Machine learning may be used in identifying retinoblastoma. It has been used with external photographs to detect leukocoria in addition to differentiating a retinoblastoma fundus from a normal fundus and also used as a model for the economic implications of ML but has not been used to distinguish retinoblastoma from pseudoretinoblastoma. For this study, a machine learning (ML) algorithm was developed to differentiate retinoblastoma cases from pseudo-retinoblastoma cases based on a single RetCam image.

[0068] The disclosure herein is based on clinical imaging that was done with the RetCam digital imaging system. De-identified images were selected from the service’s images, specifically those of retinoblastoma, pseudo-retinoblastoma, and normal pediatric eyes, excluding any patient that had received any form of ocular treatment, including radiation, chemotherapy, laser, cryotherapy, or surgery (because treated tumors have a different appearance from naive, untreated tumors). External photographs were excluded, but close-up RetCam photographs were included, including iris images if there was useful fundus imaging. Photographs that were not of the eye (such as photographs of equipment) were excluded. All images had a 4:3 aspect ratio and were of size 1600xl200px or 640x480px.

[0069] To create a dataset suitable for model training and inference, the images were augmented with a random horizontal flip. Black-pixel padding was further applied to all of the images to make them square, and the images were resized to 224x224px (standard ResNet input size).

[0070] Several neural networks (ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet- 152, and a Vision Image Transformer, or VIT) were fine-tuned, using 80% of images for training, 10% for validation, and 10% for testing (splitting into folders randomly). Two separate datasets were created for fine-tuning, one with only retinoblastoma and pseudo-retinoblastoma eyes and one with those and normal pediatric eyes. The normal eyes were included under “not retinoblastoma,” but the same test dataset was used (the one without normal images) for both sets of models to determine if the inclusion of normal images in only the training and validation sets affected model performance on abnormal eyes.

[0071] 5,566 RetCam fundus images were selected from the ophthalmic oncology service. Of these, 2,882 were of retinoblastomas, while 1,970 were of pseudoretinoblastomas, and 804 were normal. These pseudo-retinoblastoma cases include patients with Coats Disease, Persistent Hyperplastic Primary Vitreous (PHP V) / Persi stent Fetal Vasculature Syndrome (PFV), Cataract, Toxoplasmosis, Nevus of Ota, Tuberous sclerosis, Morning Glory syndrome, Microphthalmos, Pseudoglioma, Myelinated nerve fibers, Persistent Burgmeisters Papilla, Falciform fold, Epithelial cyst, Anterior segment dysgenesis, optic atrophy, Incontinentia pigmenti, chorioretinal scar, CMV retinitis, optic chiasmatic tumor, retinal hemorrhage, vitreous hemorrhage, and papilledema and dysplastic retina. Using an 80-10-10 split, a dataset containing 2,223 retinoblastoma images and 1,576 pseudo-retinoblastoma images (plus 541 normal images) was created for training, 280 retinoblastoma images and 197 pseudo-retinoblastoma images (plus 197 normal images) for validation, and 276 retinoblastoma images and 198 pseudo-retinoblastoma images for testing. ResNets were trained in parallel on this data using the hyperparameters listed below.

[0072] Hyperparameter values:• 25 Epochs• Learning Rate Decay - step of 0.1 every 7 epochs• Optimizer: SGD (Stochastic Gradient Descent) with a momentum of 0.9• Pre-trained weights: ImageNet-lK• Batch Size: 32 images / batch• Learning Rate: 1x1 O'3• Loss function: CrossEntropyLoss (Crossentropy)• VIT Base Model: vit_base_patchl6_224

[0073] Table 1 lists model performance on the test set for each of the models trained. The number that was the highest number in each category is listed in bold / italics .Table 1:

[0074] The summary of the model training history is graphically represented in FIG.6. The confusion matrix of the best model is shown in FIG. 7.

[0075] An algorithm was developed that utilizes RetCam imaging and enables clinicians to distinguish retinoblastoma from lesions that simulate retinoblastoma. Machine learning and Al have been used before in identifying retinoblastoma. For example, under one approach, a deep learning model was created using an established dataset of 400 retinoblastoma images and 400 normal fundus images. It demonstrated an accuracy of 97% on the test set. Using LIME (local interpretable model-agnostic explanations) and SHAP (SHapley Additive exPlanations), it was demonstrated that the model could reliably differentiate a fundus image of retinoblastoma from a normal fundus image. The “normal fundus images” were taken from an established library of normal adult eyes.Retinoblastoma only develops in children’s eyes (with very rare exceptions), so the control normal eyes are of interest in developing a model but not useful for clinicians. In clinical practice, differentiating a normal eye from an eye with retinoblastoma is easy and not the clinical challenge. More than 20 different conditions can simulate the fundus appearance ofretinoblastoma (Coats disease being the most common), and the clinical challenge and need for improvement is in differentiating retinoblastoma from these diverse non-malignant conditions.

[0076] In another approach, an Al model was created based on 771 fundus images of 109 eyes. Five hundred ninety demonstrated retinoblastoma, and 181 had no tumor. The sensitivity was 85%, the specificity 99%. Pseudo-retinoblastomas were not included. Their “normal” were the fellow eyes of unilateral retinoblastoma patients (children). Their dataset was mostly from heavily pigmented eyes, so its veracity in lightly pigmented eyes is unknown. Under yet another approach, a number of useful algorithms were created to detect leukocoria based on external photographs. For example, a mobile application was developed based on 52,982 external photographs of 40 children (8 with retinoblastoma) with a reported sensitivity of 90%. RetCam images were not used. This tool was designed to detect leukocoria (including retinoblastoma) but not to differentiate retinoblastoma from pseudo-retinoblastoma.

[0077] This disclosure differs as fundus images of retinoblastoma patients who had not received any treatment were compared to fundus images of pseudo-retinoblastoma that had also not had prior treatment. The goal was to create a useful, simple-to-use, reliable algorithm that would aid clinicians in the differential diagnosis of retinoblastoma. Errors in diagnosis can be fatal. If a child with retinoblastoma is thought to have a benign lesion (and therefore not treated), death can ensue.

[0078] If a child with a benign lesion (possibly treatable) is thought to have retinoblastoma and receives radiation, chemotherapy, or enucleation, loss of an eye or vision — to say nothing about treatment complications that are acceptable if the eye had contained cancer — can ensue (FIG. 8). The true incidence of errors in retinoblastoma diagnosis in 2024 is not known, but some recent series suggest an error rate of 10% in some countries. Using the algorithm may help minimize incorrect diagnoses and treatments.

[0079] Some of the limitations of this approach were also learned. Retinoblastomas in the eye are three-dimensional structures, and RetCam images are two-dimensional entities. An example of an error the algorithm made is presented in the figure. In FIG. 9, what appears to be an elevated white / creamy mass in the retina was actually a benignfibrovascular lesion just behind the lens. The model misinterpreted the first image, but correctly identified the second (as well as all other images from that eye, which were in the training set) as pseudo-retinoblastoma. This diagnosis was easily appreciated by the clinician using the indirect ophthalmoscope (which allows the viewer a 3-D view), thus clinicians were expected to use the algorithm in conjunction with indirect ophthalmoscopy.

[0080] “Field of view” seen in a RetCam image influenced results. In some cases, a RetCam image from an eye will not show the tumor as it may be outside the field captured by the photograph. The algorithm included all images taken from eyes with retinoblastoma, but images where there was no visible tumor were not filtered out. Presumably, when the algorithm is used by clinicians, they will only submit images that demonstrate pathology, eliminating this source of error.C. Computing and Network Environment

[0081] Various operations described herein can be implemented on computer systems. FIG. 10 shows a simplified block diagram of a representative server system 1000, client computing system 1014, and network 1026 usable to implement certain embodiments of the present disclosure. In various embodiments, server system 1000 or similar systems can implement services or servers described herein or portions thereof. Client computing system 1014 or similar systems can implement clients described herein. The system 1000 described herein can be similar to the server system 1000. Server system 1000 can have a modular design that incorporates a number of modules 1002 (e.g., blades in a blade server embodiment); while two modules 1002 are shown, any number can be provided. Each module 1002 can include processing unit(s) 1004 and local storage 1006.

[0082] Processing unit(s) 1004 can include a single processor, which can have one or more cores, or multiple processors. In some embodiments, processing unit(s) 1004 can include a general-purpose primary processor as well as one or more special-purpose coprocessors such as graphics processors, digital signal processors, or the like. In some embodiments, some or all processing units 1004 can be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself. In other embodiments, processing unit(s) 1004 can executeinstructions stored in local storage 1006. Any type of processors in any combination can be included in processing unit(s) 1004.

[0083] Local storage 1006 can include volatile storage media (e.g., DRAM, SRAM, SDRAM, or the like) and / or non-volatile storage media (e.g., magnetic or optical disk, flash memory, or the like). Storage media incorporated in local storage 1006 can be fixed, removable, or upgradeable as desired. Local storage 1006 can be physically or logically divided into various subunits such as a system memory, a read-only memory (ROM), and a permanent storage device. The system memory can be a read-and-write memory device or a volatile read-and-write memory, such as dynamic random-access memory. The system memory can store some or all of the instructions and data that processing unit(s) 1004 need at runtime. The ROM can store static data and instructions that are needed by processing unit(s) 1004. The permanent storage device can be a non-volatile read-and-write memory device that can store instructions and data even when module 1002 is powered down. The term “storage medium” as used herein includes any medium in which data can be stored indefinitely (subject to overwriting, electrical disturbance, power loss, or the like) and does not include carrier waves and transitory electronic signals propagating wirelessly or over wired connections.

[0084] In some embodiments, local storage 1006 can store one or more software programs to be executed by processing unit(s) 1004, such as an operating system and / or programs implementing various server functions, such as functions of the system 1000 of FIG. 10 or any other system described herein, or any other server(s) associated with system 1000 or any other system described herein.

[0085] “Software” refers generally to sequences of instructions that, when executed by processing unit(s) 1004, cause server system 1000 (or portions thereof) to perform various operations, thus defining one or more specific machine embodiments that execute and perform the operations of the software programs. The instructions can be stored as firmware residing in read-only memory and / or program code stored in non-volatile storage media that can be read into volatile working memory for execution by processing unit(s) 1004. Software can be implemented as a single program or a collection of separate programs or program modules that interact as desired. From local storage 1006 (or nonlocal storage described below), processing unit(s) 1004 can retrieve program instructions to execute and data to process in order to execute various operations described above.

[0086] In some server systems 1000, multiple modules 1002 can be interconnected via a bus or other interconnect 1008, forming a local area network that supports communication between modules 1002 and other components of server system 1000. Interconnect 1008 can be implemented using various technologies, including server racks, hubs, routers, etc.

[0087] A wide area network (WAN) interface 1010 can provide data communication capability between the local area network (interconnect 1008) and the network 1026, such as the Internet. Technologies can be used, including wired (e.g., Ethernet, IEEE 602.3 standards) and / or wireless technologies (e.g., Wi-Fi, IEEE 602.11 standards).

[0088] In some embodiments, local storage 1006 is intended to provide working memory for processing unit(s) 1004, providing fast access to programs and / or data to be processed while reducing traffic on interconnect 1008. Storage for larger quantities of data can be provided on the local area network by one or more mass storage subsystems 1012 that can be connected to interconnect 1008. Mass storage subsystem 1012 can be based on magnetic, optical, semiconductor, or other data storage media. Direct-attached storage, storage area networks, network-attached storage, and the like can be used. Any data stores or other collections of data described herein as being produced, consumed, or maintained by a service or server can be stored in mass storage subsystem 1012. In some embodiments, additional data storage resources may be accessible via WAN interface 1010 (potentially with increased latency).

[0089] Server system 1000 can operate in response to requests received via WAN interface 1010. For example, one of the modules 1002 can implement a supervisory function and assign discrete tasks to other modules 1002 in response to received requests. Work allocation techniques can be used. As requests are processed, results can be returned to the requester via WAN interface 1010. Such operation can generally be automated. Further, in some embodiments, WAN interface 1010 can connect multiple server systems 1000 to each other, providing scalable systems capable of managing high volumes of activity. Other techniques for managing server systems and server farms (collections ofserver systems that cooperate) can be used, including dynamic resource allocation and reallocation.

[0090] Server system 1000 can interact with various user-owned or user-operated devices via a wide-area network such as the Internet. An example of a user-operated device is shown in FIG. 10 as client computing system 1014. Client computing system 1014 can be implemented, for example, as a consumer device such as a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, eyeglasses), desktop computer, laptop computer, and so on.

[0091] For example, client computing system 1014 can communicate via WAN interface 1010. Client computing system 1014 can include computer components such as processing unit(s) 1016, storage device 1018, network interface 1020, user input device 1022, and user output device 1037. Client computing system 1014 can be a computing device implemented in a variety of form factors, such as a desktop computer, laptop computer, tablet computer, smartphone, other mobile computing device, wearable computing device, or the like.

[0092] Processing unit(s) 1016 and storage device 1018 can be similar to processing unit(s) 1004 and local storage 1006 described above. Suitable devices can be selected based on the demands to be placed on client computing system 1014; for example, client computing system 1014 can be implemented as a “thin” client with limited processing capability or as a high-powered computing device. Client computing system 1014 can be provisioned with program code executable by processing unit(s) 1016 to enable various interactions with server system 1000.

[0093] Network interface 1020 can provide a connection to the network 1026, such as a wide area network (e.g., the Internet), to which WAN interface 1010 of server system 1000 is also connected. In various embodiments, network interface 1020 can include a wired interface (e.g., Ethernet) and / or a wireless interface implementing various RF data communication standards such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, LTE, etc ).

[0094] User input device 1022 can include any device (or devices) via which a user can provide signals to client computing system 1014; client computing system 1014 can interpret the signals as indicative of particular user requests or information. In various embodiments, user input device 1022 can include any or all of a keyboard, touch pad, touch screen, mouse or other pointing device, scroll wheel, click wheel, dial, button, switch, keypad, microphone, and so on.

[0095] User output device 1037 can include any device via which client computing system 1014 can provide information to a user. For example, user output device 1037 can include a display-to-display image generated by or delivered to client computing system 1014. The display can incorporate various image generation technologies, e.g., a liquid crystal display (LCD), light-emitting diode (LED), including organic light-emitting diodes (OLED), projection system, cathode ray tube (CRT), or the like, together with supporting electronics (e.g., digital -to-analog or analog-to-digital converters, signal processors, or the like). Some embodiments can include a device such as a touchscreen that functions as both an input and output device. In some embodiments, other user output devices 1037 can be provided in addition to or instead of a display. Examples include indicator lights, speakers, tactile “display” devices, printers, and so on.

[0096] Some embodiments include electronic components, such as microprocessors, storage, and memory that store computer program instructions in a computer-readable storage medium. Many of the features described in this specification can be implemented as processes that are specified as a set of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processing units, they cause the processing unit(s) to perform various operations indicated in the program instructions. Examples of program instructions or computer code include machine code, such as that produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter. Through suitable programming, processing unit(s) 1004 and 1016 can provide various functionality for server system 1000 and client computing system 1014, including any of the functionality described herein as being performed by a server or client, or other functionality.

[0097] It will be appreciated that server system 1000 and client computing system 1014 are illustrative and that variations and modifications are possible. Computer systems used in connection with embodiments of the present disclosure can have other capabilities not specifically described here. Further, while server system 1000 and client computing system 1014 are described with reference to particular blocks, it is to be understood that these blocks are defined for convenience of description and are not intended to imply a particular physical arrangement of component parts. For instance, different blocks can be, but need not be, located in the same facility, in the same server rack, or on the same motherboard. Further, the blocks need not correspond to physically distinct components. Blocks can be configured to perform various operations, e.g., by programming a processor or providing appropriate control circuitry, and various blocks might or might not be reconfigurable depending on how the initial configuration is obtained. Embodiments of the present disclosure can be realized in a variety of apparatus, including electronic devices implemented using any combination of circuitry and software.

[0098] While the disclosure has been described with respect to specific embodiments, one skilled in the art will recognize that numerous modifications are possible. Embodiments of the disclosure can be realized using a variety of computer systems and communication technologies, including but not limited to the specific examples described herein. Embodiments of the present disclosure can be realized using any combination of dedicated components and / or programmable processors and / or other programmable devices. The various processes described herein can be implemented on the same processor or different processors in any combination. Where components are described as being configured to perform certain operations, such configuration can be accomplished, e.g., by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation, or any combination thereof. Further, while the embodiments described above may make reference to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware and / or software components may also be used and that particular operations described as being implemented in hardware might also be implemented in software or vice versa.

[0099] Computer programs incorporating various features of the present disclosure may be encoded and stored on various computer-readable storage media; suitable media include magnetic disk or tape, optical storage media such as compact disk (CD) or DVD (digital versatile disk), flash memory, and other non-transitory media. Computer-readable media encoded with the program code may be packaged with a compatible electronic device, or the program code may be provided separately from electronic devices (e.g., via Internet download or as a separately packaged computer-readable storage medium).

[0100] Thus, although the disclosure has been described with respect to specific embodiments, it will be appreciated that the disclosure is intended to cover all modifications and equivalents within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method of detecting retinoblastoma using images of eyes, comprising: identifying, by one or more processors, a first image of at least one first eye of a first subject prior to administration of treatment for retinoblastoma, the at least one eye having a structure of interest corresponding to an abnormality; applying, by the one or more processors, the first image to a machine learning (ML) model, the ML model established using a training dataset comprising a plurality of examples, each of the plurality of examples identifying: (i) a respective second image of at least one second eye of a second subject and (ii) a respective second classification identifying one of presence or absence of retinoblastoma in the at least second eye; determining, by the one or more processors, from applying the ML model, a first classification identifying one of presence or absence of retinoblastoma in the at least one first eye of the first subject; and storing, by the one or more processors, using one or more data structures, an association between the first subject and the classification.

2. The method of claim 1, further comprising identifying, by the one or more processors, clinical data of the first subject, the clinical data comprising at least one of: (i) an age, (ii) genetic data, (iii) family data, or (iv) a trait of the first subject; wherein applying the ML model further comprises applying the clinical data to the ML model to determine the first classification.

3. The method of claim 1, further comprising determining, by the one or more processors, from applying the ML model, a likelihood of presence of retinoblastoma in the at least one first eye; and wherein determining the first classification further comprises determining the first classification identifying the presence of retinoblastoma in the at least one first eye of the first subject, responsive to the likelihood satisfying a threshold.

4. The method of claim 3, further comprising generating, by the one or more processors, an output identifying the first subject as to be administered with a treatment forthe at least one first eye of the first subject, in response to a determination of the presence of retinoblastoma.

5. The method of claim 3, further comprising generating, by the one or more processors, an output identifying the first subject as not to be administered with a treatment for the at least one first eye of the first subject, in response to a determination of absence of retinoblastoma.

6. The method of claim 1, wherein determining the first classification further comprises comparing a likelihood of presence of retinoblastoma in the at least one first eye with a threshold, wherein the threshold is determined during establishment of the ML model.

7. The method of claim 1, further comprising providing, by the one or more processors, an output based on the association between the first subject and the first classification.

8. The method of claim 1, wherein the first image further comprises a fundus image of a back region of the at least one first eye acquired via a fundus camera from an interior or an exterior of the at least one first eye.

9. The method of claim 1, wherein the first subject is a human child under age of five years, and wherein the retinoblastoma comprises at least one of (i) unilateral retinoblastoma, (ii) bilateral retinoblastoma, or (iii) a trilateral retinoblastoma.

10. The method of claim 1, wherein applying the first image to a machine learning (ML) model comprises applying the first image to a plurality of ML models.

11. The method of claim 10, further comprising: wherein determining the first classification further comprises determining, based on applying the first image to the plurality of ML models, a plurality of likelihoods corresponding to the plurality of ML models, each of the plurality of likelihoods identifying presence of retinoblastoma in the at least one first eye of the first subject; andselecting, by the one or more processors, at least the ML model of the plurality of ML models based on the plurality of likelihoods.

12. A method of training models to detect retinoblastoma using images of eyes, comprising: identifying, by one or more processors, a training dataset comprising a plurality of examples, each of the plurality of examples comprising (i) a respective image of at least one eye of a subject prior to administration of treatment for retinoblastoma and (ii) a respective first classification identifying one of presence or absence of retinoblastoma in the at least one eye; for each example of the plurality of examples of the training dataset: applying, by the one or more processors, the respective image to a machine learning (ML) model comprising a plurality of weights to determine a second classification identifying one of presence or absence of retinoblastoma in the at least one eye; comparing, by the one or more processors, the respective first classification and the second classification; and determining, by the one or more processors, a loss metric based on comparing the respective first classification and the second classification; and updating, by the one or more processors, at least one weight of the ML model in accordance with the loss metric.

13. The method of claim 12, wherein each example of the plurality of examples further comprises clinical data comprising at least one of: (i) an age, (ii) genetic data, (iii) family data, (iv) a sex, or (v) a trait of the subject, and wherein applying the ML model further comprises applying the clinical data of each example of the plurality of examples to the ML model to determine the second classification.

14. The method of claim 12, wherein applying the ML model further comprises determining a confidence value for the second classification to identify presence of retinoblastoma in the at least one first eye for each example of the plurality of examples.

15. The method of claim 12, wherein determining the loss metric further comprises determining the loss metric as a function of the first classification and a confidence value for the second classification.

16. The method of claim 12, further comprising determining, by the one or more processors, a threshold against which to compare confidence values based on the loss metric of the ML model.

17. The method of claim 12, wherein the image in each example includes a fundus image of a back region of the at least one first eye acquired via a fundus camera from an interior or an exterior of the at least one first eye.

18. The method of claim 12, wherein the subject is a human child under age of five years, and wherein the retinoblastoma comprises at least one of: (i) unilateral retinoblastoma, (ii) bilateral retinoblastoma, or (iii) a trilateral retinoblastoma.

19. A system for detecting retinoblastoma using images of eyes, comprising: a computing system having one or more processors coupled with memory, configured to: identify a first image of at least one first eye of a first subject prior to administration of treatment for retinoblastoma, the at least one eye having a structure of interest corresponding to an abnormality; apply the first image to a machine learning (ML) model, the ML model established using a training dataset comprising a plurality of examples, each of the plurality of examples identifying: (i) a respective second image of at least one second eye of a second subject and (ii) a respective second classification identifying one of presence or absence of retinoblastoma in the at least second eye; determine, from applying the ML model, a first classification identifying one of presence or absence of retinoblastoma in the at least one first eye of the first subject; and store, using one or more data structures, an association between the first subject and the classification.

20. The system of claim 19, wherein the computing system is further configured to: identify clinical data of the first subject, the clinical data comprising at least one of:(i) an age, (ii) genetic data, (iii) family data, (iv) a sex or (v) a trait of the first subject; apply the clinical data to the ML model to determine the first classification.

21. The system of claim 19, wherein the computing system is further configured to determine, from applying the ML model, a likelihood of presence of retinoblastoma in the at least one first eye; and determine the first classification identifying the presence of retinoblastoma in the at least one first eye of the first subject, responsive to the likelihood satisfying a threshold.

22. The system of claim 21, wherein the computing system is further configured to generate an output identifying the first subject as to be administered with a treatment for the at least one first eye of the first subject, in response to a determination of the presence of retinoblastoma.

23. The system of claim 21, wherein the computing system is further configured to generate an output identifying the first subject as not to be administered with a treatment for the at least one first eye of the first subject, in response to a determination of absence of retinoblastoma.

24. The system of claim 19, wherein the computing system is further configured to compare a likelihood of presence of retinoblastoma in the at least one first eye with a threshold, wherein the threshold is determined during establishment of the ML model.

25. The system of claim 19, the first image further comprises a fundus image of a back region of the at least one first eye acquired via a fundus camera from an interior or an exterior of the at least one first eye.

26. The system of claim 19, wherein the computing system is further configured to provide an output based on the association between the first subject and the first classification.

27. The system of claim 19, wherein the computing system is further configured to apply the first image to a plurality of ML models.

28. The system of claim 27, wherein the computing system is further configured to: determine, based on an application of the first image to the plurality of ML models, a plurality of likelihoods corresponding to the plurality of ML models, each of the plurality of likelihoods identifying presence of retinoblastoma in the at least one first eye of the first subject; and select at least the ML model of the plurality of ML models based on the plurality of likelihoods.

Images