Systems and methods for detecting retinopathy in preterm infants
Through a machine learning-based detection system, the view assessment, quality assessment and attribute information extraction of retinopathy in premature infants is solved, and the problems of high cost and inaccurate detection of retinopathy screening facilities in premature infants are solved, and the problem of different Efficient, accurate and economical ROP screening for geographical location.
Patent Information
- Application Number
- CN202480001906.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-01-04
- Filing Date
- 2024-01-04
- Publication Date
- 2024-11-29
AI Technical Summary
The prior art is difficult to provide efficient, accurate and cost-effective retinopathy (ROP) screening facilities for premature infants in large populations in different geographical locations, especially in the absence of professional ophthalmologists and expensive equipment, with existing automated testing The method is mainly suitable for the detection of advanced lesions rather than early screening.
Using a machine learning-based detection system, the input eye images are preprocessed, quality assessment and attribute information extracted through view assessment, quality assessment and classification models, to detect whether there is a ROP, and visual output is generated to realize the early stage of retinopathy in premature infants. Screening. The system does not require professional equipment or skills, is suitable for portable use and can be screened on mobile devices.
Effective ROP screening in a single device is achieved, ensuring the accuracy and cost-effectiveness of detection, enabling screening without expert knowledge, and ensuring the desired accuracy and functionality on portable retinal cameras.
Smart Images

Figure CN119053280A_ABST
Abstract
Description
Background Art
[0001] The development of retinal blood vessels in infants' eyes depends on multiple factors. Among other factors, one of the most important factors is the birth time of the infant. For example, premature birth of an infant may hinder or block the development of retinal blood vessels, thereby causing abnormal development of retinal blood vessels when the infant is delivered from the mother's womb. Due to such abnormal development, a disease called retinopathy of prematurity (ROP) is caused, which is a potentially blinding disease. In the absence of early screening for ROP, severe types of ROP that eventually cause the retina to detach from the eye wall and cause blindness may result. In most cases, ROP screening can be performed under the supervision of a highly professional ophthalmologist using a bedside clinical examination device. However, such clinical procedures are not suitable for providing accurate and reliable screening facilities to a large number of people in different geographical locations at the lowest cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0002] Systems and / or methods according to examples of the present subject matter are now described with reference to the accompanying drawings, in which:
[0003] Figure 1A and Figure 1B A training system for training a retinopathy of prematurity (ROP) detection model pipeline according to an example is shown;
[0004] Figure 2 A retinopathy of prematurity (ROP) detection system for detecting the presence of ROP in an input eye according to another example is shown;
[0005] Figure 3 A method for training a ROP detection model pipeline according to an example is shown; and
[0006] Figure 4A and Figure 4B A method for detecting the presence of a ROP in an input eye image based on a trained ROP detection model pipeline according to an example is shown. DETAILED DESCRIPTION
[0007] Retinopathy of prematurity (ROP) is a potentially blinding disease that affects premature infants (particularly, premature infants weighing less than 2 kg at birth). The human eye contains a lens that is used to focus the image on the inside of the back of the eye (i.e., on the retina). The retina is covered by a network of blood vessels beneath the retina. These blood vessels usually grow rapidly in the last few weeks before the baby is born. However, because the baby is born prematurely, the growth of these blood vessels stops and may grow into unwanted parts of the eye. This may lead to the formation of scar tissue that eventually damages the retina and causes vision loss, a complication called ROP. In order to prevent the negative effects of ROP, early screening for ROP is needed.
[0008] ROP screening is a procedure that can be performed under the supervision of a professional ophthalmologist using a bedside clinical examination kit or digital image analysis tools. The screening procedure involves monitoring changes or patterns in the growth of retinal blood vessels at the back of the eye or beneath the retina. In addition, due to advances in machine learning, several machine learning algorithms have been developed to automate the detection of ROP.
[0009] However, in the case of manual inspection, manual inspection requires the presence of a professional pediatric ophthalmologist and expensive equipment with tele-ophthalmology. The availability of such professional medical practitioners and equipment is limited, and providing such facilities in a large number of tertiary care centers (especially, tertiary care centers located in rural areas) may be costly. On the other hand, automatic inspection using machine learning focuses on determining additional lesions (plusdisease) as a late stage of ROP. This method is insufficient for early detection and screening of ROP.
[0010] Therefore, there is an ongoing effort in the fields of ophthalmology and medical technology to develop systems and methods for efficient, accurate, and cost-effective ROP screening, particularly in large populations across different geographic locations.
[0011] A method for detecting the presence of ROP in an input eye is described. Detection of the presence of ROP in an input eye is performed using an input eye image corresponding to a subject being evaluated for detecting the presence of ROP. In another example, detection of the presence of ROP in an input eye may be performed using multiple input eye images corresponding to different views of the eye. Examples of such views include, but are not limited to, a temporal view, a nasal view, a disc centered view, a macula centered view, an inferior view, and a superior view. Multiple input eye images may also correspond to images of the input eye of a subject being screened. Such input eye images may be stored in a database repository or may be captured using a camera device.
[0012] In one example, an input eye image (or a set of input eye images) corresponding to an input eye that is to be subjected to screening for the presence of ROP is obtained. Once the input eye image is obtained, the input eye image may be subjected to a number of pre-processing steps including cropping, padding, resizing, and sharpening. The purpose of these pre-processing steps is to make the input eye image (or all input eye images) suitable for the view of the assessment eye image. It may be noted that performing these pre-processing steps is not essential.
[0013] Once the pre-processing step is performed, the input eye image can be processed based on the view assessment model to determine whether the input eye image has a temporal view. In another example, in the case of a set of input eye images, multiple eye images can be processed based on the view assessment model to select an input eye image with a temporal view from a set of input eye images. Examples of possible views of eye images include, but are not limited to, a temporal view, a macular view, a central view of the optic disc, an inferior view, an superior view, and a nasal view. In the example, other images with different views but without a temporal view are discarded. In one example, in the case where it is identified that no image in a set of input eye images belongs to a temporal view, a prompt message can be displayed to the user using the system. It can be noted that, in particular, having a temporal view image helps to make the processing computationally efficient and accurate.
[0014] Continuing with this example, the input eye image with the temporal view can then be subjected to a number of processing steps including cropping, padding, resizing, and sharpening. Also as described above, these processing steps are used to make the input eye image suitable for further processing stages (e.g., quality assessment) to detect the presence of ROP. Furthermore, performing these processing steps is not essential and may be omitted as appropriate.
[0015] Once the processing step is performed, the input eye image may then be processed based on the quality assessment module to determine the quality of the input eye image. If the image quality of the input eye image is acceptable, the input eye image may be used in a further processing stage to detect the presence of ROP. In an example, if the input eye image does not have acceptable quality, the user may be prompted to take the input eye image again.
[0016] Once the input eye image is determined to be acceptable based on the quality criteria, the input eye image can be processed based on the classification model to obtain attribute information of the input eye image. In an example, the attribute information corresponds to a plurality of eye image attributes of the input eye image. Examples of such eye image attributes include, but are not limited to, retinal vessel location, retinal vessel size, retinal vessel structure, presence of a demarcation line, location of a demarcation line, size of a demarcation line, presence of a ridge, location of a ridge, size of a ridge, an indicator indicating partial retinal detachment, and an indicator indicating total retinal detachment, etc.
[0017] Subsequently, based on the obtained attribute information, the classification model detects the presence of ROP and performs a binary classification of the input eye image as one of non-referral ROP and referral ROP. It may be noted that although limited examples of eye image attributes indicating the presence or absence of ROP are described above, other such examples will still be within the scope of the present subject matter. In one example, as described later, the attribute information can be used as a metric parameter for determining the presence of ROP.
[0018] In addition to the detection results of the presence of ROP in the input eye image, a visualization output may also be generated. In one example, the visualization output may take the form of an activation map. The activation map thus obtained may indicate or highlight abnormal areas in the input eye image, the activation map representing those areas or significant parts that caused the input eye to be designated as a referral ROP eye. These and other aspects are discussed in further detail later in this specification.
[0019] It may be noted that the above-mentioned determinations involving view assessment, quality assessment, obtaining attribute information, and detecting the presence of ROP may involve various models such as a view assessment model, a quality assessment model, and a classification model. In one example, each of the above-mentioned models is a machine learning-based model. In the example, the machine learning model may be a deep learning model. Although described as unique or separate models, the view assessment model, the quality assessment model, and the classification model may be implemented as a ROP detection model pipeline for detecting the presence of ROP in the subject eye. It may also be noted that, without departing from the scope of the present subject matter, a ROP detection system including multiple machine learning algorithms (such as a quality assessment model, a view assessment model, and a classification model) also includes an analysis engine for performing one or more intermediate functions such as preprocessing of an input eye image and processing of an input eye image.
[0020] The machine learning models within the ROP detection model pipeline can be trained based on various training data sets. For example, a view assessment model can be trained based on training images with different views (e.g., images with temporal views, macular views, optic disc center views, inferior views, superior views, and nasal views). Similarly, a quality assessment model can be trained based on various training images with various resolutions, contrasts, sharpness, or other such attributes. In a similar manner, a classification model can be trained based on training images associated with ROP and training images without or not associated with ROP.
[0021] The classification model may also be trained based on training attribute information, which may be obtained through clinical history, and comprehensive eye examination and research modalities, including but not limited to optical coherence tomography, visual field, intraocular pressure measurement, corneal thickness measurement, etc. In the example, the classification model includes two sub-models, namely, a binary classification model trained to detect the presence or absence of ROP and a classification classification model trained to classify eye images in various stages, which may include but are not limited to stage 1, stage 2, stage 3, stage 4, stage 5, A-ROP (Aggressive posterior ROP) and smouldering ROP. In the example, the classification classification model may be used only during training to supplement the accuracy of the binary classification model in detecting ROP. Although training has been described in the context of view assessment models, quality assessment models, and classification models, this similar training process may also be performed for other models that may be implemented within the ROP detection model pipeline. Such treatments would still fall within the scope of the present subject matter without limitation.
[0022] The present method overcomes the above technical advantages. For example, the above method can be implemented in a single device for effective ROP screening. Since no specialized equipment or skills are required, the system implementing the present method is mobile, cost-effective, and accurate for the purpose of ROP detection. For example, the implementation system allows screening without expert knowledge and can be performed on the portable retinal camera itself while ensuring the desired accuracy and functional accuracy level.
[0023] The descriptions provided above and the examples discussed further in the current specification are merely exemplary. For example, some examples may be described in which only one image is considered in the training or reasoning phase. However, without departing from the scope of the present subject matter, the current method may also be used in other instances or situations, such as instances or situations in which a set of input eye images or a set of training eye images may be used.
[0024] about Figures 1A to 4B The manner in which a model implemented within a ROP detection model pipeline is trained and used to identify the presence of ROP in an input eye is described in detail. Although aspects of the described system may be implemented in any number of different electronic devices, environments, and / or implementations, the examples are described in the context of the following (one or more) example devices. In another example, aspects of the present subject matter may also be implemented by a stand-alone device with executable instructions. It may be noted that the drawings of the present subject matter shown here are for illustrative purposes and should not be construed as limiting the scope of the claimed subject matter.
[0025] Figure 1A A training system 102 for training a model within a ROP detection model pipeline is shown, including a processor or memory (not shown). In an example, the training system 102 (referred to as system 102) can be communicatively coupled to a repository 104 via a network 106. The repository 104 can further include a training dataset 108. The training dataset 108 can include a plurality of training images that can be used to train the ROP detection model pipeline. In an example, these plurality of training images are those images that were previously taken during manual screening of subjects and annotated with corresponding ROP categories.
[0026] In another example, along with the plurality of training images, the training data set 108 may also include training attribute information and corresponding ROP categories for each of the plurality of training images. The training attribute information corresponds to the plurality of training eye image attributes. In an example, the training eye image attributes may include retinal vessel location, retinal vessel size, retinal vessel structure, presence of a dividing line, location of a dividing line, size of a dividing line, presence of a ridge, location of a ridge, size of a ridge, an indicator indicating partial retinal detachment, and an indicator indicating total retinal detachment, etc.
[0027] Although depicted as being obtained from a single repository such as repository 104, training data set 108 may also be obtained from a plurality of other sources without departing from the scope of the present subject matter. In this case, each of such a plurality of repositories may be interconnected by a network such as network 106.
[0028] The network 106 may be a private network or a public network, and may be implemented as a wired network, a wireless network, or a combination of wired and wireless networks. The network 106 may also include a collection of individual networks interconnected with each other and used as a single large network, such as the Internet. Examples of such individual networks include, but are not limited to, a Global System for Mobile Communications (GSM) network, a Universal Mobile Telecommunications System (UMTS) network, a Personal Communications Service (PCS) network, a Time Division Multiple Access (TDMA) network, a Code Division Multiple Access (CDMA) network, a Next Generation Network (NGN), a Public Switched Telephone Network (PSTN), a Long Term Evolution (LTE), and an Integrated Services Digital Network (ISDN).
[0029] The system 102 may also include instructions 110 and a training engine 112. In an example, the instructions 110 are retrieved from a memory and executed by a processor included in the system 102. The training engine 112 may be implemented as a combination of hardware and programming (e.g., programmable instructions) to implement various functions. In the examples described herein, this combination of hardware and programming may be implemented in several different ways. For example, the programming for the training engine 112 may be executable instructions such as instructions 110. Such instructions may be stored on a non-transitory machine-readable storage medium that may be directly or indirectly coupled to the system 102 (e.g., by networking means). In an example, the training engine 112 may include processing resources (e.g., a single processor or a combination of multiple processors) to execute such instructions. In this example, the non-transitory machine-readable storage medium may store instructions such as instructions 110 that implement the training engine 112 when executed by the processing resources. In other examples, the training engine 112 may be implemented as an electronic circuit.
[0030] The instructions 110, when executed by the processing resources, cause the training engine 112 to train the ROP detection model pipeline 114 based on the training data set 108. The system 102 may also include (one or more) training eye images 116, (one or more) training eye image attributes 118, and ROP categories 120. In an example, the system 102 may obtain the training data set 108 corresponding to a single training eye image from the repository 104, and information related to the training data set 108 is stored in the system 102 as (one or more) training eye images 116, (one or more) training eye image attributes 118, and ROP categories 120.
[0031] As previously described, the ROP detection model pipeline 114 (referred to as model pipeline 114) may also include multiple machine learning models. Examples of such machine learning models include deep learning models. For ease of illustration, as an example, a current method for detecting the presence of ROP is described, in which one or more deep learning models are used to perform different steps. Although the present example is described in relation to a deep learning model, other machine learning models may also be used to implement the aforementioned method. It may also be noted that any description provided in conjunction with a deep learning model is applicable to other machine learning models without limitation and without departing from the scope of the present subject matter. For the sake of brevity, such an example is not described. In conjunction with Figure 1B Further describing the manner in which multiple models within the model pipeline 114 can be trained.
[0032] Figure 1B1 depicts example deep learning models that may be implemented within the model pipeline 114. In one example, the model pipeline 114 may include a view rating model 122, a quality rating model 124, and a classification model 126. It may be noted that the model pipeline 114 may also include other deep learning models (such as Figure 1B ). It may also be possible to implement one or more models to perform a combination of one or more functions. Such variations and combinations will still be used as examples of the present subject matter without limitation.
[0033] With respect to training the view assessment model 122, the training eye image(s) 116 may be used, wherein the training eye image(s) 116 may include images with different views, such as temporal view, nasal view, disc center view, macula center view, inferior view, and superior view, etc. For training the quality assessment model 124, the training eye image(s) 116 may include images with higher resolution, contrast, clarity, or other such attributes. The quality assessment model 124 is trained to assess the quality of the input eye images so that images with low quality may be discarded and only good quality images with higher quality are considered for further processing.
[0034] Accordingly, the classification model 126 may be trained based on the training eye image(s) 116 that identifies attribute information corresponding to a plurality of eye image attributes within the training eye image(s) 116. The classification model 126 may also be trained on training eye images associated with ROP and images not associated with ROP that are part of the training eye image(s) 116. There is a category indicator, such as the ROP category 120, associated with each of the training eye image(s) 116 to indicate the state of the corresponding training eye image.
[0035] In an example, the classification model includes two sub-models, namely, a binary classification model trained to detect the presence or absence of ROP and a classification classification model trained to classify eye images in various stages (e.g., stage 1, stage 2, stage 3, stage 4, stage 5, A-ROP (aggressive posterior pole ROP), and smoldering ROP). In an example, the classification classification model may be used only during training to supplement the accuracy of the binary classification model in detecting ROP.
[0036] As will be discussed subsequently, the view assessment model 122, the quality assessment model 124, and the classification model 126, if trained, may be used to perform various tasks sequentially or in parallel, based on which the presence of a ROP in a subject eye may be determined.
[0037] As also described above, the training of the view rating model 122, the quality rating model 124, and the classification model may be performed in any order and may be performed at different times. As can be appreciated, although one or more common training datasets may be used, the training of any one deep learning model in the model pipeline 114 is independent of the training of other models.
[0038] In an example, once trained, the model pipeline 114 can be used to classify an input eye image as one of a non-referral ROP and a referral ROP. Figure 2 The manner in which the model pipeline 114 may be used to detect ROP within a subject eye is further described.
[0039] Figure 2 An environment 200 is shown utilizing a retinopathy of prematurity (ROP) detection system 202 for determining a ROP category of an input eye image 204 of a subject 206. In an example, the ROP detection system 202 (referred to as system 202) includes a mobile phone, a tablet computer, or any other portable computing device. In an example, the portable computing device attached to the system 202 is capable of taking a retinal image of an eye of a subject. The input eye image 204 may be an image of an eye of a subject 206 undergoing screening for ROP diagnosis. In an example, the input eye image 204 is a retinal image. In an example, the system 202 may analyze a plurality of eye image attributes of the input eye image 204 based on the trained model pipeline 114.
[0040] Similar to system 102, system 202 may also include instructions 208 and analysis engine 210. In an example, instructions 208 are retrieved from a memory and executed by a processor included in system 202. Analysis engine 210 may be implemented as a combination of hardware and programming (e.g., programmable instructions) to implement various functions. In the examples described herein, this combination of hardware and programming may be implemented in several different ways. For example, the programming for analysis engine 210 may be executable instructions such as instructions 208. Such instructions 208 may be stored on a non-transitory machine-readable storage medium that may be directly or indirectly (e.g., by networking means) coupled to system 202. In an example, analysis engine 210 may include processing resources (e.g., a single processor or a combination of multiple processors) to execute such instructions. In this example, a non-transitory machine-readable storage medium may store instructions such as instructions 208 that implement analysis engine 210 when executed by a processing resource. In other examples, analysis engine 210 may be implemented as an electronic circuit.
[0041] In one example, the analysis engine 210 can utilize the trained model pipeline 114 to determine whether a ROP exists in the subject's eye based on processing the input eye image 204 of the subject 206. Figure 1A and Figure 1B The model pipeline 114 is trained in the manner of the method discussed. Also as mentioned above, the model pipeline 114 can further include the trained view rating model 122, the quality rating model 124, and the classification model 126.
[0042] The system 202 may also include (one or more) input eye images 212, view types 214, attribute information 216, detection results 218, and activation maps 220. It may be noted that the above data elements are generated by the analysis engine 210 using the model pipeline 114 and in response to the execution of the (one or more) instructions 208. These aspects and further details are discussed in the following paragraphs.
[0043] In operation, an input eye image, such as an input eye image 204 of an eye of a subject 206 who is undergoing screening for the presence of ROP, may be obtained. For example, the input eye image 204 may be taken by any image sensing subsystem that may be present within the system 202. In an example, the image sensing subsystem may be a retinal camera device mounted on the system 202 itself or may be removably integrated with the system 202. In another example, a set of input eye images may be obtained rather than having a single input eye image. Each image in the set of input eye images may correspond to various possible views of the eye image. The set of input eye images also corresponds to the subject eye who is undergoing screening for the presence of ROP.
[0044] Once the input eye image 204 is obtained, the analysis engine 210 can perform a number of pre-processing steps on the input eye image 204. Examples of such pre-processing steps include, but are not limited to, cropping, padding, resizing, and sharpening. The goal of these pre-processing steps is to make the input eye image 204 suitable for further processing stages and to remove unnecessary portions of the input eye image 204. For example, cropping is used to remove or adjust the outer borders or edges of the input eye image 204 to improve the framing or composition. Specifically, unnecessary portions of the input eye image 204 are removed via cropping. Similarly, other pre-processing steps are performed to improve the suitability of the input eye image 204 for further processing stages (e.g., view assessment).
[0045] Continuing further, the input eye image 204 is further processed to assess the view of the input eye image 204. In one example, the analysis engine 210 can utilize the trained view assessment model 122 of the model pipeline 114 to determine the view type of the input eye image 204, such as the view type 214. In an example, the trained view assessment model 122 assesses various features of the input eye image 204 to determine the view of the input eye image. Examples of various possible views of the input eye image 204 include, but are not limited to, a temporal view, a nasal view, a disc center view, a macular center view, an inferior view, and a superior view. Furthermore, in one example, in the case of a set of input eye images, the view type of each input eye image can be determined.
[0046] Once the view type 214 of the input eye image 204 is determined, if the determined view type 214 is a temporal view, the input eye image 204 may be processed by the analysis engine 210 using the ROP detection model pipeline. In an example, if the input eye image does not have a temporal view, the user or subject 206 may be prompted to take other input eye images 204 by displaying an indicator on the display of the system 202, or the user or subject 206 may choose to continue with the initially taken or obtained input eye image 204. In the case of obtaining a set of input eye images, an image having a temporal view is selected and designated as the input eye image 204 among the images included in the set of input eye images.
[0047] Continuing further, the input eye image 204 having the temporal view is subjected to a plurality of processing steps. For example, the analysis engine 210 performs a plurality of processing steps on the input eye image 204 to make it suitable for further processing stages. Also as described above, the plurality of processing steps include, but are not limited to, cropping, padding, resizing, and sharpening.
[0048] Once the input eye image 204 is processed, the input eye image 204 is processed using the trained model pipeline 114 to assess the quality of the input eye image 204. In one example, the analysis engine 210 can utilize the trained quality model 124 of the model pipeline 114 to determine the quality score of the input eye image 204. In an example, the quality score depicts the acceptance level of the input eye image. For example, images with higher quality scores are accepted, and images with lower quality scores are discarded. In an example, high quality images are preferred because they include clearer feature details.
[0049] Returning to the present example, once the quality score of the input eye image 204 is determined, if the determined quality score is greater than the threshold score, the input eye image 204 may be processed by the analysis engine 210 using the classification model to detect the presence of ROP in the subject eye. In the example, if the determined quality score is less than the threshold score, the user or subject 206 may be prompted to take additional input eye images 204 by displaying an indicator on the display of the system 202, or the user or subject 206 may choose to continue with the input eye image 204 initially taken or obtained. These two examples are complementary and therefore have no impact on the scope of the present subject matter. It will be appreciated that determining the quality of the input eye image 204 may rely on various features or attributes of the input eye image 204 as detected by the quality assessment model 124. It may be noted that in the example, without departing from the scope of the present subject matter, the user 206 may choose to continue subsequent processing based on the input eye image 204 without assessing the quality of the input eye image 204.
[0050] The input eye image 204 (once determined to be acceptable as appropriate) may be further processed by the analysis engine 210 using the trained model pipeline 114 to identify attribute information such as attribute information 216 of the input eye image 204. In one example, the analysis engine 210 may utilize the trained classification model 126 of the model pipeline 114 to identify the attribute information 216 of the input eye image 204. In an example, the attribute information 216 corresponds to a plurality of eye image attributes that are used to indicate, alone or in combination, the presence or absence of a ROP in the subject eye.
[0051] To this end, the analysis engine 210 may use the classification model 126 to identify one or more eye image attributes. Examples of eye image attributes include, but are not limited to, retinal vessel location, retinal vessel size, retinal vessel structure, presence of demarcation line, demarcation line location, demarcation line size, presence of ridges, location of ridges, size of ridges, an indicator indicating partial retinal detachment, an indicator indicating total retinal detachment, and the like.
[0052] Based on the attribute information 216 thus determined using the trained classification model 126, the analysis engine 210 may further process the attribute information 216 based on the classification model 126 to determine a detection result 218 for the input eye image 204 corresponding to the patient's eye. In an example, the detection result indicates the absence or presence of ROP within the input eye image 204 of the subject eye. In another example, in the case of multiple input eye images, the analysis engine 210 may determine the detection result 218 indicating the absence or presence of ROP in the patient's eye by considering all the input eye images as a whole. Based on the detection result 218, the analysis engine 210 classifies the input eye image 204 into one of a referral ROP category and a non-referral ROP category.
[0053] It may be noted that the test result 218 thus determined may be used to provide further referral for treatment or other intervention as needed. For example, the test result 218 may indicate a diagnosis of ROP. The test result 218 may indicate one of the following states: referral for ROP or not referral for ROP. Based on the state represented by the test result 218, appropriate action may be taken. Although described as being obtained by processing the above-mentioned example of eye image attributes included in the attribute information 216, without departing from the scope of the present subject matter, the detection of the presence of ROP may be performed by considering any other eye image attributes. Such examples will still fall within the scope of the present subject matter without limitation.
[0054] Once all the processing results based on the ROI location are obtained, the resulting category identified for the input eye image 204 can then be displayed on the display device of the system 202 to indicate the ROP category of the subject being screened, so as to implement further treatment steps for curing the disease. In the example, the detection results 218 are displayed per eye, per subject, or in a combination thereof. In addition, the analysis engine 210 can also generate an activation map 220 for display on the display device of the system 202. In the example, the activation map 220 depicts a significant location within the input eye image 204 of the subject 206, which causes the detection of the ROP within the input eye image 204. In addition, the displayed activation map can also be further used by a medical practitioner to identify the location causing the disease.
[0055] In another example, the system 202 may be connected via a network ( Figure 2 The network may be a private network or a public network, and may be implemented as a wired network, a wireless network, or a combination of wired and wireless networks, and may be connected to (eg Figure 1A) network 106 is similar. All of the above disclosed steps that can be performed by the analysis engine 210 of the system 202 can be implemented or performed on behalf of the system 202 by a central computing server to reduce the computing load on the edge of the network.
[0056] Figure 3 An example method 300 for training a ROP detection model pipeline according to an example of the present subject matter is shown. The order in which the above method is described is not intended to be construed as limiting, and some of the method blocks described may be combined in a different order, or even performed simultaneously, to implement the method or an alternative method.
[0057] In addition, the above method can be implemented in the form of suitable hardware, computer-readable instructions, or a combination thereof. The steps of such a method can be performed by a system under the instructions of machine-executable instructions stored on a non-transitory computer-readable medium, or by a dedicated hardware circuit, microcontroller, or logic circuit. For example, the method can be performed by a training system such as system 102. In an implementation, the method can be performed under an "as a service" delivery model in which the system 102 operated by the supplier receives the programmable code. In this article, some examples are also intended to cover non-transitory computer-readable media (e.g., digital data storage media) that are computer-readable and encode computer-executable instructions, in which the instructions perform some or all of the steps of the above method.
[0058] In an example, a method 300 for training several deep learning models based on a training dataset, such as the training dataset 108, may be implemented by the system 102. At block 302, a training dataset for training a ROP detection model pipeline may be obtained. For example, the system 102 may obtain the training dataset 108. The training dataset 108 may be obtained through a repository, such as the repository 104. In an example, the training dataset 108 may include (one or more) training eye images (stored in the system 102 as (one or more) training eye images 116), (one or more) training eye image attributes (stored in the system 102 as (one or more) training eye image attributes 118), and ROP categories 120 to be used for training different models in the model pipeline 114. In an example, the model pipeline 114 may include a view assessment model 122, a quality assessment model 124, and a classification model 126.
[0059] At block 304, the view assessment model present in the ROP detection model pipeline may be trained. In one example, the training engine 112 may train the view assessment model 122 using the training eye image(s) 116, wherein the training eye image(s) 116 may include images with different views such as temporal view, nasal view, disc center view, macula center view, inferior view, and superior view, etc. The view assessment model 122 is thereby trained to determine the view of the input eye image.
[0060] At block 306, a quality assessment model of the ROP detection model pipeline may be trained. For example, the training engine 112 of the system 102 may train the quality assessment model 124 based on the (one or more) training eye images 116, which may include images with higher resolution, contrast, clarity, or other such attributes. The quality assessment model 124 is trained to assess the quality of the input eye images so that images with low quality may be discarded and only good quality images with higher quality are considered for further processing.
[0061] At block 308, the classification model of the ROP detection model pipeline may be trained. For example, the training engine 112 of the system 102 may train a classification model 126 based on the training eye image(s) 116, the classification model 126 identifying attribute information corresponding to a plurality of eye image attributes within the training eye image(s) 116. The classification model 126 may also be trained on training eye images associated with ROP and images not associated with ROP that are part of the training eye image(s) 116. There is a category indicator, such as a ROP category 120, associated with each of the training eye images 116 to indicate a state of the corresponding training eye image.
[0062] In an example, the classification model includes two sub-models, namely, a binary classification model trained to detect the presence or absence of ROP and a classification classification model trained to classify eye images at various stages (e.g., healthy eyes, early ROP, mid-ROP, and late ROP). In an example, these sub-models are trained sequentially or simultaneously as needed. In an example, the classification classification model can be used only during training to supplement the accuracy of the binary classification model in detecting ROP.
[0063] In an example, once trained, the model pipeline 114 can be used to classify an input eye image as one of a non-referral ROP and a referral ROP. Figure 4A and Figure 4BThe method steps involved in classifying an input eye image as one of a referred ROP and a not referred ROP are further described.
[0064] Figure 4A and Figure 4B An example method 400 is shown for classifying an input image into one of a referral ROP category and a not referral ROP category. Figure 3 Similarly, the order in which the above method is described is not intended to be construed as limiting, and some of the method blocks described may be combined in a different order to implement the method or an alternative method. Based on the present method as described in the context of the example method 400, eye image attributes of an input eye image are analyzed based on the trained model pipeline 114.
[0065] In addition, the above method 400 can be implemented in suitable hardware, computer-readable instructions, or a combination thereof. The steps of such a method can be performed by a system under the instructions of machine-executable instructions stored on a non-transitory computer-readable medium, or by a dedicated hardware circuit, a microcontroller, or a logic circuit. For example, the method can be performed by a ROP detection system such as system 202. In an implementation, the method can be performed under an "as a service" delivery model in which the system 202 operated by the provider receives the programmable code. In this document, some examples are also intended to cover non-transitory computer-readable media (e.g., digital data storage media) that are computer-readable and encode computer-executable instructions, in which the instructions perform some or all of the steps of the above method.
[0066] At block 402, an input eye image is obtained. For example, the input eye image 204 may be taken by any image sensing subsystem that may be present within the system 202. In an example, the image sensing subsystem may be a retinal camera device mounted on the system 202 itself or may be removably integrated with the system 202. In another example, a set of input eye images may be obtained rather than having a single input eye image. Each image in the set of input eye images may correspond to various possible views of an eye image. The set of input eye images also corresponds to a subject eye that is undergoing screening for detecting the presence of ROP. In another example, a database repository ( Figure 2 ) obtains the input eye image 204.
[0067] At block 404, a plurality of pre-processing steps may be performed on the input eye image thus obtained. For example, the analysis engine 210 may perform a plurality of pre-processing steps on the input eye image 204. Examples of such pre-processing steps include, but are not limited to, cropping, padding, resizing, and sharpening. The goal of these pre-processing steps is to make the input eye image 204 suitable for further processing stages and to remove unnecessary portions of the input eye image 204. For example, cropping is used to remove or adjust the outer edges of the input eye image 204 to improve the frame or composition. Specifically, unnecessary portions of the input eye image 204 are removed via cropping. Similarly, other pre-processing steps are performed to improve the adaptability of the input eye image 204 to further processing stages (e.g., view assessment).
[0068] At block 406, the view type of the input eye image is determined by processing the input eye image. For example, the analysis engine 210 may utilize the trained view assessment model 122 of the model pipeline 114 to determine the view type of the input eye image 204, such as the view type 214. In an example, the trained view assessment model 122 assesses various features of the input eye image 204 to determine the view of the input eye image. Examples of various possible views of the input eye image 204 include, but are not limited to, a temporal view, a nasal view, a disc center view, a macular center view, an inferior view, and a superior view.
[0069] At block 408, a determination may be made to determine whether the view type of the input eye image is temporal. For example, if the determined view type 214 is a temporal view, the input eye image 204 may be processed by the analysis engine 210 using the ROP detection model pipeline as will be described in a later step (from the "yes" path of block 408). However, if the input eye image 20 does not have a temporal view, the user or subject 206 may be prompted to capture or obtain other input eye images 204 by displaying an indicator on a display of the system 202 (from the "no" path of block 408), or the user or subject 206 may choose to continue with the initially captured or obtained input eye image 204. In the case of obtaining a set of input eye images, an image having a temporal view is selected and designated as the input eye image 204 among the images included in the set of input eye images.
[0070] At block 410, a plurality of processing steps may be performed on the input eye image. For example, the analysis engine 210 performs a plurality of processing steps on the input eye image 204 to make it suitable for further processing stages. Also as described above, the plurality of processing steps include, but are not limited to, cropping, padding, resizing, and sharpening.
[0071] At block 412, the input eye image may be further processed based on the quality assessment model to determine the acceptability of the input eye image. For example, the analysis engine 210 may utilize the trained quality assessment model 124 of the model pipeline 114 to determine the quality score of the input eye image 204. In an example, the quality score depicts the acceptance level of the input eye image. For example, images with higher quality scores are accepted, and images with lower quality scores are discarded. In an example, high quality images are preferred because they include clearer feature details.
[0072] At block 414, a determination is made as to whether the determined quality score of the input eye image is acceptable. For example, if the determined quality score is greater than a threshold score, the input eye image 204 may be processed by the analysis engine 210 using a classification model to detect the presence of ROP in the subject eye (from the "yes" path of block 414). However, if the determined quality score is less than the threshold score, the user or subject 206 may be prompted to capture or obtain additional input eye images 204 by displaying an indicator on a display of the system 202 (from the "no" path of block 414), or the user or subject 206 may choose to continue using the initially captured or obtained input eye image 204. These two examples are complementary and therefore have no impact on the scope of the present subject matter. It will be appreciated that determining the quality of the input eye image 204 may rely on various features or attributes of the input eye image 204 as detected by the quality assessment model 124. It may be noted that in the examples, the user 206 may choose to proceed with subsequent processing based on the input eye image 204 without assessing the quality of the input eye image 204 without departing from the scope of the present subject matter.
[0073] At block 416, the input eye image may be further processed based on the classification model to identify attribute information of the input eye image. For example, the analysis engine 210 may utilize the trained classification model 126 of the model pipeline 114 to identify attribute information 216 of the input eye image 204. In an example, the attribute information 216 corresponds to a plurality of eye image attributes that are used to indicate the presence or absence of ROP in the subject eye, either individually or in combination.
[0074] To this end, the analysis engine 210 may use the classification model 126 to identify one or more eye image attributes. Examples of eye image attributes include, but are not limited to, retinal vessel location, retinal vessel size, retinal vessel structure, presence of demarcation line, demarcation line location, demarcation line size, presence of ridges, location of ridges, size of ridges, an indicator indicating partial retinal detachment, an indicator indicating total retinal detachment, and the like.
[0075] At block 418, the determined attribute information may be further processed based on the classification model to determine a detection result indicating the presence or absence of ROP in the subject eye. For example, the analysis engine 210 may further process the attribute information 216 based on the classification model 126 to determine a detection result 218 for the input eye image 204. In the example, the detection result indicates the absence or presence of ROP within the input eye image 204 of the subject eye. Based on the detection result 218, the analysis engine 210 classifies the input eye image 204 into one of a referral ROP category and a non-referral ROP category.
[0076] It may be noted that the test result 218 thus determined may be used to provide further referral for treatment or other intervention as needed. For example, the test result 218 may indicate a diagnosis of ROP. The test result 218 may indicate one of the following states: referral for ROP or not referral for ROP. Based on the state represented by the test result 218, appropriate action may be taken. Although described as being obtained by processing the above-mentioned example of eye image attributes included in the attribute information 216, without departing from the scope of the present subject matter, the detection of the presence of ROP may be performed by considering any other eye image attributes. Such examples will still fall within the scope of the present subject matter without limitation.
[0077] Once all the processing results based on the ROI location are obtained, the resulting category identified for the input eye image 204 can then be displayed on the display device of the system 202 to indicate the ROP category of the subject 206 under screening so that further treatment steps for curing the disease can be implemented. In an example, the detection results 218 are displayed per eye, per subject, or in a combination thereof.
[0078] In addition, the analysis engine 210 may also generate an activation map 220 for display on a display device of the system 202. In an example, the activation map 220 depicts significant locations within the input eye image 204 that trigger the detection of ROP within the input eye image 204 of the subject 206. In addition, the displayed activation map may also be further used by a medical practitioner to identify the location and extent of the disease.
[0079] Although examples of the present disclosure have been described in language specific to structural features and / or methods, it should be understood that the appended claims are not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed and described as examples of the present disclosure.
Claims
1. A system, include: processor; as well as an analysis engine coupled to the processor, wherein the analysis engine is configured to: obtaining an input ocular image, wherein the input ocular image corresponds to a subject eye being evaluated for the presence of retinopathy of prematurity (ROP); Using a ROP detection model pipeline, wherein the ROP detection model pipeline is trained based on a training data set, the training data set comprising a training image associated with the ROP and training attribute information corresponding to a plurality of training eye image attributes, wherein the ROP detection model pipeline is used to: determining a view type of the input eye image; In case the view type is determined to be a temporal view, performing a plurality of processing steps on the input eye image to make the input eye image suitable for further processing; identifying attribute information from the input eye image, wherein the attribute information corresponds to a plurality of input eye image attributes; and Based on the attribute information, a detection result indicating the presence of a ROP in the subject eye is determined.
2. The system according to claim 1, in, The analysis engine is used to: A plurality of pre-processing steps are performed on the input eye image to make the input eye image suitable for view assessment, wherein the plurality of pre-processing steps include cropping, padding, resizing, and sharpening the edge of the input eye image.
3. The system according to claim 1, in, The plurality of processing steps include cropping, padding, resizing, and sharpening the edges of the input eye image.
4. The system according to claim 1, in, The analysis engine is configured to use the ROP detection model pipeline to: In case the view type of the input eye image is not a temporal view, the input eye image is discarded.
5. The system according to claim 1, in, The analysis engine is used to: obtaining a set of input eye images, wherein each image in the set of input eye images corresponds to a different view of a user's eye, wherein the views include a temporal view, a nasal view, a disc center view, a macular center view, an inferior view, and a superior view; Use the ROP detection model pipeline to: determining a view of each image in the set of input eye images; Based on the determined view, an image having a temporal view is selected from the set of input-eye images to be designated as the input-eye image.
6. The system according to claim 1, in, The ROP detection model pipeline includes a plurality of deep learning models selected from the group consisting of a view assessment model, a quality assessment model, and a classification model.
7. The system according to claim 1, in, The analysis engine is configured to use the ROP detection model pipeline to: Assessing the quality of the input eye image to generate a quality score; If it is determined that the quality score is greater than a threshold quality score, extracting the attribute information from the input eye image; or If it is determined that the quality score is less than a threshold quality score, discarding the input eye image; as well as Prompt the user to obtain or take a new input eye image.
8. The system according to claim 1, in, The analysis engine is also used to: An activation map is generated that depicts salient locations within the input eye image, the activation map causing detection of a ROP within the input eye image of the subject eye.
9. The system according to claim 1, in, The multiple input eye image attributes include retinal vessel location, retinal vessel size, retinal vessel structure, the presence of dividing lines, the location of dividing lines, the size of dividing lines, the presence of ridges, the location of ridges, the size of ridges, indicators indicating partial retinal detachment, and indicators indicating complete retinal detachment, etc.
10. A method, include: obtaining training information, the training information comprising a training eye image and training attribute information corresponding to a plurality of eye image characteristics, wherein the training eye image is associated with retinopathy of prematurity (ROP); and The ROP detection model pipeline is trained based on training information including a training eye image, training attribute information, and a ROP category of the training eye image, wherein the training attribute information corresponds to a plurality of training eye image attributes.
11. The method according to claim 10, in, The multiple training eye image attributes include retinal vessel location, retinal vessel size, retinal vessel structure, presence of dividing lines, dividing line location, dividing line size, presence of ridges, ridge location, ridge size, indicators indicating partial retinal detachment, and indicators indicating complete retinal detachment, etc.
12. The method according to claim 10, in, The ROP detection model pipeline trained based on the training eye images and corresponding training attribute information is used to determine a detection result indicative of the presence of a ROP within an input eye image of a subject eye under evaluation.
13. The method according to claim 10, in, The ROP detection model pipeline includes a plurality of deep learning models selected from the group consisting of a view assessment model, a quality assessment model, and a classification model.
14. The method according to claim 10, in, The trained ROP detection model pipeline is used to assess the quality of the input eye images to discard low-quality images.
15. The method according to claim 10, in, The trained ROP detection model pipeline is used to determine the view type of the input eye image to only accept temporal view images, wherein the input eye image has one of a temporal view, a nasal view, a disc center view, a macular center view, an inferior view, and a superior view.
Citation Information
Cited By
Premature infant retinopathy onset risk prediction method and system
CN120072319A
