BERT-style self-supervised learning convolutional neural network for disease identification from retinal images

Abstract

Systems and methods for identifying disease from a plurality of retinal images are disclosed. Systems and methods may include training a computerized image processing system comprising a CNN on a set of original unlabeled retinal images. Training the computerized image processing system may include randomly masking each retinal image in the set of original unlabeled retinal images, generating feature maps with different resolutions from each masked retinal image based on a downsampling process of the CNN, masking the feature maps adapted to different resolutions for an encoder and a decoder of the CNN, performing sparse convolution on each masked retinal image, and processing each masked retinal image through the decoder to form a set of reconstructed images. Using the trained image processing system, the system can receive an unidentified retinal image and generate a likelihood that the person associated with the unidentified retinal image is an Alzheimer's disease patient.

Description

TITLE: BERT-STYLE SELF-SUPERVISED LEARNING CONVOLUTIONAL NEURAL NETWORK FOR DISEASE IDENTIFICATION FROM RETINAL IMAGESINVENTORS: YALIN WANGXIN LIWENHUI ZHUOANA DUMITRASCUCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 655,242 filed June 3, 2024, entitled “BERT-STYLE SELF-SUPERVISED LEARNING CONVOLUTIONAL NEURAL NETWORK FOR DISEASE IDENTIFICATION FROM RETINAL IMAGES.” The disclosure of the foregoing application is incorporated herein by reference in its entirety, including but not limited to those portions that specifically appear hereinafter, but except for any subject matter disclaimers or disavowals, and except to the extent that the incorporated material is inconsistent with the express disclosure herein, in which case the language in this disclosure shall control.TECHNICAL FIELD

[0002] The present disclosure relates to imaging, and in particular to techniques for identifying disease using convolutional neural networks.BACKGROUND

[0003] Retinal fundus photography is used to diagnose various diseases. Postmortem histopathologic investigations of eyes and brains from patients with Alzheimer’s disease (AD) and clinical evaluations of patients with AD have revealed pathologic alterations in the neurosensory retina that precede and correlate with brain AD changes. Retina accumulates certain proteins and exhibits neurodegeneration, vascular amyloidosis, and increased inflammation. Retinal changes can be captured via color fundus photography, a widely accessible technology in eye care, primary care, and under resourced community settings, carrying the promise of a noninvasive and cost-effective biomarker for AD. Machine learning tools have been developed to overcome the subjectivity and low efficiency associated with manual retinal photographs analysis for disease and biomarker identification and to automate their interpretation in multiple ocular disorders. However, existing machine learning tools for this purpose often rely on large numbers of images, expert labeling, and are computationally inefficient. Accordingly, improved approaches remain desirable.SUMMARY

[0004] In an aspect, a method for identifying disease from a plurality of retinal images using a convolutional neural network (CNN) is disclosed. The method includes training a computerized image processing system comprising a CNN on a set of original unlabeled retinal images from a Biobank comprising more than 170,000 retinal fundus images. The training includes randomly masking each retinal image in the set of original unlabeled retinal images, generating feature maps with different resolutions from each masked retinal image based on a downsampling process of the CNN, masking the feature maps adapted to different resolutions for an encoder and a decoder of the CNN, the decoder comprising a U-Net decoder characterized by successive blocks with upsampling layers, performing sparse convolution on each masked retinal image, and processing each masked retinal image through the decoder to form a set of reconstructed images.

[0005] The method can further include receiving an unidentified retinal image for identifying Alzheimer's disease in a person associated with the unidentified retinal image.

[0006] The method can further include extracting, using the trained computerized image processing system, features from the unidentified retinal image to generate detailed vessel structure and binary vessel segmentation.

[0007] The method can further include inputting the detailed vessel structure and binary vessel segmentation into a U-Net encoder to extract disease diagnosis information.

[0008] The method can further include generating a likelihood that the person associated with the unidentified retinal image is an Alzheimer’s disease patient by inputting the disease diagnosis information into a new linear classifier, a fully connected layer, and a Softmax function.

[0009] The decoder can be a lightweight U-Net decoder characterized by the inclusion of four successive blocks B1-B4 with upsampling layers.

[0010] The sparse convolution can improve computational consistency by omitting all the empty voxels and focusing convolution operations on non-zero elements.

[0011] The feature maps Di can be:

[0012]

[0013] The feature maps can be created via five downsampling operations.

[0014] The feature maps set S can be:

[0015] (7) -

[0016] The method may further include applying a projection layer ‘ 'rto ensure dimensional consistency between the encoder and the decoder.

[0017] The method may further include generating a loss function by comparing the mean squared error between the masked portion of each reconstructed image and the corresponding masked region of the corresponding original image.

[0018] The new linear classifier can predict when the individual associated with the unidentified retinal image is an Alzheimer’s disease or control patient.

[0019] The fully connected layer can be a neural network layer in the CNN in which every neuron is connected to all activation units from the preceding layer, resides at a network’s end, and maps learned nonlinear features to a sample’s output space.

[0020] The Softmax function can transform a real-value vector into a probability distribution where each element's value is between 0 and 1, and the sum of all elements equals 1 and can output a probability distribution over various classes, representing the likelihood that the sample belongs to each class.

[0021] In another aspect, a computerized image processing method is disclosed. The method includes training a computerized image processing system comprising a CNN on a set of original unlabeled retinal images.

[0022] The training can include randomly masking each retinal image in the set of original unlabeled retinal images, generating feature maps with different resolutions from each masked retinal image based on a downsampling process of the CNN, masking the feature maps adapted to different resolutions for an encoder and a decoder of the CNN, the decoder comprising a U- Net decoder characterized by successive blocks with upsampling layers, performing sparse convolution on each masked retinal image, and processing each masked retinal image through the decoder to form a set of reconstructed images.

[0023] It is to be understood that any respective features / examples of each of the aspects of the disclosure as described herein may be implemented together in any appropriate combination, and that any features / examples from any one or more of these aspects may be implemented together with any of the features of the other aspect(s) as described herein in any appropriate combination to achieve the benefits as described.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] With reference to the following description and accompanying drawings:

[0025] FIG. 1 illustrates an image processing problem to be solved by an exemplary method for image processing, in accordance with an exemplary embodiment;

[0026] FIG. 2 illustrates an architecture for exemplary system and method for image processing, in accordance with an exemplary embodiment;

[0027] FIG. 3 illustrates a pre-training workflow in connection with an exemplary system and method for image processing, in accordance with an exemplary embodiment;

[0028] FIG. 4 illustrates a quality control pipeline in connection with an exemplary system and method for image processing, in accordance with an exemplary' embodiment;

[0029] FIG. 5 illustrates heat maps for Alzheimer’s disease (first row) and Parkinson’s disease (second row), demonstrating the efficacy of an exemplary method, in accordance with an exemplary' embodiment;

[0030] FIG. 6 illustrates a pipeline for a U-Net-based architecture that uses retinal vessel segmentation for image processing, in accordance w ith an exemplary embodiment:

[0031] FIG. 7 illustrates a pipeline for a bidirectional encoder-style self-supervised learning convolutional neural network (CNN) pretrained on retinal color photographs for image processing, in accordance with an exemplary embodiment;

[0032] FIG. 8 schematically illustrates a computer control system or platform programmed or otherwise configured to implement the methods provided herein, in accordance with an exemplary embodiment; and

[0033] FIG. 9 is a flow chart for a method for image processing, in accordance with an exemplary embodiment.DETAILED DESCRIPTION

[0034] The following description is of various exemplary embodiments only, and is not intended to limit the scope, applicability7or configuration of the present disclosure in any way. Rather, the following description is intended to provide a convenient illustration for implementing various embodiments including the best mode. As will become apparent, various changes may be made in the function and arrangement of the elements described in these embodiments without departing from principles of the present disclosure.

[0035] For the sake of brevity7, conventional techniques and components for mathematical processes, transforms, image manipulation, and / or the like may not be described in detail herein. Furthermore, the connecting lines shown in various figures contained herein areintended to represent exemplary' functional relationships and / or physical couplings between various elements. It should be noted that many alternative or additional functional relationships or physical connections may be present in exemplary methods and systems for imaging and / or components thereof.

[0036] Exemplary embodiments may be operative on and / or utilize computing resources of sufficient capability, for example 4 NVIDIA Al 00 80GB GPUs with an AMD EPYC 7413 24-Core Processor. Various exemplary embodiments utilize software requirements in the nature of: Python 3.6 or higher, Pytorch 1.3 or higher, and Linux distribution kernel version higher than 3.10 or Windows 7 or higher. Moreover, various exemplary embodiments utilize hardware requirements in the nature of: Intel core i3 2.0GHz processor, 4GB RAM, and CUDA version 10.0 or higher. Exemplary embodiments also support ARM-architecture mobile devices, with a minimum requirement of Cortex-A 53 or Apple A7, etc. of similar performance architecture. It will be appreciated that appropriate hardware depends at least in part on the size of the dataset for the downstream task if exemplary' weights are used for training the downstream task. Moreover, the backbone network of an exemplary pre-training model is modified based on MobileNetV2 under the convolutional neural network (CNN) architecture, so the model parameters and computational efficiency are the same as MobileNetV2. How ever, compared with a network based on a Vision Transformers (ViTs) architecture, the exemplary CNN network has a smaller number of parameters, so it has higher computational efficiency.

[0037] With the rise of artificial intelligence, the field of ophthalmic medicine is gradually benefiting from this technology. CNNs have achieved remarkable results in the assisted diagnosis of ophthalmic diseases. However, CNNs in medical imaging have faced challenges such as limited labeling and small dataset sizes. In recent years, self-supervised learning methods based on the vision Transformer (also referred to herein as “ViT”) have brought new ideas to solve the problems in the imaging field. However, the self-attention mechanism of these methods suffers from high computational complexity, and the dependence on large-scale datasets limits their application in practice. Accordingly, improved approaches are highly desirable.

[0038] With reference now to FIG. 1 through FIG. 7, in accordance with various exemplary embodiments, to overcome these challenges exemplary embodiments of the system described herein can combine CNN with generative self-supervised learning. By utilizing a large amount of unlabeled data, the present system can pre-train CNN networks and thus improve network performance. This approach not only makes full use of the available data resources but also effectively improves the accuracy and robustness of the diagnostic model.

[0039] In an exemplary7embodiment, a BERT-style self-supervised learning strategy7is disclosed and implemented through a lightweight CNN architecture, nn-MobileNet. This method fully leverages the advantages of CNNs in medical image processing and incorporates the pre-training mechanism of self-supervised learning, aiming to efficiently process large- scale unlabeled medical image datasets. The performance of the model in data representation and feature extraction is significantly enhanced by employing a sparse convolution technique to process the masked regions of the image, while preserving the original hierarchical structure of the CNN. To validate the effectiveness of the exemplary method, the system described herein can conduct pre-training on existing datasets. For example, the system may use the UK Biobank dataset, which comprises more than 170,000 retinal fundus images. Subsequently, the system can apply the pre-trained model to the classification tasks of Alzheimer’s and Parkinson’s diseases. The present model using this method can achieve improved performance on these downstream tasks.

[0040] In existing methods, convolution is widely used in 2D computer vision research, where sliding window operations are usually performed on a grid of pixels from an image. However, it is inappropriate in masked autoencoder (MAE) scenarios, because masked pixels might participate in the convolutions and lead to inconsistent results. Sparse convolution improves the computation consistency by omitting all the empty7voxels and focusing convolution operations on non-zero elements. The present system extends the sparse convolution to the self-supervised research by ensuring that convolution operations are conducted exclusively on non-masked pixels.

[0041] By adopting sparse convolution, the system described herein may7include a Bidirectional Encoder Representations from Transformers (BERT)-style self-supervised learning CNN to enrich self-supervised approaches. The BERT-style self-supervised learning CNN is applicable across a broad spectrum of medical imaging research. The exemplary method integrates with a CNN model, capitalizing on its strengths, such as precise localization and low7er data requirements.

[0042] The system may use nn-MobileNet as the core CNN architecture for the selfsupervised learning strategy. The architecture is derived from Mobile NetV2. with several key adjustments. These adjustments include modifications to the original channel configurations, the adoption of new nonlinear activation functions, and the replacement of the optimizer, as described in further detail below7. Additionally, through the introduction of numerous data enhancement techniques and the moderate use of the dropout module, the architecture achieves improved performance in various retinal disease detection tasks.

[0043] Before analyzing new retinal images, the present system may first perform pretraining on the model. In the pre-training stage of an exemplary' embodiment, the system may train on a large number of retinal fundus images (e.g., from UK Biobank) for self-supervised pre-training. The images may cover a wide range of age groups, ethnicities, and multiple diseases, including Alzheimer’s disease, Parkinson’s disease, and retina-related diseases. The model can effectively leam the representational features of retinal fundus images by pretraining the method with self-supervised learning on a large retinal image dataset.

[0044] The model may include an encoder and a decoder. Through this pre-training process, the encoder can successfully leam the representational features of the input images. After the pre-training is completed, the pre-trained encoder can be applied to downstream tasks. Specifically, when dealing with classification tasks such as Alzheimer's disease and Parkinson’s disease, the system can attach a classifier to the output of the encoder and specify the corresponding number of categories for classification. In addition to this, the encoder obtained from pre-training can also be widely used for other types of tasks, such as image segmentation and object detection. This work not only demonstrates the effectiveness of selfsupervised pre-training in improving the model’s understanding of medical images but also provides support for applying deep learning techniques to other areas of medical image analysis.

[0045] Similarly, pre-trained model weights can be used to build encoders for specific task requirements. The model weights may include parameters that represent how strongly certain aspects in each layer of the CNN affect the output. For example, a healthcare organization can utilize a small-sized labeled dataset it has collected itself, in conjunction with the pre-trained model weights, for the tasks of classifying retina-related diseases. In addition, the task of segmenting different regions of a retinal image can also be achieved by combining the encoders. The system can provide for specific decoders. Furthermore, because the exemplary method is based on a lightweight network, medical device companies can use the model on micro-devices for pathology detection of retinal images, among other applications. The pretraining method not only provides an effective pre-training strategy for research in medical image analysis but also opens new possibilities for specific applications in medical practice.

[0046] In contrast to prior approaches, in various embodiments: (1) The application of selfsupervised learning has brought new- breakthroughs to the field of medical imaging, and model performance can be improved even on many small-scale datasets. (2) The present system can implement the generative self-supervised learning method on the CNN framework. This method can combine the advantages of self-supervised learning and CNNs in the field ofmedical imaging. (3) The present system can conduct pre-training on large-scale unlabeled datasets, comprising, for example, more than 170,000 images, and subsequently achieve improved results in downstream tasks, specifically in classifying Alzheimer’s and Parkinson’s diseases.

[0047] Moreover, as compared to prior approaches, exemplary' embodiments have the following advantages compared to current alternatives: (1) By adopting the sparse convolution, the present system can utilize a BERT-style self-supervised learning CNN to enrich selfsupervised approaches. It is universally applicable across a broad spectrum of medical imaging research. (2) Compared with visual transformers (ViTs), the self-supervised learning method based on a CNN can provide advantages in capturing detailed features of images, such as edges and textures. (3) The present system can utilize a self-supervised learning method based on CNN for pre-training. Compared with most current self-supervised learning methods based on ViTs, the present system demonstrates advantages in terms of data requirements and training efficiency.

[0048] Image Processing Problem Regarding Masking

[0049] FIG. 1 illustrates an image processing problem to be solved by an exemplary method for image processing, in accordance with an exemplary embodiment. In FIG. 1, a ViT is shown on the left panel. The ViT can process non-masked patches without any changes since it can process variable-length sequences of patches. On the right panel, a CNN is shown. In contrast to the ViT, the CNN on the right panel cannot skip masks for convolutions. Simply adopting masking in CNN may lead to performance degradation. The system described herein adopts a solution to address this problem, in particular by adopting sparse convolution.

[0050] Inverted Linear Residual Bottleneck (ILRB) for nn-MobileNet Architecture

[0051] FIG. 2 illustrates an architecture for an exemplary system and method for image processing, in accordance with an exemplary embodiment. The image processing may be performed by the CNN model nn-MobileNet, as shown in FIG. 2. FIG. 2 shows the detailed architecture of the nn-MobileNet and its ILRB design. The current self-supervised learning scheme can enhance nn-MobileNet’s performance. Based on the MobileNetV2, for example, the present system may make the following architectural innovations.

[0052] First, the nn-MobileNet can modify the order of channel configurations for the ILRB in the network. This strategy aims to improve network performance by leveraging the significant impact of channel configurations.

[0053] The configuration and ordering of channels within inverted residual bottleneck (IRB) blocks are explicitly arranged to balance spatial detail extraction and semanticabstraction. The specific block arrangement, including how channels are distributed across different layers, is illustrated in the right half of FIG. 2, which reflects the proposed channel configuration strategy.

[0054] In the nn-MobileNet architecture based on MobileNetV2, the system can modify the configuration and ordering of channels within inverted residual blocks. This adjustment improves performance by aligning computational capacity with the semantic complexify across network layers. Specifically, fewer but more spatially focused channels are allocated in early layers, while deeper layers are assigned more channels to extract high-level semantic features. This redistribution reduces computational redundancy and enhances both representational efficiency and generalization.

[0055] Second, compared with conventional retinal imaging methods, the nn-MobileNet may employ a heavy data augmentation strategy, including image cropping, flipping, contrast adjustment, brightness adjustment, and sharpening of the images. This heavy data enhancement may have significant benefits for improving system performance by refining the images to the portions important to the image analysis for disease detection.

[0056] In some cases, the model may be susceptible to "overfilling” where the model may become overtrained based on its training data and thus, lead to poor performance. In order to solve the overfitting problem, the nn-MobileNet may attempt to add Dropout modules at various locations within the network to identify' the optimal placement for them as shown in FIG. 2. Dropout modules may deactivate some of the neurons in each layer, which can combat overfitting.

[0057] Furthermore, according to prior studies, smooth variants of a rectified linear unit (ReLU) activation functions can improve the performance, as show n in the ILRB block of FIG. 2. In some aspects, through experimental comparisons, ReLU6 may be used as the activation function for ILRB in the framework due to its superior performance enhancement.

[0058] Pre-Training of the CNN

[0059] FIG. 3 illustrates a pre-training workflow in connection with an exemplars' system and method for image processing, in accordance with an exemplary embodiment. FIG. 3 illustrates the workflow of the pre-training strategy, which draws on the basic principles of the BERT architecture. The system can employ image masking techniques (Step 1) and match hierarchical feature maps for masking areas (Step 2) to construct the loss functions. The system can use sparse convolution to integrate the CNN architecture (Step 3). Finally, the learned feature maps are used for downstream applications. Details of this pre-training method are as follows.

[0060] In the field of computer vision, the masked autoencoder (MAE), similar to the BERT model from the natural language processing (NLP) field, has become one of the most advanced pre-training strategies for self-supervised learning. However, the research based on MAE methods has been mainly based on ViTs. Due to their inherent differences, the encoderdecoder structure of ViTs cannot be directly substituted by CNNs until recently with the emergence of adopting CNNs for the pre-training research with the BERT-style self-supervised learning approach.

[0061] Hierarchy is widely recognized as the gold standard for vision representation systems. The hierarchical design implemented in various CNNs have significantly improved their performance. However, this principle is not utilized in ViTs. In the present disclosure, the system can adopt a classifier, such as SparK, to maintain the hierarchy inherent in CNNs and ensure that the neural network can utilize the hierarchical structure for improved representation learning.

[0062] As shown in FIG. 3, the system can begin by masking all the images from the large dataset randomly (Step 1). In one example, the size of each mask may be equal to the (mask’s height [H] adjusted by the downsampling ratio [D], mask’s width [W] adjusted by the downsampling ratio [D] of the network) (HD, WD). Then, the system can process only the unmasked visible regions of the input image. As an example, the size of the output image may be (224, 224), and the down-sampling ratio w as 32, making the mask size (7, 7).

[0063] Next, the system can generate feature maps and mask the feature maps adapted to different resolutions for the CNN encoder and decoder (Step 2). The system can use the feature maps learned from the pretraining module for classification studies. In some aspects, for quality control, 2 trained graders (A.Y. and S.S.) can exclude the retinal images exhibiting blur, low contrast, poor illumination, or artifacts.

[0064] In the present system, feature maps can be created by applying successive convolution and downsampling operations to the input image. At each downsampling stage, the spatial resolution is reduced, producing a hierarchical set of feature maps that capture progressively higher-level semantic information. Due to the use of a masked input strategy, standard dense convolution is insufficient, as it would process invalid or occluded regions. Instead, sparse convolution is employed to selectively operate only on visible, unmasked pixels, thereby improving computational efficiency and representation fidelity. These feature maps serve as intermediate representations in the decoder pathway, where they are incrementally upsampled and fused to reconstruct the final output image at the original resolution.

[0065] Finally, the system can perform sparse convolution on the masked image and restore the image through the decoder (Step 3). Using the encoder, before performing the convolution operation, the system can generate feature maps with different resolutions based on the downsampling process of the CNN. For the neural network model nn-MobileNet, a total of five downsamplings may be performed on an image with a size of H * W, where H,W are the height and width of the input image, respectively, resulting in the generation of the size of feature maps set S.

[0066] In an exemplary embodiment, a computerized image processing method, comprises training an image processing system comprising a convolutional neural network (CNN) on a set of original images. The training comprises: randomly masking each original image in the set of original images; masking feature maps adapted to different resolutions for an encoder and a decoder of the CNN; performing sparse convolution on each masked original image; and processing each masked original image through the decoder to form a set of reconstructed images.

[0067] The decoder may comprise a lightweight U-Net decoder characterized by the inclusion of, for example, four successive blocks B1-B4 with upsampling layers. The sparse convolution may improve computational consistency by omitting all the empty voxels and focusing convolution operations on non-zero elements. The feature maps Di may comprise:

[0068]

[0069] The feature maps may be created via five downsampling operations. The feature maps set S may comprise:

[0071] The method may further comprise applying a projection layerensure dimensional consistency between the encoder and the decoder. The method may further comprise generating a loss function by comparing the mean squared error between the masked portion of each reconstructed image and the corresponding masked region of the corresponding original image.

[0072] The system’s loss function utilizes a method of comparing the mean squared error (MSE) between the masked portion of the reconstructed images and the corresponding masked region in the original images. The loss function is applied during the model training phase to supervise the reconstruction process. Specifically, it calculates the mean squared error (MSE)between the masked regions of the reconstructed image and the corresponding masked regions in the original image. This loss is computed exclusively on the masked pixels, ensuring that the model focuses on accurately predicting the occluded content. The resulting loss value is then used in the backpropagation process to update the weights of both the encoder and decoder. By minimizing this loss, the network leams to generate reconstructions that are structurally consistent with the original image, particularly in the previously missing areas.

[0073] As an example of the method described above, the system may pre-train a BERT- style self-supervised CNN combined with the previously reported nn-MobileNet CNN on 178,803 unlabeled macula-centered CFPs from 87,245 UK BioBank participants with AD (1136 images, 553 subjects) and non- AD (176392 images, 86069 subjects). The system can resize the CFPs to 224 x 224 to improve the computational efficiency and use a masking rate of 60%. The system can employ sparse convolution, and hierarchical encoders and decoders in the SparK framework, generating multi-resolution feature maps with nn-MobileNet. The system can use a lightweight U-Net decoder to restore the CFPs. Then, 1600 epochs for pretraining may be set.

[0074] Next, the system can use 262 macula- or optic disc-centered CFPs from an institutional cohort of subjects with intact cognition, including amyloid-positive preclinical AD (96 images, 32 subjects) and amyloid-negative controls (166 images, 56 subjects). Amyloidpositive status may be defined as amyloid-PET standardized uptake value ratio centiloid cutoff 20. Quality analysis may be narrowed to 87 CFPs (48 left; 39 right) from 28 pre-clinical AD and 149 CFPs (75 left; 74 right) from 49 controls. The two groups can be matched for age. gender and ethnicity (p0.05).

[0075] The system can randomly divide institutional data into training and testing sets (ratio of 8:2) and employ 5-fold stratified cross validation. The model’s precision to classify pre-symptomatic AD can be done using quadratic-weighted Kappa, AUROC (Area Under the Receiver Operating Characteristic), and accuracy. Furthermore, the system can determine the features potentially distinguishing pre-symptomatic AD from controls on the generated attention heatmaps.

[0076] Quality Control for Image Processing

[0077] FIG. 4 illustrates a quality control pipeline in connection with an exemplary system and method for image processing, in accordance with an exemplary embodiment. FIG. 4 illustrates an example of the uality control pipeline as the system may filter fundus image data of AD patients, Parkinson's disease (PD) patients, and normal control subjects from the dataset.The system can filter the images exhibiting any of the conditions: blur, low contrast, poor illumination, and artifacts. The image filtration may help to improve model performance.

[0078] Image Processing Results

[0079] FIG. 5 illustrates heat maps for Alzheimer’s disease (first row) and Parkinson’s disease (second row), demonstrating the efficacy of an exemplary method, in accordance with an exemplary embodiment. FIG. 5 provides visualizations generated using a Gradient-weighted Class Activation Mapping (Grad-CAM) technique to highlight the regions of the input fundus images that contribute most significantly to the model’s predictions. These heatmaps may be computed from the final convolutional layer of the nn-MobileNet-based classification network. The highlighted areas indicate where the model may focus during inference for Alzheimer’s disease (first row) and Parkinson’s disease (second row).

[0080] Image Processing Using U-Net Architecture

[0081] FIG. 6 illustrates a pipeline for a U-Net-based architecture that uses retinal vessel segmentation for image processing (also referred to herein as Alzheimer’s Disease Vessel Analysis and Segmentation or “ADV AS”). ADV AS is a U-Net-based architecture that uses retinal vessel segmentation. The ADV AS framework may consist of 2 main steps. In the first step, the system may use a U-Net-based architecture with 2 parts, encoder (e.g., downsampling) and decoder (e.g., upsampling). The system may first segment the retinal vasculature in the retinal images from the dataset and an institutional data set and then input the segmented vessel results into the U-Net encoder for feature extraction. For vessel segmentation, the system may use the Digital Retinal Images for Vessel Extraction database to train the model and obtain its optimized weight parameters. Two outputs can be generated: the original vessel segmentation image, with detailed vessel structure (model 1); and binary vessel segmentation (denoted either 0 or 1). which enhance the clear parts of the vessel and ignore the faint parts, facilitating further analyses (model 2). Subsequently, these segmentation results can be used as inputs for further extraction of vascular features using U-Net encoders, which can perform initialization using weight from the first segmentation stage. This process may focus on extracting key information from the segmented images that could help with the disease diagnosis.

[0082] Finally, the extracted features can be fed to a new linear classifier, a fully connected layer, and a Softmax function. A fully connected layer is a neural network layer in which every neuron is connected to all activation units from the preceding layer. This layer ty pically resides at the network’s end and maps the learned nonlinear features to the sample's output space. The Softmax function is an activation function widely used for multiclass classification problems.It transforms a real -value vector into a probability' distribution where each element’s value is between 0 and 1. and the sum of all elements equals 1. The output from the Softmax function can be interpreted as the probability distribution over various classes, representing the likelihood that the sample belongs to each class. The classifier can predict whether the individual represented by the image is an AD or control patient. In addition, heatmaps can be generated in the last layer of the U-Net classifier using Gradient-weighted Class Activation Mapping (GRAD-CAM).TABLE 1. ADVAS Models (Original and Binary Vessel Segmentation) Performance for Alzheimer Disease Prediction Using Images Derived From Both Eyes vs Single EyeL, left; R right

[0083] Image Processing Using Bidirectional Encoder-Style Self-Supervised Learning Convolutional Neural Network (CNN)

[0084] FIG. 7 illustrates a pipeline for a bidirectional encoder-sty le self-supervised learning CNN pretrained on retinal color photographs for image processing (also referred to herein as Alzheimer’s Disease Retina Extraction from BERT or “ADRET”). ADRET includes bidirectional encoder representations from a transformers style self-supervised learning CNN pre-trained on a large data set of retinal color photographs from a large image dataset, such as the UK Biobank. The backbone yvas the nn-MobileNet that reported improved network performance by the following: (1 ) adjusting the order of channel configurations for the inverted linear residual bottleneck in the MobileNetV2 netyvork; (2) using a heavy data augmentation strategy through Mixup, CutMix image cropping, flipping, contrast adjustment, brightness adjustment, and sharpening; and (3) adding spatial-dropout modules at various locations within the network to identify their optimal placement in an attempt to address the over-fitting.

[0085] The system can adopt a BERT-style self-supervised learning method, as shown in FIG. 7, masking the image and then reducing it to pre-train the encoder to obtain the representative features. To improve the computational efficiency, the system can resize the input image to 224 x 224, as an example. This image resolution was chosen given ADRET waspre-trained on many fundus images, to ensure computational efficiency and the number of samples processed in each single batch, while maintaining that sufficient features can be acquired. A 224 x 224 resolution may also be used to balance the computational performance with the number of features. Next, the system can perform random masking, such as with a masking rate of 60%. The system can adopt the hierarchical design principle of the SparK framework and combine it with the nn-MobileNet to generate feature maps with different resolutions. Then, the feature maps use sparse convolution and a lightweight U-Net (such as UNet42) decoder to perform upsampling, which serves as an image reconstruction selfsupervised learning task. The system can set up 1600 epochs for pretraining. In some examples, the system may perform its processing on a single 4 NVIDIA Al 00 80GB GPUs with an AMD EPYC 7413 24-Core Processor.Table 2. ADRET and ADVAS Models Performance for Alzheimer Disease Prediction in the UK Biobank and Our Institutional Data Set

[0086] The data are mean (SD). Five-fold cross-validation method was applied in each testing data set.

[0087] AUROC is an area under the receiver operating characteristic curve.

[0088] FIG. 8 schematically illustrates a computer control system or platform programmed or otherwise configured to implement the methods provided herein. In some embodiments, the system includes a computer system 801 programmed or otherwise configured to execute executable instructions, such as instructions for performing image analysis. The computer system includes at least one central processing unit (CPU) or processor 805. The computer system includes at least one memory or storage unit 810 and / or at least one electronic storage unit 815. In some embodiments, the computer system 801 includes a communication interface 820 (e.g.. a network adapter). In some embodiments, computer system 801 may be operatively coupled to a computer network (“network7’) 830 by way of the communications interface 820. In some embodiments, an end-user device 835 is used to upload medical data, such asophthalmic images, general browsing of the database 845, or performance of other tasks. In embodiments, the database 845 is one or more databases separate from computer system 801.

[0089] In various embodiments, the memory or storage unit 810 and / or the at least one electronic storage unit 810 includes one or more tangible, non-transitory memories capable of implementing digital or programmatic logic. In various embodiments, for example, the one or more controllers are one or more of a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device, discrete gate, transistor logic, or discrete hardware components, or any various combinations thereof or the like.

[0090] In various embodiments, the memory 810 includes instructions stored thereon that, in response to execution by the processor 805, cause the computer system 801 to perform the methods provided herein.

[0091] With reference to FIG. 9, a flowchart illustrating a method 900 is provided, in accordance with various embodiments. In various embodiments, the method 900 is a method for image processing. Step 902 can include applying a machine learning framework to train a computerized image processing system comprising a CNN on a set of original unlabeled retinal images. In various embodiments, step 902 includes implementing the framework of FIG. 3. Training the image processing system may include (i) randomly masking each retinal image in the set of original unlabeled retinal images, (ii) generating feature maps with different resolutions from each masked retinal image based on a downsampling process of the CNN, (iii) masking the feature maps adapted to different resolutions for an encoder and a decoder of the CNN, the decoder comprising a U-Net decoder characterized by successive blocks with upsampling layers, (iv) performing sparse convolution on each masked retinal image, and (v) processing each masked retinal image through the decoder to form a set of reconstructed images.

[0092] The method 900 can include receiving an unidentified retinal image for identifying Alzheimer’s disease in a person associated with the unidentified retinal image (step 904). The method 900 can include extracting, using the trained computerized image processing system, features from the unidentified retinal image to generate detailed vessel structure and binary vessel segmentation, for example as shown in FIG. 6. The method 900 can include inputting the detailed vessel structure and binary vessel segmentation into a U-Net encoder to extract disease diagnosis information, for example as shown in FIG. 6.

[0093] The method 900 can include generating a likelihood that the person associated with the unidentified retinal image is an Alzheimer’s disease patient using the trained computerizedimage processing system (step 906). The method 900 may generate the likelihood by inputting the disease diagnosis information into a new linear classifier, a fully connected layer, and a Softmax function. In various embodiments, step 906 includes implementing the frameworks of FIG. 6 and FIG. 7.

[0094] Machine learning

[0095] In various embodiments, a machine learning method for analyzing medical data, for example, including retinal images (e.g., see FIG. 3 and 6-7) and eye-related data, is disclosed herein. In an exemplary embodiment, the machine learning framework disclosed herein is used to analyze retinal images (e.g., fundus images) to diagnose Alzheimer’s and / or Parkinson’s diseases or other neurodegenerative disorders or conditions. In some embodiments, the prognosis or diagnosis generated according to the systems, methods, and devices described herein includes the detection or diagnosis of a neurodegenerative disease, disorder, or condition. In some embodiments, the prognosis or diagnosis includes assessing the risk or likelihood of a neurodegenerative disease, disorder, or condition. In some embodiments, the prognosis or diagnosis includes a classification or classification of a neurodegenerative disease, disorder or condition. The disease, disorder, or condition can be selected from the group consisting of Alzheimer’s and Parkinson’s.

[0096] In various embodiments, medical imaging is used to perform the predictions or diagnoses. Examples of medical imaging include fundus photographs that can be obtained using a fundus camera that utilizes a dedicated microscope (e.g., an ophthalmoscope). The popularity of fundus photography makes it particularly suitable for rapid and accurate diagnostic screening of neurodegenerative diseases. This is especially important in areas where specialists are not readily available, such as rural areas or developing countries / low-income environments. Delays in diagnosis and / or treatment can lead to serious consequences that affect health and long-term prognosis. It is recognized in the present disclosure that one solution is to implement a computational decision support algorithm for interpreting medical imaging such as fundus images.

[0097] In various aspects, disclosed herein is a method of incorporating machine learning techniques (e.g., deep learning with convolutional neural networks) that demonstrates robust diagnostic capabilities using retinal imaging with a database of retinal images that includes a common database.

[0098] Accordingly, in some embodiments, provided herein is an artificial intelligence (Al) transfer learning framework for diagnosing common neurodegenerative diseases using a dataset of retinal images (e.g., fundus shots) that enables high accuracy diagnoses comparableto human expert performance. In some embodiments, the Al framework classifies images and generates corresponding priorities or labels for the classifications (e.g., "emergency recommendations” or "‘general recommendations”). In some embodiments, the normal image is labeled “view.” Thus, certain embodiments of the present disclosure utilize the Al framework as a triage system to generate referrals, simulating real-world applications in community environments, primary care and emergency care clinics. These embodiments can facilitate treatments that can improve outcomes and quality of life by facilitating early diagnosis and detection of disease progression, ultimately affecting a wide range of public health.

[0099] In certain aspects, disclosed herein are machine learning frameworks for generating models or classifiers for diagnosing one or more neurodegenerative diseases, disorders, or conditions. These models or classifiers can be implemented in any suitable system or device, for example, a diagnostic kiosk or a portable device, such as a smartphone with an attachable imaging device (e g., an ophthalmoscope). Non-classifier regression models are also applicable to any of the methods described herein. For example, regression analysis may be performed to generate an output indicative of the severity of a neurodegenerative disease or disorder.

[0100] An example method can include using at least one hardware processor to: receive ophthalmic image data; apply a machine learning classifier to classify the received ophthalmic image data into at least one of a plurality of classifications, the machine learning classifier trained using a domain dataset for an ophthalmic image, the ophthalmic image having been labeled with one or more of the plurality’ of classifications, wherein the plurality of classifications includes a normal classification and one or more disorder classifications, wherein the one or more disorder classifications include at least one of: Alzheimer’s or Parkinson’s; and provide a report indicative of at least one classification of the received ophthalmic image data.

[0101] While the principles of this disclosure have been shown in various embodiments, many modifications of structure, arrangements, proportions, the elements, materials and components, used in practice, which are particularly adapted for a specific environment and operating requirements may be used without departing from the principles and scope of this disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure.

[0102] The present disclosure has been described with reference to various embodiments. However, one of ordinary’ skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure. Accordingly, the specification is to be regarded in an illustrative rather than a restrictive sense, and all suchmodifications are intended to be included within the scope of the present disclosure. Likewise, benefits, other advantages, and solutions to problems have been described above with regard to various embodiments. However, benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or element.

[0103] As used herein, the terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Also, as used herein, the terms “coupled,” “coupling,” or any other variation thereof, are intended to cover a physical connection, an electrical connection, a magnetic connection, an optical connection, a communicative connection, a functional connection, and / or any other connection. When language similar to “at least one of A, B, or C” or “at least one of A, B, and C” is used in the specification or claims, the phrase is intended to mean any of the following: (1) at least one of A; (2) at least one of B; (3) at least one of C; (4) at least one of A and at least one of B: (5) at least one of B and at least one of C; (6) at least one of A and at least one of C; or (7) at least one of A, at least one of B, and at least one of C.

Claims

CLAIMSWhat is claimed is:

1. A method for identifying disease from a plurality of retinal images using a convolutional neural network (CNN), comprising: training a computerized image processing system comprising a CNN on a set of original unlabeled retinal images from a Biobank comprising more than 170,000 retinal fundus images, wherein the training comprises: randomly masking each retinal image in the set of original unlabeled retinal images, generating feature maps with different resolutions from each masked retinal image based on a downsampling process of the CNN, masking the feature maps adapted to different resolutions for an encoder and a decoder of the CNN, the decoder comprising a U-Net decoder characterized by successive blocks with upsampling layers, performing sparse convolution on each masked retinal image, and processing each masked retinal image through the decoder to form a set of reconstructed images; receiving an unidentified retinal image for identifying Alzheimer’s disease in a person associated with the unidentified retinal image; extracting, using the trained computerized image processing system, features from the unidentified retinal image to generate detailed vessel structure and binary vessel segmentation; inputting the detailed vessel structure and binary vessel segmentation into a U-Net encoder to extract disease diagnosis information; and generating a likelihood that the person associated with the unidentified retinal image is an Alzheimer’s disease patient by inputting the disease diagnosis information into a new linear classifier, a fully connected layer, and a Softmax function.

2. The method of claim 1, wherein the decoder comprises a lightweight U-Net decoder characterized by the inclusion of four successive blocks B1-B4 with upsampling layers.

3. The method of claim 1, wherein the sparse convolution improves computational consistency by omitting all the empty voxels and focusing convolution operations on non-zero elements.

4. The method of claim 1, wherein the feature maps D, comprise:

5. The method of claim 1, wherein the feature maps are created via five downsampling operations.

6. The method of claim 5, wherein the feature maps set S comprises:

7. The method of claim 1, further comprising applying a projection layerensure dimensional consistency between the encoder and the decoder.

8. The method of claim 1 , further comprising generating a loss function by comparing the mean squared error between the masked portion of each reconstructed image and the corresponding masked region of the corresponding original image.

9. The method of claim 1, wherein the new linear classifier predicts when the individual associated with the unidentified retinal image is an Alzheimer’s disease or control patient.

10. The method of claim 1, wherein the fully connected layer is a neural network layer in the CNN in which every neuron is connected to all activation units from the preceding layer, resides at a network’s end, and maps learned nonlinear features to a sample’s output space.

11. The method of claim 1, wherein the Softmax function transforms a real-value vector into a probability distribution where each element’s value is between 0 and 1, and the sum of all elements equals 1 and outputs a probability distribution over various classes, representing the likelihood that the sample belongs to each class.

12. A computerized image processing method, comprising:training a computerized image processing system comprising a CNN on a set of original unlabeled retinal images, wherein the training comprises: randomly masking each retinal image in the set of original unlabeled retinal images, generating feature maps with different resolutions from each masked retinal image based on a downsampling process of the CNN, masking the feature maps adapted to different resolutions for an encoder and a decoder of the CNN, the decoder comprising a U-Net decoder characterized by successive blocks with upsampling layers, performing sparse convolution on each masked retinal image, and processing each masked retinal image through the decoder to form a set of reconstructed images.

13. The method of claim 12, wherein the decoder comprises a lightweight U-Net decoder characterized by the inclusion of four successive blocks B1-B4 with upsampling layers.

14. The method of claim 12, wherein the sparse convolution improves computational consistency by omitting all the empty voxels and focusing convolution operations on non-zero elements.

15. The method of claim 12. wherein the feature maps Di comprise:Di = Bi(Di+l) + (i)i(S-), Vi G {4.3, 2, !}.

16. The method of claim 12, wherein the feature maps are created via five downsampling operations.

17. The method of claim 16. wherein the feature maps set S comprises:

18. The method of claim 12, further comprising applying a projection layer ^^to ensure dimensional consistency between the encoder and the decoder.

19. The method of claim 12. further comprising generating a loss function by comparing the mean squared error between the masked portion of each reconstructed image and the corresponding masked region of the corresponding original image.

20. The method of claim 12, wherein the new linear classifier predicts when the individual associated with the unidentified retinal image is an Alzheimer’s disease or control patient.

Images