System and method for simulation of fluorescein angiograms - Patent Application 20070122997

Abstract

In an aspect, a computer-implemented method and system for generating simulated fluorescein angiograms is provided. The method includes receiving color fundus images at one or more time points, generating a multi-channel two-dimensional pixel array for each time point, where two of the channels of the pixel array include pixel values ​​from the color fundus image and a third channel of the pixel array includes encoding time, generating a simulated fluorescein angiogram image for each of the time points using a generative network that takes the multi-channel two-dimensional pixel array as input and is trained using previously captured color fundus images and associated fluorescein angiograms, and outputting the simulated fluorescein angiogram image.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 484,753, filed February 14, 2023, the contents of which are incorporated herein by reference. [Background technology]

[0002] background Diabetes affects over 420 million people worldwide. Diabetic retinopathy is the most frequent microvascular complication of diabetes, with over 30% of patients at risk of partial vision loss and approximately 10% at risk of severe visual impairment. Diabetic retinopathy is a leading cause of blindness and preventable visual impairment worldwide. Diabetic retinopathy is caused by dysfunction of retinal blood vessels, and three hallmarks of the disease include: (i) the appearance of microaneurysms, small, round protrusions from retinal capillaries; (ii) leakage of the blood-retinal barrier (BRB); and (iii) retinal ischemia. The disease develops and progresses through multiple stages, with microaneurysms representing the earliest sign of vascular pathology and often appearing before the disease affects vision. As the disease progresses, the vasculature becomes leaky, potentially allowing blood components to enter the retinal extracellular space. Vascular leakage can ultimately lead to the development of edema and vision loss. Importantly, BRB leakage and leaking microaneurysms are the primary targets of treatment aimed at limiting vision loss. The appearance of ischemia (as a result of capillary occlusion) tends to represent an advanced stage of the disease, accompanied by irreversible damage to the retinal neuronal network and irreversible loss of function. The most severe stage of the disease is characterized by the appearance of retinal neovascularization and the transition from nonproliferative to proliferative diabetic retinopathy.

[0003] Clinical diagnosis of diabetic retinopathy typically involves fluorescein angiography (FA). FA allows for the identification of (i) microaneurysms, which appear as hyperfluorescent dots in the early phase of the scan; (ii) microaneurysmal leakage, which appears as fluorescein extravasation in the later phase; (iii) BRB leakage, which appears as nonvascular tissue with fluorescein accumulation in the latency period of the scan; and (iv) retinal nonperfusion, which appears as tissue not accessible to fluorescein in the early phase of the scan. Another imaging modality that may be useful in assessing diabetic retinopathy is optical coherence tomography (OCT). OCT captures vascular anatomy and blood flow and can be used to identify edema and ischemia. This technique generally has a relatively small scanning area. Furthermore, this technique typically does not identify mild vascular leakage or mild edema. Microaneurysms can be detected in images acquired using FA, OCT, or color ophthalmoscopy, but FA is generally known to be superior to the other two ophthalmoscopy techniques in visualizing microaneurysms.

[0004] Interpretation of FA has historically been the domain of retina specialists. Therefore, patients are typically referred to an ophthalmology department by their primary care physician or optician based on a patient-reported history of vision problems or poor performance on a vision test. The diagnosis of diabetic retinopathy is generally not made when the patient first contacts the health care system. Many patients with early diabetic retinopathy who could potentially benefit from early diagnostic evaluation via FA are not currently referred due to health system backlogs or because the length and invasiveness of the procedure are not considered warranted (risk / benefit ratio).

[0005] As examples, not an exhaustive list of use cases, FA can be used to evaluate retinal vein occlusion (RVO), age-related macular degeneration (AMD), hypertensive retinopathy, other retinal, neurovascular, and cardiovascular conditions, as well as other direct and indirect vascular indications. In the case of RVO, FA allows for the identification and characterization of venous occlusion, revealing areas of delayed venous filling, capillary nonperfusion, and collateral vessel development. In AMD, FA is used to detect choroidal neovascular lesions and determine their location, size, and activity.

[0006] Retina specialists use FA to determine appropriate interventions to preserve vision in patients with diabetic retinopathy, RVO, and AMD, which may include anti-vascular endothelial growth factor (anti-VEGF) therapy and / or laser photocoagulation therapy.

[0007] Addressing the limitations of FA, particularly its length and invasiveness, will increase its accessibility to patients with diabetic retinopathy, RVO, and AMD, allowing for more timely intervention and better visual outcomes. Summary of the Invention [Means for solving the problem]

[0008] overview Disclosed herein are methods and systems for simulating fluorescein angiograms. Also disclosed herein are methods and systems for training such methods and systems for simulating fluorescein angiograms.

[0009] In general, in one aspect, a method for simulating fluorescein angiograms is provided. The method receives as input a fundus image and a set of simulated fluorescein angiogram image time points. The method uses an AI model pre-trained on the fundus images and their associated fluorescein angiograms to generate a simulated fluorescein angiogram image for each of the time points.

[0010] In one aspect, a method for simulating a fluorescein angiogram is provided. The method begins by receiving a fundus image and a preselected list of desired time points. For each desired time point in the preselected list, the fundus image is provided as input to an encoder module, which then outputs to a transform module, which then outputs to a decoder module, the transform module having an input / output connection to a memory module, and the encoder module also having a skip connection for outputting directly to the decoder module. The output of the decoder module for each desired time point constitutes a predicted fluorescein angiogram image at that time point for the provided fundus image.

[0011] Implementations of the above-described method may include one or more of the following: The fundus image is a color fundus image; The AI ​​model is a generator network; The encoder module includes one or more sequentially connected encoder blocks. Each encoder block has a convolutional layer and an activation layer. The filter size of the convolutional layer is 4x4 pixels. The convolutional layer uses a stride of 2 pixels. There are at least three sequentially connected encoder blocks of increasing convolutional filter counts. The number of filters in the sequence increases by a power of 2. The number of filters can be 64, 128, or 256. The transform module includes a sequential connection of an initial convolutional layer, an initial activation layer, and one or more residual blocks. Each residual block includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, and a skip connection that jumps directly from the front of the block to the input of the second activation layer. The transform module has 3, 4, 5, 6, 7, 8, 9, or 10 residual blocks. The decoder module has the same number of decoder blocks as there are encoder blocks in the encoder module, followed by convolutional and activation layers that output the current predicted fluorescein angiography image. Each decoder block includes a transposed convolutional layer and an activation layer. Each decoder block includes an upsampling layer, a convolutional layer, and an activation layer. The decoder module can include multiple architectures of decoder blocks. There are at least three sequentially connected decoder blocks with decreasing transposed convolutional and convolutional filter counts. The number of filters in the sequence decreases by a power of two. The number of filters can be, for example, 256, 128, or 64. The transposed convolutional and convolutional layers use a stride of 2 pixels. The filter size of the transposed convolutional and convolutional layers is 4x4 pixels. The upsampling layer has a kernel size of 2x2 pixels and uses nearest neighbor interpolation. The final convolutional layer has three filters of size 3x3 and uses a stride of 1 pixel.

[0012] In one aspect, a method for analyzing simulated fluorescein angiograms is provided. The method includes providing the simulated fluorescein angiograms and corresponding time points from the method described above to a method previously used to analyze actual fluorescein angiograms (referred to as RETICAD). Implementations may include one of the following: The analysis method is described in U.S. Patent No. 9,147,246. The analysis method is described in PCT / CA2022 / 050926.

[0013] In another aspect, there is provided a method for training any of the AI ​​models described above or herein, the method comprising: providing a previously captured fundus image and an associated fluorescein angiogram as input to an untrained AI model; performing image training including constructing an image classifier network, the image classifier network outputting an array of values ​​describing regions in the simulated fluorescein angiogram determined to be simulated, and a goal of training the AI ​​model comprising reducing elements in the array of values; and performing function training including constructing a map classifier network to compare a map of vascular function determined from the previously captured associated fluorescein angiogram with a map of vascular function determined from the simulated fluorescein angiogram. In another aspect, there is provided a system for performing the above-described training method, the system comprising a processing unit and a memory storage, the processing unit in communication with the memory storage and configured to perform the training method.

[0014] In another aspect, there is provided a system for predicting fluorescein angiograms, the system including a processing unit and a memory storage, the processing unit in communication with the memory storage and configured to perform any of the methods described above.

[0015] In certain aspects, a computer-implemented method for generating simulated fluorescein angiograms is provided, the method including: receiving fundus images at one or more time points; generating a multi-channel two-dimensional pixel array for each time point, wherein one or more of the channels of the pixel array include pixel values ​​from a color fundus image and a third channel of the pixel array includes encoding time; generating a simulated fluorescein angiogram image for each of the time points using a generative network, wherein the generative network takes the multi-channel two-dimensional pixel array as input and is trained using previously captured fundus images and associated fluorescein angiograms; and outputting the simulated fluorescein angiogram image.

[0016] In a particular case of the method, the fundus image is a color fundus image.

[0017] In certain instances of the method, the method further includes determining an average intensity of each simulated fluorescein angiography image to determine the fluorescence, and outputting the simulated fluorescein angiography image having the maximum fluorescence.

[0018] In the particular case of the method, there are three channels.

[0019] In another aspect of the method, the encoding time includes a time point for the choroidal phase relative to the maximum possible time of the fluorescein angiogram.

[0020] In yet another version of the method, the encoding time is calculated using a linear or logarithmic function.

[0021] In yet another version of the method, the generator network comprises a plurality of encoder blocks for compressing each color fundus image into a feature representation.

[0022] In yet another version of the method, the generator network further includes a plurality of residual blocks for converting the feature representation of each color fundus image into a feature representation of a corresponding simulated fluorescein angiography image.

[0023] In yet another aspect of the method, the generator network further includes a plurality of decoder blocks for converting the feature representation of the simulated fluorescein angiography image into the simulated fluorescein angiography image.

[0024] In yet another version of the method, the encoder block includes a convolutional layer followed by an activation layer, the residual block includes a convolutional layer followed by an activation layer, and the decoder block includes a transposed convolutional layer followed by an activation layer or an upsampling layer followed by a convolutional layer and an activation layer.

[0025] In yet another version of the method, the residual block further includes a second convolutional layer after the activation layer, and the first convolutional layer, the activation layer, and the second convolutional layer are concatenated with inputs to the respective residual blocks passing through the second activation layer.

[0026] In yet another aspect of the method, the generator network is trained using an image classifier network that outputs an array of values ​​that describe regions in the simulated fluorescein angiography image that have been determined to be simulated, and a goal of training the generator network includes reducing elements in the array of values.

[0027] In yet another aspect of the method, the generator network is trained using a combination of image training and functional training, where the image training includes reducing elements in an array of values ​​using an image classifier network, and the functional training includes a map classifier network for comparing maps of vascular function determined from actual fluorescein angiograms with maps of vascular function determined from simulated fluorescein angiograms.

[0028] In yet another version of the method, the outputs of the image classifier network and the map classifier network are combined to determine a binary cross-entropy loss, which is minimized during training of the generator network.

[0029] In yet another aspect of the method, the map of vascular function includes one or more of a retinal perfusion map, a retinal blood flow map, a blood-retinal barrier leakage map, and leaky and non-leaking microaneurysm maps.

[0030] In another aspect, a system for generating simulated fluorescein angiograms is provided, the system comprising one or more processors and a data memory for receiving color fundus images at one or more time points; generating a multi-channel two-dimensional pixel array for each time point, wherein one or more of the channels of the pixel array include pixel values ​​from the color fundus image and a third channel of the pixel array includes encoded time values; generating a simulated fluorescein angiogram image for each of the time points using a generative network, wherein the generative network has the multi-channel two-dimensional pixel array as input and is trained using previously captured color fundus images and associated fluorescein angiograms; and outputting the simulated fluorescein angiogram image.

[0031] In certain instances of the system, the processor further determines the average intensity of each simulated fluorescein angiography image to determine fluorescence, and outputs the simulated fluorescein angiography image having the greatest fluorescence.

[0032] In another case of the system, the encoding time includes a time point for the choroidal phase relative to the maximum possible time of the fluorescein angiogram.

[0033] In other cases of systems, the encoding time is calculated using a linear or logarithmic function.

[0034] In yet another version of the system, the generator network includes a plurality of encoder blocks for compressing each color fundus image into a feature representation, a plurality of residual blocks for converting each color fundus image feature representation into a corresponding simulated fluorescein angiography image feature representation, and a plurality of decoder blocks for converting the simulated fluorescein angiography image feature representation into a simulated fluorescein angiography image.

[0035] In yet another case of the system, the encoder block includes a convolutional layer followed by an activation layer, the residual block includes a convolutional layer followed by an activation layer, and the decoder block includes a transposed convolutional layer followed by an activation layer or an upsampling layer followed by a convolutional layer and an activation layer.

[0036] In yet another aspect of the system, the generator network is trained using a combination of image training and functional training, where the image training includes an image classifier network that outputs an array of values ​​that describe regions in simulated fluorescein angiogram images determined to be simulated, where a goal of training the generator network includes reducing elements in the array of values, and where the functional training includes a map classifier network for comparing maps of vascular function determined from actual fluorescein angiograms with maps of vascular function determined from simulated fluorescein angiograms.

[0037] In yet another version of the system, the outputs of the image classifier network and the map classifier network are combined to determine a binary cross-entropy loss, which is minimized during training of the generator network.

[0038] These and other aspects are contemplated and described herein. It will be understood that the foregoing summary describes exemplary aspects of the systems and methods to aid those skilled in the art in understanding the following detailed description.

[0039] BRIEF DESCRIPTION OF THE DRAWINGS The features of the present invention will become more apparent in the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0040] [Figure 1] FIG. 1 is a schematic diagram of a generator network for simulating a fluorescein angiogram. [Figure 2] 2 shows an example of an encoder block, a residual block, and a decoder block according to the method of FIG. [Figure 3] 2 shows an example of a classifier network according to the method of FIG. [Figure 4] 1 shows an exemplary diagram of a generative network from exemplary Experiment 1. [Figure 5] FIG. 1 is a flow chart diagram of the generator and classifier network training process. [Figure 6] 1 shows an example color fundus image, four sample simulated fluorescein angiography images, and an example mean intensity time curve from an exemplary experiment. [Figure 7] 1 shows examples of blood-retinal barrier (BRB) leakage, blood flow, and perfusion calculated from simulated fluorescein angiograms from an exemplary experiment. [Figure 8] 1 is a plot of linear and logarithmic functions for calculating encoding time. [Figure 9] 1 shows an exemplary diagram of a generative network from exemplary experiment 2. [Figure 10] FIG. 1 is a flowchart diagram of the RETICAD extended generator and classifier network training process. [Figure 11]According to an exemplary experiment, we show examples of blood-retinal barrier (BRB) leakage, blood flow, and perfusion calculated in eyes with non-proliferative diabetic retinopathy and diabetic macular edema from (a) real fluorescein angiograms, (b) fluorescein angiograms simulated by a network trained with traditional generative adversarial network (GAN) training, and (c) fluorescein angiograms by a network trained with RETICAD-enhanced training. [Figure 12] According to an exemplary experiment, we show examples of blood-retinal barrier (BRB) leakage, blood flow, and perfusion calculated in an eye with age-related macular degeneration from (a) a real fluorescein angiogram, (b) a fluorescein angiogram simulated by a network trained with traditional GAN ​​training, and (c) a fluorescein angiogram by a network trained with RETICAD-enhanced training. [Figure 13] According to an exemplary experiment, we show examples of blood-retinal barrier (BRB) leakage, blood flow, and perfusion calculated in an eye with retinal vein occlusion from (a) an actual fluorescein angiogram, (b) a fluorescein angiogram simulated by a network trained with conventional GAN ​​training, and (c) a fluorescein angiogram by a network trained with RETICAD-enhanced training. [Figure 14] According to an exemplary experiment, we show examples of blood-retinal barrier (BRB) leakage, blood flow, and perfusion calculated in an eye with age-related macular degeneration and retinal vein occlusion from (a) a real fluorescein angiogram, (b) a fluorescein angiogram simulated by a network trained with traditional GAN ​​training, and (c) a fluorescein angiogram by a network trained with RETICAD-enhanced training. [Figure 15] 1 shows a computer system. DETAILED DESCRIPTION OF THE INVENTION

[0041] Note that in Figures 3 and 4, the notation "F=X" (where X is an integer) next to a block represents the number of filters used in the convolutional layer (or transposed convolutional layer, if applicable) of the block.

[0042] Detailed Description The embodiments will now be described with reference to the drawings. For simplicity and clarity of description, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or similar elements. Additionally, numerous specific details are described in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those skilled in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Moreover, the description should not be considered as limiting the scope of the embodiments described herein.

[0043] Various terms used throughout this specification may be read and understood as follows, unless the context indicates otherwise: "or" as used throughout this specification is inclusive as if written "and / or," singular articles and pronouns used throughout include their plurals and vice versa, and likewise, gendered pronouns include their corresponding pronouns, so that pronouns should not be understood as limiting what is described herein to a single gender's use, implementation, performance, etc., and "exemplary" should be understood as "exemplary" or "illustrative," and not necessarily as "preferred" over other embodiments. Further definitions of terms may be set forth herein, and these may apply to examples before and after those terms as understood from reading this description.

[0044] Any module, unit, component, server, computer, terminal, engine, or device illustrated herein that executes instructions may include or, in some cases, have access to a computer-readable medium, such as a storage medium, computer storage medium, or data storage device (removable and / or non-removable), for example, a magnetic disk, optical disk, or tape. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and that can be accessed by an application, a module, or both. Any such computer storage medium may be part of or accessible or connectable to a device. Furthermore, unless the context clearly dictates otherwise, any processor or controller described herein may be implemented as a single processor or as multiple processors. Multiple processors may be arrayed or distributed, and any processing function referred to herein may be performed by one or multiple processors, even though a single processor may be illustrated. Any method, application, or module described herein may be implemented using computer-readable / executable instructions stored or otherwise maintained by such computer-readable media and executed by one or more processors.As a non-limiting example, and referring to FIG. 15, in some embodiments, a computer 3000 has a power supply 3006, a processor 3001, an input unit 3004 for providing input to the processor, memory 3002 for use by the processor as storage for executing applications, functions, instructions, and programs and for accessing buffers and data, a communication module 3003 for sending and receiving data from, for example, a network storage device or the cloud, and a display 3005 for outputting results.

[0045] Fluorescein angiograms are invasive procedures that require the injection of dye into a patient. Advantageously, embodiments are provided herein that use non-invasive fundus images to predict fluorescein angiograms, thereby ameliorating the need for fluorescein angiograms. In some embodiments, the prediction or simulation of a fluorescein angiogram identifies patients with insufficient medical conditions to require actual treatment, thereby eliminating the need for fluorescein angiograms for those patients. In some embodiments, the prediction or simulation of a fluorescein angiogram is performed in an environment where fluorescein angiograms are not typically performed. In some embodiments, such an environment is an ophthalmologist's office. In some embodiments, such an environment is a family doctor's office. In some embodiments, such an environment is a long-term care facility. In some embodiments, such an environment is at a theater-based military medical center. In some embodiments, such an environment is a primary care clinic in a developing country. In some embodiments, the prediction or simulation of a fluorescein angiogram is performed as part of a decision to have a patient undergo fluorescein angiograms. In some embodiments, a prediction or simulation of a fluorescein angiogram is used in place of an actual fluorescein angiogram, with or without reference to other modalities such as optical coherence tomography angiography.

[0046] In some embodiments, a patient has known or suspected retinopathy and a fluorescein angiogram is simulated using the methods disclosed herein, followed by fluorescein angiogram analysis to assist in determining further action. In some embodiments, such methods are described in U.S. Patent No. 9,147,246. In some embodiments, such methods are described in PCT / CA2022 / 050926. In some embodiments, the further action is referral. In some embodiments, the further action is treatment. In some embodiments, the further action is an actual fluorescein angiogram. In some embodiments, the determination of further action is made by a trained individual, such as a retinal specialist, ophthalmologist, optometrist, optician, physician, medic, medical photography technician, or other individual at least minimally trained to make triage decisions based on fluorescein angiogram analysis.

[0047] To simulate a fluorescein angiogram, the methods disclosed herein receive a fundus image and a selection of desired time points as input to an AI model. In some embodiments, the fundus image is a color fundus image. In some embodiments, the desired time points are pre-selected by a user. In some embodiments, the desired time points are selected automatically. In some embodiments, the selected time points are selected to provide a representative sampling of the phases of a full-length fluorescein angiogram. In some embodiments, the selected time points are selected to provide a representative sampling of an early phase of the fluorescein angiogram. In some embodiments, the selected time points are selected to provide a representative sampling of a late phase of the fluorescein angiogram. In some embodiments, the selected time points are selected from a range of time points on which the AI ​​model was previously trained.

[0048] A properly trained AI model, including those described herein, can be used in the fluorescein angiogram simulation methods and systems described herein, provided the model is capable of converting fundus images into sets of simulated fluorescein angiogram images at desired time points. In some embodiments, the AI ​​model takes as input two or more fundus images from the same patient to generate multiple sets of simulated fluorescein angiogram images at desired time points.

[0049] In the particular embodiment described below, the AI ​​model is a generator model.

[0050] In some embodiments, the color mapping is a single channel (e.g., MONOCHROME1, MONOCHROME2, PALETTE COLOR). In some embodiments, there are three channels (e.g., RGB, HSV, YBR_FULL, YBR_FULL_422, YBR_PARTIAL_422, YBR_PARTIAL_420, YBR_ICT, YBR_RCT). In some embodiments, there are four channels (e.g., ARGB, CMYK). For each time point, a two-dimensional array equal in dimensions (height and width) to the color fundus image is created, and pixels are assigned a value equal to the encoding time. The two-dimensional array is then concatenated with a copy of the color fundus image. The result is a series of images (a number equal to the total number of desired time points) with the color fundus information contained in the first and second channels (e.g., red and green channels) along with the encoding time in a third channel (e.g., blue channel), which is then ready to be input into the generator network.

[0051] In some embodiments, the encoding time is calculated using a linear function (FIG. 8). This process is illustrated by the following equation:

number

[0052] In some embodiments, the encoding time is calculated using a logarithmic function (FIG. 8). This process is illustrated by the following equation:

number

[0053] An exemplary generator network implemented on one or more computer processors and memory is shown in Figure 1. The generating network used by the methods disclosed herein includes an encoder module, a transform module, a decoder module, and a memory module. Such a network can operate iteratively to simulate fluorescein angiograms, utilizing information generated from previous iterations and stored in the memory module to simulate subsequent FA images until the sequence is of sufficient quality for output.

[0054] A series of encoder blocks can be used to compress a given input image into a low-resolution feature representation (see the encoder blocks in FIG. 2). Each encoder block can include a convolutional layer followed by an activation layer. The convolutional layer has a defined kernel size, stride, and may have padding characteristics. In some embodiments, the encoder block has a kernel size of 4, a stride of 2, and "same" padding. In some embodiments, the activation layer is a leaky normalized linear unit (ReLU) with a negative slope coefficient of 0.2. Furthermore, the output from each encoder block, in some embodiments, is forwarded to a decoder module via a skip connection.

[0055] A series of residual blocks can be used to transform the feature representation of an input image into its corresponding feature representation of an FA image (see the residual blocks in FIG. 2). In some embodiments, the first block of the transformation module has a convolutional layer followed by an activation layer. In some embodiments, the first block is an encoder block with a convolutional layer with a kernel size of 4, a stride of 2, and "same" padding, and an activation layer using a leaky ReLU with a negative slope coefficient of 0.2. Subsequent elements in the transformation model may include a residual block with a convolutional layer followed by an activation layer and a second convolutional layer. The outputs from these first three layers of the residual block are concatenated with the input of the residual block, if present, and may pass through additional activation layers.

[0056] The output from the series of final residual blocks can be recorded in the memory module for subsequent image generation. In some embodiments, during a given nth cycle of FA sequence generation via the generator network, the transform module stores feature representations of the FA images in the memory module, so that during the n+1th cycle, the transform module can retrieve the feature representations of the nth cycle of simulated FA images from the memory module and concatenate them with the feature representations of the n+1th cycle of input images received from the encoder module.

[0057] In the decoder module, a series of decoder blocks can be used to convert the feature representation of the FA image into a high-resolution FA image (see decoder block type 1 or 2 in FIG. 2). Two types of decoder blocks can be used: (1) a block comprising a transposed convolutional layer followed by an activation layer, or (2) a block comprising an upsampling layer followed by a convolutional layer and an activation layer. The final block in the decoder module can be a convolutional layer (e.g., kernel size of 3, stride of 1, and "same" padding) followed by an activation layer (e.g., tanh activation function). The output from the final block is a simulated FA image. In some embodiments, the convolutional (or transposed convolutional, if applicable) layer in the decoder block has a kernel size of 4, a stride of 2, and "same" padding, and the final convolutional layer at the end of the decoder block has a kernel size of 3, a stride of 1, and "same" padding. In some embodiments, the activation layers in the decoder block use leaky ReLU with a negative slope coefficient of 0.2, and the final activation layer at the end of the decoder block uses a tanh activation function. In some embodiments, the upsampling layer uses nearest neighbor interpolation.

[0058] In some embodiments, a classifier network may be used in the process of training a generator network for use in the methods disclosed herein. An exemplary PatchGAN classifier network architecture is shown in FIG. 3. Inputting both a simulated FA image and a similar real FA image into a series of encoder blocks generates a feature representation of the given FA image. In some embodiments, the encoder block characteristics match those used in the generator network used to output a candidate simulated FA image. In some embodiments, the final block of the network includes a convolutional layer with a kernel size of 3, a stride of 1, and "same" padding, followed by an activation layer using a sigmoid function. The classifier network provides as output a two-dimensional array of values ​​that describe the region as real or simulated. Each encoder block may consist of a convolutional layer (e.g., a kernel size of 4, a stride of 2, and "same" padding) and an activation layer (e.g., a leaky ReLU with a negative gradient coefficient of 0.2). The final block of the network may consist of a convolutional layer (e.g., kernel size of 3, stride of 1, and "same" padding) and an activation layer (e.g., sigmoid function). The output of the network is an n x m array of values ​​that describe regions in the candidate simulated FA image that are considered simulated. In some embodiments, the goal of training is to reduce the value of individual elements in the array of values, or some segment thereof.

[0059] Traditional GAN ​​training Network training can be performed using conventional techniques for generative adversarial networks (FIG. 5). The losses of the generator and classifier networks are calculated using their appropriate loss equations. The generator attempts to minimize the loss of simulated FA images, and the classifier network attempts to maximize it. In some embodiments, batch normalization can be used to recenter and rescale the data for faster training. In some embodiments, dropout is used in the encoder and residual blocks to prevent overfitting, e.g., at a rate of 0.5. In some embodiments, weights are randomly initialized using a normal distribution. In some embodiments, the normal distribution assumes a mean of 0.0 and a standard deviation of 0.02. In some embodiments, Adam optimization is used. In some embodiments, Adam optimization parameters include a learning rate of 0.0002, β1 of 0.5, β2 of 0.999, and ε of 1e-07. In some embodiments, the loss equation can be defined as a weighted sum of binary cross-entropy and mean absolute error (MAE), e.g., using weights of 1 and 100, respectively. In some embodiments, the loss function of the classifier is binary cross-entropy, for example with a weight of 0.5. In some embodiments, batch normalization can be used to recenter and rescale the data for faster training, optionally using random initialization or Adam optimization with parameters as given above.

[0060] RETICAD Extended Generative Adversarial Network Training RETICAD-augmented GAN training combines two training paradigms: image training and functional training (Figure 10). During image training, the generator seeks to minimize the loss, while the image classifier network seeks to maximize it. This process is described herein. Functional training includes an additional classifier network called the map classifier. As input, these networks receive maps of vascular function computed by RETICAD. RETICAD computes maps of vascular function from real or simulated fluorescein angiograms. The maps are computed in a pixel-by-pixel analysis of the time-series data contained in the real / simulated fluorescein angiograms. The vascular function map may include, but is not intended to be an exhaustive list: (i) retinal perfusion, measured as the rate of extravascular fluorescent wash-in during the early phase of the fluorescein angiogram; (ii) retinal blood flow, measured as the rate of intravascular fluorescent wash-in during the early phase of the fluorescein angiogram; (iii) BRB leakage, measured as the rate of fluorescence accumulation in extravascular tissue over the late phase of the fluorescein angiogram; and (iv) leaky and non-leaking microaneurysms, detected as hyperfluorescent dots in the early phase of the fluorescein angiogram, with and without fluorescein extravasation in the late phase, respectively.

[0061] For a given example of functional training (FIG. 10), an actual sequence of color fundus images and their corresponding FA images is loaded into memory. For each fluorescein angiogram image, a copy of the color fundus image is made, and the time of the fluorescein angiogram image is encoded into the copy of the color fundus image. The time-encoded color fundus images are then processed by a generator to create simulated fluorescein angiograms.

[0062] The simulated fluorescein angiograms were then analyzed using the RETICAD algorithm, which calculates N maps of vascular function. The N maps are paired with the patient's corresponding color fundus images and used as training data for a set of N map classifiers. The N map classifiers can be trained using conventional classifier training techniques to recognize the N maps as simulated (y = 0).

[0063] Similarly, the actual fluorescein angiogram is then analyzed using the RETICAD algorithm, which calculates N maps of vascular function. The N maps are paired with the patient's corresponding color fundus image and used as training data for the N map classifier. The N map classifier can be trained using conventional classifier training techniques to recognize the N maps as actual (y=1).

[0064] The output of each classifier is calculated using the binary cross-entropy loss (Loss) given by the following formula: n ) is used to calculate

number

[0065] During training, the generator calculates a composite loss (Loss) given by the following formula: composite ) is attempted to be minimized.

number

[0066] In some embodiments, there may be two map classifiers with composite loss function weights of, for example, α1 = 0.5 and α2 = 0.5. The composite loss function can be used in conjunction with an optimization method, for example, gradient descent, to train the generator.

[0067] In some embodiments, batch normalization can be used to recenter and rescale the data for faster training. In some embodiments, a normal distribution is assumed with a mean of 0.0 and a standard deviation of 0.02. In some embodiments, Adam optimization is used. In some embodiments, Adam optimization parameters include a learning rate of 0.0002, β1 of 0.5, β2 of 0.999, and ε of 1e-07. In some embodiments, the loss function of the classifier is binary cross-entropy with a weight of 0.5, for example. In some embodiments, batch normalization can be used to recenter and rescale the data for faster training, optionally using random initialization or Adam optimization with parameters as given above.

[0068] Reference is now made to the following example which utilizes a generator network architecture as generally shown in FIG. 4, which was trained using the training methodology shown in FIG. 5 and further described below. [Example]

[0069] Example 1 Traditional GAN ​​training: Simulating fluorescein angiograms from color fundus images followed by analysis of the simulated fluorescein angiograms A network was designed (Figure 4) and trained using a batch size of 1 across 136 fluorescein angiograms (2,818 images) and corresponding color fundus images (136 images). The selected model was chosen from the 14th epoch of training and established as the generator network for the generation of simulated fluorescein angiograms. Temporal encoding was performed using a linear function as described herein and shown in Figure 8.

[0070] A color fundus image was selected from the test set and input into the network along with a selection of desired time points. A sequence of 255 FA images ranging from 0 to 254 seconds after injection was simulated from the color fundus image (see example time points from this series in Figure 6). This procedure was performed on multiple other color fundus images in the test set and was found to have satisfactory results suitable for use as a fluorescein angiogram simulator.

[0071] Following fluorescein angiogram simulation from color fundus images, the mean intensity of each simulated FA image was calculated, and the image with maximum fluorescence (hereafter referred to as the "reference image") was identified. The reference image was processed through a vessel segmentation algorithm to simulate a binary map of the vasculature. Each simulated FA image was masked using the binary vasculature map (i.e., pixels outside the vasculature were assigned a value of NaN), and the mean intensity was calculated for all time points. The result is a curve of mean vasculature pixel intensity (Figure 6). Four time points of interest were selected: choroidal flush, arteriovenous phase, onset of recirculation, and end of the late phase. Analysis was then generally according to the method described in PCT / CA2022 / 050926. The velocity of early-phase intensity change (i.e., blood flow) was calculated by performing pixel-wise linear regression on the images between the choroidal flush and the arteriovenous phase. Perfusion maps were calculated via binarization of the blood flow map, with pixels with values ​​greater than 0 assigned a value of 1 (perfused) and pixels with values ​​less than or equal to 0 assigned a value of 0 (non-perfused). The rate of late phase intensity change (i.e., blood-retinal barrier leakage) was calculated by performing pixel-wise linear regression on the image between the onset of recirculation and the end of the late phase. Maps were calculated and visualized using color maps, and endpoints were manually identified (Figure 7). These maps were found to correspond to maps similarly calculated from the actual fluorescein angiogram corresponding to this color fundus image.

[0072] Example 2 Traditional GAN ​​training and RETICAD-augmented GAN training: Simulation of fluorescein angiograms from color fundus images followed by analysis of the simulated fluorescein angiograms The network was designed (Figure 9) and trained using two different training methods: traditional GAN ​​training (Figure 5) and RETICAD-augmented GAN training (Figure 10). RETICAD-augmented GAN training consisted of two additional classifiers, one trained on maps of BRB leakage and the other on maps of perfusion.

[0073] The training data consisted of 136 FA image sequences (2,818 images) and corresponding color fundus images (136 images).

[0074] Color fundus images from four eyes were selected from the test set and input to the two networks along with the corresponding time points of actual fluorescein angiograms. The four selected eyes were from four patients with the following retinal diseases: nonproliferative diabetic retinopathy and diabetic macular edema (FIG. 11), age-related macular degeneration (FIG. 12), retinal vein occlusion (FIG. 13), and age-related macular degeneration with retinal vein occlusion (FIG. 14). For each patient, the corresponding figure shows three exemplary FA images: real, simulated with a network trained using conventional GAN ​​training, or simulated with a network trained using RETICAD-enhanced GAN training. The three exemplary FA images are accompanied by maps calculated from each fluorescein angiogram as described herein.

[0075] Thus, the methods disclosed herein have been shown to be of sufficient quality not only to predict or simulate fluorescein angiograms from color fundus images, but also to generate analytical outputs similar to those of actual FA. Such analytical outputs are useful for decision support in known or suspected retinopathy, age-related macular degeneration, or retinal vein occlusion. For illustrative purposes, but not by way of limitation, simulation of FA may also be used for other applications, such as neurovascular conditions, cardiovascular conditions, and other direct and indirect vascular indications.

Claims

1. 1. A computer-implemented method for generating simulated fluorescein angiograms at a set of one or more time points from fundus images, the method comprising using an AI model to generate a simulated fluorescein angiogram for each of the time points, the model having been previously trained using previously captured fundus images and associated fluorescein angiograms.

2. said generating a simulated fluorescein angiography image comprising: defining a multi-channel two-dimensional pixel array for each of the set of one or more time points, wherein two of the channels of the pixel array include pixel values ​​from the fundus image and a third channel of the pixel array includes an encoding time; generating a simulated fluorescein angiography image for each of the time points using the AI ​​model, the AI ​​model having the multi-channel two-dimensional pixel array as an input; and wherein the method comprises: outputting the simulated fluorescein angiography image; The method of claim 1 further comprising:

3. The method of claim 1 or 2, wherein the AI ​​model comprises a generator network.

4. The method according to any one of claims 1 to 3, wherein the fundus image is a color fundus image.

5. 5. The method of claim 1, further comprising determining an average intensity of each simulated fluorescein angiography image to determine fluorescence, and outputting the simulated fluorescein angiography image having maximum fluorescence.

6. The method of any one of claims 2 to 5, wherein the encoding time comprises a time point relative to the choroidal phase with respect to the maximum possible time of a fluorescein angiography sequence.

7. The method according to any one of claims 2 to 5, wherein the encoding time is calculated using a linear or logarithmic function.

8. The method of any one of claims 3 to 7, wherein the generator network includes a plurality of encoder blocks for compressing each color fundus image into a feature representation.

9. 9. The method of claim 3, wherein the generator network further comprises a plurality of residual blocks for converting the feature representation of each color fundus image into a feature representation of the corresponding simulated fluorescein angiography image.

10. 10. The method of claim 3, wherein the generator network further comprises a plurality of decoder blocks for converting the feature representation of the simulated fluorescein angiography image into the simulated fluorescein angiography image.

11. 11. The method of claim 8, wherein the encoder block comprises a convolutional layer followed by an activation layer, the residual block comprises a convolutional layer followed by an activation layer, and the decoder block comprises a transposed convolutional layer followed by an activation layer or an upsampling layer followed by a convolutional layer and an activation layer.

12. 12. The method of claim 9, wherein the residual block further comprises a second convolutional layer after the activation layer, and the first convolutional layer, the activation layer, and the second convolutional layer are coupled with inputs to the respective residual blocks that pass through a second activation layer.

13. 13. The method of any one of claims 3 to 12, wherein the generator network is trained using an image classifier network that outputs an array of values ​​that describe regions in the simulated fluorescein angiography image that have been determined to be simulated, and a goal of training the generator network includes reducing elements in the array of values.

14. 14. The method of any one of claims 3 to 13, wherein the generator network is trained using a combination of image training and functional training, the image training comprising reducing elements in the array of values ​​using the image classifier network, and the functional training comprising a map classifier network for comparing maps of vascular function determined from a sequence of images from the previously captured fluorescein angiograms with maps of vascular function determined from a sequence of simulated fluorescein angiography images.

15. 15. The method of claim 14, wherein the outputs of the image classifier network and the map classifier network are combined to determine a binary cross-entropy loss, which is minimized during training of the generator network.

16. 16. The method of claim 14 or 15, wherein the maps of vascular function include one or more of a retinal perfusion map, a retinal blood flow map, a blood-retinal barrier leakage map, and leaky and non-leaking microaneurysm maps.

17. 1. A system for generating simulated fluorescein angiograms at a set of one or more time points from fundus images, the system comprising: one or more processors; and a data memory for executing generating a simulated fluorescein angiogram for each of the time points using an AI model, the model having been previously trained using previously captured fundus images and associated fluorescein angiograms.

18. said generating a simulated fluorescein angiography image comprising: defining a multi-channel two-dimensional pixel array for each of the set of one or more time points, wherein two of the channels of the pixel array include pixel values ​​from the fundus image and a third channel of the pixel array includes an encoding time; generating a simulated fluorescein angiography image for each of the time points using the AI ​​model, the AI ​​model having the multi-channel two-dimensional pixel array as an input; and wherein the system comprises: outputting the simulated fluorescein angiography image; The system of claim 17 further comprising:

19. 19. The system of claim 17 or 18, wherein the AI ​​model comprises a generator network.

20. The system according to any one of claims 17 to 19, wherein the fundus image is a color fundus image.

21. 21. The system of claim 17, wherein the processor is further configured to determine an average intensity of each simulated fluorescein angiography image to determine fluorescence, and output the simulated fluorescein angiography image having maximum fluorescence.

22. The system of any one of claims 17 to 21, wherein the encoding time comprises a time point relative to the choroidal phase with respect to the maximum possible time of a fluorescein angiography sequence.

23. The system of any one of claims 17 to 21, wherein the encoding time is calculated using a linear or logarithmic function.

24. 24. The system of claim 17, wherein the generator network includes a plurality of encoder blocks for compressing each color fundus image into a feature representation, a plurality of residual blocks for converting the feature representation of each color fundus image into a feature representation of the corresponding simulated fluorescein angiography image, and a plurality of decoder blocks for converting the feature representation of the simulated fluorescein angiography image back to the simulated fluorescein angiography image.

25. 25. The system of claim 24, wherein the encoder block includes a convolutional layer followed by an activation layer, the residual block includes a convolutional layer followed by an activation layer, and the decoder block includes a transposed convolutional layer followed by an activation layer or an upsampling layer followed by a convolutional layer and an activation layer.

26. 26. The system of claim 24 or 25, wherein the generator network is trained using a combination of image training and functional training, the image training including an image classifier network that outputs an array of values ​​that describe regions within the simulated fluorescein angiogram images that have been determined to be simulated, a goal of training the generator network including reducing elements within the array of values, and the functional training including a map classifier network for comparing a map of vascular function determined from a sequence of images from the previously captured fluorescein angiograms with a map of vascular function determined from a sequence of simulated fluorescein angiogram images.

27. 27. The system of claim 24, wherein the outputs of the image classifier network and the map classifier network are combined to determine a binary cross-entropy loss, which is minimized during training of the generator network.

28. 1. A computer-implemented method for training an AI model to generate simulated fluorescein angiograms from fundus images, the simulated fluorescein angiograms comprising simulated fluorescein angiography images corresponding to a set of one or more time points, the AI ​​model being trained using a combination of image training and functional training, the method comprising: providing a previously captured fundus image and associated fluorescein angiogram as input to an untrained AI model; performing image training, including building an image classifier network, the image classifier network outputting an array of values ​​that describe regions in the simulated fluorescein angiography image that have been determined to be simulated, and a goal of training the AI ​​model includes reducing elements in the array of values; performing functional training including constructing a map classifier network to compare a map of vascular function determined from the previously captured associated fluorescein angiogram with a map of vascular function determined from the simulated fluorescein angiogram image; 11. A computer-implemented method comprising:

29. 1. A system for training an AI model to generate simulated fluorescein angiograms from fundus images, the simulated fluorescein angiograms comprising simulated fluorescein angiography images corresponding to a set of one or more time points, the AI ​​model being trained using a combination of image training and functional training, the system comprising: providing a previously captured fundus image and associated fluorescein angiogram as input to an untrained AI model; performing image training, including building an image classifier network, the image classifier network outputting an array of values ​​that describe regions in the simulated fluorescein angiography image that have been determined to be simulated, and a goal of training the AI ​​model includes reducing elements in the array of values; performing functional training including constructing a map classifier network to compare a map of vascular function determined from the previously captured associated fluorescein angiogram with a map of vascular function determined from the simulated fluorescein angiogram image; A system comprising one or more processors and a data memory for executing the method.

Images