Systems and methods for processing media and determining vascular changes

Abstract

Systems, methods and devices for processing a medium are disclosed. A system for processing a medium comprises a temporal localisation module configured to determine the existence of a feature in each frame of the medium and determine positional information of the feature in each frame in which the existence of the feature has been detected, a spatial stabilisation module configured to determine a region of interest which defines a template region, apply the template region to each frame and align a position of the template region with the positional information of the feature in each frame such that the position of the feature in each frame is aligned to a fixed position, and crop each frame to the template region to form a plurality of stabilised frames. The system further comprises an output module configured to output the plurality of stabilised frames.

Description

Systems and methods for processing media and determining vascular changesTechnical Field

[0001] Embodiments generally relate to systems, methods and computer readable media for processing media, such as images and / or videos, and in some embodiments, systems and methods for processing ophthalmic videos to detect vascular changes in the eye.Background

[0002] Vascular changes in the eye may be indicative of the presence or absence of physiological conditions and diseases. Such vascular changes may be present in the back of the eye, for example in the Optic Disc Region (ODR) of the retina or in the front of the eye including in the episcleral veins.

[0003] Vascular changes in the eye may be indicative of an increase in intra-ocular pressure (IOP) and the progression of diseases such as glaucoma. Further, vascular changes may be indicative of increased intracranial pressure (ICP) which can cause conditions such as migraine and result from events such as a concussion.

[0004] Vascular change evaluation may be performed by inspecting the vasculature of the eye using imaging techniques. However, the necessary images are typically captured using specialised and expensive benchtop equipment operated by trained professionals. Smartphone-based cameras are becoming more frequently used devices for visualising, for example, the retina and assessing features (for example, vascular changes such as SVPs) associated with ocular conditions. As an example, the low cost and accessibility of smartphone-based fundus cameras have allowed clinicians to perform retinal imaging in various clinical settings. However, retinal fundus images captured by hand-held devices are not robust to various real-world effects, such as noise and jittering. Manual analysis of these noisy images is laborious and relies heavily on the skill of the medical professional, thus restricting its usage in practice,presenting challenges in clinical diagnosis and resulting in decreased accuracy of evaluation.

[0005] Existing video and / or image stabilisation methods for stabilising videos of the eye require, as a starting point, high-quality videos. For example, the videos cannot contain eye blinks and drastic illumination changes. As discussed above, retinal fundus images captured by smartphones are affected by several factors, such as inexperienced operators and / or photographers, unpredictable eyeball movements, and inevitable blinks, and are therefore not suitable to have existing stabilisation methods applied. Efficiently detecting and monitoring vascular changes in the eye still remains challenging. There is, therefore, a need for improved non-invasive, quality techniques to assess vascular changes in the eye.

[0006] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each claim of this application.Summary

[0007] Some embodiments relate to a system for processing a medium, including: a temporal localisation module configured to: determine the existence of a feature in each frame of the medium; and determine positional information of the feature in each frame in which the existence of the feature has been detected; a spatial stabilisation module configured to: determine a region of interest which defines a template region; apply the template region to each frame; and align a position of the template region with the positional information of the feature in each frame such that the position of the feature in each frame is aligned to a fixed position; and crop each frame to the template region to form a plurality of stabilised frames; and an output module configured to output the plurality of stabilised frames.

[0008] In some embodiments, the medium may include a video. The medium may be a video of a region of the eye. The feature may include an optic disc region (ODR). In some embodiments, the feature may include a vein, and may include an episcleral vein on the surface of the eye.

[0009] In some embodiments, the plurality of stabilised frames may form a stabilised video clip. The temporal localisation module may include a convolution neural network (CNN) module configured to detect the existence of the feature by applying a convolutional neural network for object detection to the video. The CNN may be a Faster R-CNN model. The CNN module may be configured to extract a bounding box of the feature in each frame. The CNN module may be configured to determine the position of the feature in each frame.

[0010] The temporal localisation module may be configured to remove jitters in the footage, by drawing a polyline between the feature in each frame of the plurality of frames; and removing frames where the polyline is broken. In some embodiments, the polyline may be considered broken when the deviation from the polyline applied to the previous frame and / or the proceeding frame is above a predefined threshold.

[0011] In some embodiments, a validation module may be configured to validate the detection of the feature in each frame. The validation module may be configured to apply a circle Hough transform. The validation module may be configured to validate the visibility of the feature in a frame by: converting the frame into a greyscale image; applying an histogram equalisation to the greyscale image; dividing the pixels into a plurality of groups according to the equalised histogram, wherein each group of pixels is assigned a different grey value; recolouring the greyscale image using assigned grey value; and applying the circle Hough transform to the greyscale image; wherein the validation module is configured to determining the visibility of the feature is validated when the centre of the feature detected by the CNN module falls within the range of a circle detected by the circle Hough transform.

[0012] Each frame may include timestamp information. The system may further include a filtering module configured to identify sections of jittering by: extracting the optical flow data for each frame; calculate the variance for each frame of optical flow data; retrieve a sequence of bounding boxes for each of the plurality of frames; obtain the approximate coordinates and dimensions of the feature; draw a curve of feature moving in the plurality of frames over time; and identify sections with jittering from the curve. In some embodiments, the filtering module may be configured to extract the optical flow data for each frame by applying an optical flow TV-L1 algorithm to evaluate sharpness.

[0013] The plurality of stabilised frames may include at least 25 frames in a feature visible sequence. In some embodiments, the region of interest may substantially encompass at least part of a region defined by the feature. The spatial stabilisation module may include a template selection module configured to calculate a difference between the potential matching target and the selected template for each frame; and where the difference between the potential matching target and the selected template is lower than a predetermined threshold, determining the potential matching target as the template for each frame. The region of interest may be the feature region.

[0014] In some embodiments, a size of the template region may be acquired from the positional information of the feature. In some embodiments, the size of the template region may be larger than an average size of the feature. The system may further include a recognition module, configured to determine one or more visible spontaneous retinal venous pulsations (SVPs) in the plurality of stabilised frames. The recognition module may be configured to output one or more SVP recognition results.

[0015] Some embodiments relate to a method for processing a medium, including: detecting the existence of a feature in each frame of a plurality of frames of the medium; determining positional information of the feature in each frame in which the existence of the feature has been detected; determining a region of interest which defines a template region; applying the template region to each frame; aligning a position of the template region with the positional information of the feature in eachframe such that the position of the feature in each frame is aligned to a fixed position; and cropping each frame to the template region to form a plurality of stabilised frames.

[0016] In some embodiments, the medium may include a video. The medium may be a video of a region of the eye. The feature may include an optic disc region (ODR). The feature may include a vein, and may include an episcleral vein on the surface of the eye.

[0017] In some embodiments, detecting the existence of the feature is performed by a convolutional neural network (CNN). The convolutional neural network may be a Faster R-CNN model. The CNN may extract a bounding box of the feature in each frame. The CNN may be used to determine the position of the feature in each frame.

[0018] In some embodiments, bounding boxes are utilised to eliminate jitters in the footage, by drawing a polyline between the feature in each frame of the plurality of frames; and removing frames where the polyline is broken. The polyline may be considered broken when the deviation from the polyline applied to the previous frame and / or the proceeding frame is above a predefined threshold.

[0019] In some embodiments, the method may include validating the visibility of the feature in each frame. Validating the visibility of the feature in each frame may include applying a circle Hough transform. Validating the visibility of the feature in a frame may include: converting the frame into a greyscale image; applying an histogram equalisation to the greyscale image; dividing the pixels into a plurality of groups according to the equalised histogram, wherein each group of pixels is assigned a different grey value; recolouring the greyscale image using assigned grey value; and applying the circle Hough transform to the greyscale image; wherein the visibility of the feature is validated when the centre of the detected feature falls within the range of a circle detected by the circle Hough transform.

[0020] In some embodiments, each frame may represent a timestamp of the medium. The method may further include identifying sections of jittering by: extracting theoptical flow data for each frame; calculating the variance for each frame of optical flow data; retrieving a sequence of bounding boxes for each of the plurality of frames; obtaining the approximate coordinates and dimensions of the feature; drawing a curve of feature moving in the plurality of frames over time; and identifying sections with jittering from the curve.

[0021] In some embodiments, extracting the optical flow data for each frame may include applying an optical flow TV-L1 algorithm to evaluate sharpness. The plurality of stabilised frames may include at least 25 frames in a feature visible sequence. The region of interest may substantially encompass at least part of a region defined by the feature.

[0022] In some embodiments, determining a region of interest that defines the template region may include calculating a difference between the potential matching target and the selected template for each frame; and where the difference between the potential matching target and the selected template is lower than a predetermined threshold, determining the potential matching target as the template region for each frame.

[0023] A size of the template region may be acquired from the positional information of the feature. The size of the template region may be larger than an average size of the feature.

[0024] In some embodiments, the method may include determining one or more visible SVPs from the plurality of stablised frames. Determining one or more visible SVPs may include: obtaining one or more stablised retinal fundus video clips; processing the video clip through at least one recognition network; determining the presence of at least one SVP within the one or more stablised retinal fundus video clips; classifying the determined SVP as a grade on the basis of a measured percentile of the determination; calculating an SVP probability index on the basis of the classification; and outputting the grade and / or probability index as SVP recognition results.

[0025] Some embodiments relate to an imaging device, including: a body; a head extending perpendicularly from the top of the body, the head including: an eye piece attached to a distal end of the head; a rear aperture on the proximate end of the head, aligned with the eye piece; and an imaging lens disposed within the head and in line with the eye piece and the rear aperture; a handle extending from the underside of the head; a light source attached to the body and arranged to transmit light toward the eye piece; and a camera configured to obtain retinal fundus videos.

[0026] In some embodiments, the imaging device may include a housing attached to the body, and configured to receive a handheld device. The camera may be a camera of a handheld device, wherein an image sensor of the camera substantially aligns with the rear aperture of the head once the handheld device is received in the housing.

[0027] Some embodiments relate to a system, including: an imaging device, including: a body; a head extending perpendicularly from the top of the body, the head including: an eye piece attached to a distal end of the head; a rear aperture on the proximate end of the head, aligned with the eye piece; and an imaging lens disposed within the head and in line with the eye piece and the rear aperture; a handle extending from the underside of the head; a light source attached to the body and arranged to transmit light toward the eye piece; and a camera configured to obtain retinal fundus videos; and a processing device comprising one or more processors configured to execute instructions which, when executed by the one or more processors, cause the processing device to perform a method including: detecting the existence of a feature in each frame of a plurality of frames; determining positional information of the feature in each frame in which the existence of the feature has been detected; determining a region of interest which defines a template region; applying the template region to each frame; aligning a position of the template region with the positional information of the feature in each frame such that the position of the feature in each frame is aligned to a fixed position; and cropping each frame to the template region to form a plurality of stablised frames.

[0028] Throughout this specification the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.Brief Description of Drawings

[0029] The appended drawings merely illustrate example embodiments of the present disclosure and cannot be considered as limiting its scope.

[0030] Figure 1 is a block diagram of a system for processing ophthalmic videos, according to some embodiments;

[0031] Figure 2 is a block diagram of a temporal localisation module, according to some embodiments;

[0032] Figure 3 is an image showing frames of example noises that obstruct an ODR, according to some embodiments;

[0033] Figure 4 is a process flow diagram showing a method for validating ODR detection, according to some embodiments;

[0034] Figure 5 is a schematic diagram showing the application of a polyline to a plurality of frames, according to some embodiments;

[0035] Figure 6 is a process flow diagram showing a method for determining sections of jittering in a plurality of frames by applying an optical flow algorithm, according to some embodiments;

[0036] Figure 7 is a block diagram of a spatial stabilisation module, according to some embodiments;

[0037] Figure 8 is an image showing the optical flow variance before and after stabilisation, according to some embodiments;

[0038] Figure 9 is a diagram showing the template matching applied to a plurality of ODR-visible frames, according to some embodiments;

[0039] Figure 10 is a process flow diagram showing a method for processing ophthalmic videos, according to some embodiments;

[0040] Figure 11 A is a diagram showing the method of Figure 10 applied to a video, according to some embodiments;

[0041] Figure 1 IB is a diagram showing a visual comparison of an input video and a stablised video after the method of Figure 10 has been applied, as well as the associated polylines, according to some embodiments;

[0042] Figure 12 is a block diagram of a system for detecting retinal vascular changes, according to some embodiments;

[0043] Figure 13 is a diagram of an imaging device, according to some embodiments; and

[0044] Figure 14 is a process flow diagram showing a method for detecting the occurrence of a vascular change in one or more stabilised video clips, according to some embodiments.Description of Embodiments

[0045] Embodiments may relate to methods, systems and computer-readable media for processing media, including images and / or videos, and in particular including ophthalmic images and / or videos such as retinal images, for detecting changes in vascular activity, or vascular changes within the retina. The systems and methods described herein are configured to process a plurality of retinal fundus images in theform of a video clip so as to output a stablised video that may be used for monitoring and / or detecting vascular changes. Processing may be performed to stabilise a plurality of frames in order to focus on one or more features within the plurality of frames. The systems and methods may be used in conjunction with ophthalmoscopes configured to receive retinal fundus images. Additionally, the systems and methods may be used for the diagnosis of intracranial pressure.

[0046] In some embodiments, a system for processing a media, for example, a video, includes a temporal localisation module configured to determine the presence and position of a feature of interest within each frame of a plurality of frames of the retinal image. The temporal localisation module determines the existence of a feature in each frame of a plurality of frames and determines the positional information of the feature in each frame in which the existence of the feature has been detected. The system also includes a spatial stabilisation module configured to stabilise the retinal images. The spatial stabilisation module stabilises the images by determining a region of interest which defines a template region, matching the template region to a corresponding portion of each frame; and aligns the positional information of the feature in each frame with a position of the template region such that the position of the feature in each frame is aligned to a fixed position to form a plurality of stablised frames; and an output module configured to output the plurality of stablised frames.

[0047] The system may be configured to process media including images and / or videos. For example, the media may include medical imaging including scans, imaging, videos and the like. For example, the media may include MRI images, CT images or ultrasound images. In some embodiments, the media may be a video, or it may be a plurality of images, such as slice images or planes, taken over a predefined time interval. Media may include videos from the back of the eye or retina, which may be processed to extract features indicating vascular changes, and used to assess diseases such as glaucoma or indicators such as intracranial pressure. In some embodiments, media may include videos from the front of the eye, which may be processed to assess vascular changes, particularly blood volume changes, in the episcleral veins, and usedas a way of detecting and / or assessing physiological conditions such as intraocular pressure (IOP), non-invasively.

[0048] In some embodiments, the media may refer to videos output by dynamic imaging or dynamic video imaging, such as videos captured by an ophthalmoscope. In some embodiments, the dynamic imaging technique may use a light source to illuminate the vessels, and subtle variations in reflected light intensity from blood flow in veins are captured using a camera. The absorption of light follows the Beer-Lambert law, which states that the light absorbed by blood is proportional to the penetration of light and the concentration of haemoglobin in the blood. In the front of the eye and episcleral veins, any disruption to the aqueous humour levels mixing with the blood in the veins will directly affect haemoglobin concentration and subsequent absorption and reflection levels detected by the camera. In some embodiments, this technique may be applied to assess episcleral veins. This technique may be used to quantify volumetric blood changes in the veins.

[0049] Figure 1 is a block diagram of a system 100 for processing ophthalmic videos, according to some embodiments. The system 100 is configured to receive one or more video(s) 102, each video comprising a plurality of frames. In some embodiments, the video may be a retinal fundus video. The system includes a temporal localisation module 104, configured to determine the existence of a feature of interest in each frame of the video, and a spatial stabilisation module 106 configured to align the feature in each frame to a fixed position to generate a stabilised video.

[0050] The system comprises one or more processor(s) 108 and / or memory 110. The processor(s) 108 may comprise one or more microprocessors, central processing units (CPUs), application specific instruction set processors (ASIPs), application specific integrated circuits (ASICs) or other processors capable of reading and executing instruction code. The memory 110 may comprise one or more volatile or non-volatile memory types. For example, memory 110 may comprise one or more of random access memory (RAM), read-only memory (ROM), electrically erasable programmable readonly memory (EEPROM) or flash memory. The memory 110 is configured to storeprogram code accessible by the processor(s). The program code comprises executable program code modules. In other words, memory is configured to store executable code modules, when executed by the processors to cause the system to perform certain functionalities. For example, memory 110 may comprise a temporal localisation module 104 and / or a spatial stabilisation module 106. A code module may be a collection of computer executable or computer intelligible code, that when executed by a processor, cause the processor to perform processes, action or functionalities, such as the processes, actions or functionalities of the methods described herein.

[0051] In some embodiments, each video in the one or more video(s) 102 is a sequence of ophthalmic images that forms a video and / or a video clip. The system may be configured to process each frame which forms part of the ophthalmic or retinal videos to remove noise and / or jittering, and stabilise the plurality of frames to focus the video on a feature of interest. In some embodiments, ophthalmic video(s) to be processed may include one or more retinal videos, for example, retinal fundus videos. Reference to ophthalmic or retinal images may include reference to the frames which form part of the ophthalmic or retinal videos. Reference to ophthalmic or retinal videos may include a plurality of retinal video clips (each having their own plurality of frames). For example, a plurality of ophthalmic videos, such as retinal fundus videos, may be input into the system for processing simultaneously or sequentially. In some embodiments, each retinal fundus video includes a plurality of frames. Each frame of the plurality of frames depicts a retinal fundus image.

[0052] The occurrence of vascular changes is clinically relevant to imaging of the Optic Disc Region (ODR) in a retinal fundus image. In order to accurately identify vascular changes, it is necessary to detect the location of the ODR in retinal images. Accordingly, the systems and methods described herein for processing ophthalmic videos will be described in the context of stabilising retinal fundus images where the feature of interest is the ODR. Whilst reference will be made herein to the systems and methods in the context of processing retinal fundus images where the feature of interest is the ODR, the systems and methods may be used for processing and / or stabilising other types of ophthalmic videos and other features of interest.

[0053] Features within the ODR that may be detected include vascular changes within the ODR, and may include, but are not limited to SVPs, vascular stiffening, optic nerve bulging, retinal arterial pulsation, pulsatile ocular blood flow, retinal arterial changes such as arteriovenous ratios, vasodilation, retinal arteriolar narrowing, retinal vascular tree and retinal vasculature.

[0054] Videos may be obtained by a clinical device, benchtop device, and / or handheld device. The videos may be obtained through an imaging device associated with, or part of, an ophthalmoscope, for example, a smartphone camera attached to a portable ophthalmoscope. Retinal images, including retinal fundus videos, may be collected in a clinical setting, ambulatory setting and / or home setting. In some embodiments, the videos may be RGB videos, and may be in the range of 0.75 to 30 seconds. In some embodiments, videos input into the system may be at least 25 frames per second. Videos may be a range of orientations and / or sizes depending upon the imaging device used to obtain the videos. For example, the video frames may have a rectangular shape with a side length of 1800 pixels when obtained by a smartphone. Videos may have a resolution of 640 x 640 pixels. Videos may be stored in various formats including AVI, MP4, MOV, Wmv, Mkv, AVCHD, Flash, WebM, VOB, M4V, and / or RMVB.

[0055] The temporal localisation module 104 is configured to distinguish the input video images into ODR- visible images and ODR-invisible images. The temporal localisation module may then validate the ODR-visible images, and collect video clips that exclusively consist of ODR-visible images. In these collected video clips, the temporal localisation module 104 may then be configured to calculate the segments of the video clips with relatively stable ODR motion, for example, segments where the ODR does not deviate significantly from a predetermined range. The module may then be configured to filter out the blurred images from these segments by applying an optic flow method. The temporal localisation module 104 may then be configured to clip the segments for processing by the spatial stabilisation module 106 at a later stage. Effectively, the temporal localisation module 104 is configured to filter out ODR- invisible frames which have been affected by noises such as blinking, low illuminationand the like, and identify the positions of the ODR within the retinal videos and / or individual frames.

[0056] Figure 2 is a block diagram of the temporal localisation module, according to some embodiments. The temporal localisation module 104 includes a CNN module 202, configured to output bounding boxes 204 around the detected ODR for each frame of a video, and output ODR-visible frames 206. The temporal localisation module 104 may further include a validation module 208, a smoothing module 210, and a filtering module 212. The temporal localisation module 104 may be configured to output ODR- visible video clips 214, which include a plurality of frames where each frame has a detected ODR visible in the frame’s image. The ODR-visible video clips may include all or part of each of the one or more input videos(s) 102. In some embodiments, the ODR-visible frames output from the CNN module are run through at least one of the validation module 208, smoothing module 210 and the filtering module 212 before being collected and output as ODR-visible video clips 214. In some embodiments, the ODR visible frames may be automatically collated into ODR-visible clips by the temporal localisation module 104.

[0057] The temporal localisation module 104 is configured to determine, from a plurality of input retinal fundus frames, which frames include a visible ODR (ODR visible-images), and which frames do not include a visible ODR (ODR-invisible images). That is, the system determines in which frames a feature of interest, for example, the ODR is present and detectable, thereby determining the timestamp in the video image that the visible ODR occurs. Frames which include too much noise, distortion and / or jittering may result in no visible ODR being detected. Figure 3 is a diagram showing example frames taken from clinical retinal fundus videos which include the types of noise that results in an obstruction to an ODR, or features within the ODR, that prevent it from being detected. The noises may include, but are not limited to, floor noise, jittering blur, spot blocking relevant features for vascular change detection, bad illumination, lens obstruction, wrong shooting distance, contaminated lens, eyelid blink and other distortions where the direct cause may be unknown. Insome embodiments, the temporal localisation module is configured to remove the ODR-invisible images from the plurality of frames.

[0058] Detection of ODR-visible images is performed by the temporal localisation module by applying an artificial neural network for object detection. This may include a convolutional neural network (CNN). In some embodiments, the temporal localisation module 104 includes a CNN module 202 that applies the artificial neural network to detect the ODRs within each frame of a video. Due to the large amount of irregular noise in retinal fundus videos, some conventional detection methods cannot be applied. In order to accurately obtain the timestamps in each video when the ODR is visible, that is, which frames contain a visible ODR, a CNN model is applied to the plurality of frames by the CNN module. In some embodiments, the CNN model may be a Faster R-CNN neural network to detect the ODR in each frame of the video. In other embodiments, the detection network may include at least one the following models Fast R-CNN, R-CNN, 3D CNN, CNN, and / or YOLO.

[0059] The CNN module 202 may output the temporal timestamps 216 when the ODR appears in the temporal sequence of the video. That is, the frames in which the ODR appears will be determined. The application of CNN by the CNN module 202 to determine the presence of an ODR in a frame is advantageous as it is more likely to identify the ODR than applying a manual approach. With the temporal timestamps 216, the video may be segmented into one or more video clips, where each clip contains a plurality of frames with a visible ODR for a predetermined duration. In some embodiments, a predetermined duration includes substantially sufficient time to identify avascular change. For example, the predetermined duration may be at least 25 frames. The detection by the CNN module 202 for better observation of the ODR in each retinal fundus video and improved detection of vascular changes.

[0060] Additionally, in some embodiments, the CNN module 202 is configured to provide the positional information 218 of the ODR in each frame 206 of the video 102, which is particularly useful when applying template matching for stabilising the video clips 214 at a later stage. In some embodiments, in addition to determining the presenceof the ODR in the frame, the CNN module 202 may determine the positional information 218 of the ODR within in each frame. The positional information 218 of the ODR may include coordinates of the ODR, and / or may include applying a bounding box 204 to the ODR detection within each frame. In some embodiments, the coordinates of the ODR are the coordinates of the bounding box, or the positional information 218 may be determined from the coordinates of the bounding boxes 204. In some embodiments, detection of ODR-visible frames 206, and / or detection of frames where a feature of interest is present, may be performed by the CNN module 202 based on trained input.

[0061] Referring again to Figure 2, the temporal localisation module may include a validation module 208, a smoothing module 210, and a filtering module 212. The validation module 208 may be configured to receive ODR-visible frames and validate the detection of the ODR performed the CNN module 202. The validation module 208 may be configured to perform validation of ODR visibility by applying a Hough transform to assess the output indicating an ODR-visible image. The Hough transform may be a circle Hough transform. In some embodiments, the validation module 208 is configured to apply a circle Hough transform to the ODR-visible frames 204 after the detection by the CNN module 202 to improve the accuracy of detection of frames with a visible ODR. Using the timestamp information 216 which indicates which frames have a visible ODR, the circle Hough transform effectively validates the detection result of the CNN module 202.

[0062] Figure 4 is a process flow diagram showing a method 400 for validating the ODR detection. In some embodiments, the validation module 208 is configured to perform the method 400 to validate the ODR detection by the CNN module 202. Method 400 validates the ODR detection for each individual frame of the ODR visible frames 204 by converting a frame into a greyscale image, and applying a histogram equalisation. The pixels are then divided into eight groups according to the equalised histogram. Each group of pixels is assigned a different grey value, and the pixels in each group use the assigned grey value to recolour the greyscale image. In some embodiments, the new greyscale image may be similar to a gradient image. A circleHough transform is then applied to the new greyscale image, such that the existence and position of the circle are detected more accurately. If the centre of the ODR detected by the CNN module 202 falls within the range of the circle detected by the Hough transform, then the validation module 208 confirms that the frame contains an ODR at the timestamp associated with the frame around that Hough circle. In some embodiments, where the CNN module has provided multiple ODR positions in a frame, for example, where a plurality of ODR detections have been determined for a single frame, the validation module 208 may be configured to filter out incorrect predictions of the plurality of ODR detections, for example, by providing the most probable match of the plurality of detections. In some embodiments, the validation module 208 may be configured to validate the ODR detection which has the closest overlap with the circle detected by the circle Hough transform.

[0063] Once the validation module 208 validates the visible ODR frames, the temporal localisation module may use this information to collate sequential frames to form video clips that exclusively consist of ODR-visible frames 206. That is, video clips in which the ODR, or feature of interest, is visible in each and every frame. In some embodiments, the temporal localisation module 104 may be configured to use the validations from the validation module 208 to collate ODR-visible clips, without processing the ODR visible frames through the smoothing module 210 or the filtering module 212.

[0064] Referring back to Figure 2, in some embodiments, the smoothing module 210 may be configured to calculate smooth segments from the plurality of frames 206 of the retinal fundus videos 102. The smoothing module 210 is configured to take the results of the CNN model which contain the detected positional information 218 of the ODR in the plurality of frames 206 and draw the positions for each frame onto an uneven polyline.

[0065] Figure 5 is a schematic diagram showing the application of the polyline 500 to example ODR positions 502a to 502f in example frames 504a to 504f, according to some embodiments. The smoothing module uses the detected positions 502a to 502f ofthe ODR in each frame 504a to 504f to draw each position frame by frame onto an uneven polyline 500. The polyline 500 may be applied in segments between each ODR detection 502a to 502f for each frame 504a to 504f, respectively. The polyline 500 extends through each frame and the smoothing module 210 will gradually calculate the variance of the curve value in a sliding window applied to calculated trajectories of the ODR positions. Due to the limited frame rate of a smartphone camera, jittering often results in blur. An increase in eye vibrations will lead to a proportional increase in the output of the sliding window calculation, and vice versa. The smoothing module 210 is configured to draw each frame onto the polyline 500, and the polyline 500 may then be fitted into a smooth curve along with the sliding window progress. The polyline reflects the movement of the ODR throughout the plurality of frames. Broken parts of the polyline 500, such as shown in frame 504d, may indicate that the ODR is not detected or clear enough in a particular frames. An ODR position which has a greater deviation from the rest of the polyline, for example 502c in frame 504c is indicative of a higher amplitude of jittering. These oscillations have a high likelihood of resulting in blurred images.

[0066] Referring back to Figure 2, the smoothing module 210 may also be configured to filter noise from the plurality of frames using bounding boxes 204 that are output from the CNN model. In some embodiments, bounding boxes 204 are applied by the CNN module and can be used to eliminate jittering from the plurality of frames by determining the positional information 218 of the ODR. Due to the inherent noise present in the retinal fundus videos and the size variance of detected boxes 204 applied by the CNN module, the detected bounding boxes 204 may present frequent fluctuations. For example, the bounding boxes 204 for ODR detection may be larger on some frames and smaller on other frames as the CNN module detects the presence and position of the ODR. However, since the fluctuation around the ODR is relatively small compared to the relative sizes of bounding boxes within the frames, the temporal localisation module is still configured to obtain positional information 218 of the ODR within each ODR- visible frame 206. Thus, the smoothing module 210 may be configured to screen out large jitters within the plurality of frames 206. By eliminating large jitters and deviations in the position of the ODR, the temporal localisation modulepreserves a continuous plurality of frames, where the ODR is continually shown, in ODR- visible video clips 214. Frames which show large amounts of jittering, for example where the positional information of the ODR deviates from a base position by greater than a predetermined threshold, may be removed from the plurality of frames 206.

[0067] In some embodiments, the temporal localisation module may be configured to collate ODR-visible video clips 214 using the frames modified by the smoothing module, without passing them through the filtering module 212. In some embodiments, the temporal localisation module 104 may retrieve a sequence of bounding boxes 204 from the detection results for each video and use this in combination with the polylines. Although these bounding boxes may have jitters within their positions, the bounding boxes can provide approximate positional information 218, for example coordinates and dimensions, of the ODR. From the bounding boxes 204, the temporal localisation module 104 can draw a curve of the ODR bounding boxes moving in the video frames over time to identify sections with high or low jittering.

[0068] The temporal localisation module 104 may include a filtering module 212. In some embodiments, the filtering module 212 may be configured to apply an optic flow screening algorithm to screen out the effects of optic flow. Optic flow is the motion pattern generated at an eye that is moving relative to the environment. Due to the presence of noise within retinal fundus images, it is difficult for conventional gradientbased algorithms to represent the sharpness of the ODR in each frame. Under certain circumstances, a picture that looks very blurry to the naked eye may be considered to have the highest sharpness in the calculation. As such, the filtering module 212 is configured to apply an optical flow algorithm to evaluate the sharpness in the high- noise frames. In some embodiments, the algorithm may be an optical flow TV-L1 algorithm. The optic flow TV-L1 algorithm is advantageous to use as it is relatively insensitive to changes in brightness and has certain robustness to background noise.

[0069] Figure 6 is a process flow diagram showing a method 600 for determining sections of jittering in a plurality of frames by applying an optical flow algorithm,according to some embodiments. In some embodiments, the filtering module 212 is configured to perform the method 600 to determine sections of jittering in a plurality of ODR- visible frames. The method 600 includes, at 610 extracting the optical flow data for each frame of the plurality of ODR-visible frames. The variance for each frame of optical flow data is then calculated at 620. The method 600 then retrieves, at 630, a sequence of bounding boxes for each of the plurality of frames, and at 640, obtains the approximate coordinates and dimensions of the ODR from the bounding boxes. A curve of ODR moving in the plurality of frames over time is then drawn at 650, and sections of jittering are determined from the curve at 660. As optical flow reflects motion information, the optical flow information of clear images with high floor noise mainly reflects the oscillating motion of noise points. In some embodiments, where the range of this oscillation is small and irregular, it will not be recorded as a substantial difference from the optical flow, resulting in a small variance. An impermanent blur in the frame will be expressed as a large movement, which leads to the variance of the distribution of the optical flow value being larger.

[0070] In some embodiments, the temporal localisation module 104 may be configured to edit retinal fundus videos. In some embodiments, this may be performed be an editing module. The editing module may be configured to apply automatic video editing to input videos 102, wherein after obtaining a series of ODR-visible timestamps from a plurality of frames, the information obtained from the CNN module 202, validation module 208, smoothing module 210, and filtering module 212 can be used to remove and / or edit frames from the plurality of frames. In some embodiments, the editing module is designed to clip the retinal fundus videos into predetermined lengths, to generate video clips which represent at least one complete appearance of avascular change. For example, this may include at least 25 frames in an ODR-visible sequence (which corresponds to 1 second of a video). This would allow for the dynamic observation of vascular changes in line with the cardiac cycle. In some embodiments, the editing module may be located within the temporal localisation module 104, configured to receive input of ODR-visible frames and output ODR-visible clips. It may further be configured to receive data from the validation module, smoothing module of filtering module. Alternatively, the editing module may be external and / orseparate from the temporal localisation module, but configured to receive the ODR- visible clips 214.

[0071] Figure 7 is a block diagram of the spatial stabilisation module, according to some embodiments. The spatial stabilisation module 106 includes a template selection module 700, a specular spot filtering module 702 and a template matching module 704. To narrow down the adverse effects of jittering, the spatial stabilisation module is configured to stabilise the position of the ODR, or any determined feature of interest, within an ODR-visible video clip. In some embodiments, the ODR-visible video clips are received from the temporal localisation module 104. The spatial stabilisation module 106 is configured to determines where and when the ODR is present to align the ODR to a fixed position. In some embodiments, the determination of when the ODR is present in a frame is performed by using the timestamp information for each frame obtained by the temporal localisation module. To determine the location of the ODR, the spatial stabilisation module is configured to apply a template-matching algorithm to identify the position of the ODR in each frame by aligning with a reference template. To perform template matching accurately, it is important to select a good reference template. However, noises present in smartphone retinal fundus videos result in difficulties in using generic template matching algorithms. Accordingly, in some embodiments, the spatial stabilisation module 106 includes a template selection module and / or a specular spots filtering module, along with a template matching module to improve the accuracy of template matching in each frame in order to stabilise the ODR. In some embodiments, the spatial stabilisation module 106 is configured to perform a noise aware template matching method. The method determines the position of the ODR in each video clip to allow stabilisation of the ODR-visible video clips.

[0072] Template-based matching requires a template to be selected as a reference. The template selection module 700 is configured to select both a size 706 of the template 708, and a region 710 for the template. The template selection module is configured to perform a template-matching method to determine a region of interestpotential matching target, and calculating the difference, pixel-wise, between the potential matching target and the selected template. Increasing the difference results in a better matching template. It is also important to choose a template from a region that is significantly different from the other parts of the image. The region of interest then defines the template 708 to which each frame will be mapped.

[0073] In some embodiments, the template selection module 700 is configured to determine the size 712 of the template 708 from the ODR-visible clips 214 and the bounding boxes 204. Since the ODR is the brightest area in a retinal fundus image, is unique to retina images, and includes a unique distribution of internal blood vessels, in some embodiments the template selection module 700 applies a substantially rectangular template 708 centred on the ODR. In other embodiments, the template 808 may be another shape, for example, the template 708 may be substantially circular to match the ODR, or may have dimension that match the video resolution of the retinal fundus videos. In some embodiments, the rectangular template 708 has dimensions 712 that extend beyond the ODR diameter. In some embodiments, the ODR diameter may be estimated from the sizes of bounding boxes 204 detected by the temporal localisation module. For example, the position 710 of the template 708 may be acquired from coordinates of the bounding boxes 204 obtained by the temporal localisation module 104, and the size 212 of the template 708 may be acquired from the dimensions of the bounding boxes 204. In some embodiments, the size 712 of the template may be modified to be slightly larger than the average diameter of the detected ODR. The template size 712 may be modified to contain more or less blood vessels, increasing the distance between an ideal matching result and other potential possible options. That is, the template size 712 may be modified to capture more potential matches.

[0074] After the template selection module determines the size 712 and region 710 for the template, the template selection module is then configured to determine which selected frame 706 of the plurality of frames should be adopted as the preferred or ideal template 708. In some embodiments, the template selection module 700 is configured to select the template 708 from an individual frame 706 of the plurality of frames that does not have blur. Considering the irregular noise in a retinal fundus video, it may bedifficult to calculate sharpness via conventional gradient-based algorithms such as Laplacian sharpness measurements. However, the blurred frames have the effect of enlarging the jitters of the bounding boxes 204 identified by the CNN module. Accordingly, in some embodiments, the trajectory of the bounding box 204 centres within a sliding window can be used for each video clip to select the smoothest period of the video.

[0075] In some embodiments, the smoothest period of the video may be determined by the smoothing module and / or the filtering module of the temporal localisation module. The smoothest period of the video clip may be stored or labelled for reference by the template selection module 700. The variance of optical flow determined by the filtering module may be used to determine the quality of each frame in terms of sharpness. Figure 8 shows an example of the optical flow before and after stabilisation. On the left of Figure 8, the optical flow of an original retinal fundus video along X and Y dimensions, respectively. On the right of Figure 8, there is shown an optical flow of the stabilised video along X and Y dimensions, respectively. The smoother flow indicates that the video has fewer jittering frames and / or noises. In some embodiments, the frame with the lowest variance of the optic flow will be considered as the sharpest image in the most smooth period of the video clip, and will be selected as the frame 706 for the template 708. In some embodiments, the smoothest period of the video and the selection of the sharpest frame may be determined before or after determining the size 712 and / or region 710 of the template by the template selection module 700.

[0076] In some embodiments, the spatial stabilisation module 106 may include a specular spots filtering module 702. Some handheld video-capturing devices utilise one or more external light source(s) to better observe fundus retina. For example, smartphone fundus imaging devices use built-in photoflash or external light sources to provide illumination. In some embodiments, the external light sources may include a LED light source. This light source can result in specular spots on the eyeballs due to the reflection. When the incoming light shines on a patient’s eyeball, the eyeball will reflect it, which will appear as a white specular spot in any video or images that aretaken. As the light sources normally emit white light, the reflected white spots in RGB images present higher values in each of the RGB channels.

[0077] Specular spots protrude especially in the blue (B) and green (G) channels. Accordingly, the specular spots filtering module 702 is configured to select an appropriate global threshold on the B and G channel, and apply mean filtering to minimise the interference of specular light spots during template matching. In some embodiments, the specular spots filtering is only applied by the spatial stabilisation module 106, rather than the temporal localisation module 104 to avoid adding or deleting information from the frames or video clips at an earlier stage. In other embodiments, the specular sports filtering module 702 may be included as part of the temporal localisation module to pre-filter the retinal images.

[0078] A template matching module 704 is then able to apply the template 708 to each frame of the ODR-visible video clips 214. Following template matching by the template matching module, a set of coordinates for the ODR location is obtained from the positional information determined by the temporal localisation module. The ODR coordinates for each frame are aligned with the template coordinates to fix the positions of the ODR in the video, such that stabilised retinal fundus videos 716 are generated.

[0079] Figure 9 is a diagram showing the template matching applied to a plurality of ODR-visible frames, according to some embodiments. In S900, each frame 904 has a visible ODR 906. In S 901, a template 908 is matched to the ODR 906 in each frame 904 by aligning the coordinates of the template 908 with the coordinates of the ODR. In S902, each frame 904 is cropped to the size of the template 908, thereby generating a plurality of cropped frames 910 in which the ODR 906 is in a fixed position. The plurality of frames 910 then form a stabilised video clip.

[0080] Referring to Figure 7, the spatial stabilisation module 106 may further include a cropping module 714, configured to crop the frames to which the template 708 has been applied by the template matching module 704. In some embodiments, each of the frames may be cropped to the size 712 of the template 708 in order to enable focus onthe ODR and potential vascular change-related information. The cropping by the spatial stabilisation module 106 is advantageous as it enables a stable ODR video that is lightweight with a smaller size compared to the original video. The processed video eliminates jittering, removes frames in which the ODR is difficult to see, and retains the ODR visibility and its surrounding information. In some embodiments, the system is configured to automatically crop video clips of 640x640 pixels from the original retinal fundus videos having a size of 1800 X 1800 pixels. This not only saves storage by reducing redundant information within the videos, but also emphasises the area where vascular changes are commonly detectable, eliminating information which is not useful for further analysis.

[0081] A multitude of types of vascular changes may be determined from the processed videos using image analysis techniques to extract one or more features and / or indicators of vascular change. In some embodiments, after processing the videos, each video may be decomposed into a plurality of different branches, each branch representing different features and / or indicators of vascular change. In some embodiments, the videos may be decomposed into a plurality of different branches by applying an artificial intelligent model. In some embodiments, a fully-connected convolution and recurrent neural network is incorporated in each branch to extract the spatial and temporal features of the input processed videos in order to output the features and / or indicators of vascular change. These branches may include, but are not limited to:• a pulsatility index, in which the percentile pulse amplitude of a vein such as the central retinal vein or the episcleral veins is quantified by dividing the difference between the peak and trough of the maximum vein dilation by the average vein calibre, and then multiplying by 100;• vascular stiffening, in which the retinal pulse wave velocity is used as a marker of vascular stiffening. Retinal artery diameter waveforms at pre-set proximal and distal sites are extracted and the transit time calculated from the phase delay between frequency components. Retinal pulse wave velocity may be calculated by dividing the distance between proximal and distal sites by the transit time.Percentile retinal pulse wave velocity may be calculated by dividing retinal pulse wave velocity by mean arterial calibre and then multiplying by 100;• Optic nerve bulging, in which, using the Kaggle dataset, the model branch is trained to classify the input images as class 0 or 1 (that is, normal or papilledema);• Arteriovenous ratio, in which the average artery to venous (i.e., arteriovenous) ratio was measured 1- and 2-disc diameters from the optic disc edge;• Fractal dimension, in which the retinal vascular fractal dimension is a measure of vasculature branching pattern complexity. The FracLac plugin of imageJ may be used to produce ground truth fractal dimension for the input images.

[0082] The output features and / or indicators of vascular change can then be used in a variety of applications, including to use with recognition algorithms and / or to produce diagnostics. In some embodiments, the application and use case will depend upon the type of output features and indicators determined, and the disease / physiological condition. In some embodiments, the features will be considered during determination of diagnostics, but some may have more weight in the final classification. For example, determination of SVP features will have greater influence on ICP estimation than other feature or indicators of vascular change.

[0083] Figure 10 is a process flow diagram showing a method 1000 for processing ophthalmic videos, according to some embodiments. The method 1000 includes, at 1010, detecting the existence of a feature in each frame of the ophthalmic video, and at 1020 determining positional information of the feature in each frame in which the existence of the feature has been detected. Then, at 1030, determining a region of interest which defines a template region; and at 1040 matching the template region to a corresponding portion of each frame. The positional information of the feature in each frame is aligned at 1050 with a position of the template region such that the feature in each frame is aligned to a fixed position to generate a plurality of stablised frames. At 1060 the plurality of stablised frames are output as a stablised video. The method 1000may be configured to be performed by the system 100 of Figure 1 or the system 1200 of Figure 12.

[0084] Figure 11A is a diagram showing the method 1000 of Figure 10 applied to an input video 1100. In Figure 11 A, the template 1102 is shown aligned to a detected ODR position on three ODR- visible frames 1104. The first frame 1106 and the last frame 1108 are ODR-invisible frames and are cut from the video. The ODR- visible frames 1104 have the remainder of the frame outside of the template region cropped, resulting in three stablised frames 1110 that form a stablised video clip 1112 which each show the ODR in a fixed position.

[0085] Figure 11B is a diagram showing a visual comparison of the input video, the stablised video after the method 1000 of Figure 1 has been applied, and the associated polylines. The original input video 1114 shown on the top row has been stablised using the method shown in Figure 10, which results in a plurality of stablised frames 1116 shown in the second row. The uneven polyline depicting the ODR route in the original input video is shown at 1118, and can be compared with the even polyline 1120 depicting the ODR route in the stablised video, which is a substantially straight line.

[0086] Figure 12 is a block diagram of a system 1200 for detecting retinal vascular changes, according to some embodiments. The system 1200 includes an imaging device 1202 and one or more external computing device(s) 1204 in communication with a network 1206. The network 1206 is further in communication with system 1200. The processing device includes one or more processor(s) 1210 and memory 1212. The processing device includes a temporal localisation module 1214, a spatial stabilisation module 1216 and a recognition module 1218.

[0087] In some embodiments, the imaging device may be an ophthalmoscope. The imaging device may be a direct or indirect ophthalmoscope, and may be a benchtop device, portable device or handheld device. In some embodiments, the imaging device may include a smartphone adapter.

[0088] Figure 13 is a diagram of an imaging device 1300, according to some embodiments. The imaging device includes a body 1302, a head 1303 extending perpendicularly from the top of body 1302. A handle 1304 extends from the base 1308 of the body 1302 and from the underside of the head 1303. The handle 1304 enables manual holding and positioning of the device near an eye to be examined (not shown). The body 1302 includes at least one primary support 1306 on the base 1308 of the handle 1304 to enable the imaging device to stand upright. The primary support 1306 may be provided on a proximate end of the handle 1304, having a substantially polygonal shape which extends perpendicularly to the length of the base 1308. A secondary support (not shown) may be provided on a distal end of the base 1308 for additional support. In some embodiments, the imaging device may only require one support. In other embodiments, the base 1308 may be substantially planar to support the device to stand upright.

[0089] The head 1303 includes an a eye piece 1312, attached to the distal end of the body 1302, a rear aperture (not shown) formed on the proximate end of the head 1303, the rear aperture aligned eye piece 1312, and an imaging lens (not shown) disposed within the head 1303 and in line with the eye piece 1312 for magnifying an image of the retina being illuminated by light from a light source. The lens may be a 20 dioptre bio-ophthalmic lens. It may produce a 46 / 60 degree field of view and may have a 202 image magnification. The head further includes a power source (not shown) for powering a light source and other processing components of the imaging device. The imaging device further includes a light source 1314, arranged for transmitting light toward the eye piece 1312, such that the retina of the eye being examined is illuminated when the eye piece 1312 is positioned near and aimed toward the eye’s pupil.

[0090] In some embodiments, the light source may include a fixed light source. In some embodiments, the device may include a light flickering module incorporated into the hardware configured to transmit flickering light. The flickering light transmitted from the light flickering module can induce vasodilation in retinal blood vessels. In some embodiments, the flicker frequency is within a range of 5 Hz and 20 Hz. In some embodiments, the light flickering module has a flickering frequency of 5Hz. In someembodiments, the recording is 90 seconds in duration with a pattern of 20 second flickering light followed by 10 second non flickering light, which may be repeated at least 3 times. Vasodilation occurs due to the activation of neural and / or metabolic mechanisms that result in the relaxation of the smooth muscle in the vessel walls, leading to an increase in vessel diameter. If the flickering light module is in use, and causes a vascular change in the form of vasodilation, it may be used to study and / or document the degree of vasodilation.

[0091] The device includes a device housing 1316 attached to the body and a device adapter 1318, wherein a portable device such as a smartphone 1320 is configured to be mounted within the housing 1316 such that the camera of the smartphone aligns with the rear aperture of the head 1303. The smartphone may be configured to connect to the adapter 1318 such that a camera (not shown) of the smartphone 1320 is optically coupled with the imaging lens to generating digital image data, for example, in the form of retinal fundus videos, of the magnified image of the eye to be examined.

[0092] The camera may include a CMOS, CCD or other type of digital image sensor, an image processor (not shown) configured to convert the signals from the image sensor into a digital image for display or recording. The imaging lens disposed in the head 1303 focuses light being reflected back from the eyepiece 1312 on to the image sensor. The smartphone 1320 may include a display 1322 and / or user interface for displaying a retinal image of the eye being examined. The camera displays processed images of the retina of the eye to be examined on display 1322. The smartphone may include one or more processor(s) to store the received images, write the received images to a memory card or other digital storage device within or attached to the smartphone 1320.

[0093] In some embodiments, the smartphone 1320 may further be configured to transmit retinal fundus videos obtained by the imaging device 1300 to another device or system, including, but not limited to, a computing device, server, cloud, network, desktop, laptop, database, tablet, another smartphone, or a storage device. The retinal fundus videos may be transmitted over a wireless and / or wired connection. The videosmay be transferred for processing by a system via a network in communication with both the imaging device and the system. For example, the imaging device may be configured to obtain retinal fundus videos, and transmit these to the system 1200 shown in Figure 12 for stabilisation and detection of vascular changes. In some embodiments, the imaging device is automatically configured to transmit retinal fundus videos to system 1200, which then automatically processes the videos to generate stabilised retinal fundus videos.

[0094] Referring back to Figure 12, the recognition module 1218 is configured to take one or more stabilised ODR-visible video clip(s) output by the spatial stabilisation module 1216, and detect the presence of retinal vascular changes within each video clip. In some embodiments, the recognition module 1218 is configured to decompose the ODR-visible video clip(s) into a plurality of different branches, each having their own layers. Each branch represents different features used to indicate vascular changes. For example, there may be five different branches relating to pulsatility index, vascular stiffening, optic nerve bulging, arteriovenous ratio, and fractal dimension. By working with stabilised ODR-visible video clip(s), rather than the original retinal fundus videos obtained by the imaging device 1202, the stabilised clip(s) ensure a continuous occurrence of ODR in the videos, while having ODR and vascular change in a stable position. It also eliminates more of the noise which can affect the recognition models applied by the recognition module. This enables the plurality of indicators of vascular change to be readily identified.

[0095] Figure 14 is a process flow diagram showing a method 1400 for detecting the occurrence of vascular changes in one or more stabilised video clips, according to some embodiments. The recognition module is configured to perform the method 1400 to detect the presence of vascular changes within stabilised retinal fundus images. Method 1400 includes obtaining one or more stabilised retinal fundus video clip(s) at 1410, processing the videos through at least one recognition network at 1420, determining the presence of at least one vascular change within the one or more stabilised retinal fundus video clip(s) at 1430, classifying the determined vascular change as a grade on the basis of their measured percentile at 1440, calculating a vascular change probability index onthe basis of the classified grade at 1450, and then at 1460, outputting the vascular change recognition results. The recognition network may be a deep learning model trained to analyse the input retinal video clips and to classify each image. In some embodiments, the recognition network model may include at least one of I3D, X3D, TSN, RESNET3D, VTN. The vascular change recognition results may include additional information about the vascular changes, for example, where the vascular changes are SVPs, additional information may include the size of the venous pulse in pixels and the location at which they occur in the videos. In some embodiments, the vascular change recognition results include a measured percentile of the vascular change amplitude for each detected vascular change. The grades may include: 0 (no percentile pulse), 1 (0-33% pulse), 2 (34%-66% pulse), 3 (67%-100% pulse).

[0096] In some embodiments, the deep learning model that forms the recognition network is a fully connected convolutional (CNN) and recurrent neural network (RNN) model. The recognition network may use a pre-trained VGG16 and Long Short-Term Memory (LSTM) architecture. The depth of the network may be 16 layers (for example, including 13 convolutional and 3 fully connected). In some embodiments, a Softmax function is used at the final layer to generate avascular change probability index for each of the grades. In some embodiments, the probability index may be shown for each individual indicator of vascular change. The accuracy of the model for binary classification (that is, the presence or absence of vascular changes) is greatly improved over running raw or original retinal fundus videos through the recognition network.

[0097] In some embodiments, for training and / or validation purposes, the occurrence of a vascular change may be labelled in predetermined stablised video clips(s). Due to the fixed ODR position in the output stablised video clips, the positions where the vascular change appears can readily be located, so as to further annotate the start and / or end timestamps of vascular change occurrences and count the number of vascular changes in each video clip. According to these annotations, the video clips may be reorganised and the labels modified. For a video labelled as vascular change-positivebut having vascular change-negative segments, the vascular change-negative segments may be clipped out and assigned with a label of vascular change-negative.

[0098] The systems and methods described herein to process retinal videos and determine the presence of vascular changes may be used to track progression of glaucoma or may be used to non-invasively assess intracranial pressure, an important marker to examine in conditions such as traumatic brain injury, intracranial hypertension, hydrocephalus and the like. The systems, methods and devices described herein may be used for rapid and non-invasive assessment of intracranial pressure (ICP) in a number of settings, without requiring specialist clinical equipment, or extensive training to take retinal fundus image, but instead using dynamic imaging (i.e., real-time video imaging) of the blood vessels located on the eye’s surface, to assess intraocular pressure changes.

[0099] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

CLAIMS:

1. A system for processing a medium, including: a temporal localisation module configured to: determine the existence of a feature in each frame of the medium; and determine positional information of the feature in each frame in which the existence of the feature has been detected; a spatial stabilisation module configured to: determine a region of interest which defines a template region; apply the template region to each frame; align a position of the template region with the positional information of the feature in each frame such that the position of the feature in each frame is aligned to a fixed position; and crop each frame to the template region to form a plurality of stablised frames; and an output module configured to output the plurality of stablised frames.

2. The system according to claim 1, wherein the medium includes a video.

3. The system according to claim 2, wherein the medium is a video of a region of the eye.

4. The system according to any one of claims 1 to 3, wherein the feature includes an optic disc region (ODR).

5. The system according to any one of claims 1 to 4, wherein the feature includes a vein, including an episcleral vein on the surface of the eye.

6. The system according to any one of claims 1 to 5, wherein the plurality of stabilised frames form a stabilised video clip.

7. The system according to any one of claims 1 to 6, wherein the temporal localisation module includes a convolution neural network (CNN) module configured to detect the existence of the feature by applying a convolutional neural network for object detection to the video.

8. The system according to claim 7, wherein the CNN is a Faster R-CNN model.

9. The system according to claim 7 or claim 8, wherein the CNN module is configured to extract a bounding box of the feature in each frame.

10. The system according to any one of claims 7 to 9, wherein the CNN module is configured to determine the position of the feature in each frame.

11. The system according to any one of claims 1 to 10, wherein the temporal localisation module is configured to remove jitters in the footage, by: drawing a polyline between the feature in each frame of the plurality of frames; and removing frames where the polyline is broken.

12. The system according to claim 11, wherein the polyline is considered broken when the deviation from the polyline applied to the previous frame and / or the proceeding frame is above a predefined threshold.

13. The system according to any one of claims 1 to 12, further including a validation module configured to validate the detection of the feature in each frame.

14. The system according to claim 13, wherein the validation module is configured to apply a circle Hough transform.

15. The system according to claim 14, wherein the validation module is configured to validate the visibility of the feature in a frame by: converting the frame into a greyscale image; applying a histogram equalisation to the greyscale image; dividing the pixels into a plurality of groups according to the equalised histogram, wherein each group of pixels is assigned a different grey value; recolouring the greyscale image using assigned grey value; and applying the circle Hough transform to the greyscale image; wherein the validation module is configured to determine the visibility of the feature is validated when the centre of the feature detected by the CNN module falls within the range of a circle detected by the circle Hough transform.

16. The system according to any one of claims 1 to 15, wherein each frame includes timestamp information.

17. The system according to any one of claims 1 to 16, further including a filtering module configured to identify sections of jittering by: extracting the optical flow data for each frame; calculating the variance for each frame of optical flow data; retrieving a sequence of bounding boxes for each of the plurality of frames; obtaining the approximate coordinates and dimensions of the feature;drawing a curve of feature moving in the plurality of frames over time; and identifying sections with jittering from the curve.

18. The system according to claim 17, wherein the filtering module is configured to extract the optical flow data for each frame by applying an optical flow TV-L1 algorithm to evaluate sharpness.

19. The system according to any one of claims 1 to 18, wherein the plurality of stablised frames includes at least 25 frames in a feature visible sequence.

20. The system according to any one of claims 1 to 19, wherein the region of interest substantially encompasses at least part of a region defined by the feature.

21. The system according to any one of claims 1 to 20, wherein the spatial stabilisation module includes a template selection module configured to calculate a difference between the potential matching target and the selected template for each frame; and where the difference between the potential matching target and the selected template is lower than a predetermined threshold, determining the potential matching target as the template for each frame.

22. The system according to any one of claims 1 to 21, wherein the region of interest is the region defined by the feature.

23. The system according to any one of claims 1 to 22, wherein a size of the template region is acquired from the positional information of the feature.

24. The system according to any one of claims 1 to 23, wherein a size of the template is larger than an average size of the feature.

25. The system according to any one of claims 1 to 24, wherein the system further includes a recognition module, configured to determine one or more visible spontaneous retinal venous pulsations (SVPs) in the plurality of stablised frames.

26. The system according to claim 25, wherein the recognition module is configured to output one or more SVP recognition results.

27. A method for processing a medium, including: detecting the existence of a feature in each frame of a plurality of frames of the medium; determining positional information of the feature in each frame in which the existence of the feature has been detected; determining a region of interest which defines a template region; applying the template region to each frame;aligning a position of the template region with the positional information of the feature in each frame such that the position of the feature in each frame is aligned to a fixed position; and cropping each frame to the template region to form a plurality of stablised frames.

28. The system according to claim 27, wherein the medium includes a video.

29. The system according to claim 28, wherein the medium is a video of a region of the eye.

30. The method according to any one of claims 27 to 29, wherein the feature includes an optic disc region (ODR).

31. The system according to any one of claims 27 to 29, wherein the feature includes a vein, including an episcleral vein on the surface of the eye.

32. The method according to any one of claims 27 to 31, wherein the detecting the existence of the feature is performed by a convolutional neural network (CNN).

33. The method according to claim 32, wherein the convolutional neural network is a Faster R-CNN model.

34. The method according to claim 32 or claim 33, wherein the CNN extracts a bounding box of the feature in each frame.

35. The method according to any one of claims 32 to 34, wherein the CNN is used to determine the position of the feature in each frame.

36. The method according to claim 34, wherein bounding boxes are utilised to eliminate jitters in the footage, by: drawing a polyline between the feature in each frame of the plurality of frames; and removing frames where the polyline is broken.

37. The method according to claim 36, wherein the polyline is considered broken when the deviation from the polyline applied to the previous frame and / or the proceeding frame is above a predefined threshold.

38. The method according to any one of claims 27 to 37, further including validating the visibility of the feature in each frame.

39. The method according to claim 38, wherein validating the visibility of the feature in each frame includes applying a circle Hough transform40. The method according to claim 39, wherein validating the visibility of the feature in a frame includes: converting the frame into a greyscale image; applying a histogram equalisation to the greyscale image; dividing the pixels into a plurality of groups according to the equalised histogram, wherein each group of pixels is assigned a different grey value; recolouring the greyscale image using assigned grey value; and applying the circle Hough transform to the greyscale image; wherein the visibility of the feature is validated when the centre of the detected feature falls within the range of a circle detected by the circle Hough transform.

41. The method according to any one of claims 27 to 40, wherein each frame represents a timestamp of the medium.

42. The method according to any one of claims 27 to 41, further including identifying sections of jittering by: extracting the optical flow data for each frame; calculating the variance for each frame of optical flow data; retrieving a sequence of bounding boxes for each of the plurality of frames; obtaining the approximate coordinates and dimensions of the feature; drawing a curve of feature moving in the plurality of frames over time; and identifying sections with jittering from the curve.

43. The method according to claim 42, wherein extracting the optical flow data for each frame includes applying an optical flow TV-L1 algorithm to evaluate sharpness.

44. The method according to any one of claims 27 to 43, wherein the plurality of stablised frames includes at least 25 frames in a feature visible sequence.

45. The method according to any one of claims 27 to 44, wherein the region of interest substantially encompasses at least part of a region defined by the feature.

46. The method according to any one of claims 27 to 45, wherein determining a region of interest that defines the template region includes calculating a difference between the potential matching target and the selected template for each frame; and where the difference between the potential matching target and the selected template is lower than a predetermined threshold, determining the potential matching target as the template region for each frame.

47. The method according to any one of claims 27 to 46, wherein a size of the template region is acquired from the positional information of the feature.

48. The method according to claim 47, wherein the size of the template region is larger than an average size of the feature.

49. The method according to any one of claims 27 to 48, further including determining one or more visible SVPs from the plurality of stablised frames.

50. The method according to any one of claim 49, wherein determining one or more visible SVPs includes: obtaining one or more stablised retinal fundus video clips; processing the video clip through at least one recognition network; determining the presence of at least one SVP within the one or more stablised retinal fundus video clips; classifying the determined SVP as a grade on the basis of a measured percentile of the determination; calculating an SVP probability index on the basis of the classification; and outputting the grade and / or probability index as SVP recognition results.

51. An imaging device, including: a body; a head extending perpendicularly from the top of the body, the head including: an eye piece attached to a distal end of the head; a rear aperture on the proximate end of the head, aligned with the eye piece; and an imaging lens disposed within the head and in line with the eye piece and the rear aperture; a handle extending from the underside of the head; a light source attached to the body and arranged to transmit light toward the eye piece; and a camera configured to obtain retinal fundus videos.

52. The device according to claim 51, wherein the imaging device further includes a housing attached to the body, configured to receive a handheld device.

53. The device according to claim 52, wherein the camera is a camera of a handheld device, wherein an image sensor of the camera substantially aligns with the rear aperture of the head once the handheld device is received in the housing.

54. A system, including: an imaging device, including: a body; a head extending perpendicularly from the top of the body, the head including: an eye piece attached to a distal end of the head; a rear aperture on the proximate end of the head, aligned with the eye piece; and an imaging lens disposed within the head and in line with the eye piece and the rear aperture; a handle extending from the underside of the head; a light source attached to the body and arranged to transmit light toward the eye piece; and a camera configured to obtain retinal fundus videos; and a processing device comprising one or more processors configured to execute instructions which, when executed by the one or more processors, cause the processing device to perform a method including: detecting the existence of a feature in each frame of a plurality of frames; determining positional information of the feature in each frame in which the existence of the feature has been detected; determining a region of interest which defines a template region; applying the template region to each frame; aligning a position of the template region with the positional information of the feature in each frame such that the position of the feature in each frame is aligned to a fixed position; and cropping each frame to the template region to form a plurality of stablised frames.

Images