Processing of multi-modal retinal images

Abstract

The present disclosure relates to processing of multi-modal retinal images. A method of processing image data defining a fundus image of an eye to include supplemental information about a specified feature in the fundus image is provided, comprising: specifying (S10) a feature in the fundus image; receiving (S20) OCT data of a C-scan of the retina; selecting (S30) a subset of the OCT data, the subset of the OCT data representing a volumetric image of a portion of the retina at a location on the retina corresponding to a location of the specified feature in the fundus image; processing (S40) the selected subset to generate supplemental image data as supplemental information, the supplemental image data indicating a variation of a measured reflectivity of the eye along a depth direction of the retina in the selected subset; and combining (S50) the image data and the supplemental image data such that an image defined by the combined data provides an indication of the variation at the specified feature.

Description

Technical Field

[0002] The examples in this paper generally relate to the field of retinal image processing, and more specifically, to the processing of reflective images of the retina to include supplementary information derived from optical coherence tomography (OCT) data acquired from the retina.

[0003] background

[0004] For example, two-dimensional images of the fundus acquired via fundus cameras or scanning laser ophthalmoscopy (SLO) are widely used to detect various eye diseases and systemic diseases in subjects. While these imaging methods tend to have sensitivity that varies with retinal depth, their depth specificity tends to be low, making the depth of features observed in retinal fundus images often indeterminate. Because some common eye diseases, such as age-related macular degeneration, diabetic retinopathy, and retinal vein occlusion, are associated with features such as hyper-reflective dots, which also have a similar appearance in retinal fundus images, clinicians often find it difficult to differentiate these diseases solely by examining retinal fundus images. Artifacts in retinal fundus images, such as those caused by atrophy of the hyperreflective internal limiting membrane (ILM) and retinal pigment epithelium (RPE), can also confuse the identification of disease-related features such as exudative, RPE-related, and vitreous hyperreflective dots. Further information about features of interest already identified on retinal fundus images can be obtained by examining OCT thickness maps covering the features of interest.

[0005] Overview

[0006] According to a first exemplary aspect of this document, a computer-implemented method is provided for processing image data of a fundus image defining a portion of the retina of the eye to include supplementary information about specified features in the fundus image. The method includes specifying features in the fundus image, receiving C-scan optical coherence tomography (OCT) data of that portion of the retina, and selecting a subset of the OCT data representing a volumetric image of a portion of the retina at a location on the retina corresponding to the location of the specified feature in the fundus image. The method further includes processing the selected subset of the OCT data to generate supplementary image data as supplementary information, the supplementary image data indicating variations in the eye's measured reflectance along the depth direction of the retina at the selected subset of the OCT data. The method further includes generating combined image data by combining the image data with the supplementary image data, such that the combined image data defined by the combined image data provides an indication of variations in the eye's measured reflectance at the specified feature at the selected subset of the OCT data.

[0007] In a computer-implemented method according to the first example aspect, combined image data can be generated by replacing the pixel values ​​of a subset of pixels in fundus image data with pixel values ​​of supplementary image data, such that the combined image data defines a combined image that provides an indication of changes in the measured reflectance of the eye at a selected subset of OCT data at a specified feature. In some example embodiments, combined image data can be generated by replacing the pixel values ​​of a subset of pixels in fundus image data with pixel values ​​of supplementary image data, which defines a sub-region of the fundus image located at a specified feature in the fundus image, such that the combined image data provides an indication of changes in the measured reflectance of the eye at a selected subset of OCT data at the specified feature. In some other example embodiments, the supplementary image data may define a graph indicating changes in the measured reflectance of the eye at a selected subset of OCT data along the depth direction of the retina, or more specifically, define a graph indicating how the measured reflectance of the eye at a selected subset of OCT data changes along the depth direction of the retina. In these other example embodiments, composite image data can be generated by replacing the pixel values ​​of a subset of the fundus image data with the pixel values ​​of the supplementary image data, such that the composite image defined by the composite image data includes a graphic overlaid on the fundus image to provide an indication of changes in the reflectance of the eye measured at a selected subset of the OCT data at a specified feature.

[0008] Additionally or alternatively, according to an example embodiment, the computer-implemented method according to the first example aspect may further include displaying a fundus image and a cursor on a display, such that the cursor can be controlled to move on the displayed fundus image via a signal from a user input device, wherein a feature in the fundus image is specified by recording the value of a first position indicator in response to a feature specification command, the first position indicator indicating the display position of the cursor on the displayed fundus image.

[0009] The method in this example embodiment may further include: processing OCT data to generate an OCT en-face image of that portion of the retina; and displaying the en-face OCT image together with a fundus image on a display such that a cursor can be controlled to move on the displayed en-face OCT image via a signal from a user input device, wherein a subset of the OCT data is selected based on the value of a second position indicator, the second position indicator indicating the display position of the cursor when the cursor has been guided by a signal from the user input device to cover a portion of the displayed en-face OCT image corresponding to a specified feature in the displayed fundus image.

[0010] Optionally, features in the fundus image can be automatically specified using a feature extraction algorithm. In this case, the computer-implemented method may further include: displaying the fundus image and a feature position indicator indicating the location of the specified feature in the fundus image on a display; processing OCT data to generate an OCT frontal image of that portion of the retina; and displaying the OCT frontal image and a cursor together with the fundus image on the display, such that the cursor can be controlled to move on the displayed OCT frontal image via a signal from a user input device, wherein a subset of the OCT data is selected based on the value of a second position indicator indicating the cursor's display position when the cursor has been guided by a signal from the user input device to cover a portion of the displayed frontal image, the position of the portion of the displayed OCT frontal image corresponding to the location of the specified feature in the fundus image indicated by the feature position indicator.

[0011] Optionally, when the features in the fundus image are automatically specified by the feature extraction algorithm, a subset of the OCT data can be selected by applying a geometric transformation to the location of the features already specified by the feature extraction algorithm in the fundus image. This geometric transformation maps the location in the fundus image to the corresponding A-scan location in the OCT data.

[0012] In any of the computer-implemented methods described above, the feature may be one of a point and a hyperreflective point in a fundus image, the feature having a pathological cause or being caused by reflection from the internal limiting membrane of the retina. For example, the feature may have a pathological cause, including one of the following: blood leakage, exudation, drusen, atrophy and / or nevus in the retina, and atrophy of the retinal pigment epithelium.

[0013] In the above text, a selected subset of OCT data can be processed to generate supplementary image data using the following scheme 1, scheme 2 or scheme 3.

[0014] Option 1:

[0015] To generate supplementary image data, a selected subset of OCT data is processed as follows: multiple anatomical layers of the eye, including one or more retinal layers, are detected within the selected subset of OCT data; a corresponding sum is calculated for each of at least two of the detected anatomical layers by summing the values ​​of data elements in the subset of OCT data within the anatomical layers; for each of the at least two detected anatomical layers, a corresponding ratio is calculated between the sum calculated for that anatomical layer and the sum of all data elements in the subset of OCT data within the at least two detected anatomical layers; and color information is generated as supplementary image data based on an ordered sequence of the calculated ratios, wherein the calculated ratios are arranged in order of the corresponding anatomical layers in the eye, and the color information defines the color of the ordered sequence of the calculated ratios that will be displayed in the combined image and identify the color, such that the color indicates the variation of the measured reflectance of the eye along the depth direction of the retina in the selected subset of OCT data. For example, three anatomical layers in a selected subset of OCT data can be detected, and color information can be generated by assigning corresponding weights to each of the red, green, and blue components of the color to be displayed in the combined image according to the corresponding one of the calculated ratios in an ordered sequence of calculated ratios.

[0016] Option 2:

[0017] To generate supplementary image data, a selected subset of OCT data is processed as follows: multiple anatomical layers of the eye, including one or more retinal layers, are detected within the selected subset of OCT data; a corresponding sum value is calculated for each detected anatomical layer by summing the values ​​of data elements from the subset of OCT data within the anatomical layers; based on the calculated sum value, the anatomical layer among the detected anatomical layers that makes a major contribution to the measurement of reflectivity of the eye is selected; and graphic image data defining the pattern identifying the selected anatomical layer is generated as supplementary image data.

[0018] Option 3:

[0019] A selected subset of OCT data represents a volumetric image of a portion of the retina with a predetermined type of feature, and the selected subset of OCT data is processed to generate supplementary image data by: training a model to determine the depth of a predetermined type of feature in the depth direction of the retina through supervised learning of examples of OCT data of pathological regions of at least one other retina, each example of OCT data comprising a single OCT A scan or two or more adjacent OCT A scans, and each pathological region having a corresponding feature of the predetermined type, wherein the indication of the corresponding depth of the corresponding feature in the depth direction of the retina in each example of OCT data is specified by the user during training; processing the selected subset of OCT data using the trained model to determine the depth of the feature in the depth direction of the retina; and generating one of (i) graphic image data and (ii) color information as supplementary image data, the graphic image data defining a graphic indicating the determined depth of the feature and to be overlaid on the fundus image (10) to indicate the location of the feature in the combined image (40), the color information defining a color to be displayed at the location of the feature in the combined image and indicating the determined depth of the feature.

[0020] According to a second exemplary aspect of this document, a computer program comprising computer program instructions is also provided, which, when executed by a computer, cause the computer to perform the methods described above. The computer program may be stored on a non-transitory computer-readable storage medium, or it may be carried by a signal.

[0021] According to a third exemplary aspect of this document, an apparatus is also provided for processing image data of a fundus image defining a portion of the retina of the eye to include supplementary information about specified features in the fundus image. The apparatus includes a feature designation module arranged to designate features in the fundus image, and a receiver module arranged to receive C-scan OCT data of the portion of the retina. The apparatus also includes a selection module arranged to select a subset of OCT data representing a volumetric image of a portion of the retina at a location on the retina corresponding to the location of the specified feature in the fundus image. The apparatus further includes a supplementary image data generation module arranged to process the selected subset of OCT data to generate supplementary image data as supplementary information, indicating variations in the reflectance of the eye measured along the depth direction of the retina within the selected subset of OCT data. The apparatus also includes a combined image data generation module arranged to generate combined image data by combining the image data with the supplementary image data, such that the combined image data defines a combined image that provides an indication of variations at the specified feature.

[0022] In the apparatus according to the third example aspect, the combined image data generation module can be arranged to generate combined image data by replacing the pixel values ​​of a subset of pixels in fundus image data with pixel values ​​of supplementary image data, such that the combined image defined by the combined image data provides an indication of changes in the measured reflectance of the eye at a selected subset of OCT data at a specified feature. In some example embodiments, the combined image data generation module can be arranged to generate combined image data by replacing the pixel values ​​of a subset of pixels in fundus image data with pixel values ​​of supplementary image data, which defines a sub-region of the fundus image located at a specified feature in the fundus image, such that the combined image defined by the combined image data provides an indication of changes in the measured reflectance of the eye at a selected subset of OCT data at the specified feature. In some other example embodiments, the supplementary image data generation module can be arranged to generate supplementary image data that defines a graph indicating changes in the measured reflectance of the eye at a selected subset of OCT data along the depth direction of the retina, or more specifically, defines a graph indicating how the measured reflectance of the eye at a selected subset of OCT data changes along the depth direction of the retina. In these other example embodiments, the combined image data generation module may be arranged to generate combined image data by replacing the pixel values ​​of a subset of the fundus image data with the pixel values ​​of the supplementary image data, such that the combined image defined by the combined image data includes a graphic overlaid on the fundus image to provide an indication of changes in the reflectance of the eye measured at a selected subset of the OCT data at a specified feature. Brief description of the attached diagram

[0024] Exemplary embodiments will now be explained in detail by way of non-limiting example only, with reference to the accompanying drawings described below. Unless otherwise indicated, similar reference numerals appearing in different figures in the drawings may denote the same or functionally similar elements.

[0025] Figure 1 This is a schematic diagram of an apparatus for processing image data according to a first example embodiment.

[0026] Figure 2 This is a schematic diagram of a display showing fundus images and frontal OCT images in the first example embodiment.

[0027] Figure 3 This is a schematic diagram of OCT data and its subset selected by the selection module of the first example embodiment.

[0028] Figure 4 An example implementation of a programmable signal processing hardware in the first example embodiment of this article is shown.

[0029] Figure 5This is a flowchart illustrating the process by which an apparatus of a first example embodiment processes image data of a fundus image that defines a portion of the retina of the eye to include supplementary information about specified features in the fundus image.

[0030] Figure 6 This is a flowchart illustrating the process by which the supplementary image data generation module of the first example embodiment generates supplementary image data.

[0031] Figure 7 An image of the fundus (Image A), a processed version of the fundus image in which specified features are highlighted (Image B), and a combined image (Image C) generated by the apparatus of the first example embodiment are shown.

[0032] Figure 8 It shows along Figure 7 The horizontal line shown in image C is a cutoff of scanned image B.

[0033] Figure 9 This is a schematic diagram of an apparatus for processing image data according to a second example embodiment.

[0034] Figure 10 This is a flowchart illustrating the process by which the supplementary image data generation module of the second example embodiment generates supplementary image data.

[0035] Figure 11 A first example of a combined image generated by the apparatus of the second example embodiment is shown.

[0036] Figure 12 A second example of a combined image generated by the apparatus of the second example embodiment is shown.

[0037] Figure 13 This is a schematic diagram of an apparatus for processing image data according to a third exemplary embodiment.

[0038] Figure 14 This is a flowchart illustrating the process by which the supplementary image data generation module of the third example embodiment generates supplementary image data.

[0039] Detailed description of example embodiments

[0040] The following describes a method, apparatus, and computer program for processing retinal fundus image data. This can help clinicians assess and differentiate the distribution of features in common eye diseases (e.g., diabetic retinopathy, age-related macular degeneration, and retinal vein occlusion) that have similar appearances in retinal fundus images, and can help avoid misinterpretation of retinal fundus images caused by specular artifacts. The techniques described herein can allow for the use of additional information in OCT data to more clearly render features (e.g., bright spots) in retinal fundus images. In some example embodiments, the information can be fully understood by the user without reviewing the OCT data. Furthermore, in some example embodiments, the user can be notified whether features in the retinal fundus image are consistent with features in the OCT, thereby helping to avoid misinterpretation of artifactual spots, etc., in retinal fundus images.

[0041] The exemplary embodiments described herein will now be described in more detail with reference to the accompanying drawings.

[0042] First Example Implementation

[0043] Figure 1 This is a schematic diagram of an apparatus 100 for processing image data according to a first exemplary embodiment. As will be described in more detail below, the apparatus 100 is arranged to process received image data D of a fundus image 10 that defines a portion of the retina of the eye. F To include supplementary information about specified features 12 in the fundus image 10, such as Figure 2 As shown, the fundus image 10 and designated features 12 are displayed on the display 14 (e.g., a visual display unit, such as a computer monitor) along with a user-controlled cursor 16.

[0044] A fundus image of the retina (also referred to herein as a retinal fundus image) 10 can be acquired through any fundus imaging procedure in which a two-dimensional representation of the three-dimensional (semi-transparent) retinal tissue is projected onto an imaging plane of a fundus imaging device (not shown) using light collected from the retina. The fundus image can be acquired by illuminating the retina with a global illumination or a scanning beam provided by the fundus imaging device. The light reflected from the retina is collected by a receiver of the fundus imaging device, and its position-dependent intensity is detected and then converted into image data D representing the two-dimensional fundus image 10. F Therefore, the term “fundus imaging” as used in this paper refers to any process that produces a two-dimensional image of a portion of the retina, where the image pixel values ​​represent the corresponding intensity of light collected from the retina, and contrasts with OCT imaging (discussed below).

[0045] Fundus image 10 can be acquired using one of a variety of fundus imaging devices known to those skilled in the art, including but not limited to fundus cameras and scanning laser ophthalmoscopy (SLO). Combined with filters, these types of fundus imaging devices can be used to acquire monochrome or autofluorescence images of the fundus, and, in the case of intravenous contrast agents, to acquire fundus fluorescein angiography, indocyanine green angiography, etc. Therefore, fundus imaging encompasses a variety of forms / techniques, including monochrome fundus photography, color fundus photography, scanning laser ophthalmoscopy (SLO), adaptive optics SLO, fluorescein angiography, and indocyanine angiography.

[0046] Fundus imaging devices can have a relatively narrow field of view, around 30°–55°, which is typical in conventional fundus imaging, or they can be wide-field-of-view fundus imaging devices with a field of view of approximately 100°. As a further alternative, fundus imaging devices can be ultra-wide field-of-view (UWF) devices with a field of view of approximately 200°. TM Fundus imaging devices, such as the Optos California manufactured by Optos plc. TM system.

[0047] like Figure 1 As shown, the apparatus 100 includes a feature designation module 110, which is arranged to designate features 12 in a fundus image 10 received from a fundus imaging device. Feature 12 may be an image of a structure with relatively high reflectivity (whether inside the retina or in another part of the imaged eye), and therefore may be a portion of the fundus image 10 exhibiting higher reflectivity than the area surrounding that portion of the fundus image 10. Alternatively, feature 12 may be an image of a structure with relatively low reflectivity (whether inside the retina or in another part of the imaged eye), and therefore may be a portion of the fundus image 10 exhibiting lower reflectivity than the area surrounding that portion of the fundus image 10. Although for convenience, the description of the apparatus 100 is given herein by reference to a single designated feature 12, it should be understood that multiple features in the fundus image 10 may be designated, and the operations described below may be performed based on each of these designated features.

[0048] Feature 12 may be, for example, a dot feature whose extent in the fundus image is small compared to the size of the fundus image. Such dot features may have pathological causes, such as blood leakage, exudation, drusen, atrophy, and / or nevi in ​​the retina. Dot features associated with these pathologies have very similar characteristic appearances (in terms of their size and the difference in brightness between the dot features and their surroundings in fundus image 10), and it is difficult for a clinician to distinguish them solely by examining fundus image 10. They are also difficult to distinguish from dot features with non-pathological causes, such as floaters in the vitreous humor, reflections from the retinal surface (e.g., the internal limiting membrane), or reflections from imperfections in the imaging system (e.g., dust), which have a similar appearance in the fundus image to the aforementioned pathological dot features.

[0049] For example, a dot feature can be a highly reflective dot / focal point frequently observed in retinal fundus images. A highly reflective focal point may have pathological causes such as blood leakage, exudation, drusen, atrophy and / or nevus in the retina, or atrophy of the retinal pigment epithelium, or non-pathological causes such as reflections from the internal limiting membrane of the retina. In all these cases, highly reflective dots have very similar characteristic appearances and are difficult to distinguish by examining fundus images alone.10

[0050] Feature designation module 110 can designate feature 12 in fundus image 10 in one of a variety of different ways. For example, in this example embodiment, feature designation module 110 causes fundus image 10 and cursor 16 to be displayed on display 14. The display position of cursor 16 on the displayed fundus image 10 can be controlled by a signal from user input device 18 (e.g., computer mouse, touchpad, etc.). In this example embodiment, feature 12 in fundus image 10 is designated by recording the value of a first position indicator in response to a feature designation command. This first position indicator indicates the display position of cursor 16 when the cursor is signaled to cover the displayed fundus image 10. Feature designation module 110 can cause graphic 19 to cover fundus image 10 at the position indicated by the first position indicator.

[0051] As in this example embodiment, the feature designation command can be provided by a user, such as by a user operating the user input device 18 (e.g., a mouse click in the case of a user input device provided as a computer mouse, or a tap in the case of a user input device provided as a touchpad). The feature designation command can also be provided by the user operating another user input device, such as pressing a key on a computer keyboard. Alternatively, the feature designation command can be generated by the feature designation module 110 in response to the fulfillment of predetermined conditions, such as the expiration of a predetermined time period initiated by displaying an instruction on the display 14 instructing the user to designate features of interest in the displayed fundus image 10.

[0052] The feature designation module 110 does not require such human interaction to designate feature 12, and in alternative example embodiments, the designation can be performed automatically using one of a variety of feature extraction algorithms known to those skilled in the art, such as “Automated detection of age-related macular degeneration in color fundus photography: a systematic review” by Pead, E., Megaw, R., Cameron, J., Fleming, A., Dhillon, B., Trucco, E. and MacGillivray, T., published in Survey of Ophthalmology, 64(4), 2019, pp. 498-511, or “A review on exudates detection methods for diabetic macular degeneration” by Joshi, S. and Karule, PT, published in Biomedicine & Pharmacotherapy, 97, 2018, pp. 1454-1460. "retinopathy", or as described in "A review on recent developments for detection of diabeticretinopathy" published in Scientifica in 2016 by Amin, J., Sharif, M. and Yasmin, M.

[0053] The device 100 also includes a receiver module 120, which is arranged to receive a C-scan of a portion of the retina acquired by an OCT imaging system (not shown). Figure 3 Optical OCT data D (shown in Figure 20) OCTThe device 100 also includes a selection module 130, which is arranged to select OCT data D. OCT subset d OCT The subset d OCT This represents a volumetric image of a portion of the retina at a location corresponding to the specified feature 12 in fundus image 10 (which is composed of...). Figure 3 The definition of voxels in scan set 22 (as in the example). Figure 3 As illustrated schematically, C scan 20 consists of a series of B scans 24, where each B scan 24 consists of a series of A scans 26. The value of each data element in A scan 26 provides an indication of the eye's measured reflectance at a corresponding location in the depth direction of the retina, which corresponds to along A scan 26 (i.e., along...). Figure 3 The position of the data element (in the z-axis direction) in scan A26. The position of scan A26 along the y-axis and x-axis directions of the data element array forming scan C20 corresponds to the position on the retina of the measurement performed by scan A26 acquired by the OCT imaging system.

[0054] Used to obtain OCT data D OCTThe OCT imaging system can be of any type known to those skilled in the art, such as a point-scan OCT imaging system, which acquires OCT images by scanning the eye region laterally with a laser beam. Alternatively, the OCT imaging system can be a parallel acquisition OCT imaging system, such as full-field OCT (FF-OCT) or line-field OCT (LF-OCT), which provides a high A-scan acquisition rate (up to tens of MHz) by illuminating a region or line on the sample instead of scanning a single point on the eye. In FF-OCT, a two-dimensional region of the eye is simultaneously illuminated, and a photodetector array (such as a high-speed charge-coupled device (CCD) camera) simultaneously captures the lateral position across that region. In the case of a full-field OCT imaging system, it can take the form of, for example, full-field time-domain OCT (FF-TD-OCT) or full-field swept-source OCT (FF-SS-OCT). In FF-TD-OCT, the optical length of the reference arm can vary during scanning to image regions at different depths within the eye. Therefore, each frame captured by the high-speed camera in FF-TD-OCT corresponds to an eye slice at a corresponding depth within the eye. In FF-SS-OCT, the sample region is illuminated across the entire field using a swept-frequency source that emits light with wavelengths varying over time. When the wavelength of the swept-frequency source is swept within the range of light wavelengths, the high-speed camera can generate a spectrum for each camera pixel that correlates reflectance information with light wavelength. Therefore, each frame captured by the camera corresponds to the reflectance information of a single wavelength of the swept-frequency source. When acquiring a frame for each wavelength of the swept-frequency source, the C-scan of that region can be obtained by performing a Fourier transform on the spectrum sample generated by the camera. In Line Field OCT (LF-OCT), a line of illumination can be provided to the sample, and the B-scan can be acquired during imaging. For example, Line Field OCT can be classified as Line Field Temporal Domain OCT (LF-TD-OCT), Line Field Sweeping Source OCT (LF-SS-OCT), or Line Field Spectral Domain OCT (LF-SD-OCT).

[0055] The OCT imaging system for acquiring OCT data and the fundus imaging device for acquiring fundus images can be separate devices. However, it should be noted that fundus images and OCT data can be acquired using a single multimodal retinal imaging system, such as the Silverstone system manufactured by Optos plc, which combines a UWF retinal imaging device and a UWF-guided sweep source OCT scanner.

[0056] As in this example embodiment, the received OCT data D OCT subset d OCTThe selection module 130 can make a selection based on input from a user who has reviewed the OCT data D displayed on the monitor 14. OCT The two-dimensional representation of the OCT data D was determined. OCT The position in the representation corresponds to the position of the specified feature 12 in the reference image 10 also displayed on the display 14, and the guide cursor 16 overlays the OCT data D. OCT The position is determined in the representation.

[0057] More specifically, in this example embodiment, the selection module 130 processes the received OCT data D. OCT This process generates an OCT frontal image of that portion of the retina. The OCT frontal image 30 is a projection of the (3D) C-scan 20 image onto the same 2D plane that will be observed in the fundus image 10 of the same portion of the retina. The generation of the OCT frontal image 30 may include summing, weighting, or maximizing the data elements (voxels) of the C-scan along the depth axis (z) of the OCT C-scan 20.

[0058] The selection module 130 causes the OCT frontal image 30 and the fundus image 10 to be displayed together on the display 14, for example... Figure 2 As shown next to the fundus image 10, the cursor 16 can be controlled by a signal from the user input device 18 to move over the displayed OCT frontal image 30. Although the display 14 is provided in the form of a single visual display unit in this example embodiment, the display 14 may alternatively be provided in the form of a first monitor (or other visual display unit) and a second monitor (or other visual display unit, not necessarily of the same kind as the first visual display unit), which are arranged to simultaneously display the fundus image 10 and the OCT frontal image 30 to the user, respectively.

[0059] In this example embodiment, in response to a feature specification command of the type described above, the selection module 130 selects OCT data D based on the value of the second position indicator. OCT subset d OCTThe second position indicator indicates the display position of cursor 16 when cursor 16 has been guided by a signal from user input device 18 to cover a portion 32 of the displayed OCT frontal image 30, which portion 32 is determined by the user (based on a comparison of retinal features in the displayed images 10 and 30) to correspond to a specified feature 12 in the displayed fundus image 10. OCT data D can be selected based on the value of the second position indicator by mapping the value of the second position indicator to an A scan of OCT C scan 20 with corresponding (x, y) coordinates and selecting an A scan, or by selecting an A scan along with a predefined arrangement of adjacent (nearby) A scans (e.g., the m nearest neighboring A scans mapped to and selected in the xy plane of C scan 20, where m is an integer, preferably greater than 4, and more preferably greater than 8). OCT subset d OCT As OCT data D OCT subset d OCT .

[0060] However, it should be noted that OCT data D OCT subset d OCT Instead of being defined by a predefined arrangement of mapped and selected A-scans and adjacent A-scans, it can be defined instead by a set of A-scans surrounded by a contour in the xy-plane of C-scan 20, defined by a straight line segment linking the A-scans of C-scan 20, which has been selected based on the corresponding value of a second position indicator in response to multiple feature specification commands issued by the user as he / she moves the cursor 16 around the boundary of a feature of interest in the fundus image 10. Therefore, OCT data D OCT subset d OCT This can correspond to an area of ​​the OCT front image 30 that has a predefined shape (as in this example embodiment) or a user-defined shape.

[0061] Selection module 130 allows graphic 34 to be overlaid on the OCT frontal image 30 at the position indicated by the second position indicator. Graphic 34 may be the same as graphic 19, or may differ from graphic 19, for example, where different colors and / or shapes of graphic 34 provide better visibility on the OCT frontal image 30. Similarly, the appearance of cursor 16 when overlaying the displayed fundus image 10 may differ (in shape and / or color) from its appearance when overlaying the displayed frontal image 30.

[0062] In the alternative example embodiment described above, the feature designation module 110 automatically designates feature 12 in the fundus image 10 using a feature extraction algorithm, and the selection module 130 allows a feature location indicator (e.g., in the form of graphic 19) to be overlaid on the displayed fundus image 10 to indicate the location of the designated feature 12 in the fundus image 10. In this case, as described above, OCT data D can be selected based on the value of the second location indicator. OCT subset d OCT The second position indicator indicates the display position of the cursor 16 when the cursor 16 has been guided by a signal from the user input device 18 to cover a portion of the displayed OCT frontal image 30, the position of which is determined by the user (based on a comparison of retinal features in the displayed images 10 and 30) to correspond to the position in the fundus image 10 indicated by the feature position indicator.

[0063] However, it should be noted that the location of the specified feature 12 in fundus image 10 corresponds to that in OCT data D. OCT subset d in OCT The correspondence between the positions does not need to be determined by the user through examining the displayed fundus image 10 and the displayed OCT frontal image of the common part of the retina. In some example embodiments, based on the position of specified feature 12 in fundus image 10, the OCT data D can be automatically determined by using predetermined geometric transformations. OCT inner subset d OCT The predetermined geometric transformation maps the position in the fundus image 10 to the corresponding A-scan position in the OCT data, and can be applied by the selection module 130 to the position of feature 12 already specified by the feature extraction algorithm in the fundus image 10, in order to identify the corresponding A-scan position in the C-scan 20 of A-scan 26, which will be included in the subset d of the OCT data. OCT In (optionally, together with one or more adjacent A scans in C scan 20).

[0064] Geometric transformations can be determined using one of many different methods. For example, they can be determined by transforming the image data D defining the fundus image 10. F Geometric transformations are determined by registering the image data with the OCT frontal image 30 generated as defined above, without displaying the OCT frontal image 30 to the user. Any intensity- and / or feature-based registration algorithm known to those skilled in the art can be used to determine the geometric transformations, thereby determining their position in the fundus image 10 and in the OCT C scan 20 (by...). Figure 3A point-to-point correspondence is established between the lateral positions (defined along the x-axis and y-axis). It should be noted that this registration algorithm can operate directly on the two-dimensional dataset defining fundus image 10 and the three-dimensional dataset defining OCT C scan 20, rather than on the two-dimensional dataset defining fundus image 10 and the two-dimensional dataset of OCT frontal image 30 derived from the three-dimensional dataset of OCT C scan 20.

[0065] In some example embodiments, features in the fundus image 10 are specified by the feature specification module 110 and OCT data D is selected by the selection module 130. OCT subset d OCT The selection can be performed automatically (i.e., without any user input). Therefore, in such an example embodiment, the feature specification module 110 can use a feature extraction algorithm to specify the image data D. F The image data D of the fundus image 10 is defined as a portion of feature 12, and the selection module 130 can apply the above geometric transformation to the image data D of the fundus image 10. F Select the location of the specified portion of the OCT data D OCT subset d OCT In these example embodiments, the specification of feature 12 and a subset d of the OCT data are performed. OCT The option to not display the image data of the fundus image 10 to the user. F Or OCT data D OCT It allows for any representation and requires no user input.

[0066] Refer again Figure 1 The device 100 also includes a supplementary image data generation module 140-1, which is arranged to process OCT data D. OCT The selected subset d OCT To generate supplementary image data D SI As supplementary information, this supplementary image data D SI Indicates the selected subset d of the OCT data OCT The device 100 measures the change in reflectivity of the eye along the depth direction of the retina (i.e., the direction in which light enters the retina). The device 100 also includes a combined image data generation module 150, which is arranged to generate image data D... F With supplementary image data D SI Combining to generate combined image data D CI This makes the combined image data D CI The defined combined image provides OCT data at specified features. OCT The selected subset d OCTThe image data generation module 140-1 and the combined image data generation module 150 are described in more detail below.

[0067] Figure 4 This is a schematic diagram of programmable signal processing hardware 200, which can be configured to perform the operation of the apparatus 100 of the first example embodiment.

[0068] Programmable signal processing device 200 includes a communication interface (I / F) 210 for communicating with a fundus imaging device and an OCT imaging system (or with image data D capable of generating a fundus image 10). F OCT data of the subject's retina D OCT The aforementioned combined imaging system communicates with it to receive image data D from the fundus image 10. F and OCT data D OCT The signal processing apparatus 200 also includes a processor (e.g., a central processing unit, CPU) 220, working memory 230 (e.g., random access memory), and an instruction store 240 storing a computer program 245 comprising computer-readable instructions that, when executed by the processor 220, cause the processor 220 to perform various functions of the apparatus 100 described herein. The working memory 230 stores information used by the processor 220 during execution of the computer program 245. The instruction store 240 may include a ROM preloaded with computer-readable instructions (e.g., in the form of electrically erasable programmable read-only memory (EEPROM) or flash memory). Alternatively, the instruction store 240 may include RAM or a similar type of memory, and the computer-readable instructions of the computer program 245 may be input to the instruction store 240 from a computer program product (such as a non-transitory computer-readable storage medium 250 in the form of a CD-ROM, DVD-ROM, etc., or a computer-readable signal 260 carrying computer-readable instructions). In any case, when executed by processor 220, computer program 245 causes processor 220 to perform methods for processing image data to include the supplementary information described herein. However, it should be noted that, optionally, apparatus 100 may be implemented in non-programmable hardware, such as in application-specific integrated circuits (ASICs).

[0069] Figure 5 This is a flowchart illustrating a method, through which... Figure 1 The device 100 processes image data D defining the fundus image 10. F This includes supplementary information about specified features 12 in the fundus image 10.

[0070] exist Figure 5In process S10, as described above, the feature designation module 110 designates the image data D received from the fundus imaging device. F Feature 12 in fundus image 10, which defines a portion of the retina.

[0071] exist Figure 5 In process S20, as described above, receiver module 120 receives OCT data D from an OCT C scan of that portion of the retina. OCT Although process S20 is shown as in Figure 5 The process S10 occurs after the process in the middle, but it should be understood that the order in which these processes are executed can be reversed, and at least some of these processes can be executed simultaneously.

[0072] exist Figure 5 In process S30, as described above, the selection module 130 selects the received OCT data D. OCT subset d OCT The subset d OCT A volumetric image representing a portion of the retina at a location corresponding to the location of a specified feature 12 in the fundus image 10.

[0073] exist Figure 5 In process S40, the supplementary image data generation module 140-1 processes the OCT data D. OCT The selected subset d OCT To generate supplementary image data D SI As supplementary information, this supplementary image data D SI Indication in OCT data D OCT The selected subset d OCT The variation of reflectance of the eye along the depth direction of the retina is measured. In other words, supplementary image data D SI Indication such as from OCT data D OCT The selected subset d OCT The voxels in the image indicate how the eye's reflectance varies along the depth direction of the retina. (Supplementary image data D) SI It can plot data such as OCT data D. OCT The selected subset d OCT The voxels in the image indicate the variation of the eye's reflectance with position along the depth direction of the retina. (Supplementary image data D) SI Therefore, a profile of reflectance distribution along the depth direction of the retina can be provided.

[0074] In this example embodiment, the supplementary image data generation module 140-1, by now referring to Figure 6 The described method processes OCT data D OCT The selected subset dOCT To generate supplementary image data D SI .

[0075] exist Figure 6 In process S42, the supplementary image data generation module 140-1 generates OCT data D OCT The selected subset d OCT Multiple anatomical layers are detected in the OCT image. Anatomical layers are different anatomically distinct structures of the eye that overlap with each other and can be distinguished on the depth axis of the OCT image due to their different light diffusion properties. Anatomical layers include those present in the posterior segment of the eye, including one or more layers of the retina. Each layer has an inner surface and an outer surface (relative to the vitreous body of the eye). The retina can be divided into 10 layers, namely: (1) internal limiting membrane (ILM); (2) nerve fiber layer (NFL); (3) ganglion cell layer (GCL); (4) inner plexiform layer (IPL); (5) nuclear layer (INL); (6) outer plexiform layer (OPL); (7) outer nuclear layer (ONL); (8) external lateral membrane (OLM); ​​(9) photosensitive layer (PL); and (10) retinal pigment epithelium (RPE) monolayer. The supplementary image data generation module 140-1 can detect OCT data D using one of the various types of algorithms known to those skilled in the art for ocular layer segmentation. OCT The selected subset d OCT The anatomical layer in the data is reviewed in R. Kafieh et al., “A Review of Algorithms for Segmentation of Optical Coherence Tomography from Retina”, J Med Signals Sens, Jan–Mar 2013; 3(1):45–60. Therefore, the anatomical layer segmentation algorithm is used in process S42 to generate layer identification information, which identifies the OCT data D. OCT The selected subset d OCT The boundaries of the anatomical layers detected (i.e., the inner and outer surfaces) can be used to identify OCT data belonging to each detected anatomical layer. OCT The selected subset d OCT The corresponding set of data elements (voxels) in the data.

[0076] exist Figure 6 In process S44, the supplementary image data generation module 140-1 generates a subset of OCT data d from the corresponding anatomical layer. OCTThe values ​​of the data elements (voxels) in the data are summed to calculate the corresponding sum for each detected anatomical layer. In other words, the supplementary image data generation module 140-1 calculates the corresponding sum for each detected anatomical layer by summing the values ​​of the OCT data D. OCT subset d OCT Positioning along A scan to be located in anatomical layer L i The voxel values ​​between the inner and outer surfaces are summed to obtain the values ​​for the n detected anatomical layers L1...L1 that have been identified using anatomical layer identification information. n Each anatomical layer L i Generate the sum value S i , where n is an integer greater than or equal to 2.

[0077] exist Figure 6 In process S46, the supplementary image data generation module 140-1 generates the detected anatomical layers L1...L n Each anatomical layer L i Calculated as anatomical layer L i The calculated sum S i With the detected anatomical layers and OCT data D OCT subset d OCT The sum of all data elements S 总 (Right now The corresponding ratio r between ) i .

[0078] exist Figure 6 In process S48, the supplementary image data generation module 140-1 generates data based on the calculated ratio r1 to r n The ordered sequence generates color information (as supplementary image data D). SI ), where the calculated ratios r1 to r n Arranged in the same order as the corresponding anatomical layers in the posterior segment of the eye. Color information is generated to define the colors to be displayed in the combined image, uniquely identifying the calculated ratios r1 to r2. n An ordered sequence. Therefore, at the location of specified feature 12, the color displayed in the composite image indicates the OCT data D. OCT The selected subset d OCT The variation of the eye's reflectance along the depth direction of the retina was measured.

[0079] It should be noted that Figure 6 Processes S44 to S48 do not require processing of OCT data D. OCT The selected subset d OCT In Figure 6 Instead of operating on all the anatomical layers detected in process S42, it is possible to operate on at least two of these anatomical layers. Figure 6Processes S44 to S48 operate on one or more detected anatomical layers that form part of the retina, and may also operate on one or more anatomical layers that do not form part of the retina (e.g., subretinal layers including the choroid and pathological subretinal fluid).

[0080] For example, in the current example embodiment, in Figure 6 In process S42, the supplementary image data generation module 140-1 detects the first anatomical layer L1, the second anatomical layer L2, and the third anatomical layer L3. Figure 6 In process S44, the supplementary image data generation module 140-1 generates the corresponding sum values ​​S1, S2, and S3 of the anatomical layers L1, L2, and L3, and (in Figure 6 In process S46, the corresponding ratios r1, r2, and r3 of the anatomical layers L1, L2, and L3 are calculated. For example, r1 is calculated as S1 / (S1+S2+S3). In this example... Figure 6 In process S48, the supplementary image data generation module 140-1 assigns a corresponding weight w to each of the red component R, green component G, and blue component B of the color to be displayed in the combined image based on the corresponding one of the calculated ratios r1, r2, or r3 in the ordered sequence r1; r2; r3. R w G or w B This is used to generate color information. For example, in which... Figure 6 In the example where the ratios calculated in process S46 are r1 = 0.1, r2 = 0.85, and r3 = 0.05, it indicates that the anatomical layer L2 corresponds to the OCT data D. OCT subset d OCT The data element values ​​in the image represent the measured reflectance, which makes a major contribution. Assigning corresponding weights of 0.1:0.85:0.05 to the RGB color components results in the color to be predominantly green in the combined image. This indicates to the observer of the combined image that anatomical layer L2 makes a major contribution to the measured reflectance of specified feature 12. In another example, displaying yellow in the combined image would indicate that anatomical layers L1 and L2 make major contributions to the measured reflectance. In yet another example, displaying cyan in the combined image would indicate that layers L2 and L3 make major contributions to the measured reflectance.

[0081] Refer again Figure 5 In process S50, the combined image data generation module 150 generates image data D by... F With supplementary image data D SI Combining to generate combined image data D CI This makes the combined image data D CIThe defined composite image provides an indication of changes at specified feature 12. The composite image data generation module 150 can be arranged to generate the composite image data using supplementary image data D. SI Replace fundus image data with pixel values ​​D F The pixel values ​​of a subset of pixels are used to generate combined image data D. CI This makes the combined image data D CI The defined combined image provides OCT data D at specified feature 12. OCT The selected subset d OCT The image data generation module 150 can be arranged to generate a composite image data by measuring changes in the reflectance of the eye. In an example embodiment similar to this example embodiment, the composite image data generation module 150 can be arranged to generate a composite image data by using supplementary image data D. SI Replace fundus image data with pixel values ​​D F The pixel values ​​of a subset of pixels are used to generate combined image data D. CI This subset of pixels defines a subregion of the fundus image located at the position of a specified feature 12 in the fundus image 10, such that the combined image data D CI The defined combined image provides OCT data D at specified feature 12. OCT The selected subset d OCT An indication of changes in the reflectivity of the eye as measured. More specifically, as in this example embodiment, the combined image data generation module 150 can modify image data D... F This makes the fundus image 10 located in the same position as the OCT frontal image 30, showing data from the OCT data D. OCT The selected subset d OCT The pixel value at the corresponding pixel position of the data is represented by the supplementary image data generation module 140-1 as OCT data D. OCT subset d OCT The pixel values ​​of a given color are replaced to generate combined image data D. CI The aforementioned correspondence between the pixel positions in the fundus image 10 and the OCT frontal image 30 can be determined through the geometric transformations described above.

[0082] Figure 7 Image A in the image is an example of fundus image 10, which shows that in Figure 7 Several features of interest are highlighted in image B. The boundary curves of the features of interest shown in image B are based on the output of the feature extraction algorithm. Figure 7Image C is an example of composite image 40, in which the regions of fundus image 10 highlighted in image B are colored according to color information generated by supplementary image data generation module 140-1. In image C, regions 41, 42, and 43 are predominantly blue, regions 44 and 45 are predominantly red, region 46 is predominantly white, and the remaining regions are predominantly green.

[0083] These colored indication areas 41-43 are associated with retinal features closest to the (inner) retinal surface (particularly within the inner and outer neuroretina), which is the first anatomical layer detected in this case. Figure 8 It shows along Figure 7 The horizontal line in image C is used to crop scan image B. Corresponding to... Figure 7 The mid-horizontal line passes through areas 41 and 42. Figure 8 The retinal region in the middle is near the inner surface of the retina (facing) Figure 8 (top) than Figure 8 The peripheral region of the midretina has a higher reflectivity.

[0084] exist Figure 7 In the middle, the predominantly green areas are associated with features of the deeper retinal layer detected in this example, which is the retinal pigment epithelium (RPE) in this case. Regions 44 and 45 involve even deeper features furthest from the retinal surface and are located in the choroid or pathological subretinal fluid, which were detected in the third anatomical layer in this example.

[0085] Second Example Implementation

[0086] Figure 9 This is a schematic diagram of an apparatus 300 for processing image data according to a second exemplary embodiment. The apparatus 300 of the second exemplary embodiment differs from the apparatus 100 of the first exemplary embodiment in the arrangement of the supplementary image data generation module 140-2, which is arranged to generate different types of supplementary image data D'. SI In all other respects, the apparatus 300 of the second example embodiment is identical to the apparatus 100 of the first example embodiment as described above.

[0087] In this example embodiment, the supplementary image data generation module 140-2 processes the OCT data D. OCT The selected subset d OCT To generate supplementary image data D' through a method SI This method is referenced above. Figure 6 Variations of the described method, and will now be referenced. Figure 10 Describe the method.

[0088] Figure 10 In processes S42 and S44 and Figure 6 The processes S42 and S44 are the same, so they will not be described here.

[0089] exist Figure 10 In process S47, the supplementary image data generation module 140-2 selects detection layers L1 to L2 based on the sum value calculated in process S44. n An anatomical layer (e.g., the retina) that makes a major contribution to the reflectance measured in the detection layer where the sum has already been calculated. As in this example embodiment, the supplementary image data generation module 140-2 can select the anatomical layer that makes the major contribution by selecting the layer with the largest calculated sum among the detection layers where the sum has already been calculated.

[0090] exist Figure 10 In process S49, the supplementary image data generation module 140-2 generates graphic image data as supplementary image data D'. SI The graphic image data defines a graphic that identifies the selected anatomical layer, for example, by naming the layer with a text label (e.g., in full or abbreviated form). However, the graphic does not require such a separate identifier for the selected anatomical layer and may include, for example, a pattern that an observer can use to identify the selected anatomical layer by referring to a legend also displayed on the screen (which shows the association between different forms of the graphic (e.g., patterns) and the corresponding named anatomical layers).

[0091] The combined image data generation module 150 is arranged to generate image data D F Combined with graphic image data to generate combined image data D' CI This makes the combined image an annotated version of the fundus image 10, wherein graphics are overlaid on the fundus image 10 (and preferably shaped, for example, having pointing features such as arrows) to indicate the location of the designated feature 12. Figure 11 and Figure 12 An example of this annotated version of fundus image 10 is shown in the figure.

[0092] Therefore, in some example embodiments similar to this exemplary embodiment, the combined image data generation module 150 can be arranged to generate image data using supplementary image data D' SI The pixel values ​​of a subset of the fundus image data are replaced with the pixel values ​​of the subset of pixels in the fundus image data to generate the combined image data D'. CI This makes the combined image data D' CIThe defined composite image provides an indication of the variation in the measured reflectance of the eye at a selected subset of the OCT data at specified feature 12. The supplementary image data generation module 140-2 can be arranged to generate supplementary image data that defines a graph indicating the variation in the measured reflectance of the eye at a selected subset of the OCT data along the depth direction of the retina, or more specifically, a graph indicating how the measured reflectance of the eye at a selected subset of the OCT data varies along the depth direction of the retina. In an example embodiment similar to this example embodiment, the composite image data generation module 150 can be arranged to generate supplementary image data D' SI The pixel values ​​of a subset of the fundus image data are replaced with the pixel values ​​of the subset of pixels in the fundus image data to generate the combined image data D'. CI This makes the combined image data D' CI The defined composite image includes a graph overlaid on the fundus image 10 to provide an indication of changes in the reflectance of the eye within a selected subset of the OCT data at specified features 12.

[0093] exist Figure 11 In the image, the label "INL" is overlaid on the fundus image to indicate the location of features in the inner retinal layer (INL), while the label "GCL" is overlaid to indicate the location of features in the ganglion cell layer (GCL). The label "None" is overlaid on the fundus image to indicate that the relevant feature in the fundus image is not located in any layer of the retina.

[0094] exist Figure 12 In the image, the label “RPE” is overlaid on the fundus image to indicate the location of features in the retinal pigment epithelium (RPE) located in the retina, while the label “ILM” is overlaid on the fundus image to indicate the location of features in the internal limiting membrane (ILM) located between the retina and the vitreous body.

[0095] Third Example Implementation

[0096] Figure 13 This is a schematic diagram of an apparatus 400 for processing image data according to a third exemplary embodiment. The apparatus 400 of the third exemplary embodiment differs from the apparatuses of the first and second exemplary embodiments in the arrangement of the supplementary image data generation module 140-3, which is arranged to generate the supplementary information described in those exemplary embodiments in a different manner. In all other respects, the apparatus 400 of the third exemplary embodiment is identical to the apparatus 100 of the first exemplary embodiment and the apparatus 300 of the second exemplary embodiment as described above.

[0097] In the current example embodiment, OCT data DOCT The selected subset d OCT A volumetric image representing a portion of the retina (possibly other than, for example, a portion beneath the outer surface of the retina) having predetermined type characteristics, and processed by a supplementary image data generation module 140-3, to be used by now referencing Figure 14 The described method generates supplementary image data D SI .

[0098] exist Figure 14 In process S41, a model for determining the depth of a predetermined type of feature in the depth direction of the retina is trained through supervised learning of examples of OCT data from at least one other pathological region of the retina. Each of these examples of OCT data includes a single OCT A scan or two or more adjacent OCT A scans, and each of the pathological regions has a corresponding feature of a predetermined type. When the model is trained, an indication of the corresponding depth of the corresponding feature in the depth direction of the retina in each example of the OCT data is specified by the user. The location of the retinal layer in the A scan can also be provided to the model, and during training, given an A scan (or a set of adjacent A scans) and layers as input, the model can learn to output the depth of the feature.

[0099] The depth of a feature can be specified in several ways. Depth is typically defined relative to one or more retinal layers. For example, depth can be defined by the number of pixels, or as a linear measurement relative to one of the retinal layers (assuming the pixel size has been pre-estimated). The innermost or outermost retinal surface can provide a suitable reference point for such a measurement. Alternatively, depth can be defined relative to multiple retinal layers that have been identified automatically. In this case, depth can be indicated by the name of the layer in which the feature resides, and optionally, by the displacement of the feature relative to the inner or outer surface of the layer. Furthermore, depth can be indicated by the name of the layer in which the feature resides and the unitless normalized displacement of the feature relative to the inner and outer surfaces of the layer, for example, such that 0 indicates the inner surface of the layer and 1 indicates the outer surface.

[0100] exist Figure 14 In process S43, the supplementary image data generation module 140-3 processes the OCT data D using a model that has already been trained in process S41. OCT The selected subset d OCT This is to determine the depth of the feature in the depth direction of the retina.

[0101] exist Figure 14In process S45, the supplementary image data generation module 140-3 generates either (i) graphic image data or (ii) color information as supplementary image data. The graphic image data defines a graphic that indicates a certain depth of a feature and will be overlaid on the fundus image 10 to indicate the position of the feature in the combined image 40. The color information defines a color that will be displayed at the position of the feature in the combined image and indicates a certain depth of the feature. The graphic image data and color information can be generated as described above.

[0102] The apparatus described below, E1 to E12, is illustrated with reference to the exemplary embodiments herein.

[0103] E1. An apparatus 100 for processing image data of a fundus image 10 defining a portion of the retina of the eye to include supplementary information about specified features 12 in the fundus image 10, the apparatus comprising:

[0104] Feature specification module 110 is arranged to specify features in fundus image 10;

[0105] Receiver module 120, which is arranged to receive optical coherence tomography (OCT) data D of the C-scan 20 of this portion of the retina. OCT ;

[0106] Select module 130, which is arranged to select OCT data D OCT subset d OCT The subset d OCT A volumetric image of a portion of the retina at a location corresponding to the location of a specified feature 12 in the fundus image 10;

[0107] Supplementary image data generation module 140-1, which is arranged to process OCT data D OCT The selected subset d OCT To generate supplementary image data D SI As supplementary information, this supplementary image data D SI Indication in OCT data D OCT The selected subset d OCT The variation of the eye's reflectance along the depth direction of the retina; and

[0108] Combined image data generation module 150, which is arranged to generate image data D F With supplementary image data D SI Combining to generate combined image data D CI This makes the combined image data D CI The defined composite image provides an indication of the changes at 12 specified features.

[0109] E2. The apparatus according to E1, wherein the feature designation module 110 is arranged to display a fundus image 10 and a cursor 16 on a display 14, such that the cursor 16 can be controlled by a signal from a user input device 18 to move on the displayed fundus image 10, and to designate a feature 12 in the fundus image 10 by recording the value of a first position indicator in response to a feature designation command, the first position indicator indicating the display position of the cursor 16 on the displayed fundus image 10.

[0110] E3. The apparatus according to E2, wherein the selection module 130 is arranged to process OCT data D OCT An OCT frontal image 30 of that portion of the retina is generated and displayed on the display 14 together with the fundus image 10. The cursor 16 can be controlled to move over the displayed OCT frontal image 30 via a signal from the user input device 18, and OCT data D can be selected based on the value of the second position indicator. OCT subset d OCT The second position indicator indicates the display position of the cursor 16 when the cursor 16 has been guided by a signal from the user input device 18 to cover a portion of the displayed frontal image 30 that corresponds to a specified feature 12 in the displayed fundus image 10.

[0111] E4. The apparatus according to E1, wherein the feature designation module 110 is arranged to automatically designate features 12 in the fundus image 10 using a feature extraction algorithm.

[0112] E5. The apparatus according to E4, wherein the selection module 130 is arranged as follows:

[0113] The fundus image 10 and the feature position indicator 19, which indicates the position of the specified feature 12 in the fundus image 10, are displayed on the display 14.

[0114] Processing OCT data D OCT To generate an OCT frontal image 30 of that portion of the retina;

[0115] The OCT frontal image 30 and cursor 16 are displayed on the display 14 together with the fundus image 10, so that the cursor 16 can be controlled to move on the displayed OCT frontal image 30 via signals from the user input device 18; and

[0116] Select OCT data D based on the value of the second position indicator. OCT subset d OCTThe second position indicator indicates the display position of the cursor 16 when the cursor 16 is guided by a signal from the user input device 18 to cover a portion of the displayed OCT frontal image 30, the position of the portion of the displayed OCT frontal image 30 corresponding to the position of the designated feature 12 indicated by the feature position indicator 19 in the fundus image 10.

[0117] E6. The apparatus according to E4, wherein the selection module 130 is arranged to select OCT data D by applying geometric transformations to the positions of features 12 already specified by the feature extraction algorithm in the fundus image 10. OCT subset d OCT This geometric transformation maps the location in fundus image 10 to OCT data D. OCT The corresponding A-scan position in the diagram.

[0118] E7. The apparatus according to any one of E1 to E6, wherein the feature 12 is one of the points and high-reflectivity points in the fundus image 10, the feature 12 having a pathological cause or being caused by reflection from the internal limiting membrane of the retina.

[0119] E8. The device according to E7, wherein the feature 12 has a pathological cause, which includes one of blood leakage, exudation, drusen, atrophy and / or nevus in the retina and atrophy of the retinal pigment epithelium in the retina.

[0120] E9. The apparatus according to any one of E1 to E8, wherein the supplementary image data generation module 140-1 is arranged to process OCT data D in such a way as to OCT The selected subset d OCT To generate supplementary image data D SI :

[0121] In OCT data D OCT The selected subset d OCT Multiple anatomical layers of the S42 eye were examined.

[0122] By analyzing OCT data from the anatomical layers... OCT subset d OCT The values ​​of the data elements are summed, and the corresponding sum value of S44 is calculated for each of at least two of the detected anatomical layers;

[0123] For each of at least two of the detected anatomical layers, S46 calculates the sum value calculated for that anatomical layer and the sum of the sums in the at least two detected anatomical layers in the OCT data D. OCT subset d OCT The corresponding ratios between the sums of all data elements; and

[0124] Based on the ordered sequence of the calculated ratios, S48 color information is generated as supplementary image data D. SI The calculated ratios are arranged in order of corresponding anatomical layers in the eye. Color information is defined and displayed in the composite image, identifying the colors of the ordered sequence of calculated ratios, such that the color indicates the color in the OCT data. OCT The selected subset d OCT The variation of the eye's reflectance along the depth direction of the retina was measured.

[0125] E10. The apparatus according to E9, wherein the supplementary image data generation module 140-1 is arranged as follows:

[0126] In OCT data D OCT The selected subset d OCT Three anatomical layers were detected in the middle; and

[0127] Color information is generated by assigning a corresponding weight to each of the red, green, and blue components of the color to be displayed in the composite image, based on the corresponding one of the calculated ratios in an ordered sequence of calculated ratios.

[0128] E11. The apparatus according to any one of E1 to E8, wherein the supplementary image data generation module 140-2 is arranged to process OCT data D in such a way as to OCT The selected subset d OCT To generate supplementary image data D SI :

[0129] In OCT data D OCT The selected subset d OCT The eye is examined in multiple anatomical layers.

[0130] By analyzing OCT data from the anatomical layers... OCT subset d OCT The values ​​of the data elements are summed to calculate the corresponding sum for each detected anatomical layer;

[0131] Based on the calculated sum, the anatomical layers that provide the main contribution to the reflectivity measurements of the eye among the tested anatomical layers are selected; and

[0132] Generate graphic image data that defines the selected anatomical layer as supplementary image data D. SI ,and

[0133] Among them, by using image data D F Combined with graphic image data to generate composite image data D CI This allows the graphic to overlay the fundus image 10 at a specified feature 12 in the combined image.

[0134] E12. The apparatus according to any one of E1 to E8, wherein the OCT data D OCT The selected subset d OCT This represents a volumetric image of a portion of the retina with features of a predetermined type, and the supplementary image data generation module 140-3 is arranged to generate supplementary image data D in the following manner. SI :

[0135] Supervised learning is conducted on examples of OCT data from at least one other pathological region of the retina to train a model for determining the depth of a predetermined type of feature in the retina. Each of the examples of OCT data includes a single OCT A scan or two or more adjacent OCT A scans, and each of the pathological regions has a corresponding feature of a predetermined type. The indication of the corresponding depth of the corresponding feature in the retina in each of the examples of OCT data is specified by the user during training.

[0136] Processing OCT data using a trained model D OCT The selected subset d OCT To determine the depth of features in the retina; and

[0137] Generate one of the following as supplementary image data:

[0138] Graphical image data, defined as a figure indicating the determined depth of features in the retina and to be overlaid on fundus image 10 to indicate the location of features in combined image 40, and

[0139] Color information is defined as the color that is displayed at the location of features in a composite image and indicates a specific depth of the features in the retina.

[0140] The example aspect described herein avoids the limitations associated with retinal fundus image processing, particularly those rooted in computer technology. Specifically, some common eye diseases exhibit similar appearances in fundus images, which can be difficult to distinguish from artifacts such as specular imaging. With the example aspect described herein, additional information from OCT data can be used to more clearly present features in fundus images, allowing users to fully understand them without reviewing the OCT data, and helping them avoid misinterpretations of artifacts such as specks in fundus images. Furthermore, by leveraging the aforementioned capabilities of the example aspect described herein, which is rooted in computer technology, the example aspect described herein improves upon computers and computer processing / functionality, and also improves at least the field of retinal image analysis.

[0141] In the foregoing description, exemplary aspects have been described with reference to several exemplary embodiments. Therefore, the specification should be considered illustrative rather than restrictive. Similarly, the accompanying drawings, which highlight the functionality and advantages of exemplary embodiments, are presented merely for illustrative purposes. The architecture of the exemplary embodiments is flexible and configurable enough that it can be utilized (and navigated) in ways other than those shown in the drawings.

[0142] In one exemplary embodiment, the software embodiments of the examples presented herein may be provided as computer programs or software, such as one or more programs having instructions or sequences of instructions included or stored in an article of art (e.g., a machine-accessible or machine-readable medium, an instruction store, or a computer-readable storage device, each of which may be non-transitory). Programs or instructions on non-transitory machine-accessible media, machine-readable media, instruction stores, or computer-readable storage devices can be used to program computer systems or other electronic devices. Machine or computer-readable media, instruction stores, and storage devices may include, but are not limited to, floppy disks, optical disks, and magneto-optical disks, or other types of media / machine-readable media / instruction stores / storage devices suitable for storing or transmitting electronic instructions. The techniques described herein are not limited to any particular software configuration. They may be applied in any computing or processing environment. As used herein, the terms “computer-readable,” “machine-accessible medium,” “machine-readable medium,” “instruction store,” and “computer-readable storage device” shall include any medium capable of storing, encoding, or transmitting instructions or sequences of instructions for execution by a machine, computer, or computer processor and causing the machine / computer / computer processor to perform any of the methods described herein. Furthermore, it is common in this art to refer to software as taking an action or causing a result in one or another form (e.g., program, procedure, process, application, module, unit, logic, etc.). This expression is merely a simplified way of stating that the processing system executes the software to cause the processor to perform actions to produce a result.

[0143] Some embodiments can also be implemented by preparing application-specific integrated circuits, field-programmable gate arrays, or by interconnecting appropriate networks of conventional component circuits.

[0144] Some embodiments include computer program products. A computer program product may be one or more storage media, instruction stores, or storage devices on or therein storing instructions that can be used to control or cause a computer or computer processor to perform any of the processes described in the example embodiments herein. Storage media / instruction stores / storage devices may, by way of example and without limitation, include optical discs, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory, flash memory cards, magnetic cards, optical cards, nanosystems, molecular memory integrated circuits, RAID, remote data storage / archiving / warehouse devices, and / or any other type of device suitable for storing instructions and / or data.

[0145] Some implementations stored on any of one or more computer-readable media, instruction stores (multiple instruction stores), or storage devices (multiple storage devices) include hardware for controlling a system and software for enabling a system or microprocessor to interact with a human user or other entity using the results of the exemplary embodiments described herein. Such software may, without limitation, include device drivers, operating systems, and user applications. Finally, as described above, such computer-readable media or storage devices also include software for performing the exemplary aspects described herein.

[0146] The system's programming and / or software includes software modules for implementing the processes described herein. In some example embodiments herein, the modules comprise software, but in other example embodiments herein, the modules comprise hardware or a combination of hardware and software.

[0147] While various exemplary embodiments have been described above, it should be understood that they are presented in an exemplary and not limiting manner. It will be apparent to those skilled in the art that various changes in form and detail may be made. Therefore, this disclosure should not be limited to any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

[0148] Furthermore, the purpose of this abstract is to enable patent offices and the general public, as well as scientists, engineers, and practitioners in the art who are particularly unfamiliar with patent or legal terminology or wording, to quickly determine the nature and essence of the technical disclosure of this application based on a cursory examination. The abstract is not intended to limit the scope of the exemplary embodiments presented herein in any way. It should also be understood that the processes recited in the claims need not be performed in the order presented.

[0149] While this specification contains numerous details of specific embodiments, these should not be construed as limiting, but rather as descriptions of features specific to the particular embodiments described herein. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as acting in a particular combination and even initially claimed in this way, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may be for a sub-combination or a variation thereof.

[0150] In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0151] Some illustrative embodiments and examples have now been described. It is clear that the foregoing embodiments are illustrative rather than limiting, and have been given by way of example. Specifically, although many examples presented herein relate to specific combinations of apparatus or software elements, these elements can be combined in other ways to achieve the same purpose. Actions, elements, and features discussed in connection with only one embodiment are not intended to exclude similar roles from those embodiments or other embodiments.

[0152] The apparatus and computer program described herein may be implemented in other specific forms without departing from their characteristics. The foregoing embodiments are illustrative and not intended to limit the systems and methods described. Therefore, the scope of the apparatus and computer program described herein is indicated by the appended claims rather than the foregoing description, and variations falling within the meaning and scope of the equivalents of the claims are included therein.

Claims

1. An image data (D) processing of a fundus image (10) defining a portion of the retina of the eye. F A computer-implemented method including supplementary information about specified features (12) in the fundus image (10), the method comprising: Specify (S10) a feature (12) at a location in the fundus image (10); Receive (S20) optical coherence tomography (OCT) data of C-scan (20) of said portion of the retina (D) OCT ) Select (S30) the OCT data (D) OCT subset of ) (d OCT ), the subset (d OCT ) represents a volumetric image of a portion of the retina at a location corresponding to the location of the specified feature (12) in the fundus image (10); Processing (S40) the OCT data (D OCT The selected subset (d) OCT To generate supplementary image data (D) SI As supplementary information, the supplementary image data (D) SI ) indicates in the OCT data (D OCT The selected subset (d) OCT How the reflectivity of the eye, as indicated in the diagram, varies along the depth direction of the retina; and By using the image data (D F ) and the supplementary image data (D SI (S50) Combined to generate (S50) combined image data (D) CI ), by using the supplementary image data (D) SI The pixel values ​​of the image data (D) are replaced with the pixel values ​​of the image data (D). F The pixel values ​​of a subset of pixels of the combined image data (D) make the combined image data (D) CI The combined image (40) defined by the given information provides the OCT data (D) at the specified feature (12). OCT The selected subset (d) OCT The measurement of the change in reflectivity of the eye as described in the document indicates this.

2. The computer-implemented method according to claim 1 further includes: The fundus image (10) and cursor (16) are displayed on the display (14), and the cursor (16) is controlled by a signal from the user input device (18) to move on the displayed fundus image (10). The feature (12) in the fundus image (10) is specified by recording the value of a first position indicator in response to a feature specification command. The first position indicator indicates the display position of the cursor (16) on the displayed fundus image (10).

3. The computer-implemented method according to claim 2 further includes: Processing the OCT data (D OCT ) to generate an OCT frontal image (30) of said portion of the retina; and The OCT frontal image (30) and the fundus image (10) are displayed together on the display (14), so that the cursor (16) can be controlled by a signal from the user input device (18) to move on the displayed OCT frontal image (30). The OCT data (D) is selected based on the value of the second position indicator. OCT subset of ) (d OCT The second position indicator indicates the display position of the cursor (16) when the cursor (16) has been guided by a signal from the user input device (18) to cover a portion of the displayed OCT frontal image (30) that corresponds to a specified feature (12) in the displayed fundus image (10).

4. The computer-implemented method according to claim 1, wherein, The features (12) in the fundus image (10) are automatically specified by the feature extraction algorithm.

5. The computer-implemented method according to claim 4, further comprising: The fundus image (10) and a feature position indicator (19) indicating the position of the specified feature (12) in the fundus image (10) are displayed on the display (14); Processing the OCT data (D OCT ) to generate an OCT frontal image (30) of said portion of the retina; and The OCT frontal image (30) and cursor (16) are displayed together with the fundus image (10) on the display (14), such that the cursor (16) can be controlled by a signal from the user input device (18) to move on the displayed OCT frontal image (30). The OCT data (D) is selected based on the value of the second position indicator. OCT subset of ) (d OCT The second position indicator indicates the display position of the cursor (16) when the cursor (16) has been guided by a signal from the user input device (18) to cover a portion of the displayed OCT frontal image (30), the position of the portion of the displayed OCT frontal image (30) corresponding to the position of the specified feature (12) indicated by the feature position indicator (19) in the fundus image (10).

6. The computer-implemented method according to claim 4, wherein, The OCT data (D) is selected by applying geometric transformations to the locations of features (12) already specified by the feature extraction algorithm in the fundus image (10). OCT subset of ) (d OCT The geometric transformation maps the location in the fundus image (10) to the OCT data (D). OCT The corresponding A scan position in ).

7. The computer-implemented method according to any of the preceding claims, wherein, The feature (12) is one of the points and high reflectivity points in the fundus image (10), and the feature (12) has a pathological cause or is caused by reflection from the internal limiting membrane of the retina.

8. The computer-implemented method according to claim 7, wherein, The feature (12) has a pathological cause, which includes one of the following: blood leakage, exudation, drusen, atrophy and / or nevus in the retina, and atrophy of the retinal pigment epithelium in the retina.

9. The computer-implemented method according to any of the preceding claims, wherein, The OCT data (D) is processed in the following manner. OCT The selected subset (d) OCT ) to generate the supplementary image data (D SI ): In the OCT data (D OCT The selected subset (d) OCT (S42) Detect multiple anatomical layers of the eye, the anatomical layers including one or more retinal layers; By analyzing the OCT data (D) in the anatomical layer OCT subset of ) (d OCT The values ​​of the data elements are summed to calculate the corresponding sum value for each of at least two of the detected anatomical layers (S44); For each of at least two of the detected anatomical layers, calculate (S46) the sum calculated for said anatomical layer and the sum of the sums in the at least two of the detected anatomical layers in the OCT data (D OCT subset of ) (d OCT The corresponding ratios between the sums of all data elements in ) and Based on the ordered sequence of the calculated ratios, color information (S48) is generated as the supplementary image data (D). SI The calculated ratios are arranged in the order of the corresponding anatomical layers of the eye, and the color information defines the colors that will be displayed in the combined image and identify the ordered sequence of the calculated ratios, such that the colors indicate the OCT data (D... OCT The selected subset (d) OCT The variation of the eye's reflectance along the depth direction of the retina, as described in the figure.

10. The computer-implemented method according to claim 9, wherein... In the OCT data (D OCT The selected subset (d) OCT The three anatomical layers (S42) were detected in the test, and The color information is generated (S48) by assigning a corresponding weight to each of the red, green, and blue components of the color to be displayed in the combined image according to the corresponding one of the calculated ratios in the ordered sequence of the calculated ratios.

11. The computer-implemented method according to any one of claims 1-8, wherein, The OCT data (D) is processed in the following manner. OCT The selected subset (d) OCT ) to generate the supplementary image data (D SI ): In the OCT data (D OCT The selected subset (d) OCT The eye is detected in multiple anatomical layers, including one or more retinal layers; By analyzing the OCT data (D) in the anatomical layer OCT subset of ) (d OCT The values ​​of the data elements are summed to calculate the corresponding sum for each anatomical layer in the detected anatomical layers; Based on the calculated sum, the anatomical layer that makes the major contribution to the measured reflectivity of the eye is selected from the detected anatomical layers; and Generate graphic image data that defines the selected anatomical layer as the supplementary image data (D). SI ).

12. The computer-implemented method according to any one of claims 1-8, wherein, The OCT data (D) OCT The selected subset (d) OCT The supplementary image data (D) represents a volumetric image of a portion of the retina having features of a predetermined type, and is processed in the following manner to generate the supplementary image data. SI ): Supervised learning is performed on examples of OCT data from at least one other pathological region of the retina to train a model for determining the depth of a feature of the predetermined type in the depth direction of the retina. Each of the examples of OCT data includes a single OCT A scan or two or more adjacent OCT A scans, and each of the pathological regions has a corresponding feature of the predetermined type. The indication of the corresponding depth of the corresponding feature in the depth direction of the retina in each of the examples of OCT data is specified by the user during the training. The OCT data (D) is processed using a trained model. OCT The selected subset (d) OCT ), to determine the depth of the feature in the depth direction of the retina; and Generate one of the following as the supplementary image data: Graphical image data, defined as a figure indicating the determined depth of the feature and to be overlaid on the fundus image (10) to indicate the position of the feature in the combined image (40), and Color information, which is defined to display and indicate the color of the feature at the location of the feature in the combined image at a determined depth.

13. A non-transitory computer-readable storage medium (250) storing a computer program (245) including computer program instructions, which, when executed by a computer processor (220), cause the computer processor (220) to perform the method according to any one of the preceding claims.

14. An image data (D) for processing fundus images (10) that define a portion of the retina of the eye. F A means (100) for including supplementary information about a specified feature (12) in the fundus image (10), the means comprising: A feature designation module (110) is arranged to designate a feature at a location in the fundus image (10); Receiver module (120), which is arranged to receive optical coherence tomography (OCT) data (D) of the C-scan (20) of said portion of the retina. OCT ) The selection module (130) is configured to select the OCT data (D OCT subset of ) (d OCT ), the subset (d OCT ) represents a volumetric image of a portion of the retina at a location corresponding to the location of the specified feature (12) in the fundus image (10); Supplementary image data generation module (140-1), which is arranged to process the OCT data (D OCT The selected subset (d) OCT To generate supplementary image data (D) SI As supplementary information, the supplementary image data (D) SI ) indicates in the OCT data (D OCT The selected subset (d) OCT How the reflectivity of the eye, as indicated in the diagram, varies along the depth direction of the retina; and Combined image data generation module (150), which is arranged to generate the image data (D) F ) and the supplementary image data (D SI Combining to generate combined image data (D) CI ), by using the supplementary image data (D) SI The pixel values ​​of the image data (D) are replaced with the pixel values ​​of the image data (D). F The pixel values ​​of a subset of pixels of the combined image data (D) make the combined image data (D) CI The combined image defined provides the OCT data (D) at the specified feature (12). OCT The selected subset (d) OCT The measurement of the change in reflectivity of the eye as described in the document indicates this.

Images