Noise reduction in retinal images

Abstract

To generate an image that holds a texture related to the retina from which noise is removed.SOLUTION: A computer mounting method which processes an image sequence in one area of the retina and generates an averaging image in the area includes: determining each offset between a reference image and each comparison image with respect to each set of a reference image selected from the image sequence and each of comparison images being the remaining images in the sequence; comparing each offset with threshold to determine whether the offset is smaller than the threshold; selecting each comparison image in each set whose offset is determined to be smaller than the threshold; and using the selected comparison image so as to generate an averaging image in the area. The threshold is determined so that an averaging image shows more texture characteristics in the area of the retina than a reference averaging image generated from the image in the image sequence.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Exemplary embodiments of this specification relate to techniques for suppressing noise in images of the retina of the eye, and more particularly to processing an image sequence of a region of the retina to generate a denoised image of that region. [Background technology]

[0002] Retinal imaging using various modalities is widely used to detect pathological and age-related physiological changes in the retina of the eye. In all cases, successful detection of such changes requires retinal images to be in focus, with as little noise and artifacts as possible. However, for patient safety, retinal imaging devices generally use low-power illumination sources, which can result in a lower signal-to-noise ratio (SNR) compared to imaging of other tissues. Furthermore, noise is inherent in some imaging techniques. For example, optical coherence tomography (OCT) images tend to contain speckle noise, which can reduce contrast and make it difficult to distinguish boundaries between strongly scattering structures. Fundus autofluorescence (FAF) images, which originate from the detection of low-intensity fluorescence, often have a low SNR due to a combination of photon noise and amplifier (readout) noise. [Overview of the project] [Problems that the invention aims to solve]

[0003] Noise can be suppressed by adapting the image capture process, for example, by using averaging techniques that record multiple image frames, align them relative to each other, and average the aligned images to produce a denoised image with a low level of noise averaged from the component images. Alternatively, noise can be suppressed by post-capture image processing techniques, such as using convolutional neural networks (CNNs) or other supervised machine learning algorithms. CNNs or other supervised machine learning algorithms are trained using averaged (denoised) images to remove noise from captured retinal images. However, both of these approaches tend to suppress or lose valuable low-contrast anatomical features, including retinal textures, in the final denoised image. Therefore, it is desirable to provide a technique that generates denoised retinal images while preserving retinal textures. [Means for solving the problem]

[0004] A first exemplary embodiment of this specification provides a computer implementation method for processing an image sequence of a region of the retina of an eye to generate an averaged image of the region. The computer implementation method includes: determining an offset between a reference image and each comparison image for each combination of a reference image selected from the image sequence and each comparison image which is an image from the rest of the sequence; comparing each determined offset to an offset threshold to determine if the offset is less than an offset threshold; selecting each comparison image in each combination where it is determined that the offset between the reference image and each comparison image is less than an offset threshold; and using the selected comparison images to generate an averaged image of the region. The offset threshold is defined such that, if the image sequence includes at least one image offset from the reference image by an offset greater than the threshold and images offset from the reference image by respective offsets less than the threshold, the averaged image exhibits more texture showing the structure of the first region of the retina than a reference averaged image generated from the images in the image sequence.

[0005] A second exemplary aspect of this specification provides a computer implementation method for training a machine learning algorithm for filtering noise from retinal images. The computer implementation method includes: generating ground truth training data by processing each sequence of a plurality of sequences of retinal images to generate each averaged retinal image, each averaged retinal image being generated according to the computer implementation method of the first exemplary aspect; generating training input data by selecting each image from each of the image sequences; and using the ground truth training data and training input data to train a machine learning algorithm for filtering noise from retinal images.

[0006] Furthermore, according to a third exemplary embodiment of this specification, a computer program is provided which, when executed by at least one processor, causes at least one processor to execute at least one method of the first exemplary embodiment or the second exemplary embodiment described above. The computer program may be stored on a non-temporary computer-readable storage medium (e.g., a computer hard disk or CD, etc.) or may be carried by computer-readable signals.

[0007] Furthermore, according to a fourth exemplary aspect of this specification, a data processing device is provided which is configured to process an image sequence of a region of the retina of an eye and generate an averaged image of the region. The data processing device includes at least one processor and at least one memory for storing computer-readable instructions. When executed by at least one processor, the computer-readable instructions cause at least one processor to determine the respective offset between a reference image selected from the image sequence and each comparison image which is an image from the rest of the sequence; to compare each determined offset with an offset threshold to determine if the offset is less than an offset threshold; to select the respective comparison image in each combination where it is determined that the respective offset between the reference image and each comparison image is less than an offset threshold; and to use the selected comparison image to generate an averaged image of the region. The offset threshold is defined such that, when the image sequence includes at least one image offset by a threshold greater than the reference image and images offset by smaller than the threshold for each of the reference images, the averaged image exhibits more texture showing the structure of the first region of the retina than the reference averaged image generated from the images in the image sequence.

[0008] Furthermore, according to a fifth exemplary embodiment of this specification, an ophthalmic imaging system is provided which includes an ophthalmic imaging device arranged to acquire an image sequence of a region of the retina of the eye, and a data processing device according to a fourth exemplary embodiment arranged to process the image sequence acquired by the ophthalmic imaging device to generate an averaged image of the retinal region.

[0009] Next, exemplary embodiments will be described in detail with reference to the attached drawings, as they are not limiting examples. Unless otherwise noted, similar reference numerals indicate identical or functionally similar elements in other drawings. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of an ophthalmic imaging system 100 including an ophthalmic imaging device 10 and a data processing device 60 according to an exemplary embodiment of this specification. [Figure 2] This is a schematic diagram of an exemplary process in which a data processing device 60 processes an image sequence 20 of a region of the retina of the eye and generates an averaged image 70 of that region. [Figure 3] This is a schematic diagram of an exemplary implementation of the data processing device 60 in the programmable signal processing hardware 300. [Figure 4] This flowchart illustrates a method according to a first exemplary embodiment, in which a data processing device 60 processes an image sequence of a region of the retina and generates an averaged image of that region. [Figure 5] An example of an FAF image is shown, along with a 1.3-pixel displacement at the edge of the image resulting from rotating it 0.3 degrees relative to its center. [Figure 6A] An example of an FAF image is shown. [Figure 6B] The averaged FAF image is shown, obtained by averaging the FAF image shown in Figure 6A with another FAF image rotated and offset by 4.5 degrees from the image in Figure 6A. [Figure 6C]The averaged FAF image is shown, obtained by averaging the FAF image shown in Figure 6A with another FAF image having a negligible rotational offset relative to the image in Figure 6A. [Figure 7] This is a schematic diagram of the segmentation of the second image sequence 15 for generating image sequence 20. [Figure 8] This flowchart illustrates a method according to a second exemplary embodiment, in which a data processing device 60 processes an image sequence of a region of the retina and generates an averaged image of that region. [Figure 9A] An exemplary FAF image acquired by the ophthalmic imaging device 10 is shown. [Figure 9B] An example of an averaged image 70 obtained by averaging 10 FAF images, including the FAF image in Figure 9A, selected from an FAF image sequence according to the method of the second exemplary embodiment is shown. [Figure 10] This flowchart illustrates a method according to a third exemplary embodiment, in which a data processing device 60 trains a machine learning algorithm to filter noise from retinal images. [Figure 11] In a third exemplary embodiment, this is a flowchart illustrating how the processor 320 generates each of the averaged retinal images from each sequence of retinal images. [Modes for carrying out the invention]

[0011] [First exemplary embodiment] Figure 1 is a schematic diagram of an ophthalmic imaging system 100 according to an exemplary embodiment, which includes an ophthalmic imaging device 10 positioned to acquire an image sequence 20 of a common region 30 of the retina 40 of an eye 50. The ophthalmic imaging system 100 further includes a data processing device 60 positioned to process images from the image sequence 20 to generate an averaged image 70 of the region 30 of the retina 40.

[0012] As schematically shown in Figure 2, the data processing device 60 is configured to generate averaged images 70 by first selecting images from the image sequence 20 according to one or more predefined criteria. These images are first selected from the image sequence 20 by reference image I Ref And each image from sequence 20 considered for selection (comparative image I in this specification) Comp For each combination of (referred to as), reference image I Ref and comparison image I Comp The selection may be made by determining the respective offsets between and . Each determined offset may be compared to an offset threshold to determine if the offset is less than the offset threshold. As an example, one or more predefined criteria may be used for comparison image I Comp and reference image I Ref The criterion may include that the magnitude of the determined translational offset t between and is less than a predetermined translational offset threshold T. Additionally or alternatively, one or more predefined criteria may be used for comparison image I. Comp and reference image I Ref The criterion may include that the magnitude of the determined rotational offset (i.e., relative rotation) Δφ between the selected comparison image 25 is smaller than a predetermined rotational offset threshold Θ. The selected comparison image 25 is aligned with respect to each other by the data processing device 60, which then calculates the average of the aligned images such that, for example, each pixel value of each pixel in the resulting averaged image 70 is the average (e.g., arithmetic mean) of the pixel values ​​of the pixels at the corresponding positions in the aligned selected image 25 that correspond to each common position in the region 30 of the retina 40. Alternatively, the data processing device 60 may calculate the average of the aligned images such that, for example, each pixel value of each pixel in the resulting averaged image 70 is the weighted average of the pixel values ​​of the pixels at the corresponding positions in the aligned selected image 25 that correspond to each common position in the region 30 of the retina 40.

[0013] In this process (an example of which will be described later), when the image sequence 20 includes at least one image offset by an offset greater than a threshold from the reference image I Ref and an image offset by each offset less than the threshold from the reference image I Ref , the resulting averaged image 70 is generated from some or all of the images of the image sequence 20 such that the images are selected, without being selected in the manner described herein by one or more predefined criteria, to show more texture than a reference averaged image 75 generated from the images of the image sequence 20. In this context, the texture is an anatomical texture related to the retina and shows the anatomical structure in the region 30 of the retina 40. For example, when the image acquired by the ophthalmic imaging device 10 is a fundus autofluorescence (FAF) image, as in this exemplary embodiment, the structure can be defined by the spatial distribution of the phosphors throughout the region 30 of the retina 40. As another example, when the image acquired by the ophthalmic imaging device 10 is an OCT image, the structure can include the physical structure of one or more layers of the retina 40 in the region 30 of the retina 40. As a further example, when the image acquired by the ophthalmic imaging device 10 is a reflectance image (e.g., a red, blue, or green reflectance image) of the region 30 of the retina 40, the structure may include the upper surface of the retina in the region 30 such that the texture is a physical texture of the surface reflecting the topography of the surface.

[0014] Furthermore, the averaged image 70 tends to have a higher SNR than a single image from the image sequence 20. This is because the selected images 25 are averaged, partially canceling out noise components within the selected images 25. Thus, the averaged image 70 produced by the data processing device 60 of the exemplary embodiment is a denoised image of region 30 of the retina 40, and retains more texture than a conventional denoised image (e.g., image 75) produced by averaging images in the sequence 20 that have not been selected according to one or more criteria described herein. Thus, the image processing techniques described herein can produce denoised and anatomically accurate retinal images in which important clinical data is preserved. Such denoised images are not only valuable to clinicians for assessing the health of the retina 40, but also valuable for training AI denoising algorithms that suppress noise while reducing or avoiding the texture loss in denoised images that often occurs when using conventionally trained artificial intelligence (AI) denoising algorithms, as will be described in detail below.

[0015] To suppress noise while preserving the texture of the averaged image 70, the processing of the image sequence 20 must consider the following classes of imperfections in the image acquisition process and their effects on the acquired image. Firstly, the reference image I Ref and comparison image I CompThese distortions may be associated with affine or non-affine transformations (e.g., translation, rotation, or distortion in one or more directions), which can result from retinal movement perpendicular to the imaging axis during image acquisition, rotation of the retina around the imaging axis, or retinal movement resulting from changes in the eye's line of sight. Secondly, acquired images may contain non-affine distortion, which can result from rapid eye movements such as horizontal saccades occurring during image capture. Thirdly, some images may contain one or more localized areas or patches of minute distortion (minute shifts), which can result from variations in the imaging system (e.g., beam scanning systems including polygon mirrors or other scanning elements) or from resulting optical path variations between different image captures. Optical path variations can result from deviations from scanning linearity and / or mirror imperfections, among other factors.

[0016] The ophthalmic imaging device 10 may be any type of FAF imaging device well known to those skilled in the art (e.g., a fundus camera, a confocal scanning laser ophthalmoscope (SLO), or an ultra-wide-field imaging device such as Optos®'s Daytona®), as in this exemplary embodiment, and is positioned to capture a sequence of FAF images of the same region of the retina 30. The FAF imaging device often uses short-to-mid-wavelength visible light excitation to collect emission within a predefined spectral band (e.g., within a wavelength range of 500-750 nm) to form a brightness map that reflects the distribution of lipofuscin, the dominant fluorescence present in the retinal pigment epithelium (RPE). However, other excitation wavelengths, such as near-infrared, may be used to detect additional fluorescent dyes such as melanin. FAF is useful for evaluating a variety of diseases involving the retina and RPE.

[0017] However, the form of the ophthalmic imaging device 10 is not so limited. In other exemplary embodiments, the ophthalmic imaging device 10 may take the form of an OCT imaging device such as a wavelength-swept OCT (SS-OCT) imaging device or a spectral-range OCT (SD-OCT) imaging device positioned to acquire OCT images of a region 30 of the retina 40, such as B-scans, C-scans, and / or En-face OCT images. In yet another exemplary embodiment, the ophthalmic imaging device 10 may include an SLO, fundus camera, etc., positioned to capture reflectance images (e.g., red, green, or blue images) of a region 30 of the retina 40.

[0018] Figure 1 shows that the ophthalmic imaging device 10 acquires a sequence of 10 images, but this number is merely an example. The number of images in the image sequence 20 is not limited to 10 and may be more or less. Also, note that region 30 is a portion of the retina 40 within the field of view (FoV) of the ophthalmic imaging device 10, and may be smaller than the portion of the retina 40 that extends into the FoV. As will be explained in more detail below, region 30 may be a larger segment of the retina 40 in which the ophthalmic imaging device 10 can be positioned to capture an image, with each image spanning the field of view of the ophthalmic imaging device 10. Therefore, each image in the image sequence 20 may be a segment of each larger retinal image (e.g., a rectangular image tile) of the retina 40 captured by the ophthalmic imaging device 10.

[0019] The data processing device 60 may be provided in any suitable form, such as programmable signal processing hardware 300 of the type schematically shown in Figure 3. The programmable signal processing device 300 includes a communication interface (I / F) 310 for receiving an image sequence 20 from an ophthalmic imaging device 10 and outputting an averaged image 70. The signal processing hardware 300 further includes a processor (e.g., a central processing unit, CPU and / or a graphics processing unit, GPU) 320, working memory 330 (e.g., random access memory), and an instruction store 340 that stores a computer program 345 containing computer-readable instructions that, when executed by the processor 320, cause the processor 320 to perform various functions of the data processing device 60 described herein. The working memory 330 stores reference image I Ref , comparison image I Comp1 ~I Comp9The instruction store 340 stores information used by the processor 320 during the execution of the computer program 345, such as the calculated image offset, image offset threshold, selected comparison image 25, determined similarity, similarity threshold, and the results of various intermediate processing described herein. The instruction store 340 may include ROM (for example, in the form of electrically erasable programmable read-only memory (EEPROM) or flash memory) in which computer-readable instructions are pre-stored. Alternatively, the instruction store 340 may include RAM or a similar type of memory, and the computer-readable instructions of the computer program 345 can be input from a non-temporary computer-readable storage medium 350 in the form of a CD-ROM, DVD-ROM, or a computer program product such as a computer-readable signal 360 that carries computer-readable instructions. In either case, when the computer program 345 is executed by the processor 320, it causes the processor 320 to perform the functions of the data processing device 60 described herein. Therefore, the data processing device 60 of an exemplary embodiment may include a computer processor 320 (or two or more similar processors) and a memory 340 (or two or more similar memories) that stores computer-readable instructions, which, when executed by the processor(s), cause the processor(s) to process an image sequence 20 to generate an averaged image 70 of a region 30 of the retina 40 as described herein.

[0020] However, it should be noted that the data processing device 60 may alternatively be implemented as non-programmable hardware such as an ASIC, FPGA, or other dedicated integrated circuit performing the functions of the data processing device 60 described herein, or as a combination of such non-programmable hardware and programmable hardware as described with reference to Figure 3. The data processing device 60 may be provided as a standalone product or as part of an ophthalmic imaging system 100.

[0021] Next, the process by which the data processing device 60 of this exemplary embodiment processes the image sequence 20 to generate an averaged image 70 of the retina region 30 will be described with reference to Figure 4.

[0022] In process S10 of Figure 4, the processor 320 of the data processing device 60 selects a reference image I from the image sequence 20. Ref And each comparison image I from the remaining images of image sequence 20. Comp For each combination, reference image I Ref and comparison image I Comp Determine the respective offsets between them. Each offset is the reference image I in the combination. Ref Comparison image I for each combination Comp Rotation and / or combination of reference image I Ref Comparison image I for each combination Comp It may include translation.

[0023] Reference Image I Ref The image sequence 20 can be selected in various different ways. Reference image I Ref This may be, for example, an image appearing at a predetermined position in the image sequence 20, for example, at the beginning, end, or preferably at an intermediate position in the image sequence 20 (for example, in the middle of the sequence). Alternatively, it may be a reference image I Ref The reference image I may be randomly selected from the images in the image sequence 20. However, in order to more reliably achieve as many selected images 25 as possible, and thereby make noise suppression in the averaged image 70 (derived from the selected images 25) more effective, Ref From the images in image sequence 20, the reference image I Ref The image is preferably selected such that it has the largest number (or is equal to) the remaining images in the sequence that are offset by an amount smaller than a predetermined offset threshold from the reference image I. RefFor example, a base image can be selected from the image sequence 20 to function as a base image by determining the respective translational offsets between the base image and each of the remaining images in the image sequence 20, for example, by determining the position of each peak in the cross-correlation calculated between the base image and each of the remaining images. The translational offsets thus determined can be represented as a set of points in a two-dimensional plot. For each point in this plot, the number of remaining points in a circle of radius T centered on that point is determined. Once this has been done for all points in the plot, the point with the largest (or equal) number of remaining points in a circle centered on that point is selected. Then, from the image sequence, the image represented by the point whose offset from the base image is selected is the reference image I. Ref It is selected as such.

[0024] In this way, the processor 320 determines the translational offset between each of the remaining images in sequence 20 (i.e., images other than the base image) and all other images in the sequence, compares the determined translational offset with an offset threshold T to identify all other images in the sequence whose translational offset from the image is less than the translational threshold T, and determines the count of each identified image by counting them. Next, the processor 320 uses the determined count to identify the image with the highest (or equal) determined count among the remaining images in sequence 20 as the base image I Ref Identify as such.

[0025] In the process S10 of Figure 4, as in this exemplary embodiment, the processor 320 processes the reference image I Ref and comparison image I from some or all of the remaining images of image sequence 20. Comp For each combination, reference image I Ref and comparison image I Comp The respective translational offsets between and , and reference image I Ref and comparison image IComp The respective rotational offsets between and may be determined. However, in another exemplary embodiment, the processor 320 determines the reference image I Ref and comparison image I from the remaining images of image sequence 20. Comp For each combination, reference image I Ref and comparison image I Comp Each translational offset between, or reference image I Ref and comparison image I Comp You may decide on one (but not both) of the rotational offsets between them.

[0026] Translational and rotational offsets between images can be determined using various techniques, such as those described herein or those well known to those skilled in the art. For example, if relative rotation between images in image sequence 20 is negligible, then reference image I Ref and each comparison image I Comp Each translation offset t between is the reference image I Ref Comparison Image I Comp This may be determined by calculating the cross-correlation between the two points, and the position of the peak in the calculated cross-correlation is used to determine the respective translational offset t. Due to the self-similarity of the images in the image sequence 20, a relatively broad peak may occur in the calculated cross-correlation; therefore, the images may be preprocessed (e.g., using an edge detector) before cross-correlation to sharpen the peak and allow for a more accurate determination of the translational offset t.

[0027] As an alternative, reference image I Ref and each comparison image I Comp Each translational offset t between is first the reference image I Ref Comparison Image I Comp This may also be determined by calculating the normalized cross-power spectrum of F Ref F Comp * / |F Ref F Comp * It may also be expressed as |, where F RefThis is reference image I Ref This is the discrete Fourier transform of F Comp * This is image I Comp This is the complex conjugate of the discrete Fourier transform of . Next, we calculate the inverse Fourier transform of the normalized cross-power spectrum and compare it to image I. Comp Reference image I Ref Shift the reference image I by Δx pixels in the x-axis direction. Ref This provides a delta function that has its maximum value at (Δx, Δy), which is shifted by Δy pixels in the y-axis direction.

[0028] The cross-correlation approach described above, as in this exemplary embodiment, uses reference image I Ref Comparison Image I Comp This can be extended to allow relative rotations between images in the image sequence 20 by determining the respective translation offset t between each of the multiple rotation versions of the reference image I. Ref Comparison Image I Comp This can be done by calculating the respective cross-correlation between each rotation version of [image]. For example, but not limited to, each comparison image I Comp The image is rotated in 0.1-degree increments between -6 and +6 degrees (regardless of whether the above preprocessing was performed), and each comparison image I Comp This generates 120 rotated versions of the original (unrotated) comparison image I. Comp (Together with) Reference Image I Ref It is cross-correlated with the following. The angle range of -6 to +6 degrees and the 0.1 degree increments are shown as examples, and other ranges and / or angle increments can also be used. Reference image I Ref The rotation (if any) applied to the cross-correlated image is taken to generate the cross-correlation with the highest peak among the calculated 121 cross-correlation peaks, relative to the reference image I. Ref Comparison Image I Comp An index of the rotational offset between them may be provided. On the other hand, the position of the highest cross-correlation peak is the reference image I Ref Comparison Image I Comp This may be taken to provide an index of the translational offset t between and .

[0029] Reference Image I Ref Comparison Image I Comp The rotational offset Δφ and translational offset t between the reference image I may alternatively be calculated in the Fourier domain. This approach relies on two properties of the Fourier transform. First, the frequency spectrum is always centered at the origin, regardless of the displacement of the background image. Second, the rotation of an image always results in a rotation corresponding to its Fourier transform. In another approach, the reference image I Ref and comparison image I Comp It is converted to polar coordinates. The image I(x,y) is converted to I(r,θ), and the discrete Fourier transform of the image F(u,v) becomes F(ρ,φ), and as a result,

[0030]

number

[0031] I Comp I Ref If an angle β is given, then I Comp (r,θ)=I Ref It can be seen that (r, θ+β). This means the following:

[0032]

number

[0033] The above formula is equivalent to the following formula.

[0034]

number

[0035] Therefore, the following can be said:

[0036]

number

[0037] These two characteristics can separate the rotational and translational components of image alignment and solve each independently. For a set of images that are shifted and rotated relative to each other, by taking their Fourier transforms, the rotational shift between the images can be separated. The magnitude of the frequency components generated by this Fourier transform is treated as if it were the pixel values of a new set of images that only require rotational alignment. Once the rotational correction is obtained, it is applied to the underlying image for alignment. Next, the translational parameters are determined using the same algorithm. Thus, for the reference image I Ref and each comparison image I Comp the respective rotational offset Δφ between them can be determined by calculating the inverse Fourier transform of the normalized cross-power spectrum of the polar transformation between the reference image I Ref and each comparison image I Comp and. Further details of this approach are described in the paper "Medical Image Registration Using the Fourier Transform" by J. Luce et al., International Journal of Medical Physics, Clinical Engineering and Radiation Oncology, 2024.3, pp49 - 55 (February 2014), the content of which is hereby incorporated by reference in its entirety into this specification.

[0038] In the process S10 of FIG. 4, to determine the offset between the reference image I Ref and the comparison image I Comp other intensity-based algorithms may alternatively be used, including those based on similarity measures other than cross-correlation, such as mutual information, sum of squared intensity differences, or ratio image uniformity (among others).

[0039] Referring again to FIG. 4, in process S20, the processor 320 compares each determined offset with an offset threshold value to determine whether the offset is smaller than the offset threshold value. The processor 320 may compare each translational offset t with a translational offset threshold value T to determine whether the translational offset t is smaller than the translational offset threshold value T, as in this exemplary embodiment. The processor 320 may compare each rotational offset Δφ with a rotational offset threshold value Θ to determine whether the rotational offset Δφ is smaller than the rotational offset threshold value Θ, as in this exemplary embodiment. In other exemplary embodiments, the reference image I Ref and each comparison image I Comp between the translational offsets, or the reference image I Ref and each comparison image I Comp between the rotational offsets, if any of them is determined by the processor 320 for each combination between the reference image I Ref and each comparison image I Comp from the remaining images of the image sequence 20 (in process S10 of FIG. 4), the processor 320 may optionally determine in process S20 of FIG. 4 whether either the translational offset t is smaller than the translational offset threshold value T or the rotational offset Δφ is smaller than the rotational offset threshold value Θ. The setting of the offset threshold values T and Θ will be described below.

[0040] In process S30 of FIG. 4, the processor 320 selects each comparison image I Ref in each combination in which it is determined that each offset between the reference image I Comp and each comparison image I Comp is smaller than the offset threshold value. For any combination in which it is determined that each offset between the reference image I Ref and each comparison image I​​​​​​​In the process S20 of Figure 4, as in this exemplary embodiment, the processor 320 determines that each translational offset t is smaller than the translational offset threshold T, and each rotational offset Δφ is smaller than the rotational offset threshold Θ, and for each combination, the comparison image I Comp You may choose to select reference image I Ref and comparison image I Comp Each translational offset between, or reference image I Ref and comparison image I Comp Any of the rotational offsets between the reference image I Ref and comparison image I from the remaining images of image sequence 20. Comp For each combination, when determined by the processor 320 (in processing S10 of Figure 4), the processor 320, in processing S20 of Figure 4, determines whether each translational offset t is smaller than the translational offset threshold T, or whether each rotational offset Δφ is smaller than the rotational offset threshold Θ, for each combination, the respective comparison image I Comp You may choose this option.

[0042] In the process S40 of Figure 4, the processor 320 generates an averaged image 70 of the retina 40 region 30 using the selected comparison image 25. The processor 320, as in this exemplary embodiment, uses the selected comparison image 25 (and optionally a reference image I Ref Using the selected image 25 (and optionally the reference image I), RefThe processor 320 generates an averaged image 70 of region 30 by aligning the images relative to each other, and then averages the aligned images such that, for example, each pixel value of the averaged image 70 is the average (e.g., arithmetic mean) of the pixel values ​​of the pixels at the corresponding positions in the aligned images corresponding to each common position in region 30 of the retina 40, thereby generating an averaged image 70 of the first region 30. Alternatively, the processor 320 may generate an averaged image 70 of the first region 30 by averaging the aligned images such that, for example, each pixel value of the averaged image 70 is the weighted average of the pixel values ​​of the pixels at the corresponding positions in the aligned selected images 25 corresponding to each common position in region 30 of the retina 40.

[0043] The offset thresholds T and Θ are set such that the image sequence 20 has a translational offset t greater than the translational offset threshold T, and / or a rotational offset Δφ greater than the rotational offset threshold Θ, relative to the reference image I Ref Image offset from, and reference image I Ref If the averaged image 70 includes at least one image whose respective translational and rotational offsets are less than T and Θ, respectively, then the averaged image 70 will show more texture indicating the structure of region 30 of the retina 40 than the reference averaged image 75 generated from the images in the image sequence 20.

[0044] The images compared in processing S20 of Figure 4 often differ due to non-affine transformations, which tend to produce deformations that increase as the offset between images (whether translational or rotational) increases, so the reference image I Ref Comparison image I, offset below the offset threshold. Comp By selecting this option, a higher correlation between the image contents and therefore an improved preservation of the texture of the averaged image 70 can be achieved. Any affine frame offset (typically translation and / or rotation) of the selected image 25 can be effectively corrected by the image alignment performed in process S40 of Figure 4 so that the aligned images are highly correlated.

[0045] The amount of texture in averaged image 70 can be quantified using an algorithm such as that described in the paper “Detection of Textured Areas in Images Using a Disorganization Indicator Based on Component Counts”, J. Electronic Imaging, 17.043003, 2008 by R. Bergman et al., which is incorporated herein by reference in full. The texture detector presented in this paper is based on the intuition that texture in natural images is “disorganized”. The measure used to detect texture examines the structure of local regions of the image. This structural approach enables the detection of both structured and unstructured textures at many scales. Furthermore, it distinguishes between edges and textures, and between textures and noise. The results of the automated detection are shown in the above paper to be consistent with human classification of the corresponding image regions. The amount of texture in averaged image 70 and reference averaged image 75 can be compared by comparing the regions of these images that have been designated as “texture” by the algorithm.

[0046] The inventors found that in processing S40 of Figure 4, when interpolation is used to align the selected images 25 relative to each other, at least a portion of the texture present in the images acquired by the ophthalmic imaging device 10 tends to be removed, and that by aligning the selected images 25 without interpolation, i.e., without interpolating between any pixel values ​​of the selected images 25 to align the images, more of the original texture can be retained in the averaged image 70. Avoiding interpolation in processing S40 not only improves the retention of this clinical data in the averaged image 70, but also significantly improves the effectiveness of the machine learning (ML) denoising algorithm in distinguishing texture features from noise when the machine learning (ML) denoising algorithm is trained using the averaged image 70 generated without interpolation.

[0047] Therefore, as in this exemplary embodiment, the selected comparison images 25 may be used by the processor 320 to generate an averaged image 70 of region 30 by first aligning the selected comparison images 25 relative to each other. In this process, one of the selected images 25 is selected as the base image, and each of the remaining selected images is sequentially aligned with the base image to generate a set of aligned images. When one of the remaining images is aligned with the base image, the pixel values ​​of the first image of this pair are redistributed according to the geometric transformation between the image coordinate systems of the pair; that is, the pixel values ​​of the first image are maintained but reassigned to different pixels of the first image according to the geometric transformation required to align the images, and no interpolation is performed. Next, the processor 320 generates an averaged image 70 of region 30 by averaging the aligned images such that, for example, the pixel value of each pixel in the averaged image 70 is the average (e.g., arithmetic mean) of the pixel values ​​of the pixels in the aligned selected images 25 that correspond to each common position in region 30 of the retina 40. Alternatively, the processor 320 may generate the averaged image 70 by averaging the aligned images such that each pixel value of each pixel in the averaged image 70 is a weighted average of the pixel values ​​of the pixels at the corresponding positions in the aligned selected images 25 that correspond to each common position in the region 30 of the retina 40.

[0048] When interpolation is avoided in processing S40 of Figure 4, the respective geometric transformations between the image coordinate systems of each pair of selected comparison images 25 consist of (i) a first translation of each by a first integer number of pixels along the first pixel array direction in which the pixels of the selected comparison image 25 are arranged, and / or (ii) a second translation of each by a second integer number of pixels along the second pixel array direction in which the pixels of the selected comparison image 25 are arranged, where the second direction is orthogonal to the first direction. Therefore, neither image rotates in the pair of images being aligned. When this constraint is used in the alignment process, the rotation offset threshold Θ used in processing S20 of Figure 4 must be small enough that the image rotation required to compensate for the rotation offset Δφ between the pair of images results in a displacement at the edge of the image frame that is smaller than the resolution of the ophthalmic imaging device 10. For example, in the case of an FAF imaging device, the resolution is usually limited by the device's point spread function. Therefore, the rotation offset threshold Θ is set so that the rotation required for alignment, which produces a displacement smaller than the width of the point spread function along the edges of the image frame, is smaller than Θ (in this case, the comparison image of the pair of images will be selected in S30 of Figure 4). On the other hand, the rotation required for alignment, which produces a displacement larger than the width of the point spread function along the edges of the image frame, is larger than Θ (in this case, the comparison image of the pair of images will not be selected in S30 of Figure 4). As an example, if the size of the images in the sequence is 512 × 512 pixels and the width of the point spread function of region 30 is 1.5 pixels as shown in Figure 5, setting Θ = 0.3 degrees results in a maximum displacement of tan(0.3) - 256 = 1.3 pixels (at the edges of the image frame) due to the rotation for alignment, which is smaller than the width of the point spread function.

[0049] Figure 6B shows an averaged image obtained by averaging the exemplary FAF image shown in Figure 6A with another FAF image rotated 4.5 degrees from the image in Figure 6A. For comparison, Figure 6C shows another averaged image obtained by averaging the FAF image shown in Figure 6A with another FAF image having a negligible rotational offset from the image in Figure 6A. As can be seen from the comparison of Figure 6B and Figure 6C, the averaged image in Figure 6C shows anatomical details of the enhanced region R that are not present in the averaged image in Figure 6B.

[0050] In some cases, a high proportion of the images in sequence 20 may have intra-frame distortion and / or local micro-distortion regions of the type described above (regions with small misalignments between captured retinal features). In such cases, in processing S30 of Figure 4, a relatively low proportion of images in image sequence 20 may be selected for averaging to generate the averaged image 70, which may result in ineffective noise suppression in the averaged image 70. In such cases, it may be beneficial to adapt the processing in Figure 4 by including a preliminary process (before S10) that generates each image in image sequence 20 by segmenting each image of the second image sequence 15 of the second region of the retina 40, as schematically shown in Figure 7. This results in the images of region 30 becoming segments 17 of each image in the second image sequence 15. Therefore, the image sequence 20 may be a sequence of image segments of a second image sequence 15 acquired by an ophthalmic imaging device 10 that images a larger area of ​​the retina 40 including region 30, where each image segment 17 has its own position within each image of the second image sequence 15, and its position is the same as the position of all other image segments within each image of the second image sequence 15. Limiting the subsequent image processing operations in the process of Figure 4 to segments of the second image sequence 15 results in a relatively high proportion of image segments being selected in process S30 for averaging to generate an averaged image 70, which may result in less noise in the averaged image 70 compared to when the images of the second image sequence 15 are processed as a whole according to the process described above with reference to Figure 4. The averaged image 70 based on such selection of image segments has less noise, but it only covers a portion of the area of ​​the retina 40 covered by the images in the second image sequence 15. Therefore, multiple other image segments 17 at corresponding locations in the second image sequence 15 can be processed in the same way to generate an additional averaged image that collectively covers a wider area of ​​the second region of the retina 40.

[0051] Each image in the second image sequence 15 may similarly be segmented into a one-dimensional or two-dimensional array of rectangular image tiles (for example, a two-dimensional array of square image tiles as shown in Figure 7), and each set of image tiles at a corresponding position from the segmented image may be processed according to the process described above with reference to Figure 4 to generate a corresponding averaged image tile. The averaged image tiles may then be joined to generate an averaged image of a second region of the retina, which can exhibit varying levels of noise suppression among the component averaged image tiles. The averaged image tiles are also useful for training machine learning denoising algorithms that more effectively distinguish noise from retinal structures, and thus can generate denoised retinal images that retain more of the texture present in the originally acquired images, as will be discussed later. It should be noted that the two-dimensional orthogonal array of square image segments 17 in Figure 7 is illustrative only, and the image segments 17 do not need to be the same size or shape, nor do they need to be in a regular arrangement. [Second exemplary embodiment]

[0052] Figure 8 is a flowchart illustrating a method according to a second exemplary embodiment, in which a data processing device 60 can process an image sequence 20 to generate an averaged image 70 of region 30.

[0053] Processes S10, S20, and S40 in Figure 8 are the same as those in Figure 4, and therefore no further explanation is given here. The method of the second exemplary embodiment differs from the method of the first exemplary embodiment in that it includes additional processes S15 and S25, as well as a modified version of process S30 (labeled S30' in Figure 8), which will be described below.

[0054] In processing S15 of Figure 8, the processor 320 processes the reference image I Ref and comparison image I Comp For each combination of the above, the reference image I is aligned with respect to each other using the respective offsets (multiple offsets are possible), for example, the respective translational offset t and the respective rotational offset Δφ in the first exemplary embodiment described above.Ref and comparison image I Comp Determine the similarity between each combination and the reference image I. Ref and comparison image I Comp Each translational offset t between is, as in the first exemplary embodiment, reference image I Ref and each comparison image I Comp This may be determined by calculating the cross-correlation between them. In this case, the reference image I is aligned with respect to each other using their respective translational offsets t. Ref and comparison image I Comp The similarity between each may be determined by determining the maximum value of the calculated cross-correlation. Note that in some exemplary embodiments, the order of processes S10 and S15 may be reversed. Thus, if the processor 320 processes reference image I Ref and comparison image I Comp Once the cross-correlation between them is calculated, the processor 320 may determine the maximum value of the calculated cross-correlation before determining the translation (xy) offset corresponding to its maximum value. The reference image I, when aligned relative to each other using each offset(s), Ref and comparison image I Comp It should be noted that the similarity between each image does not need to be derived from the calculated cross-correlation between the images, but may instead be obtained, for example, by calculating the sum of squared differences or cross-information based on the images.

[0055] In process S25 of Figure 7, the processor 320 compares each similarity determined in process S15 with a first similarity threshold and determines whether the determined similarity is greater than the first similarity threshold. Note that in some exemplary embodiments, the order of processes S20 and S25 may be reversed.

[0056] In the process S30' shown in Figure 8, the processor 320 processes the reference image I Ref and comparison image I CompHowever, in each combination that is determined to have an offset smaller than the offset threshold and a similarity greater than the first similarity threshold when aligned relative to each other using the respective offsets (i.e., each combination that satisfies the conditions of S30 in Figure 4), each comparison image I Comp Select the latter. The additional selection criterion for the latter is comparison image I, which has in-frame non-affine distortion. Comp Comparative image I with and / or minute distortion (i.e., displacement of retinal features limited to one or more localized areas) Comp However, this helps to avoid being selected for averaging to generate averaged image 70. Such images are reference image I Ref Even if the offset(s) between the reference image I is determined to be smaller than the offset threshold(s), Ref This is because the correlation with it tends to be relatively low. Therefore, the additional processing S15 and S25, as well as the correction processing S30' in Figure 8, can help avoid the loss of valuable clinical data such as texture in the averaged image 70. The first similarity threshold can be set by trial and error, for example, while observing the effect of adjusting the first similarity threshold on the amount of texture present in the averaged image 70.

[0057] Figure 9A shows an exemplary FAF image acquired by the ophthalmic imaging device 10. For comparison, Figure 9B shows an example of an averaged image 70 obtained by averaging 10 FAF images, including the FAF image in Figure 9A, the 10 FAF images being selected from a sequence of FAF images according to the method described herein with reference to Figure 8. Figure 9B shows that the averaged image retains most of the texture of the FAF image in Figure 9A while being less noisy than the FAF image in Figure 9A (this is most evident in the light gray areas of the image).

[0058] Referring again to Figure 8, in process S25, the processor 320 may also compare each determined similarity to a second similarity threshold to determine if the determined similarity is less than the second similarity threshold, where the second similarity threshold is greater than the first similarity threshold. In this case, the processor 320 considers the reference image I Ref and comparison image I Comp However, during that time, each offset smaller than the offset threshold and reference image I Ref and comparison image I Comp When and are aligned relative to each other using their respective offsets, each combination is determined to have a similarity greater than the first similarity threshold and less than the second similarity threshold, and each comparison image I CompBy selecting this option, a modified version of process S30' may be executed. Thus, comparison images can be selected for averaging to generate an averaged image 70 using the same criteria as in the first exemplary embodiment, plus the additional criterion that the similarity is within a predefined range of values, i.e., between the first similarity threshold and the second similarity threshold. The inventors have found that excluding comparison images that result in an excessively high determined similarity (determined by comparison with an appropriate value of the second similarity threshold set by trial and error) and, similarly, comparison images that result in a determined similarity that is not sufficiently high (determined by comparison with an appropriate value of the first similarity threshold set by trial and error) from selection improves the preservation of texture in the averaged image 70. While we do not wish to be bound by theory, the reasons for the above can be understood as follows: Every imaging area on the retina has a certain amount of information, and each single image can be considered to capture only a portion of that information space due to factors such as subtle changes in illumination, changes in the scanning system (such as subtle changes in the line start position and associated optical paths), and other complex effects such as the way light is scattered from the layers of the retina. Statistically, images from regions with slightly different informational content tend to produce an averaged image that reconstructs a larger proportion of the total available information compared to the individual images. Conversely, highly correlated images with exactly the same informational content will not produce an averaged image that contains more information than the individual images, but will still contain noise. [Third exemplary embodiment]

[0059] Figure 10 is a flowchart illustrating a method according to a third exemplary embodiment in which a data processing device 60 trains a machine learning (ML) algorithm to filter noise from retinal images. The ML algorithm may be any type of supervised ML algorithm known to those skilled in the art that, once trained to perform this task using labeled training data, is suitable for removing one or more different types of noise from FAF images or other types of retinal images. The ML algorithm may consist of a convolutional neural network (CNN), as in this exemplary embodiment. RSThakur et al., “Image De-Noising With Machine Learning: A Review”, IEEE Access, Vol. 9, 2021, pp. 93338-93363 (the entire content of which is incorporated herein by reference), describes various state-of-the-art machine learning-based image denoisers, including dictionary-learning models, convolutional neural networks, and generative adversarial networks, for a wide range of noises, including Gaussian noise, impulse noise, Poisson noise, mixed noise, and real-world noise. The ML algorithm may be any of the various types described in this paper, or any other known to those skilled in the art.

[0060] In the process S100 of Figure 10, the processor 320 of the data processing device 60 generates ground truth training data by processing each sequence 20 of a plurality of sequences of retinal images as described in any of the above exemplary embodiments or their modified versions to generate their respective averaged retinal images.

[0061] Figure 11 is a flowchart summarizing how, in a third exemplary embodiment, the processor 320 generates each averaged retinal image from each sequence of retinal images.

[0062] In the processing S110 shown in Figure 11, the processor 320 processes the reference image I Ref And each comparison image I is an image from the remaining images in sequence 20.Comp For each combination, reference image I Ref and comparison image I Comp Determine the respective offsets between them.

[0063] In the process S120 shown in Figure 11, the processor 320 compares each determined offset with an offset threshold and determines whether the offset is smaller than the offset threshold.

[0064] In the processing S130 shown in Figure 11, the processor 320 processes the reference image I Ref and comparison image I Comp In each combination where the respective offset between and was determined to be smaller than the offset threshold, the respective comparison image I Comp Select this option.

[0065] In the process S140 shown in Figure 11, the processor 320 generates averaged images 70 for each region 30 using the selected comparison images 25.

[0066] As explained above, the offset threshold is set so that sequence 20 is reference image I Ref At least one image offset by an offset greater than the threshold, and reference image I Ref When including images offset by an offset amount smaller than a threshold, the averaged image 70 is determined to show more texture indicating the structure of the first region 30 of the retina 40 than the reference averaged image generated from the images in the image sequence 20.

[0067] Referring again to Figure 10, in process S200, the processor 320 generates training input data by selecting each image from each of the image sequences.

[0068] Next, in process S300 shown in Figure 10, the processor 320 uses the ground truth training target data and the training input data to train an ML algorithm that filters noise from retinal images using techniques well known to those skilled in the art.

[0069] Some of the exemplary embodiments described above are summarized in the following numbered sections E1 to E14.

[0070] E1 A data processing device 60 arranged to process a first image sequence 20 of a region 30 of the retina 40 of an eye 50 to generate an averaged image 70 of the region 30, comprising at least one processor 320 and at least one memory 340 for storing computer-readable instructions, wherein when the computer-readable instructions are executed by at least one processor 320, the at least one processor 320 is... Reference image I selected from the first image sequence 20 Ref And each comparison image I is an image from the remaining images of the first image sequence 20. Comp For each combination, reference image I Ref and comparison image I Comp Determine the respective offsets between them; To determine if an offset is smaller than the offset threshold, compare each determined offset with the offset threshold; Reference Image I Ref and comparison image I Comp In each combination where the respective offset between and was determined to be smaller than the offset threshold, the respective comparison image I Comp Select; The selected comparison image 25 is used to generate an averaged image 70 of region 30. The offset threshold is set so that the first image sequence 20 is compared with the reference image I Ref At least one image offset by an offset greater than the threshold, and reference image I Ref When including images offset by an offset amount smaller than a threshold, the averaged image 70 is determined to show more texture indicating the structure of the first region 30 of the retina 40 than the reference averaged image generated from the images in the first image sequence 20. Data processing device 60.

[0071] E2 Reference Image I Ref and comparison image I Comp Each offset determined for each combination includes a translation offset. When a computer-readable instruction is executed by at least one processor 320, it is sent to at least one processor 320. To determine if an offset is smaller than the offset threshold, each determined offset is compared to the offset threshold by comparing each translation offset to the translation offset threshold; For each combination in which the respective translational offset was determined to be smaller than the translational offset threshold, the comparison image I Comp By selecting Reference Image I Ref and comparison image I Comp In each combination where the respective offset between and was determined to be smaller than the offset threshold, the respective comparison image I Comp Allow the user to select E1 data processing device 60.

[0072] E3 When a computer-readable instruction is executed by at least one processor 320, at least one processor 320 is given a reference image I in each combination. Ref and comparison image I Comp The respective translational offsets between them are Reference Image I Ref and comparison image I Comp Calculating cross-correlation using; and Reference Image I Ref and comparison image I Comp To calculate the inverse Fourier transform of the normalized cross-power spectrum calculated using, The data processing device 60 for E2 determines this by one of the following methods.

[0073] E4 Reference Image I Refand comparison image I Comp Each offset determined for each combination includes a rotational offset. When a computer-readable instruction is executed by at least one processor 320, it is sent to at least one processor 320. To determine if an offset is smaller than an offset threshold, each determined offset is compared to the offset threshold by comparing each rotational offset to the rotational offset threshold; For each combination in which the rotation offset was determined to be smaller than the rotation offset threshold, the comparison image I Comp By selecting Reference Image I Ref and comparison image I Comp In each combination where the respective offset between and was determined to be smaller than the offset threshold, the respective comparison image I Comp Allow the user to select A data processing device 60, one of E1 through E3.

[0074] E5 When a computer-readable instruction is executed by at least one processor 320, at least one processor 320 is given a reference image I in each combination. Ref and comparison image I Comp Each rotational offset between them is Each comparison image I Comp Calculating cross-correlation using a rotated version of; and Reference Image I Ref And, calculating the inverse Fourier transform of the normalized cross-power spectrum calculated using the polar transform of each comparison image, The data processing device 60 of E4 determines this by one of the following methods.

[0075] E6 When a computer-readable instruction is executed by at least one processor 320, it is sent to at least one processor 320. Aligning selected comparison images 25 relative to each other, wherein aligning each pair of selected comparison images 25 includes redistributing the pixel values ​​of one of the paired images according to the respective geometric transformations between the image coordinate systems of the paired images; By averaging the aligned images, an averaged image 70 of the first region is generated, A data processing device 60, one of E1 to E5, generates an averaged image of the first region using the selected comparison image 25.

[0076] E7 The respective geometric transformations between the image coordinate systems of each pair of selected comparison images 25 are: The first translation of each first integer number of pixels along the first pixel array direction in which the pixels of the selected comparison image 25 are arranged; and The second translation of each second integer number of pixels along the second pixel array direction in which the pixels of the selected comparison image 25 are arranged, A data processing device 60 for E6, comprising at least one of the following.

[0077] E8 When a computer-readable instruction is executed by at least one processor 320, it further sends instructions to at least one processor 320. Reference Image I Ref and comparison image I Comp For each combination, the reference image I is obtained when it is aligned with the others using the respective offsets. Ref and comparison image I Comp Determine the similarity of each; To determine if the determined similarity is greater than the first similarity threshold, each determined similarity is compared to the first similarity threshold. When a computer-readable instruction is executed by at least one processor 320, it sends a reference image I to at least one processor 320. Ref and comparison image I CompHowever, in each combination that is determined to have an offset smaller than the offset threshold and a similarity greater than the first similarity threshold when aligned relative to each other using the respective offsets, each comparison image I Comp Including allowing selection, A data processing device 60, one of E1 through E7.

[0078] E9 When a computer-readable instruction is executed by at least one processor 320, it causes at least one processor 320 to further compare each determined similarity with a second similarity threshold to determine if the determined similarity is less than a second similarity threshold, and if the second similarity threshold is greater than the first similarity threshold, When a computer-readable instruction is executed by at least one processor 320, it sends a reference image I to at least one processor 320. Ref and comparison image I Comp However, in each combination that is determined to have an offset smaller than the offset threshold, and a similarity that is greater than the first similarity threshold and less than the second similarity threshold when aligned relative to each other using the respective offsets, the respective comparison image I Comp Including allowing selection, E8 data processing device 60.

[0079] E10 When a computer-readable instruction is executed by at least one processor 320, it is sent to at least one processor 320. Reference image I for each combination Ref and comparison image I Comp Each translational offset between them is given by reference image I Ref and comparison image I Comp The cross-correlation is determined by calculating it using and Reference image I when aligned relative to each other using their respective translational offsets. Ref and comparison image IComp The degree of similarity between each is determined by determining the maximum value of the calculated cross-correlation. The data processing device 60 is E8 if it is dependent on E2, and E9 if it is dependent on E2.

[0080] E11 A data processing device 60, one of E1 to E10, which, when a computer-readable instruction is executed by at least one processor 320, causes at least one processor 320 to generate each image of the first image sequence 20 by segmenting each image of the second image sequence of the second region of the retina 40 such that the image of the first region 30 is a segment of each image of the second image sequence.

[0081] E12 A data processing device 60, E1 to E11, wherein the first image sequence 20 includes an autofluorescence image sequence of a first region 30 of the retina 40 of the eye 50.

[0082] E13 Once the computer-readable instructions are executed by at least one processor 320, the processor 320 is further instructed to train a machine learning algorithm to filter noise from retinal images. By processing each sequence 20 of multiple sequences of retinal images and generating their respective averaged retinal images, ground truth training data is generated, and each of the averaged retinal images is, Reference Image I Ref And each comparison image I is an image from the remaining images in sequence 20. Comp For each combination, reference image I Ref and comparison image I Comp To determine the respective offsets between; To determine if an offset is smaller than the offset threshold, each determined offset is compared to the offset threshold; Reference Image I Ref and comparison image I CompIn each combination where the respective offset between and was determined to be smaller than the offset threshold, the respective comparison image I Comp To choose; To generate averaged images 70 for each region 30, the selected comparison image 25 is used, Generated by, The offset threshold is set so that sequence 20 is reference image I Ref At least one image offset by an offset greater than the threshold, and reference image I Ref When including images offset by an offset amount smaller than a threshold, the averaged image 70 is determined to show more texture indicating the structure of the first region 30 of the retina 40 than the reference averaged image generated from the images in the image sequence 20; Training input data is generated by selecting each image from each part of the image sequence; To train a machine learning algorithm that filters noise from retinal images, we use ground truth training data and training input data. A data processing device 60, one of E1 through E12.

[0083] E14 An ophthalmic imaging device 10 is positioned to acquire an image sequence 20 of a region 30 of the retina 40 of the eye 50; A data processing device 60, one of E1 to E13, is arranged to process the image sequence 20 acquired by the ophthalmic imaging device 10 and generate an averaged image 70 of a region 30 of the retina 40. An ophthalmic imaging system 100 including the above.

[0084] In the preceding description, exemplary embodiments are described with reference to several exemplary embodiments. Therefore, this specification should be considered exemplary and not limiting. Similarly, the drawings highlighting the functions and advantages of exemplary embodiments are presented for illustrative purposes only. The architecture of the exemplary embodiments is flexible and configurable in various ways so that it may be used in ways other than those shown in the accompanying drawings.

[0085] Some aspects of the examples presented herein may, in an exemplary embodiment, be provided as computer programs or software, such as one or more programs having instructions or instruction sequences, contained in or stored on a manufactured article such as a machine-accessible or machine-readable medium, an instruction store, or a computer-readable storage device, each of which may be non-temporary. Programs or instructions on a non-temporary machine-accessible medium, machine-readable medium, instruction store, or computer-readable storage device may be used to program a computer system or other electronic device. The machine or computer-readable medium, instruction store, and storage device may include, but is not limited to, floppy disks, optical disks, and magneto-optical disks, or other types of media / machine-readable media / instruction stores / storage devices suitable for storing or transmitting electronic instructions. The technologies described herein are not limited to any particular software configuration. They are applicable to any computer environment or processing environment. As used herein, the terms “computer-readable,” “machine-accessible medium,” “machine-readable media,” “instruction store,” and “computer-readable storage device” include any medium capable of storing, encoding, or transmitting instructions or instruction sequences for execution by a machine, computer, or computer processor, thereby causing the machine / computer / computer processor to perform any of the methods described herein. Furthermore, in the art, it is common to consider software in one or another form (e.g., a program, procedure, process, application, module, unit, logic, etc.) as something that takes action or produces a result. Such expressions are merely abbreviations indicating that the execution of software by a processing system causes a processor to perform an action and produce a result.

[0086] Furthermore, some or all of the functions of the data processing device 60 may be implemented by preparing application-specific integrated circuits, field-programmable gate arrays, or by interconnecting a suitable network of conventional component circuits.

[0087] Computer program products may be provided in the form of one or more storage media, instruction stores, or memory devices, which store or contain instructions that can be used to control a computer or computer processor, or to cause a computer or computer processor to perform any of the procedures of the exemplary embodiments described herein. Storage media / instruction stores / memory devices may include, but are not limited to, optical discs, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory, flash cards, magnetic cards, optical cards, nanosystems, molecular memory integrated circuits, RAIDs, remote data storage / archives / warehousing, and / or other types of devices suitable for storing instructions and / or data.

[0088] Some implementations involve software stored in one or more computer-readable media, instruction stores, or storage devices, which control the system's hardware and enable the system or microprocessor to interact with a human user or other mechanism using the results of the exemplary embodiments described herein. This software may include, but is not limited to, device drivers, operating systems, and user applications. Ultimately, these computer-readable media or storage devices further include software for performing exemplary embodiments of the invention, as described above.

[0089] The programming and / or software of this system includes software modules for performing the procedures described herein. In some exemplary embodiments herein, the modules include software, and in other exemplary embodiments herein, the modules include hardware, or a combination of hardware and software.

[0090] Although various exemplary embodiments of the present invention have been described above, it should be understood that these are merely examples and do not limit the invention in any way. It will be obvious to those skilled in the art that various modifications can be made to the shape and details. Therefore, the present invention should not be limited to any of the exemplary embodiments described above, but should be defined solely by the following claims and their equivalents.

[0091] Furthermore, the purpose of the abstract is to enable the Patent Office and the general public, particularly scientists, engineers, and those skilled in the art who are not familiar with patent or legal terminology, to quickly grasp the nature and essence of the technical disclosure of this application through a single reading. The abstract is not intended to limit in any way the scope of the exemplary embodiments presented herein. It should also be understood that the procedures described in the claims do not need to be performed in the order presented.

[0092] This specification includes details of many specific embodiments, but these should not be construed as limiting any invention or claim, but rather as descriptions of features specific to the particular embodiments described herein. Certain features described in separate embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, features may be described above as operating in a particular combination, and may even be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from that combination, and the claimed combination may be directed towards subcombinations or variations of subcombinations.

[0093] In some situations, multitasking or parallel processing may be advantageous. Furthermore, the separation of various components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0094] While several exemplary embodiments and aspects have been described, it is clear that these are illustrative and presented for illustrative purposes only. In particular, many of the examples presented herein involve specific combinations of apparatus or software elements, but these elements can be combined in other ways to achieve the same objective. Actions, elements, and features discussed only in relation to one embodiment are not intended to be excluded from similar roles in other embodiments or other multiple embodiments.

Claims

1. A computer implementation method for processing a first image sequence (20) of a first region (30) of the retina (40) of an eye (50) to generate an averaged image (70) of the first region (30), wherein A reference image (I) selected from the first image sequence (20) Ｒｅｆ ) and each comparison image (I Ｃｏｍｐ For each combination with the above reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ (S10) Determine the respective translational offsets (t) between ) and In order to determine whether the translational offset (t) is smaller than the translational offset threshold (T), the magnitude of each determined translational offset (t) is compared with the translational offset threshold (T) (S20). The aforementioned reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ In each combination in which the respective translational offset (t) between ) is determined to be smaller than the translational offset threshold (T), the respective comparison images (I Ｃｏｍｐ ) to select (S30), (S40) includes using the selected comparison image (25) to generate the averaged image (70) of the first region (30), The translational offset threshold (T) is such that when the first image sequence (20) includes at least one image offset by a translational offset greater than the translational offset threshold (T) from the reference image (I Ｒｅｆ ), and an image offset by each translational offset smaller than the translational offset threshold (T) from the reference image (I Ｒｅｆ ), the averaged image (70) is defined to show more texture indicating the structure of the first region (30) of the retina (40) than a reference averaged image (75) generated from the images within the first image sequence (20). Computer implementation method.

2. The reference image (I) in each combination Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ The respective translational offsets (t) between ) are The aforementioned reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ Calculate the cross-correlation using ), and The aforementioned reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ ) Calculate the inverse Fourier transform of the normalized cross power spectrum calculated using ), The computer implementation method according to claim 1, determined by any one of the following (S10).

3. The aforementioned reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ For each combination of ), determine the rotational offset (Δφ) between the reference image (I Ref) and the respective comparison image (I Comp), and In order to determine whether the rotational offset (Δφ) is smaller than the rotational offset threshold (Θ), the magnitude of each rotational offset (Δφ) is compared with the rotational offset threshold (Θ). It further includes, The selection (S30) is performed in each combination in which it is determined that each rotation offset (Δφ) is smaller than the rotation offset threshold (Θ), and the respective comparison images (I Ｃｏｍｐ ) including selecting, The computer implementation method according to claim 1.

4. The aforementioned reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ For each combination of ), determine the rotational offset (Δφ) between the reference image (I Ref) and the respective comparison image (I Comp), and The process includes comparing the magnitude of each rotation offset (Δφ) with the rotation offset threshold (Θ) to determine whether the rotation offset (Δφ) is smaller than the rotation offset threshold (Θ), The selection (S30) is performed in each combination in which it is determined that each rotation offset (Δφ) is smaller than the rotation offset threshold (Θ), and the respective comparison images (I Ｃｏｍｐ ) including selecting, The computer implementation method according to claim 2.

5. The reference image (I) in each combination Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ The respective rotational offsets (Δφ) between ) are The comparison images (I Ｃｏｍｐ Calculating cross-correlation using a rotated version of ), and The aforementioned reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ To calculate the inverse Fourier transform of the normalized cross-power spectrum calculated using the pole transform of ), The computer implementation method according to claim 3, determined by any one of the following.

6. The selected comparison image (25) is, Aligning the selected comparison images (25) relative to each other, wherein aligning each pair of selected comparison images (25) includes redistributing the pixel values ​​of one of the images in the pair according to the respective geometric transformations between the image coordinate systems of the images in the pair, The averaged image (70) of the first region (30) is generated by averaging the aligned images. A computer implementation method according to any one of claims 1 to 5, used to generate the averaged image (70) of the first region (30).

7. The geometric transformation between the image coordinate systems of each pair of selected comparison images (25) is as follows: The translation of each first integer number of pixels along the first pixel array direction in which the pixels in the selected comparison image (25) are arranged, and The pixels of the selected comparison image (25) are arranged in a second pixel array direction that is orthogonal to the first pixel array direction, and each second integer number of pixels is translated accordingly. The computer implementation method according to claim 6, comprising at least one of the following.

8. The aforementioned reference image (I Ｒｅｆ ) and comparison images of each (I Ｃｏｍｐ For each combination of ), the reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ Determining the similarity of each of them (S15), In order to determine whether the determined similarity is greater than the first similarity threshold, each determined similarity is compared with the first similarity threshold (S25), It further includes, The above selection is the reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ When the respective offsets are aligned with each other, in each combination that is determined to have a similarity greater than the first similarity threshold, the respective comparison images (I Ｃｏｍｐ This includes selecting (S30'), The computer implementation method according to any one of claims 1 to 5.

9. The process further includes comparing each determined similarity to a second similarity threshold in order to determine whether the determined similarity is less than a second similarity threshold, wherein the second similarity threshold is greater than the first similarity threshold, and the selection is made based on the reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ When the respective offsets are aligned with each other, in each combination that is determined to have a similarity greater than the first similarity threshold and less than the second similarity threshold, the respective comparison images (I Ｃｏｍｐ The computer implementation method according to claim 8, which includes selecting ).

10. The reference image (I) in each combination Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ The respective translational offset (t) between the reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ It is determined by calculating the cross-correlation using ) The reference image (I) when aligned relative to each other using the respective translational offsets (t) Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ The respective similarities between ) are determined by determining the maximum value of the calculated cross-correlation. The computer implementation method according to claim 8.

11. The computer implementation method according to any one of claims 1 to 5, further comprising generating each image of the first image sequence (20) by segmenting each image of the second image sequence of the second region of the retina (40) such that the image of the first region (30) is a segment of each image of the second image sequence.

12. The computer implementation method according to any one of claims 1 to 5, wherein the first image sequence (20) includes an autofluorescence image sequence of the first region (30) of the retina (40) of the eye (50).

13. A computer implementation method for training a machine learning algorithm that filters noise from retinal images, The method involves generating ground truth training data by processing each sequence of multiple sequences of retinal images to generate an averaged retinal image for each, wherein each averaged retinal image is generated according to the computer implementation method described in any one of claims 1 to 5 (S100), Training input data is generated by selecting each image from each part of the image sequence (S200), To train the machine learning algorithm that filters noise from retinal images, the ground truth training target data and the training input data are used (S300), Computer implementation methods including

14. A computer program (345) including computer-readable instructions, which, when executed by at least one processor (320), causes the at least one processor (320) to perform the method according to any one of claims 1 to 5.

15. A data processing device (60) arranged to process an image sequence (20) of a region (30) of the retina (40) of an eye (50) to generate an averaged image (70) of the region (30), comprising at least one processor (320) and at least one memory (340) for storing computer-readable instructions, wherein when the computer-readable instructions are executed by the at least one processor (320), the at least one processor (320) is informed Reference image (I) selected from the aforementioned image sequence Ｒｅｆ ) and each comparison image (I Ｃｏｍｐ For each combination with the above reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ Determine the respective translational offset (t) between ) To determine whether the translational offset (t) is smaller than the translational offset threshold (T), the magnitude of each determined translational offset (t) is compared with the translational offset threshold (T). The aforementioned reference image (I Ｒｅｆ ) and the respective comparison images (I Ｃｏｍｐ In each combination in which the respective translational offset (t) between ) is determined to be smaller than the translational offset threshold (T), the respective comparison images (I Ｃｏｍｐ ) Select The selected comparison image (25) is used to generate the averaged image (70) of the region (30). The translational offset threshold (T) is determined when the image sequence (20) is compared with the reference image (I Ｒｅｆ ) at least one image offset by a translation offset greater than the translation offset threshold (T) and the reference image (I Ｒｅｆ If the averaged image (70) includes images offset by a translation offset amount smaller than the translation offset threshold (T), the averaged image (70) is determined to show more texture indicating the structure of the region (30) of the retina (40) than a reference averaged image (75) generated from the images in the image sequence (20). Data processing device (60).

16. An ophthalmic imaging device (10) positioned to acquire an image sequence (20) of a region (30) of the retina (40) of an eye (50); A data processing device (60) according to claim 15, arranged to process the image sequence (20) acquired by the ophthalmic imaging device (10) to generate an averaged image (70) of the region (30) of the retina (40); An ophthalmic imaging system (100) including the above.

Images