Deep Learning for Optical Coherence Tomography Segmentation

Abstract

A system and method are presented that provides a machine learning model for segmenting optical coherence tomography (OCT) images. A first OCT image is acquired and then labeled using a graph search algorithm with identified boundaries associated with different tissues within the first OCT image. Portions of the labeled first OCT image are extracted to generate a first plurality of image tiles. A second plurality of image tiles is generated by manipulating at least one image tile from the first plurality of image tiles (e.g., by rotating and / or flipping at least one image tile). A machine learning model is trained using the first plurality of image tiles and the second plurality of image tiles. Segmentation is performed in the second OCT image using the trained machine learning model.

Description

Technical Field

[0001] The present disclosure relates to image processing, and more particularly, to using a machine learning model to perform biomedical image segmentation according to various embodiments of the present disclosure.

Background Art

[0002] In certain biomedical fields such as ophthalmology, images of a patient's body part (e.g., the eye), such as x-ray images, optical coherence tomography (OCT) images, etc., can be captured and analyzed to determine a patient's diagnosis. When analyzing an image, automatic segmentation of elements within the image can be converted into quantitative measurements that are useful for both diagnostic methods and surgical guidance in qualitative images. However, automatic image segmentation can be difficult. For example, artifacts that appear on an image such as speckle can cause the continuous thin boundary between different types of tissue within an OCT image to become discontinuous, which makes it difficult to automatically distinguish between different types of tissue within the OCT image. Additionally, complex pathological conditions can also make image segmentation difficult.

[0003] Conventional segmentation algorithms rely on a clear description of the problem and detailed steps (e.g., clear rules provided by the algorithm designer) to solve this problem. This approach works well for images obtained from normal subjects (patients without disease) where the anatomical structure follows rules that can be established from a standard human database. However, in the case of subjects with different diseases, the anatomical structure can be significantly different from the normal state, making OCT image segmentation difficult. For example, the boundary between different types of tissue inside a person's eye with a pathological condition may not follow the normal eye pattern. Therefore, there is a need in the art to provide an effective mechanism for automatically segmenting OCT images.

Summary of the Invention

Means for Solving the Problems

[0004] According to some embodiments, the system includes a non-transitory memory and one or more hardware processors configured to read instructions from the non-transitory memory and cause the system to perform operations including: acquiring an optical coherence tomography (OCT) image; determining edges in the OCT image based on an edge detection algorithm; generating a plurality of image tiles based on the OCT image; generating a plurality of additional image tiles by operating on at least one image tile of the plurality of image tiles; and training a machine learning model for predicting edges in the OCT image based on the plurality of image tiles and the plurality of additional image tiles.

[0005] According to some embodiments, the method includes: acquiring a biomedical image; determining boundaries of different tissues in the biomedical image based on an edge detection algorithm; generating a first plurality of image tiles based on the biomedical image; generating a second plurality of image tiles by operating on at least one image tile of the first plurality of image tiles; and training, by one or more hardware processors, a machine learning model for segmenting the biomedical image based on the first plurality of image tiles and the second plurality of image tiles.

[0006] According to some embodiments, a non-transitory machine-readable medium storing machine-readable instructions, the machine-readable instructions being executable to cause a machine to perform operations including: obtaining an optical coherence tomography (OCT) image; analyzing the OCT image, at least in part based on an edge detection algorithm; generating a first plurality of image tiles based on the analysis of the OCT image; generating a second plurality of image tiles by manipulating at least one image tile of the first plurality of image tiles; and training a machine learning model for segmenting the OCT image based on the first plurality of image tiles and the second plurality of image tiles.

[0007] To more fully understand the present technology, its features, and its advantages, reference is made to the following description in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figures 6A-6B

Figure 7

[0009] In the drawings, elements having the same reference numerals have the same or similar functions.

[0010] This description and the accompanying drawings showing aspects, embodiments, implementations, or modules of the invention should not be construed as limiting, as the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some cases, well-known circuits, structures, or techniques are not illustrated or described in detail so as not to obscure the present invention. Similar numbers in two or more figures represent the same or similar elements.

[0011] In this description, specific details are shown that describe some embodiments consistent with the present disclosure. Numerous specific details are shown to provide a complete understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative rather than limiting. Those skilled in the art may implement other elements not specifically described herein but within the scope and spirit of this disclosure. Additionally, to avoid unnecessary repetition, one or more features illustrated and described in connection with one embodiment may be incorporated into other embodiments, unless specifically stated otherwise or unless the embodiment would not function with one or more of the features.

[0012] The following techniques include a system and method for providing a machine learning model for automatically segmenting an image (e.g., a biomedical image such as an x-ray image, an optical coherence tomography (OCT) image, etc.), where the machine learning model is trained using training data artificially generated based on the manipulation of existing training images. As described above, OCT image segmentation can be difficult due to factors such as artifacts (e.g., speckles) appearing in the image and the complex pathological conditions of the patient. So far, conventional algorithms such as graph search algorithms have been used (e.g., implemented on a computer) to perform OCT image segmentation. However, these algorithms can only be effective when performing segmentation on images of normal patients (e.g., patients without disease) whose anatomical structures follow rules (or patterns) that can be established from a standard human database. These algorithms may not be effective when performing segmentation on OCT images containing artifacts and / or OCT images obtained from patients with pathological conditions, as the anatomical structures of these patients with complex pathological conditions may follow different patterns or may not follow any pattern at all.

[0013] In some embodiments, the machine learning model may be configured to perform segmentation of an image (e.g., an x-ray image, an OCT image, etc.). The machine learning model can be trained using training data such as images of patients acquired in the past. The advantage of using a machine learning model such as a convolutional neural network (CNN), which outperforms conventional algorithms for performing biomedical image segmentation, is that the machine learning model does not rely on explicit rules regarding how to segment the image. More precisely, with sufficient training data, the machine learning model can derive rules on its own and continuously evolve (e.g., modify and / or correct the rules) based on new training data. Therefore, when given a large amount of high-quality training data, the machine learning model can be trained to accurately and effectively segment the image. So far, the generation of training data has required a human operator to manually analyze and label the boundaries of different tissue types within existing images in order to determine the ground truth. However, generating training data in this way is not only cumbersome and error-prone, but also difficult to generate a large amount of training data due to the manual labor involved. Since the performance of the machine learning model depends greatly on the amount and quality of the training data used to train the machine learning model, the performance of the machine learning model will probably be impaired when the training data is generated in this way.

[0014] Accordingly, according to various embodiments of the present disclosure, a training system can be provided for automatically generating a large amount of high-quality training data for training a machine model configured to perform image segmentation. In some embodiments, the training system can acquire training images. The training images may be existing images obtained from patients in the past. In some embodiments, the training data system may use conventional algorithms (e.g., graph search algorithms) and manual or semi-automatic annotation to analyze and label the training images (e.g., by identifying boundaries (also called edges) of different types of tissues such as different layers of the eye in the training images). As described herein, one drawback of using conventional algorithms to segment images is that conventional algorithms may not be effective in segmenting images with a significant amount of artifacts (e.g., speckles) or images of patients with different pathological conditions (e.g., different eye diseases). Therefore, the training data generated using conventional algorithms may be limited to only clean images (e.g., images without a significant amount of artifacts) and images of normal patients. To expand the training data to cover images of patients with various pathological conditions, the training system of some embodiments can artificially generate additional training data by manipulating the existing training images.

[0015] In some embodiments, the training system may obtain image tiles (e.g., patches) from each training image. Different embodiments may use different techniques for obtaining image tiles from the training images. In some embodiments, the training system may divide the training image into a plurality of tiles. For example, from an image having a size of 160×40 pixels, the training system may divide the image into 64 tiles of the same size (10×10 pixels). In some embodiments, the training system can obtain the device attributes (e.g., the memory size of the graphics processing unit) of a device configured to generate training data, and based on the device attributes, divide the image. For example, the training system can determine a tile size that does not exceed the memory size of the device's graphics processing unit, and then divide the image into tiles based on that tile size.

[0016] In some embodiments, the training system can also perform one or more analyses on the image, and based on the one or more analyses, divide the image. For example, the training system may perform pixel analysis to determine portions of the image that do not contain relevant data (e.g., portions of the image that contain background or blank data). In this regard, the training system may analyze the pixel values of each pixel in the image to determine portions of the image that have adjacent pixels with substantially similar (or identical) pixel values (e.g., spatial frequencies within those portions that are below a threshold). Next, the training system may exclude (e.g., remove) the above portions of the image before dividing the image into a plurality of tiles.

[0017] In some embodiments, instead of dividing an image into a plurality of tiles, the training system may generate image tiles by extracting different portions of the image from the image. For example, the training system may generate a virtual window having a determined tile size (e.g., 10×10 pixels). The training system may place the virtual window at an initial position of the image (e.g., the upper left corner). The training system may analyze a portion of the image within the virtual window to determine whether the portion of the image exceeds a relevance threshold. When it is determined that the portion of the image exceeds the relevance threshold, the training system may extract the portion of the image as an image tile. On the other hand, when it is determined that the portion of the image does not exceed the relevance threshold, the training system may ignore the portion of the image. In some embodiments, the training system may determine whether the portion of the image exceeds the relevance threshold based on one or more factors such as whether the spatial frequency of the portion of the image exceeds a spatial frequency threshold, whether the portion of the image contains labeled data (e.g., includes a portion of the boundary of different tissues labeled by a graph search algorithm), and the like.

[0018] After extracting (or ignoring) a portion of the image within the virtual window, the training system may move the virtual window to another position to cover another portion of the image (e.g., move the virtual window to the right, down, etc. by a predetermined number of pixels). The training data system may continue to analyze different portions of the image covered by the virtual window and extract portions that exceed the relevance threshold. Based on a predetermined movement of the virtual window, different portions of the image covered by the virtual window may or may not overlap with each other such that the image tiles extracted from the image may partially overlap. Each image tile extracted from the image may be a separate piece of training data for training a machine learning model. By independently analyzing different portions of the image and extracting only relevant portions, the quality of the training data may be significantly improved.

[0019] Since the initial training images are images of normal patients (e.g., patients without disease), the tiles obtained from these images are not yet able to provide sufficient training for the machine learning model for patients with different types of medical conditions. Thus, in some embodiments, the training system may generate additional training data corresponding to patients with various medical conditions by manipulating the image tiles. For example, the training system may adjust the orientation of each tile (which effectively changes the orientation of the boundaries of the layers in the image tile) to simulate images of patients with various medical conditions. In some embodiments, the training system may generate additional tiles by rotating each tile by one or more rotations, where each additional tile corresponds to rotating the tile by a predetermined degree of rotation. For example, the training system may be configured to rotate each tile by 90 degrees, 180 degrees, and 270 degrees. Thus, for each original image tile, the training system can generate three additional tiles (e.g., three additional pieces of training data) - a first additional tile corresponding to rotating the original tile by 90 degrees, a second additional tile corresponding to rotating the original tile by 180 degrees, and a third additional tile by rotating the original tile by 270 degrees. In other embodiments, different degrees of rotation may be used and / or different numbers of additional tiles may be generated.

[0020] In some embodiments, instead of rotating the tiles or in addition to rotating the tiles, the training system can also generate additional tiles by flipping each original tile and each additional tile along an axis (e.g., a horizontal axis, a vertical axis, etc.). For example, by flipping a tile along its horizontal axis and vertical axis, the training system can generate two additional tiles based on the given tile. In an example where the training system rotates each original tile by 90 degrees, 180 degrees, and 270 degrees and then flips the original tile and the rotated tiles along the horizontal axis and vertical axis, the training system can generate 11 additional tiles based on the original tile. Thus, the training system can increase the amount of training data by 11 times, and the additional training data can cover examples of patients with various medical conditions. Further, the training system can also generate additional image tiles by generating different versions of the same image tile, where each version includes artifacts (e.g., different amounts of speckle, etc.) added to the image tile.

[0021] The training system can then use the generated training data (e.g., the tiles and the additional tiles) to train a machine learning model. In some embodiments, the machine learning model can be implemented as a deep convolutional neural network. When training the machine learning model, each piece of training data (e.g., each tile) is first downsampled through a set of convolutional layers and then upsampled through a corresponding set of convolutional layers. By downsampling and upsampling the training data, the machine learning model can be trained to identify the boundaries of tissues in OCT images. After training, the machine learning model can be used to identify the boundaries of tissues in new OCT images of patients. In some embodiments, the machine learning model can be retrained periodically using new training data. For example, when new OCT images are acquired, the training system can be configured to generate training data using the methods described herein and retrain the machine learning model using the newly generated training data.

[0022] Figure 1 shows a system 100 in which a training system as described herein can be implemented, according to some embodiments. System 100 includes a biometrics analysis platform 102 coupled to one or more eye care professional (ECP) devices (such as ECP devices 130, 140, and 150) via a network 115. In some examples, network 115 may include one or more switching devices, routers, local area networks (e.g., Ethernet (registered trademark)), wide area networks (e.g., the Internet), and the like.

[0023] Each of the ECP devices (e.g., ECP devices 130, 140, and 150) may include a user interface (UI) application and an ECP identifier. For example, ECP device 130 includes a UI application 132 and an ECP identifier 134. UI application 132 may be used by a corresponding ECP (e.g., ECP 170) to interact with biometrics analysis platform 102. For example, UI application 132 may be a web browser or a client application (e.g., a mobile application). Eye care professional (ECP) 170 may access a graphical user interface (GUI), such as a web page, generated and / or provided by biometrics analysis platform 102 via UI application 132. ECP identifier 134 is an identifier that uniquely identifies ECP 170 from among a plurality of ECPs serviced by lens selection platform 102.

[0024] The biometrics analysis platform 102 includes a user interface (UI) server 103, a biometrics analysis engine 106, a training module 107, and an image segmentation model 108. The interface server 103 is configured to provide a user interface (e.g., a graphical user interface (GUI), etc.) on the ECP devices 130, 140, and 150 in some embodiments, and through this user interface, an ECP such as the ECP 170 can interact with the biometrics analysis platform 102. For example, the UI server 103 of some embodiments may include a web server that provides a website associated with the lens selection platform 102. The UI server 103 can generate and / or save one or more interactive web pages that can be presented on the ECP device by a UI application (e.g., the UI application 132). In another example, the UI server 103 may include an application server that interacts with a client application (e.g., the UI application 132) through a protocol (e.g., the REST protocol, etc.).

[0025] The image segmentation model 108 may be a machine learning model (e.g., a convolutional neural network, etc.) configured to perform segmentation on an image (e.g., identify boundaries of different tissues on the image). The training module 107 may be configured to train the image segmentation model 108 by generating training data using the techniques disclosed herein. The training module 107 may obtain an image of a patient (e.g., an OCT image of the patient's eye). The training module 107 may use a conventional algorithm (e.g., a graph search algorithm) to analyze and label the boundaries of different tissues on the image. Next, the training module 107 may artificially generate additional training data using the techniques disclosed herein. For example, the training data module 107 can obtain tiles from an image (e.g., by dividing the image or extracting tiles from the image), and each tile can be manipulated (e.g., by changing the orientation of the tile, adding artifacts to the tile, etc.) to generate additional training data. Next, the training module 107 may use the generated training data to train the image segmentation model 108. After training the image segmentation model 108, the image segmentation model 108 may be used by the biometrics analysis engine 106 to augment an image (e.g., an OCT image).

[0026] In some embodiments, the ECP (e.g., ECP 170) may provide image data of a patient's eye (e.g., an OCT image) through a user interface provided by a UI application (e.g., UI application 132) and a UI server 103. For example, the ECP 170 may use the diagnostic device 160 to capture an image of the patient's eye (e.g., an OCT image). In some embodiments, the ECP device 130 may be coupled to the diagnostic device 160 such that the ECP device 130 can automatically retrieve an image from the diagnostic device and transmit the image to the biometrics analysis platform 102 by the UI server 103.

[0027] In some embodiments, upon receiving an image, the biometrics analysis engine 106 may analyze the image and provide a diagnosis and / or other information regarding the patient's eye to the ECP 170 based on the image. For example, the biometrics analysis engine 106 may use a trained image segmentation model 108 to identify the boundaries of different tissues (e.g., different corneal layers) within the image. Next, the biometrics analysis engine 106 may augment the image by highlighting the identified boundaries within the image and present the augmented image on the ECP device 130. The augmented image may assist the ECP 170 in the diagnosis and / or surgical guidance of the patient. In some embodiments, the biometrics analysis engine 106 may provide further recommendations based on the image, such as the selection of an intraocular lens or contact lens for the patient, by analyzing the augmented image.

[0028] Figure 2 shows a training module according to various embodiments of the present disclosure. As shown, the training module 107 includes a segmentation module 202 and a tile generation module 204. The training module 107 can use the segmentation module 202 to analyze and label an existing image (e.g., image 222), for example, by using a graph search algorithm. Next, the training module 107 can use the tile generation module 204 to obtain image tiles from each labeled image as training data for training the image segmentation model 108. For example, the tile generation module 204 can divide the image 222 into image tiles (e.g., image tiles 224a - 224d, also referred to as original image tiles 224a - 224d). Next, the tile generation module 204 can operate on the original image tiles 224a - 224d to generate additional image tiles. In some embodiments, the tile generation module 204 can rotate each of the original image tiles 224a - 224d multiple times by different degrees of rotation to generate additional image tiles. Further, the tile generation module 204 can also flip each of the original image tiles 224a - 224d and each of the additional image tiles along an axis (e.g., horizontal axis, vertical axis, etc.) to generate additional image tiles for training the image segmentation model 108.

[0029] FIG. 3 shows a process 300 for training an image segmentation model configured to perform segmentation on an image, according to an embodiment of the present disclosure. In some embodiments, process 300 may be performed by training module 107 and / or biometrics analysis engine 106. Process 300 begins by obtaining a first optical coherence tomography (OCT) image. For example, training module 107 may obtain training images such as existing images obtained from a patient in the past (e.g., existing OCT images obtained from a patient's eye). In some embodiments, the existing images can be obtained from one or more ECP devices such as ECP devices 130, 140, and 150. For example, an ECP (e.g., ECP 170) may capture an OCT image of a patient (using a diagnostic device such as diagnostic device 160). The ECP may send the OCT image to biometrics analysis platform 102 for analysis, e.g., to perform segmentation on the image.

[0030] Next, the process 300 performs segmentation on the first OCT image (in step 310) using an algorithm and generates image tiles from the first OCT image (in step 315). For example, the segmentation module 202 of the training module 107 can use a conventional algorithm (e.g., a graph search algorithm) to analyze and label the acquired image (e.g., by identifying boundaries of different types of tissue such as different layers of the eye in the training image). FIG. 4A shows an exemplary OCT image 402 that can be acquired from the ECP device 130. In this example, the OCT image 402 is an image of a patient's eye, specifically, images of different corneal layers of the eye. For example, the OCT image 402 can show an eye including layers 422 and 424. As shown, due to artifacts and other issues with the image 402, the boundaries of layers 422 and 424 may not be very sharp and / or may be discontinuous. Therefore, the segmentation module 202 can use a graph search algorithm to identify the boundaries of different layers. As shown in the image 402, by using a graph search algorithm, the segmentation module 202 can highlight the layer boundaries including boundaries 432 and 434 for layer 422 and boundaries 436 and 438 for layer 424.

[0031] Next, the training module 107 may use labeled images (e.g., labeled OCT image 402) as training images for training the image segmentation module 108. As shown in FIG. 4A, layers 422 and 424 exhibit one or more patterns with prominent features. For example, layer 422 has a wave pattern including a plurality of peaks and valleys, and each period of the wave has a prominent feature or characteristic (e.g., amplitude, thickness, etc.). Any portion of layer 422 may include distinct characteristics for training the image segmentation model 108. Similarly, layer 424 includes discontinuous patches of elongated-shaped elements, and each of these elements may have prominent features or characteristics for training the image segmentation model 108. Thus, in some embodiments, instead of using the entire image 402 as a training data piece, the training module 107 may obtain tiles (or patches) of the image 402 as training data.

[0032] When obtaining image tiles from a training image (e.g., image 402), different embodiments may use different techniques. In some embodiments, the tile generation module 204 of the training module 107 may divide the training image into a plurality of tiles. For example, if image 402 has a size of 160×40 pixels, the tile generation module 204 may divide image 402 into 64 tiles of the same size (10×10 pixels). As shown in FIG. 4A, the tile generation module 204 may use virtual lines 412-420 to divide image 402 into a plurality of tiles such as tiles 442-448. In some embodiments, the tile generation module 204 can obtain the device attributes (e.g., the memory size of the graphics processing unit) of a device (e.g., a computer server such as the biometrics analysis platform 102) configured to generate training data, and based on the device attributes, the image can be divided. For example, the tile generation module 204 can determine a tile size that does not exceed the memory size of the graphics processing unit of the device (e.g., 8GB, 16GB, etc.), and then, based on the tile size, divide image 402 into tiles such that each tile may have a size that does not exceed the memory size of the graphics processing unit.

[0033] In some embodiments, the tile generation module 204 can also perform one or more analyses on the image 402 and, based on the one or more analyses, can divide the image. For example, the tile generation module 204 may perform pixel analysis to determine portions of the image that do not contain relevant data (e.g., portions of the image that contain background or blank data). In this regard, the tile generation module 204 can analyze the pixel values of each pixel in the image to determine portions of the image that have adjacent pixels with substantially similar (or identical) pixel values (e.g., spatial frequencies within those portions that are below a threshold). Next, the tile generation module 204 can exclude (e.g., remove) the portions of the image before dividing the image into a plurality of tiles. For example, the tile generation module 204 can determine that portion 450 of the image 402 does not have relevant data based on the low spatial frequencies and lack of labeled data (e.g., labeled boundaries) within portion 450. Thus, the tile generation module 204 can remove portion 450 from the image 402 before dividing the image 402 into tiles.

[0034] In some embodiments, instead of dividing an image into a plurality of tiles, the tile generation module 204 may generate image tiles by extracting different portions of the training image (e.g., image 402). For example, the tile generation module 204 may provide a virtual window having a determined tile size (e.g., 10×10 pixels) on the image. The tile generation module 204 may place the virtual window at the first position of the image (e.g., the upper left corner). The tile generation module 204 may analyze the portion of the image in order to determine whether the portion of the image within the virtual window exceeds a relevance threshold. When it is determined that the portion of the image exceeds the relevance threshold, the tile generation module 204 may extract the portion of the image as an image tile. On the other hand, when it is determined that the portion of the image does not exceed the relevance threshold, the tile generation module 204 may ignore the portion of the image. In some embodiments, the tile generation module 204 may determine whether the portion of the image exceeds the relevance threshold based on one or more factors such as whether the spatial frequency of the portion of the image exceeds a spatial frequency threshold, whether the portion of the image contains labeled data (e.g., includes a part of the boundary of different tissues labeled by a graph search algorithm), and the like.

[0035] Figure 4B shows, for example, a virtual window 462 provided on an image 402 by a tile generation module 204. The virtual window 462 is provided at an initial position (e.g., the upper left corner) covering a first image portion 472 of the image 402. The tile generation module 204 may analyze the image portion 472 of the image 402 to determine whether the image portion 472 within the virtual window 462 exceeds a relevance threshold. For example, the tile generation module 204 may analyze the pixel values of the image portion 472 to determine whether the spatial frequency exceeds a predetermined threshold. The tile generation module 204 may also determine whether labeled data (e.g., identified boundaries based on a graph search algorithm) is included within the image portion 472. Next, the tile generation module 204 may determine whether the image portion 472 of the image 402 exceeds the relevance threshold, for example, based on the spatial frequency of the image portion 472 of the image 402 and / or the presence of labeled data. When it is determined that the image portion 472 exceeds the relevance threshold, the tile generation module 204 may extract the image portion 472 of the image 402 as an image tile. On the other hand, when it is determined that the image portion 472 of the image 402 does not exceed the relevance threshold, the tile generation module 204 may ignore the image portion 472. In this example, since the image portion 472 includes a part of the labeled boundary 432, the tile generation module 204 can determine that the image portion 472 exceeds the relevance threshold, and thus can extract the image portion 472 from the image 402.

[0036] After extracting (or ignoring) a portion of the image within the virtual window, the tile generation module 204 may move the virtual window to another location to cover another portion of the image (e.g., move the virtual window to the right, down, etc. by a predetermined number of pixels). For example, as shown in FIG. 4B, after extracting or ignoring the image portion 472, the tile generation module 204 may move the virtual window 462 to the right by a predetermined number of pixels (e.g., 5 pixels) to cover a second image portion 474 of the image 402. The tile generation module 204 may continue to analyze different portions of the image covered by the virtual window and extract portions that exceed a relevance threshold. Based on a predetermined movement of the virtual window, different portions of the image coverage by the virtual window may overlap with each other or may not overlap with each other such that the image tiles extracted from the image may partially overlap. In this example, the image portions 472 and 474 partially overlap with each other. Each image tile extracted from the image may serve as a separate piece of training data for training the image segmentation model 108. By analyzing different portions of the image independently and extracting only relevant portions, the quality of the training data may be significantly improved.

[0037] FIG. 5 shows exemplary image tiles 502-512 obtained from an image 402, either by dividing the image 402 or extracting tiles from the image 402 using a virtual window. Each image tile (also referred to as an original image tile) can be used as a piece of training data for training the image segmentation model 108. However, as described herein, one drawback of using conventional algorithms to segment an image is that conventional algorithms may not be effective when segmenting images with a significant amount of artifacts (e.g., speckle), or images of patients with different pathological conditions (e.g., different eye diseases). Therefore, the training data (e.g., original image tiles) generated using conventional algorithms may be limited (e.g., only images without a significant amount of artifacts and images of normal patients are labeled). Thus, in some embodiments, the training module 107 may artificially generate additional training data corresponding to patients with various medical conditions by manipulating the original image tiles.

[0038] Referring again to FIG. 3, process 300 generates additional training images by changing the orientation of the tiles (at step 320). For example, training module 107 may manipulate the original image tiles by adjusting the orientation of each original image tile (which effectively changes the orientation of the identified boundaries of the layers in the image tile) to simulate images of patients with various medical conditions. In some embodiments, training module 107 may generate additional tiles by rotating each original image tile by one or more rotations, and each additional tile corresponds to rotating the tile by a predetermined degree of rotation. For example, training module 107 may manipulate each original image tile by rotating it 90 degrees, 180 degrees, and 270 degrees. As shown in FIG. 5, training module 107 may generate an additional image tile 520a by rotating the original image tile (e.g., image tile 510) 90 degrees. Training module 107 may also generate an additional image tile 520b by rotating image tile 510 180 degrees. Training module 107 may also generate an additional image tile 520c by rotating image tile 510 270 degrees. Thus, in this example, for each original image tile, training module 107 can generate three additional tiles (e.g., three additional pieces of training data) based on rotating the original image tile - a first additional tile corresponding to rotating the original tile 90 degrees, a second additional tile corresponding to rotating the original tile 180 degrees, and a third additional tile by rotating the original tile 270 degrees. In other embodiments, different degrees of rotation may be used and / or a different number of additional tiles may be generated. For example, more additional tiles can be generated by rotating the original image tile by additional degrees of rotation.

[0039] In some embodiments, instead of or in addition to rotating the tiles, the training module 107 can also generate additional tiles by flipping each original tile and each additional tile along an axis (e.g., a horizontal axis, a vertical axis, etc.). For example, by flipping a tile along its horizontal axis and vertical axis, the training module 107 can generate two additional tiles based on the given tile. As shown in FIG. 5, the training module 107 can generate an additional image tile 520d by flipping the image tile 510 along the vertical axis 530. The training module 107 can also generate another additional image tile 520e by flipping the image tile 510 along the horizontal axis 525. In some embodiments, the training module 107 can also generate additional image tiles by flipping the image tiles 520a - 520c. Thus, in an example where the training data system rotates each original tile by 90 degrees, 180 degrees, and 270 degrees and then flips the original tile and the rotated tiles along the horizontal axis and vertical axis, the training module 107 can potentially generate 11 additional tiles based on the original tile. As a result, the training module 107 can increase the amount of training data by 11 times, and the additional training data can cover examples of patients with various medical conditions. Further, the training module 107 can generate additional tiles by deriving different versions of the same image tile (e.g., adding different amounts of artifacts to the same image tile).

[0040] The training module 107 can then use the generated training data (e.g., the above tiles and the additional tiles) to train the image segmentation model 108. In some embodiments, the segmentation model 108 may be implemented as a deep convolutional neural network using the techniques described in the document titled "U-Net: Convolutional Networks for Biomedical Image Segmentation" by Ronneberger et al., which is incorporated herein by reference in its entirety. As described by Ronneberger, when training the image segmentation model 108, each training data piece (e.g., each image tile) is first downsampled through a set of convolutional layers and then upsampled through a corresponding set of convolutional layers. By downsampling and upsampling the training data, the image segmentation model 108 can be trained to identify the boundaries of tissues within the OCT image. After training, the image segmentation model 108 can be used to identify the boundaries of tissues in a new OCT image of a patient. In some embodiments, the image segmentation model 108 can be periodically retrained using new training data. For example, when new OCT images are acquired, the training module 107 can be configured to generate training data using the methods described herein and use the newly generated training data to retrain the image segmentation model 108.

[0041] Referring back to FIG. 3, process 300 receives a second OCT image (at step 330) and performs segmentation on the second OCT image using a trained machine learning model (at step 335). For example, the biometrics analysis engine 106 may receive an image from one of, for example, ECP devices 130, 140, and 150 via the UI server 103. The biometrics analysis engine 106 may use the image segmentation model 108 to identify the boundaries of different layers (e.g., different types of tissue) within the image. In some embodiments, the biometrics analysis engine 106 can divide the image into image tiles, each having a predetermined size (e.g., a size determined to generate image tiles for training the image segmentation model 108). The biometrics analysis engine 106 may provide the image tiles one by one to the image segmentation model 108 to obtain the identification of the boundaries of different layers (e.g., different types of tissue) within the image tiles.

[0042] In some embodiments, the biometrics analysis engine 106 may augment the image by emphasizing different layers or the boundaries of different layers within the image and present the augmented image to the ECP device. In some embodiments, the biometrics analysis engine 106 can perform further analysis on the image based on the identified layers and present a report (e.g., a recommendation for the type of intraocular lens or contact lens for the patient, etc.) on the ECP device.

[0043] FIGS. 6A and 6B are diagrams of a processing system according to some embodiments. Although FIGS. 6A and 46B show two embodiments, those skilled in the art will readily understand that other system embodiments are also possible. According to some embodiments, the processing system of FIGS. 6A and / or 6B may represent a computing system that may be included in the biometrics analysis platform 102 and one or more such as ECP devices 130, 140, and 150.

[0044] FIG. 6A shows a computing system 600 in which the components of the system 600 communicate electrically with each other using a bus 605. The system 600 includes a processor 610 and a system bus 605 that couples various system components including a memory (e.g., PROM, EPROM, FLASH-EPROM, and / or other memory chips or cartridges) in the form of a read-only memory (ROM) 620, a random access memory (RAM) 625, etc. to the processor 610. The system 600 may further include a cache 612 of high-speed memory that is directly connected to, in proximity to, or integrated as part of the processor 610. The system 600 may access data stored in the ROM 620, the RAM 625, and / or one or more storage devices 630 via the cache 612 for high-speed access by the processor 610. In some embodiments, the cache 612 may provide a performance boost that avoids latency when the processor 610 accesses data from the memory 615, the ROM 620, the RAM 625, and / or one or more storage devices 630 previously stored in the cache 612. In some embodiments, one or more storage devices 630 store one or more software modules (e.g., software modules 632, 634, 636, etc.). The software modules 632, 634, and / or 636 can control and / or be configured to control the processor 610 to perform various operations such as the processes of method 300. Also, although the system 600 shows only one processor 610, it will be understood that the processor 610 may represent one or more central processing units (CPUs), multi-core processors, microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), tensor processing units (TPUs), etc. In some embodiments, the system 600 may be implemented as a stand-alone subsystem, and / or as a board added to a computing device, or as a virtual machine.

[0045] To enable a user to interact with system 600, system 600 includes one or more communication interfaces 640 and / or one or more input / output (I / O) devices 645. In some embodiments, the one or more communication interfaces 640 may include one or more network interfaces, network interface cards, etc. to provide communication according to one or more network and / or communication bus standards. In some embodiments, the one or more communication interfaces 440 may include an interface for communicating with system 600 via a network such as network 115. In some embodiments, the one or more I / O devices 645 may include one or more user interface devices (e.g., keyboard, pointing / selection device (e.g., mouse, touchpad, scroll wheel, trackball, touch screen, etc.), audio device (e.g., microphone and / or speaker), sensor, actuator, display device, etc.).

[0046] Each of the one or more storage devices 630 may include a non-transitory and non-volatile storage device such as that provided by a hard disk, optical medium, solid state drive, etc. In some embodiments, each of the one or more storage devices 630 may be located in the same location as system 600 (e.g., local storage device) and / or may be located remotely from system 600 (e.g., cloud storage device).

[0047] FIG. 6B shows a computing system 650 based on a chipset architecture that may be used in performing any of the methods described herein (e.g., method 300 and / or 510). System 650 may include software, firmware, and / or a processor 655 representing any number of physical and / or logically distinct resources that may also perform other computations, such as one or more CPUs, multi-core processors, microprocessors, microcontrollers, DSPs, FPGAs, ASICs, GPUs, TPUs, etc. As shown, processor 655 may be assisted by one or more chip sets 660, which may also include one or more CPUs, multi-core processors, microprocessors, microcontrollers, DSPs, FPGAs, ASICs, GPUs, TPUs, coprocessors, codec (CODEC), etc. As shown, one or more chip sets 660 interface with processor 655 along with one or more of one or more I / O devices 665, one or more storage devices 670, memory 675, bridge 680, and / or one or more communication interfaces 690. In some embodiments, one or more I / O devices 665, one or more storage devices 670, memory, and / or one or more communication interfaces 690 may correspond to similarly named counterparts of FIG. 6A and system 600.

[0048] In some embodiments, bridge 680 may provide an additional interface to provide system 650 access to one or more user interface (UI) components, such as one or more keyboards, pointing / selection devices (e.g., mouse, touchpad, scroll wheel, trackball, touch screen, etc.), audio devices (e.g., microphone and / or speaker), display devices, etc. According to some embodiments, system 600 and / or 650 may provide a graphical user interface (GUI) suitable for assisting a user (e.g., a surgeon and / or other medical personnel) during execution of the process of method 200.

[0049] The method according to the above embodiments may be implemented as executable instructions stored in a non-transitory and tangible machine-readable medium. When the executable instructions are executed by one or more processors (e.g., processor 610 and / or processor 655), the process of method 300 may be caused to be performed by the one or more processors. Some common forms of machine-readable media that may include the process of method 300 are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, other magnetic media, CD-ROMs, other optical media, punch cards, paper tapes, other physical media having patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, other memory chips or cartridges, and / or other media adapted for reading by processors or computers.

[0050] An apparatus for implementing the method according to these disclosures can include hardware, firmware, and / or software, and may take any of various form factors. Typical examples of such form factors include laptops, smartphones, small form factor personal computers, personal digital assistants, and the like. Some of the functionality described herein may also be embodied in peripheral devices and / or add-in cards. Such functionality may also, as a further example, be implemented on a circuit board from among different chips or different processes executed in a single device.

[0051] FIG. 7 is a diagram of a multi-layer neural network 700 according to some embodiments. In some embodiments, the neural network 700 may represent a neural network used to implement a machine learning model for performing segmentation on images such as OCT images as described herein. The neural network 700 processes input data 710 using an input layer 720. In some examples, the input data 710 may correspond to input data provided to one or more models and / or training data provided to one or more models during updates used to train one or more models. The input layer 720 includes a plurality of neurons used to condition the input data 710 by scaling, clipping, etc. Each neuron in the input layer 720 generates an output that is fed into the input of a hidden layer 731. The hidden layer 731 includes a plurality of neurons that process the output from the input layer 720. In some examples, each neuron in the hidden layer 731 generates an output, which is then propagated through one or more additional hidden layers ending in a hidden layer 739. The hidden layer 739 includes a plurality of neurons that process the output from the previous hidden layer. The output of the hidden layer 739 is fed into an output layer 740. The output layer 740 includes one or more neurons used to condition the output from the hidden layer 739 by scaling, clipping, etc. The architecture of the neural network 700 is merely exemplary, and it should be understood that other architectures are possible, including neural networks having only one hidden layer, neural networks without an input layer and / or an output layer, neural networks having a recurrent layer, etc.

[0052] In some embodiments, each of the input layer 720, the hidden layers 731 - 739, and / or the output layer 740 includes one or more neurons. In some embodiments, each of the input layer 720, the hidden layers 731 - 739, and / or the output layer 740 may include the same number or different numbers of neurons. In some embodiments, each neuron takes a combination of its inputs x (e.g., weighted sum using a trainable weight matrix W), adds an optional trainable bias b, and applies an activation function f to generate an output a, as shown in Equation 1. In some embodiments, the activation function f may be a linear activation function, an activation function with upper and / or lower bounds, a logarithmic sigmoid function, a hyperbolic tangent function, a rectified linear unit function, etc. Activation functions such as the rectified linear unit (ReLU) activation function may likewise be non-linear. In some embodiments, each neuron may have the same or different activation functions. a = f(Wx + b).....................(1)

[0053] In some embodiments, the neural network 700 may be trained using supervised learning, which is a combination of training data (e.g., biometric data of a patient, etc.) that includes a combination of input data and ground truth (e.g., expected) output data (e.g., lens products selected by ECP for a patient in the past, etc.). The difference between the output of the neural network 700 generated using the input data for the input data 710 and the output data 750 generated by the neural network 700 compared to the ground truth output data. The difference between the generated output data 750 and the ground truth output data may then be fed back to the neural network 700 to correct various trainable weights and biases. In some embodiments, backpropagation techniques using algorithms such as the stochastic gradient descent algorithm may be used to feed back the difference. In some embodiments, multiple sets of combinations of training data may be presented to the neural network 700 multiple times until the overall loss function (e.g., mean squared error based on the difference of each training combination) converges to an acceptable level.

[0054] Exemplary embodiments have been shown and described, but a wide range of modifications, changes, and substitutions are envisioned in the foregoing disclosure, and in some cases, some features of the embodiments may be utilized without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the following claims, and it is appropriate that the claims be construed broadly in a manner consistent with the scope of the embodiments disclosed herein. According to aspect (1), a system comprising: a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to acquire an optical coherence tomography (OCT) image; determine edges within the OCT image based on an edge detection algorithm or manual annotation; generate a plurality of image tiles based on the OCT image; generate a plurality of additional image tiles by manipulating at least one of the plurality of image tiles; train a machine learning model for predicting edges within the OCT image based on the plurality of image tiles and the plurality of additional image tiles; and one or more hardware processors configured to perform operations including those described above; A system comprising the above. According to aspect (2), the manipulation of at least one of the image tiles includes at least one of rotating at least one of the image tiles or flipping at least one of the image tiles along an axis. According to aspect (3), the plurality of additional image tiles includes image tiles corresponding to rotating at least one of the image tiles by 0 degrees, 90 degrees, 180 degrees, and 270 degrees. According to aspect (4), the plurality of additional image tiles includes image tiles corresponding to flipping at least one of the image tiles along at least one of a vertical axis or a horizontal axis. According to aspect (5), the edge detection algorithm includes a graph search algorithm. According to aspect (6), the machine learning model includes a deep convolutional neural network. According to aspect (7), the operations further include predicting edges within a second OCT image using the trained machine learning model. According to aspect (8), a method comprising: acquiring a biomedical image; determining boundaries of different tissues within the biomedical image based on an edge detection algorithm; generating a first plurality of image tiles based on the biomedical image; generating a second plurality of image tiles by manipulating at least one of the first plurality of image tiles; Training, by one or more hardware processors, a machine learning model for segmenting a biomedical image based on the first plurality of image tiles and the second plurality of image tiles. A method including this. According to aspect (9), the determined boundary corresponds to the anterior corneal layer of the eye in the biomedical image. According to aspect (10), the generation of the first plurality of image tiles includes dividing the biomedical image into image patches. According to aspect (11), it further includes analyzing one or more characteristics of the biomedical image, and the biomedical image is divided into the first plurality of image tiles based on the analysis. According to aspect (12), the generation of the first plurality of image tiles analyzing a plurality of different parts of the biomedical image; selecting, from the plurality of different parts, a subset of the parts of the biomedical image that exceed a relevance threshold; including this. According to aspect (13), the analysis of the plurality of different parts includes determining whether a part of the biomedical image from the plurality of different parts includes a boundary determined by the edge detection algorithm. According to aspect (14), at least two of the second plurality of image tiles partially overlap. According to aspect (15), a non-transitory machine-readable medium storing machine-readable instructions, the machine-readable instructions causing a machine to acquire an optical coherence tomography (OCT) image; analyze the OCT image based at least in part on an edge detection algorithm; generate a first plurality of image tiles based on the analysis of the OCT image; generate a second plurality of image tiles by operating at least one of the first plurality of image tiles; train a machine learning model for segmenting the OCT image based on the first plurality of image tiles and the second plurality of image tiles; A non-transitory machine-readable medium executable to cause the machine to perform operations including this. According to aspect (16), the analysis of the OCT image includes identifying edges in the OCT image. According to aspect (17), the operations determine some edges identified within each of the first plurality of image tiles; Further comprising selecting at least one of the first plurality of tiles to generate the second plurality of image tiles based on the several edges identified within each image tile. According to aspect (18), the operation of at least one of the image tiles includes at least one of rotating at least one of the image tiles or inverting at least one of the image tiles along an axis. According to aspect (19), the second plurality of image tiles includes image tiles corresponding to rotating at least one of the image tiles at 0 degrees, 90 degrees, 180 degrees, and 270 degrees. According to aspect (20), the second plurality of image tiles includes image tiles corresponding to inverting at least one of the image tiles along at least one of a vertical axis or a horizontal axis.

Claims

1. A system comprising: a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to: acquire an optical coherence tomography (OCT) image of a patient's eye; determine edges in the OCT image based on an edge detection algorithm or manual annotation; generate a plurality of image tiles based on the OCT image; generate a plurality of additional image tiles by operating on at least one of the plurality of image tiles; generate a plurality of additional image tiles by generating different versions of the same image tile, each version including added artifacts; train a machine learning model to identify edges in the OCT image based on the plurality of image tiles and the plurality of additional image tiles; use the trained machine learning model to identify edges in the OCT image; enhance the OCT image by highlighting the identified edges in the OCT image; analyze the enhanced OCT image to provide recommendations regarding the selection of an intraocular lens or contact lens suitable for the patient; one or more hardware processors configured to perform operations including; including; wherein the operation on at least one of the image tiles includes at least one of rotating at least one of the image tiles or flipping at least one of the image tiles along an axis; wherein the plurality of additional image tiles includes image tiles corresponding to rotating at least one of the image tiles by 0 degrees, 90 degrees, 180 degrees, and 270 degrees; wherein the plurality of additional image tiles includes image tiles corresponding to flipping at least one of the image tiles along at least one of a vertical axis or a horizontal axis; A system.

2. The system of claim 1, wherein the edge detection algorithm includes a graph search algorithm.

3. The system of claim 1, wherein the machine learning model includes a deep convolutional neural network.

4. The system of claim 1, wherein the operation further comprises predicting an edge in a second OCT image using the trained machine learning model. **Claim 5** A computer-implemented method performed in a system according to any one of claims 1 to 4, the method comprising: acquiring a biomedical image; determining boundaries of different tissues in the biomedical image based on the edge detection algorithm; generating a first plurality of image tiles based on the biomedical image; generating a second plurality of image tiles by operating on at least one of the first plurality of image tiles; training, by one or more hardware processors, the machine learning model for segmenting the biomedical image based on the first plurality of image tiles and the second plurality of image tiles; A method comprising the steps of: **Claim 6** The method of claim 5, wherein the determined boundary corresponds to an anterior corneal layer of an eye in the biomedical image. **Claim 7** The method of claim 5, wherein the generating of the first plurality of image tiles comprises dividing the biomedical image into image patches. **Claim 8** The method of claim 7, further comprising analyzing one or more characteristics of the biomedical image, wherein the biomedical image is divided into the first plurality of image tiles based on the analysis. **Claim 9** The generating of the first plurality of image tiles comprises: analyzing a plurality of different portions of the biomedical image; selecting, from the plurality of different portions, a subset of portions of the biomedical image that exceed a relevance threshold; The method of claim 5, comprising the steps of: **Claim 10** The method of claim 9, wherein the analyzing of the plurality of different portions comprises determining whether a portion of the biomedical image from the plurality of different portions includes a boundary determined by the edge detection algorithm. **Claim 11** The method of claim 5, wherein at least two of the second plurality of image tiles partially overlap. **Claim 12** A non-transitory machine-readable medium storing machine-readable instructions, the machine-readable instructions causing a machine to: acquire an optical coherence tomography (OCT) image of a patient's eye; analyze the OCT image based at least in part on an edge detection algorithm; Generating a first plurality of image tiles based on the analysis of the OCT image; Generating a second plurality of image tiles by operating on at least one image tile of the first plurality of image tiles; Training a machine learning model for identifying edges in the OCT image based on the first plurality of image tiles and the second plurality of image tiles; Identifying edges in the OCT image using the trained machine learning model; Enhancing the OCT image by highlighting the edges identified in the OCT image; Analyzing the enhanced OCT image to provide recommendations regarding the selection of an intraocular lens or contact lens suitable for a patient; being executable to cause an operation including; wherein the operation of at least one of the image tiles includes at least one of rotating at least one of the image tiles or inverting at least one of the image tiles along an axis; generating the second plurality of image tiles includes generating different versions of the same image tile, each version including an added artifact; the second plurality of image tiles includes image tiles corresponding to rotating at least one of the image tiles at 0 degrees, 90 degrees, 180 degrees, and 270 degrees; the second plurality of image tiles includes image tiles corresponding to inverting at least one of the image tiles along at least one of a vertical axis or a horizontal axis; A non-transitory machine-readable medium.

13. The non-transitory machine-readable medium according to claim 12, wherein the analysis of the OCT image includes identifying edges in the OCT image.

14. The operation is determining some edges identified within each of the image tiles in the first plurality of image tiles; selecting at least one of the image tiles from the first plurality of image tiles to generate the second plurality of image tiles based on the some edges identified within each of the image tiles. The non-transitory machine-readable medium according to claim 13, further including.

Images