Multi-label fluid segmentation and classification system, and multi-label fluid segmentation and classification method

Abstract

A multi-label fluid segmentation and classification system according to an embodiment of the present invention comprises: a collection unit for collecting optical coherence tomography (OCT) images; a memory that stores a multi-label fluid segmentation and classification program for segmenting and classifying multi-label fluids representing body fluids accumulated in a plurality of spaces between a plurality of retinal layers included in the OCT image; and a processor for executing the program so as to classify and segment the multi-label fluids in the OCT image, wherein the processor includes a training unit for training, with the multi-label fluid classification and segmentation method, a deep learning model, which extracts features from the OCT image and simultaneously performs, by using the features, classification prediction for the multi-label fluids and segmentation for each of the multi-label fluids.

Description

Multi-label fluid segmentation and classification system and multi-label fluid segmentation and classification method

[0001] The present invention relates to a multi-label fluid segmentation and classification system and a multi-label fluid segmentation and classification method, and more particularly, to a system and method for segmenting and classifying multi-label fluid representing body fluid accumulated in multiple spaces of multiple retinal layers included in an optical coherence tomography (OCT) image.

[0002] In recent years, the convergence of deep learning and medical imaging has sparked a paradigm shift in ophthalmology, heralding a new era of automated and precise diagnostic capabilities.

[0003] Optical coherence tomography (OCT), a cornerstone of modern ophthalmology, can provide unparalleled insight into ocular anatomy and pathology. By harnessing the power of deep learning algorithms, OCT images surpass traditional manual analysis, enabling rapid, accurate, and standardized diagnoses that can transform patient care.

[0004] OCT images can provide high-resolution images of each layer of the retina. Ophthalmologists can interpret OCT images to detect changes in eye structure and investigate various pathologies, such as age-related macular degeneration (AMD), epiretinal membranes (ERM), and macular edema (ME). Therefore, accurate fluid segmentation and classification in OCT images can play a crucial role in effective diagnosis and treatment planning.

[0005] Previous multi-task learning research on medical image processing has primarily relied on the encoder-decoder framework, with the UNet architecture widely adopted. However, this approach can struggle to handle the long-range dependencies in OCT images. In particular, small convolutional kernels can engage local regions, causing each layer to focus on local feature patterns rather than global features.

[0006] The purpose of the present invention is to solve the above problems, and to provide a multi-label fluid segmentation and classification system and a multi-label fluid segmentation and classification method that can efficiently segment and classify fluids in multiple spaces included in an OCT image.

[0007] In order to achieve the above object, according to an embodiment of the present invention, a multi-label fluid segmentation and classification system includes a collection unit that collects an optical coherence tomography (OCT) image, a memory in which a multi-label fluid segmentation and classification program that segments and classifies multi-label fluid representing body fluid accumulated in a plurality of spaces between a plurality of retinal layers included in the OCT image is stored, and a processor that executes the program to classify and segment the multi-label fluid from the OCT image, and the processor includes a learning unit that extracts features from the OCT image and simultaneously performs classification prediction for the multi-label fluid and segmentation for each of the multi-label fluids using the features to teach a method of classifying and segmenting the multi-label fluid.

[0008] The deep learning model may include a normalization module that normalizes the OCT image, a feature extractor that extracts a feature map from the OCT image, a classification branch that compresses the feature map and supplies the compressed feature map to two fully connected layers to generate a classification prediction for the multi-label fluid, and a segmentation branch that upsamples the feature map and segments each of the multi-label fluids through a segmentation head.

[0009] The above feature extractor may include a residual block that performs a residual convolution operation, and a transformer encoder that is combined with the residual block and captures a local context and a long-range context.

[0010] The above learning unit can train the deep learning model based on a first loss function according to the classification task of the classification branch and a second loss function according to the division task of the division branch.

[0011] According to another embodiment of the present invention, a multi-label fluid segmentation and classification method is provided, wherein a processor executes a multi-label fluid segmentation and classification program that segments and classifies multi-label fluid representing body fluid accumulated in a plurality of spaces between a plurality of retinal layers included in an OCT image stored in a memory, and classifies and segments the multi-label fluid from the OCT image, the method comprising the steps of: collecting an optical coherence tomography (OCT) image; extracting features from the OCT image; and teaching a deep learning model, which simultaneously performs classification prediction for the multi-label fluid and segmentation for each of the multi-label fluids using the features, a method of classifying and segmenting the multi-label fluid.

[0012] The method may further include a step of normalizing the OCT image, a step of extracting a feature map from the OCT image, a step of compressing the feature map and supplying the compressed feature map to two fully connected layers to generate a classification prediction for the multi-label fluid, and a step of upsampling the feature map and segmenting each of the multi-label fluids through a segmentation head.

[0013] The step of extracting the above feature map may include a step of a residual block performing a residual convolution operation, and a step of a transformer encoder combined with the residual block capturing a local context and a long-range context.

[0014] The above learning step may include a step of learning the deep learning model based on a first loss function according to a classification task and a second loss function according to a segmentation task.

[0015] According to the present invention, an accurate and stable process for segmenting and classifying multi-class fluids using OCT images can be provided.

[0016] According to the present invention, it is possible to assist ophthalmologists in diagnosing and treating ophthalmic diseases, save time required for diagnosis, improve accuracy of diagnosis, reduce burden on patients, and reduce costs for patients.

[0017] According to the present invention, not only can the diagnostic accuracy be improved, but the cost and time required for managing retinal diseases can be reduced, effective treatment and intervention can be achieved through early diagnosis of the disease, and ultimately, the prognosis and quality of life of patients can be improved.

[0018] FIG. 1 is a block diagram schematically illustrating the configuration of a multi-label fluid segmentation and classification system according to one embodiment of the present invention.

[0019] FIG. 2 is an example of an OCT image according to one embodiment.

[0020] FIG. 3 is an image showing three types of fluids included in an OCT image according to one embodiment.

[0021] Figure 4 is a block diagram schematically showing the detailed configuration of the processor illustrated in Figure 1.

[0022] FIG. 5 is an exemplary diagram illustrating a normalization operation of an OCT image according to one embodiment.

[0023] Figure 6 is an example showing a detailed configuration of the feature extractor illustrated in Figure 4.

[0024] FIG. 7 is a block diagram schematically illustrating the overall framework of a multi-label fluid segmentation and classification operation according to one embodiment.

[0025] FIG. 8 is a flowchart of a multi-label fluid segmentation and classification method according to one embodiment.

[0026] The present invention can be implemented with various modifications without departing from the spirit or scope of the invention, and may have one or more embodiments. Furthermore, the embodiments described in the "Specific Description for Carrying Out the Invention" and "Drawings" sections of the present invention are examples intended to specifically illustrate the invention and do not limit or restrict the scope of the invention.

[0027] Accordingly, anything that a person with ordinary knowledge in the technical field to which the present invention belongs can easily infer from the “specific contents for carrying out the invention” and “drawings” of the present invention can be interpreted as falling within the scope of the present invention.

[0028] In addition, the size and shape of each component shown in the drawing may be exaggerated for the purpose of explaining the embodiment, and do not limit the size and shape of the invention actually implemented.

[0029] Unless specifically defined, terms used in the specification of the present invention may have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains.

[0030] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0031] FIG. 1 is a block diagram schematically illustrating the configuration of a multi-label fluid segmentation and classification system according to one embodiment of the present invention.

[0032] Referring to FIG. 1, a multi-label fluid segmentation and classification system (1) may include a multi-label fluid segmentation and classification server (10) and a user terminal (20). The multi-label fluid segmentation and classification server (10) may include a collection unit (100), a memory (200), and a processor (300).

[0033] The collection unit (100) can collect optical coherence tomography (OCT) images (hereinafter, "OCT images"). The collection unit (100) can communicate with a user terminal (20) via a wired or wireless network and receive OCT images from the user terminal (20). The collection unit (100) can collect a plurality of OCT images, and the collected plurality of OCT images can include annotated masks for fluid segmentation and labels for fluid type classification. The collection unit (100) can preprocess an OCT image data set including the masks and labels in the collected plurality of OCT images.

[0034] Here, the labels for fluid type classification can indicate the type of each fluid as classified by experts such as ophthalmologists. The fluid type can be one of the following: intraretinal fluid (IRF), subretinal fluid (SRF), pigment epithelial detachment (PED), etc.

[0035] FIG. 2 is an example of an OCT image according to one embodiment.

[0036] Referring to FIG. 2, an OCT image can represent multiple layers included in a subject's retina. The OCT image illustrated in FIG. 2 can include, from top to bottom of FIG. 2, a nerve fiber layer (FIG. 2(a)), a ganglion cell layer (FIG. 2(b)), an inner plexiform layer (FIG. 2(c)), an inner nuclear layer (FIG. 2(d)), an outer plexiform layer (FIG. 2(e)), an outer nuclear layer (FIG. 2(f)), an external limiting membrane (FIG. 2(g)), an ellipsoid region (FIG. 2(h)), an interdigitated retinal pigment epithelium (RPE) (FIG. 2(i)), an RPE / Bruch's membrane complex (FIG. 2(j)), and a choroid (FIG. 2(k)).

[0037] FIG. 3 is an image showing three types of fluids included in an OCT image according to one embodiment.

[0038] Referring to Figure 3, an OCT image can represent three types of fluid using multiple layers. The three types of fluid may include, for example, intraretinal fluid (IRF), subretinal fluid (SRF), and pigment epithelial detachment (PED).

[0039] Intraretinal fluid (IRF) may consist of continuous fluid-filled spaces containing columns of tissue. These spaces may appear as discrete, hyporeflective scisoidal pockets on OCT images, and are therefore sometimes referred to as intraretinal scisoidal fluid.

[0040] Subretinal fluid (SRF) may represent an accumulation of clear or lipid-rich exudate in the subretinal space, i.e., between the neurosensory retina and the underlying retinal pigment epithelium (RPE).

[0041] Pigment epithelial detachment (PED) is specific to AMD and can manifest as detachment of the retinal pigment epithelium (RPE, the bright band at the base of the retina) with fluid accumulation overlapping the remaining retina from Bruch's membrane. It can present in three subtypes: serial, fibrovascular, or drusen. Age-related macular degeneration (AMD) is a condition in which the macula, responsible for central vision, deteriorates over time. AMD can be characterized by lesions such as drusen, pigmentary changes, geographic atrophy, and neovascularization.

[0042] The memory (200) may include volatile memory and / or non-volatile memory. The memory (200) may store, for example, commands or data related to the collection unit (100) and the processor (300), one or more programs and / or software, an operating system, etc., to implement and / or provide operations, functions, etc. provided by the multi-label fluid segmentation and classification system (1).

[0043] The program stored in the memory (200) may include a multi-label fluid segmentation and classification program (hereinafter, “multi-label fluid segmentation and classification program”) that segments and classifies multi-label fluid representing body fluid accumulated in multiple spaces of multiple retinal layers included in an OCT image. The multi-label fluid segmentation and classification program may provide a service for segmenting and classifying multi-label fluid included in an OCT image.

[0044] Therefore, in one embodiment, the multi-label fluid segmentation and classification program can classify multi-label fluids representing each of a plurality of fluids (e.g., IRF, SRF, PED) classified and segmented from an OCT image by the processor (300), and segment each multi-label fluid.

[0045] The processor (300) can execute a multi-label fluid segmentation and classification program to classify and segment the multi-label fluid from the OCT image. The processor (300) can perform a transformer-based multi-task learning method for segmenting and classifying the multi-label fluid from the OCT image. The processor (300) can generate result data representing each multi-label fluid as a result of executing the multi-label fluid segmentation and classification program and transmit the result data to the user terminal (20). The processor (300) can control the operation of each of the collection unit (100) and the memory (200). The processor (300) can be a computing device. The processor (300) can include at least one of a processing unit (Processor) such as an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a microcontroller, and a microprocessor.

[0046] Figure 4 is a block diagram schematically showing the detailed configuration of the processor illustrated in Figure 1.

[0047] Referring to FIG. 4, the processor (300) may include a deep learning model (310) and a learning unit (320). The deep learning model (310) may extract features from an OCT image and, using the extracted features, simultaneously perform classification prediction for multi-label fluids and segmentation for each of the multi-label fluids. The learning unit (320) may teach the deep learning model (310) a method for classifying and segmenting multi-label fluids.

[0048] The deep learning model (310) may include a normalization module (311), a feature extractor (312), a classification branch (313), and a segmentation branch (314).

[0049] The normalization module (311) can normalize OCT images. The normalization module (311) can adjust the size of the OCT image to a predetermined size and normalize it. For example, the normalization module (311) can normalize the OCT image to fit various systems such as Cirrus, Spectralis, and Topcon.

[0050] FIG. 5 is an exemplary diagram illustrating a normalization operation of an OCT image according to one embodiment.

[0051] Referring to FIG. 5, the normalization module (311) can adjust the size of the OCT image ((a) of FIG. 5) to a size of 512*512 and generate a normalized OCT image ((b) of FIG. 5) by removing the upper and lower portions except for the middle region representing multiple layers of the retina representing multi-label fluid.

[0052] The feature extractor (312) can extract a feature map from an OCT image (e.g., an OCT image normalized by the normalization module (311). The feature extractor (312) can extract important features from the OCT image normalized by the normalization module (311) and remove unnecessary images. The encoder of the feature extractor (312) can effectively capture long-range dependencies by incorporating a self-attention mechanism across the entire portion, thereby extracting better features.

[0053] The encoder portion of the feature extractor (312) can be implemented by including a ResFormer block. In the following description, it is assumed that the feature extractor (312) is implemented by a ResFormer block, but this is only an example and the invention is not limited thereto.

[0054] Figure 6 is an example showing a detailed configuration of the feature extractor illustrated in Figure 4.

[0055] Referring to FIG. 6, the feature extractor (312) may include a residual block (3121) and a transformer encoder (3122).

[0056] The residual block (3121) can perform a residual convolution operation.

[0057] The transformer encoder (3122) can be combined with the residual block (3121). The transformer encoder (3122) can capture both local context and long-range context.

[0058] The transformer encoder (3122) can combine the result of layer normalization and multi-head self-attention with the residual generated by the residual block (3121) through a convolution operation on the input of the encoder, and output the result by merging the result of layer normalization and feed forward layer with the result of the combined result (for example, the result of combining the result of layer normalization and multi-head self-attention with the residual block (3121) and the residual). The transformer encoder (3122) can repeatedly perform attention performance, residual connection, normalization, etc. to extract a feature map that captures both local context and long-range context.

[0059] The feature map output by the converter encoder (3122) can be input to the classification branch (313) and the division branch (314) illustrated in FIG. 4.

[0060] The classification branch (313) can compress the feature map and supply the compressed feature map to two fully connected layers to generate a classification prediction for multi-label fluid.

[0061] The segmentation branch (314) can upsample the feature map and segment each multi-label fluid through the segmentation head. The segmentation branch (314) can upsample the feature map using transposed convolutions to increase the spatial dimension while reducing the number of channels for the extracted feature map. For example, the segmentation branch can upsample the feature map to 512*512, which is the original size of the normalized OCT image. The segmentation head of the segmentation branch (314) can generate segmentation masks.

[0062] FIG. 7 is a block diagram schematically illustrating the overall framework of a multi-label fluid segmentation and classification operation according to one embodiment.

[0063] Referring to FIG. 7, the feature extractor (312) may include multiple layers including a Patch Merging and ResFormer block (e.g., a transformer encoder (3122 of FIG. 6)).

[0064] Referring to FIG. 7, the classification branch (313) can classify each of the three types of multi-label fluids, for example, IRF, SRF, and PED, by adding the global average pooling (GAP) for the output of the ResFormer block (B4) of the last layer among the multiple layers of the feature extractor (312) and the GAP for the output of the ResFormer block (B3) of the previous layer of the last layer.

[0065] Referring to FIG. 7, the segmentation branch (314) upsamples the feature map using the Patch Expanding and Decoder Block of the skip-connected decoder with the ResFormer block of the feature extractor (312), and segments each of the three types of multi-label fluids, for example, IRF, SRF, and PED, through the segmentation head.

[0066] The learning unit (320) may use a first loss function (hereinafter, “first loss function”) according to the classification task of the classification branch (313) and a second loss function (hereinafter, “second loss function”) according to the division task of the division branch (314) to update the deep learning model (310). The learning unit (320) may train the deep learning model (310) based on the first loss function and the second loss function.

[0067] The first loss function may be a multi-label loss function according to the classification task of the classification branch (313) according to [Mathematical Formula 1] below.

[0068]

[0069] Here L ml is a multi-label loss function, y is the correct value for the input of the classification branch (313), and is the output value of the classification branch (313), and C may be the number of labels indicated by the multi-label fluid. For example, if the multi-label fluid segmentation and classification system (1) according to one embodiment classifies and segments three types of fluids, IRF, SRF, and PED, C may be 3.

[0070] The classification branch (313) can use accuracy (Acc), sensitivity (Sen), and specificity (Spe) for each class (label) in classification tasks to evaluate performance. Additionally, the micro-average (μ-average) of each indicator can be utilized to more accurately represent overall performance.

[0071] The micro-average accuracy (μACC) can be determined by calculating the accuracy after counting the number of true negatives (TN), true positives (TP), false negatives (FN), and false positives (FP) across all classes, as in [Mathematical Formula 2] below.

[0072]

[0073] The micro-average sensitivity (μSen) can be determined by adding up the number of false negatives and true positives of all classes and then calculating the sensitivity, as in [Mathematical Equation 3] below.

[0074]

[0075] The micro-average specificity (μSpe) can be determined by summing the number of false positives and true negatives across all classes and then calculating the specificity, as in [Mathematical Equation 4] below.

[0076]

[0077] The second loss function may be a dice loss function according to the division operation of the division branch (314) according to [Mathematical Formula 5] below. The dice loss function may be a function using the dice score (DSC) and the Jaccard index (JI).

[0078]

[0079]

[0080]

[0081] Dice here loss is a dice loss function, DSC is a dice coefficient, GT is a ground truth, and PR can represent a predicted region. TP can be a true positive, FP a false positive, and FN a false negative. As DSC approaches 1, the closer the actual and predicted are, and at this time, the dice loss function can approach 0. The learning unit (320) can perform optimization learning so that the dice loss function approaches 0.

[0082] The learning unit (320) can train the deep learning model (310) so that each of the first loss function value and the second loss function value approaches 0.

[0083] One embodiment can be seen to achieve superior performance across both segmentation and classification tasks compared to existing multi-task learning methods. Furthermore, one embodiment can outperform existing single-task learning methods for segmentation. Furthermore, one embodiment can also demonstrate superior performance compared to well-known backbones such as EfficientNet, Inception V3, Vision Transformer, and Swin Transformer.

[0084] Therefore, the multi-label fluid segmentation and classification system (1) according to one embodiment can outperform other existing methods in both segmentation and classification tasks. Furthermore, collaborative learning that considers both the first loss function and the second loss function can improve performance in both segmentation and classification compared to single-task (e.g., segmentation or classification) learning methods.

[0085] FIG. 8 is a flowchart of a multi-label fluid segmentation and classification method according to one embodiment.

[0086] Hereinafter, descriptions of each component of the multi-label fluid segmentation and classification system (1) that overlap with the descriptions above may be omitted. A multi-label fluid segmentation and classification method according to one embodiment may be a method of classifying and segmenting multi-label fluids from an OCT image by having a processor (300) execute a multi-label fluid segmentation and classification program stored in a memory (200).

[0087] Referring to FIG. 8, the collection unit (100) can collect OCT images (S100).

[0088] The processor (300) can normalize the OCT image (S200).

[0089] The processor (300) can extract features from a normalized OCT image (S300).

[0090] The processor (300) can classify multi-label fluids using the extracted features (S400).

[0091] The processor (300) can segment each multi-label fluid using the extracted features (S500).

[0092] The processor (300) can train a deep learning model (310) that simultaneously performs classification and segmentation for multi-label fluids based on the loss functions of each of the classification branch (313) and the segmentation branch (314) (S600).

[0093] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0094] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0095] While the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications may be made within the scope of the detailed description and the attached drawings, as long as the scope of the invention is not deviated from or the effects are not impaired. Furthermore, it is obvious that such embodiments fall within the scope of the present invention.

Claims

1. A collection unit that collects optical coherence tomography (OCT) images; A memory storing a multi-label fluid segmentation and classification program that segments and classifies multi-label fluid representing fluid accumulated in multiple spaces between multiple retinal layers included in the OCT image; and A processor for executing the above program to classify and segment the multi-label fluid from the OCT image, The above processor, A learning unit that extracts features from the OCT image and uses the features to simultaneously perform classification prediction for the multi-label fluid and segmentation for each of the multi-label fluids, and teaches a method of classifying and segmenting the multi-class fluids to a deep learning model. Multi-label fluid segmentation and classification system.

2. In paragraph 1, The above deep learning model is, A normalization module for normalizing the above OCT image; A feature extractor for extracting a feature map from the above OCT image; A classification branch that compresses the above feature map and feeds the compressed feature map to two fully connected layers to generate a classification prediction for the multi-label fluid; and A segmentation branch that upsamples the above feature map and segments each of the multi-label fluids through a segmentation head, Multi-label fluid segmentation and classification system.

3. In paragraph 2, The above feature extractor is, A residual block that performs residual convolution operations; and A transformer encoder combined with the residual block and capturing local context and long-range context, Multi-label fluid segmentation and classification system.

4. In paragraph 2, The above learning department, Training the deep learning model based on a first loss function according to the classification task of the classification branch and a second loss function according to the division task of the division branch. Multi-label fluid segmentation and classification system.

5. A method for classifying and segmenting multi-label fluid from an OCT image by executing a multi-label fluid segmentation and classification program that segments and classifies multi-label fluid representing body fluid accumulated in multiple spaces between multiple retinal layers included in an OCT image stored in a memory, A step of collecting optical coherence tomography (OCT) images; A step of extracting features from the OCT image; and A step of teaching a deep learning model a method of classifying and segmenting the multi-label fluid by simultaneously performing classification prediction for the multi-label fluid and segmentation for each of the multi-label fluids using the above features, A multi-label fluid segmentation and classification method.

6. In paragraph 5, A step of normalizing the above OCT image; A step of extracting a feature map from the above OCT image; A step of compressing the above feature map and providing the compressed feature map to two fully connected layers to generate a classification prediction for the multi-label fluid; and Further comprising the step of upsampling the above feature map and segmenting each of the multi-label fluids through a segmentation head. A multi-label fluid segmentation and classification method.

7. In paragraph 6, The step of extracting the above feature map is: A step in which a residual block performs a residual convolution operation; and A transformer encoder combined with the residual block comprises a step of capturing local context and long-range context. A multi-label fluid segmentation and classification method.

8. In paragraph 6, The above learning steps are: A step of training the deep learning model based on a first loss function according to a classification task and a second loss function according to a segmentation task, A multi-label fluid segmentation and classification method.

Images