Method for annotating biological tissue image with lesion type and computing system for performing same

Abstract

Provided are a method for annotating a portion corresponding to a tissue in a biological tissue image with a predetermined grade according to a severity grade system or a predetermined histological classification of a lesion, and a computing system for performing the same. An aspect of the present invention provides a method including the steps of: displaying a biological tissue image; receiving, by the computing system, a selection command of an annotation type which is one of a plurality of lesion types, wherein the lesion type means a predetermined grade according to a severity grade system or a predetermined histological classification of a lesion; receiving a designation command for a user designated area which is a partial area of the displayed biological tissue image; specifying an annotation area which is a part corresponding to a biological tissue in the user designated area; reflecting the annotation type to a part corresponding to the annotation area of a pixel-unit diagnosis result of the biological tissue image, wherein the pixel-unit diagnosis result of the biological tissue image includes a lesion type allocated to each pixel constituting the biological tissue image; and visualizing the pixel-unit diagnosis result and displaying same together with the biological tissue image.

Description

Method for annotating lesion types in biological tissue images and computing system for performing the same

[0001] The present invention relates to a method for annotating lesion types in biological tissue images and a computing system for performing the same. More specifically, the present invention relates to a method for annotating a portion of a biological tissue image corresponding to a tissue with a grade according to a predetermined severity grading system or a histological classification of a predetermined lesion, and a computing system for performing the same.

[0002]

[0003] In medicine, pathology, which involves interpreting actual specimens, is crucial for accurately diagnosing and predicting the prognosis of serious diseases. Conventional pathology diagnostic methods involve visual observation and interpretation of diagnostic pathology slides produced from specimens through an optical microscope by a pathologist. The dawn of digital pathology began with the use of a computer-connected microscope camera to convert pathology slides into digital images, which are then viewed and interpreted on a monitor. Recently, the advent of digital slide scanners has led to the widespread adoption of this method, which converts entire pathology slides into a single digital image, which is then viewed and interpreted on a computer monitor. Digital pathology-based pathology, which interprets scanned images on a monitor, is replacing the traditional method of visually examining specimen slides through an optical microscope.

[0004] Meanwhile, for the diagnosis of serious diseases such as cancer, part or all of the tissue is collected, a biopsy image is created, and the severity of the disease is assessed through histopathological examination. There is an agreed-upon system for reporting histological severity for each disease, and these systems grade and report the severity based on the morphological characteristics of the tissue. For example, the Gleason grading system for prostate cancer classifies tissue patterns into grades from 1 to 5 based on the morphological characteristics of the glandular tissue, and the overall severity is graded based on the proportion of each grade area in the entire tissue. In addition, breast cancer severity can be graded using histological grade. Histopathological examination of tissue images can also be used to histologically classify lesions. For example, it can be used to distinguish between invasive and intraepithelial breast lesions, or between invasive and precancerous lesions (atypical adenomatous (glandular) hyperplasia) in lung lesions.

[0005] In the diagnosis of biological tissue images, such as pathology slide images, a key part of the diagnostic process involves a specialist annotating specific lesion types (either grades according to a given severity grading system or histological classifications of specific lesions) on the pathology slide images. Typically, annotation involves a specialist designating a region of the biological tissue image displayed on the screen and annotating that region with a specific lesion type. However, because lesion types exist only in biological tissue, it is necessary to avoid annotating regions that are not biological tissue. However, because the boundaries of biological tissues are often irregular and complex, it is difficult for humans to accurately annotate lesion regions according to the boundaries of biological tissue. Therefore, a technology is needed that can accurately annotate regions where lesions exist, even in such an annotation environment.

[0006]

[0007] The technical problem to be achieved by the present invention is to provide a method for easily annotating a grade according to a predetermined severity grading system or a histological classification of a predetermined lesion in a portion corresponding to a tissue among a biological tissue image, and a computing system for performing the method.

[0008]

[0009] According to one aspect of the present invention, a method is provided, comprising: a step of a computing system displaying a biological tissue image; a step of the computing system receiving a command to select an annotation type, which is one of a plurality of lesion types, wherein the lesion type means a grade according to a predetermined severity grading system or a histological classification of a predetermined lesion; a step of the computing system receiving a command to designate a user-specified area, which is a portion of the displayed biological tissue image; a step of the computing system specifying an annotation area, which is a portion of the user-specified area corresponding to biological tissue; a step of the computing system reflecting the annotation type in a portion corresponding to the annotation area of ​​a pixel-wise diagnosis result of the biological tissue image, wherein the pixel-wise diagnosis result of the biological tissue image includes a lesion type assigned to each pixel constituting the biological tissue image; and a step of the computing system visualizing the pixel-wise diagnosis result and displaying it together with the biological tissue image.

[0010] In one embodiment, the step of specifying an annotation region corresponding to a biological tissue among the user-specified regions may include the step of the computing system generating a tissue mask image of the biological tissue image, wherein each pixel of the tissue mask image is assigned a binary value indicating whether the corresponding pixel of the biological tissue image corresponds to biological tissue; and the step of specifying an annotation region corresponding to a biological tissue among the user-specified regions based on the tissue mask image.

[0011] In one embodiment, the step of specifying an annotation region corresponding to a biological tissue among the user-specified regions based on the tissue mask image may include the steps of: generating a designated region mask corresponding to the user-specified region, wherein each pixel of the designated region mask is assigned a binary value indicating whether a pixel of the biological tissue image corresponding thereto is included in the user-specified region; and performing a logical AND operation on each pixel of the tissue mask image and the designated region mask image.

[0012] In one embodiment, the step of reflecting the annotation type in a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image may include the step of assigning the annotation type to a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image to which no lesion type has been assigned yet; and the step of selecting and assigning either the lesion type that has already been assigned or the annotation type according to predetermined selection criteria to a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image to which a lesion type different from the annotation type has already been assigned.

[0013] In one embodiment, the lesion type may be any one of a Gleason pattern according to the Gleason grading system for prostate cancer, a tumor grade according to the histologic grading system for breast cancer, a classification of invasive carcinoma lesions and intraepithelial carcinoma lesions of the breast, and a classification of invasive carcinoma lesions and precancerous lesions of the lung.

[0014] According to another aspect of the present invention, a computer program is provided, which is installed in a data processing device and recorded on a recording medium for performing the above-described method.

[0015] According to another aspect of the present invention, a computer-readable recording medium having recorded thereon a computer program for performing the above-described method is provided.

[0016] According to another aspect of the present invention, a computing system is provided, comprising: a processor; and a memory storing a computer program, wherein the computer program, when executed by the processor, causes the computing system to perform the above-described method.

[0017] According to another aspect of the present invention, there is provided a computing system comprising: a display module for displaying a biological tissue image; a receiving module for receiving a command to select an annotation type, which is one of a plurality of lesion types, and a command to designate a user-specified area, which is a portion of the displayed biological tissue image, wherein the lesion type means a grade according to a predetermined severity grading system or a histological classification of a predetermined lesion; a specific module for specifying an annotation area, which is a portion of the user-specified area corresponding to biological tissue; and an annotation module for reflecting the annotation type in a portion corresponding to the annotation area of ​​a pixel-by-pixel diagnosis result of the biological tissue image, wherein the pixel-by-pixel diagnosis result of the biological tissue image includes a lesion type assigned to each pixel constituting the biological tissue image; wherein the display module, when the annotation type is reflected in a portion corresponding to the annotation area of ​​the pixel-by-pixel diagnosis result of the biological tissue image, visualizes the pixel-by-pixel diagnosis result and displays it together with the biological tissue image.

[0018] In one embodiment, the computing system further includes a tissue mask image generation module that generates a tissue mask image corresponding to the biological tissue image, wherein each pixel of the tissue mask image is assigned a binary value indicating whether the corresponding pixel of the biological tissue image corresponds to biological tissue, wherein the specific module can specify an annotation area corresponding to biological tissue among the user-specified areas based on the tissue mask image.

[0019] In one embodiment, the specific module generates a designated area mask corresponding to the user-specified area, wherein each pixel of the designated area mask is assigned a binary value indicating whether the corresponding pixel of the biological tissue image is included in the user-specified area, and performs a logical AND operation on each pixel of the tissue mask image and the designated area mask image to specify an annotation area corresponding to the biological tissue among the user-specified areas.

[0020] In one embodiment, the annotation module may assign an annotation type to a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image to which no lesion type has been assigned yet, and may select and assign either the already assigned lesion type or the annotation type according to a predetermined selection criterion to a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image to which a lesion type different from the annotation type has already been assigned.

[0021]

[0022] According to the technical idea of ​​the present invention, a method for easily annotating a grade according to a predetermined severity grading system or a histological classification of a predetermined lesion in a portion corresponding to a tissue in a biological tissue image and a computing system for performing the same can be provided.

[0023]

[0024] In order to more fully understand the drawings cited in the detailed description of the present invention, a brief description of each drawing is provided.

[0025] FIG. 1 is a drawing for explaining a schematic system configuration for implementing a method for annotating a biological tissue image according to an embodiment of the present invention.

[0026] Figure 2 is a flowchart for explaining an annotation method according to one embodiment of the present invention.

[0027] Figures 3a to 3e are drawings showing an example of an annotation UI.

[0028] Figure 4 is a flowchart illustrating an example of a specific process of step S140 of Figure 2.

[0029] Figure 5 is a drawing showing an example of a tissue mask image.

[0030] FIG. 6 is a flowchart illustrating an example of a process in which a computing system according to one embodiment of the present invention generates a tissue mask image corresponding to a biological tissue image.

[0031] FIG. 7 is a block diagram illustrating a logical configuration of a computing system (100) according to one embodiment of the present invention.

[0032] FIG. 8 is a block diagram illustrating the physical configuration of a computing system according to one embodiment of the present invention.

[0033]

[0034] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention. In describing the present invention, detailed descriptions of related known technologies will be omitted if they are deemed to obscure the gist of the present invention.

[0035] Terms such as "first," "second," etc. may be used to describe various components, but these components should not be limited by these terms. Terms such as "first," "second," etc. do not denote a particular order and are used solely to distinguish one component from another.

[0036] The terminology used in this application is solely for the purpose of describing specific embodiments and is not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0037] In this specification, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0038] Additionally, in the present specification, when a component "transmits" data to another component, it means that the component may transmit the data directly to the other component, or may transmit the data to the other component via at least one other component. Conversely, when a component "directly transmits" data to another component, it means that the data is transmitted from the component to the other component without going through the other component.

[0039] Hereinafter, the present invention will be described in detail, focusing on embodiments thereof, with reference to the attached drawings. The same reference numerals in each drawing represent the same components.

[0040] FIG. 1 is a diagram for explaining a schematic system configuration for implementing a method for annotating a lesion type in a biological tissue image according to an embodiment of the present invention (hereinafter referred to as the “annotation method”).

[0041] Referring to FIG. 1, an annotation method according to the technical idea of ​​the present invention can be performed by a computing system (100). The computing system (100) provides a GUI (Graphical User Interface) for annotating a lesion type in a biological tissue image, and can provide a function through which a user can easily annotate a specific lesion type in a biological tissue image, particularly in a portion corresponding to the biological tissue. The lesion type refers to a grade according to a predetermined severity grading system or a histological classification of a predetermined lesion, which will be described later.

[0042] A biological tissue image may be an image generated from a slide of a pathological specimen tissue stained using a predetermined staining method. The pathological specimen tissue may be a biopsy taken from various organs of the human body or a surgically excised biological tissue. In one embodiment, the pathological specimen tissue may be a biological tissue collected through a core needle biopsy. Meanwhile, the biological tissue image may be a digital slide image of a stained pathological specimen or a portion of a digital slide image. The pathological specimen slide may be a portion of a sliced ​​pathological specimen. The digital slide image of a pathological specimen may be generated by slicing a pathological specimen to produce a glass slide, staining the glass slide with a predetermined staining agent, and digitizing the glass slide.

[0043] In one embodiment, a computing system (100) may be installed on a predetermined server (10) to implement the technical idea of ​​the present invention. The server (10) refers to a data processing device having a computing capability for implementing the technical idea of ​​the present invention. Generally, a person skilled in the art will be able to easily infer that not only a data processing device that can be accessed by a client through a network, but also any device capable of performing a specific service, such as a personal computer or a mobile terminal, can be defined as a computing device.

[0044] According to one embodiment, the computing system (100) may need to directly determine the pixel-wise diagnosis result of a biological tissue image (e.g., Gleason pattern for each pixel), and in this case, the computing system (100) may use a pre-trained deep learning model (200). The deep learning model (200) may be an artificial neural network that receives a biological tissue image or a patch obtained by dividing a biological tissue image into a certain size as input, and outputs a result value for determining the pixel-wise diagnosis result of the input image (e.g., probability corresponding to each Gleason pattern, etc.).

[0045] The above deep learning model (200) is a neural network artificially constructed based on the operating principles of human neurons, including a multilayer perceptron model, and may refer to a set of information expressing a series of design specifications defining an artificial neural network. The above deep learning model (200) may include, but is not limited to, a convolutional neural network (CNN) widely known in the fields of artificial intelligence, machine learning, or deep learning.

[0046] The server (10) including the computing system (100) may be a system that stores or applies results (e.g., annotation information of a biological tissue image, etc.) output by the computing system (100). For example, the server (10) may be a system for generating a digital document including a biological tissue image in which a lesion type is annotated by the computing system (100), or a diagnostic assistance system for providing information necessary for a doctor to diagnose a disease, but is not limited thereto. The deep learning model (200) and the computing system (100) may be provided in one physical device, i.e., the server (10), but may be provided in different physical devices depending on the embodiment, and various modifications may be possible as needed, as an average expert in the technical field of the present invention will be able to easily infer.

[0047] In one embodiment, the server (10) and / or the computing system (100) may communicate with a user terminal (20). For example, the computing system (100) may provide a GUI for annotating a lesion type in a biological tissue image to the user terminal (20), and the user terminal (20) may transmit a command / signal, etc. for annotating a specific lesion type in a specific area of ​​the biological tissue image to the computing system (100).

[0048] The user terminal (20) may be a data processing device with computing capabilities and may be equipped with a display device for displaying visual information such as biological tissue images or a GUI. The user terminal (20) may include, for example, a personal computer such as a desktop or laptop, as well as a mobile device such as a cell phone or tablet PC.

[0049] Meanwhile, in FIG. 1, an example is shown in which the computing system (100) is implemented in the form of a subsystem installed in a server (10), but it is obvious that, depending on the embodiment, the computing system (100) may be implemented as an independent single system. In this case, the computing system (100) may be a computing system that is a data processing device having a computing capability for implementing the technical idea of ​​the present invention, and may generally include not only a server that is a data processing device that can be accessed by a client through a network, but also a computing device such as a personal computer or a mobile terminal.

[0050] Additionally, depending on the embodiment, the computing system (100) may be implemented in a form included in a user terminal (20).

[0051] Figure 2 is a flowchart for explaining an annotation method according to one embodiment of the present invention.

[0052] The above computing system (100) can display a biological tissue image (S110).

[0053] The above biological tissue image may be a full slide image or a portion of a full slide image created by digitally scanning a slide of biological tissue collected through a biopsy or surgery, etc.

[0054] In one embodiment, the computing system (100) may receive a biological tissue image from the user terminal (20), or may acquire a biological tissue image from a database or storage device provided by the server (10) or the computing system (100), and the computing system (100) may display the acquired biological tissue image.

[0055] According to an embodiment, the computing system (100) may further obtain pixel-by-pixel diagnostic results of the biological tissue image, and may visualize and display the pixel-by-pixel diagnostic results together with the biological tissue image. In this case, the pixel-by-pixel diagnostic results of the biological tissue image may include the type of lesion for each pixel constituting the biological tissue image.

[0056] In this specification, the lesion type may mean a grade according to a severity grading system of a given disease, such as cancer, or may mean a histological classification of a given lesion.

[0057] For example, the above lesion type may be a Gleason pattern according to the Gleason grading system for prostate cancer. The Gleason pattern is a representative grading system indicating the severity of prostate cancer tissue, and is classified into benign or patterns of 1 to 5 depending on the histological differentiation. A classification of 1 indicates differentiation close to normal and low malignancy, whereas a score of 5 indicates cancer with the most severe malignancy. The Gleason pattern used in the present invention can be divided into grades 1 to 5 indicating prostate cancer and benign for each pixel, but in some cases, it can be divided into grades 3 to 5 and benign.

[0058] Another example of lesion type is the tumor grade according to the histologic grading system for breast cancer. In addition to grades that indicate severity, such as Gleason pattern or histologic grade of breast cancer, as mentioned above, lesion type can also refer to a histologic classification of a given lesion, such as the classification of invasive versus non-invasive breast lesions, or invasive versus precancerous lesions (atypical adenomatous hyperplasia) in the lung.

[0059] The pixel-wise diagnosis result of the biological tissue image may be in the form of a two-dimensional array of the same size as the biological tissue image. For example, if the biological tissue image has a size of W pixels × H pixels, the pixel-wise diagnosis result of the biological tissue image may be a two-dimensional array of W × H, and each element of the array may be assigned a pixel-wise diagnosis result of the pixel corresponding to that element.

[0060] For example, if the lesion type is a Gleason pattern, each element of the array may be assigned a value indicating the corresponding Gleason pattern if the pixel corresponding to the element corresponds to Gleason pattern 1 to 5, and a value indicating positivity (e.g., 0) may be assigned otherwise.

[0061] In an embodiment where the lesion type is a histological classification of a breast lesion, i.e., a classification of an invasive lesion and an intraepithelial carcinoma lesion, each element of the array may be assigned a value (e.g., 1) corresponding to the classification if the pixel corresponding to the element is classified as an invasive lesion, a value (e.g., 2) corresponding to the classification if the pixel is classified as an intraepithelial carcinoma, and a value indicating positivity (e.g., 0) in other cases.

[0062] However, depending on the embodiment, each element of the array may be implemented such that if the pixel corresponding to the element does not belong to the tissue region, a value indicating that it is not a tissue region is assigned, and if it belongs to the tissue region, a pixel-by-pixel diagnostic result of the pixel corresponding to the element is assigned. For example, each element of the array may be assigned -1 if the pixel corresponding to the element belongs to the non-tissue region, 0 if it is benign, and otherwise a value corresponding to the lesion type.

[0063] For convenience of explanation, below, an embodiment is described in which the same positive value (e.g., 0) is assigned to both cases in which a pixel belongs to a non-organized area and cases in which it is positive.

[0064] In one embodiment, the initial pixel-by-pixel diagnosis results of the biological tissue image acquired by the computing system (100) may all be filled with values ​​corresponding to positivity (e.g., 0), and the lesion types may be gradually filled in as the annotation method according to the technical idea of ​​the present invention is performed. Alternatively, the initial pixel-by-pixel diagnosis results of the biological tissue image acquired by the computing system (100) may be in a state where some or all of the pixel-by-pixel diagnosis results are already filled in, as in the embodiment below.

[0065] In one embodiment, the pixel-by-pixel diagnostic results of the biological tissue image may be input into the computing system (100) together with the biological tissue image as a result of manual annotation by a pathologist.

[0066] According to an embodiment, the computing system (100) may analyze the biological tissue image to directly obtain a pixel-wise diagnosis result of the biological tissue image. In this case, the computing system (100) may obtain a pixel-wise diagnosis result of the biological tissue image using a pre-trained deep learning model (200-1). More specifically, the computing system (100) may input each of the biological tissue images or patches divided into a grid shape into an input layer of the deep learning model (200-1) and determine the type of lesion for each pixel of the biological tissue image based on an output value output from an output layer of the deep learning model (200-1). In this case, the deep learning model (200-1) may be an artificial neural network that has been trained in advance to determine the type of lesion for each pixel of the input image, and may be an artificial neural network that has been trained in advance to receive an image through an input layer and output a probability that each pixel included in the input image corresponds to each lesion type.

[0067] In one embodiment, the computing system (100) may display a GUI (hereinafter referred to as an “annotation UI”) for enabling a user to annotate a lesion type in a biological tissue image. The annotation UI may display a biological tissue image.

[0068] In one embodiment, the computing system (100) may cause the annotation UI to be displayed on the user terminal (20). To this end, the computing system (100) may transmit information for rendering the annotation UI to the user terminal (20).

[0069] Fig. 3a is a diagram illustrating an example of an annotation UI. Fig. 3a is merely an example illustrating the simplest form of an annotation UI, and the annotation UI may have a different form than that illustrated in Fig. 3a and may, of course, have more GUI components.

[0070] Referring to FIG. 3a, the annotation UI (1000) may include a GUI component (e.g., an image panel, etc.) that displays a biological tissue image (1100).

[0071] Meanwhile, according to an embodiment, the computing system (100) can visualize the pixel-by-pixel diagnosis results of the pre-determined biological tissue image and display them together with the biological tissue image. For example, the computing system (100) can visualize the pixel-by-pixel diagnosis results by assigning a unique color to each lesion type and displaying each pixel with a unique color corresponding to the lesion type assigned to the pixel. FIG. 3A illustrates an example in which visualized results (1300-1 to 1300-4) of the pixel-by-pixel diagnosis results are displayed together with the biological tissue image.

[0072] Meanwhile, the annotation UI (1000) may further include a GUI component (e.g., radio button, drop-down list, etc.) for selecting an annotation type among a plurality of lesion types.

[0073] The annotation type selected by the user may be any one of the above lesion types. FIG. 3a illustrates an example in which Gleason pattern 3 is selected as an annotation type through an annotation type selection UI (1200), which is a GUI component for selecting any one of benign and Gleason patterns 3 to 5.

[0074] Referring again to FIG. 2, the computing system (100) can receive a command to select an annotation type among a plurality of lesion types (S120). For example, if an annotation type among a plurality of lesion types is selected through an annotation type selection UI (1200) displayed on the user terminal (20), the user terminal (20) can transmit information indicating that the annotation type has been selected to the computing system (100).

[0075] In addition, the computing system (100) can receive a command to designate a user-specified area, which is a portion of the displayed biological tissue image (S130).

[0076] Fig. 3b illustrates an example in which a user designates a user-specified region through an annotation UI (1000). As illustrated in Fig. 3b, a user can designate a portion (1400) of a biological tissue image (1100) displayed on the annotation UI (1000). Then, the user terminal (20) can transmit a region designation command including information about the user-specified region (1400), which is a portion of the biological tissue image designated by the user, to the computing system (100).

[0077] Referring again to FIG. 2, the computing system (100) can specify an annotation area corresponding to a biological tissue among the user-specified areas (S140).

[0078] Figure 4 is a flowchart illustrating an example of a specific process of step S140 of Figure 2.

[0079] Referring to FIG. 4, the computing system (100) can generate a tissue mask image of the biological tissue image to specify an annotation area (S141). Each pixel of the tissue mask image is assigned a binary value indicating whether the corresponding pixel of the biological tissue image corresponds to biological tissue. For example, the computing system (100) can generate a tissue mask image as illustrated in FIG. 5.

[0080] FIG. 6 is a flowchart illustrating an example of a process in which a computing system (100) according to one embodiment of the present invention generates a tissue mask image corresponding to a biological tissue image.

[0081] Referring to FIG. 6, the computing system (100) can convert a biological tissue image into an HSV (Hue-Saturation-Value) model (S210). The HSV model is a model that specifies a specific color using the coordinates of hue, saturation, and value, and the method of converting RGB into HSV is widely known.

[0082] The computing system (100) can perform binarization on the S space of a biological tissue image converted into an HSV model to generate a first binarization result. The S space is a space composed of saturation values ​​of the HSV model.

[0083] The above computing system (100) can utilize Otsu thresholding as an image binarization method. Otsu thresholding is a clustering-based image thresholding technique used in the fields of computer vision and image processing.

[0084] The computing system (100) may perform binarization on the 1-V space of the biological tissue image converted into the HSV model to generate a second binarization result (S220). The V space is a brightness value (Value) space of the HSV model (i.e., a matrix having a size of w×h and composed of values ​​(brightness values) of the V channel (w is the width of the biological tissue image, h is the height of the biological tissue image)), and the 1-V space may be a space obtained by subtracting the value of the V channel from a matrix having a size of w×h and filled with 1.

[0085] Meanwhile, the computing system (100) can generate a tissue mask image based on the first binarization result and the second binarization result (S240).

[0086] The first binarization result and the second binarization result may include a binary value (e.g., 0 or 1 or 0 or 255) corresponding to each pixel of the biological tissue image, and the computing system (100) determines, for each pixel of the biological tissue image, if the binary value of the first binarization result corresponding to the pixel or the binary value of the first binarization result corresponding to the pixel is 1 (or 255), that the pixel on the tissue mask image corresponding to the pixel is a tissue pixel (a pixel corresponding to biological tissue), and otherwise (i.e., if the binary value is 0), that the pixel on the tissue mask image corresponding to the pixel is a non-tissue pixel (a pixel that does not correspond to biological tissue), thereby generating a tissue mask image corresponding to the biological tissue image. In summary, the computing system (100) may generate a tissue mask image through a logical OR operation between the image binarization result for the S space and the image binarization result for the 1-V space.

[0087] In addition to the HSV and Otsu thresholding methods described above, various methods for generating binary tissue masks can be applied to the present invention. For example, the computing system (100) may generate tissue mask images using a pre-trained deep learning model (e.g., a convolutional neural network).

[0088] Referring again to FIG. 4, after generating a tissue mask image of the biological tissue image, the computing system (100) can specify an annotation area, which is a portion corresponding to the biological tissue among the user-specified areas, based on the tissue mask image (S142). That is, the computing system (100) can specify a portion corresponding to the biological tissue in the tissue mask image that overlaps with the user-specified area as the annotation area.

[0089] In one embodiment, the computing system (100) generates a designated area mask corresponding to the user-specified area, and performs a logical AND operation on each pixel of the tissue mask image and the designated area mask image to specify an annotation area corresponding to a biological tissue among the user-specified area. At this time, each pixel of the designated area mask is assigned a binary value indicating whether the corresponding pixel of the biological tissue image is included in the user-specified area.

[0090] Referring again to FIG. 2, the computing system (100) can reflect the annotation type in a portion corresponding to the annotation area of ​​the pixel-by-pixel diagnosis result of the biological tissue image (S150), and can visualize the pixel-by-pixel diagnosis result and display it together with the biological tissue image (S160).

[0091] The computing system (100) may assign the annotation type to a portion of the pixel-unit diagnostic result of the biological tissue image, if the portion corresponds to the annotation area and no lesion type has been assigned to the portion.

[0092] FIG. 3c is a drawing showing an example in which the computing system (100) visualizes a state in which the annotation type (Gleason pattern 3) is reflected in a specific annotation area when the user has specified a user-specified area and annotation type as in FIG. 3b.

[0093] Meanwhile, the computing system (100) can select and assign either the lesion type already assigned or the annotation type to a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image, to which a lesion type different from the annotation type has already been assigned, according to a predetermined selection criterion.

[0094] FIG. 3d is a drawing showing a state in which a user additionally selects an annotation type (Gleason pattern 4) through an annotation UI in a state similar to FIG. 3c and designates a user-specified area (1500) in which the pattern is to be reflected.

[0095] When an annotation type and a user-specified area are specified as in FIG. 3d, the computing system (100) can specify an annotation area in the manner described above and reflect the annotation type (Gleason pattern 4) in the specified annotation area, and a reflected example is illustrated in FIG. 3e.

[0096] In the example of Figure 3e, for an area (1600-1) among the annotation areas to which a lesion type has not yet been assigned, the annotation type selected by the user (i.e., Gleason pattern 4) is annotated.

[0097] However, in the case of an area (S1600-2) where a lesion type has already been annotated, either the previously annotated lesion type or the lesion type to be newly annotated must be selected. In this case, the computing system (100) can make a selection based on a predetermined selection criterion.

[0098] The selection criteria may vary depending on the embodiment. For example, if the previously annotated lesion type is a lesion type determined by the computing system (100) itself, the computing system (100) may preferentially select a pattern directly specified by the user.

[0099] Alternatively, the computing system (100) may select a lesion type to be annotated according to a predetermined priority (e.g., priority of pattern 5 > priority of pattern 4 > priority of pattern 3). Alternatively, the computing system (100) may receive a selection from the user of either a lesion type already assigned or an annotation type, and may annotate the lesion type selected by the user. Of course, there may be various other selection criteria. Fig. 3e illustrates an example in which Gleason pattern 4 is annotated in a region (S1600-2) in which a lesion type was already annotated.

[0100] FIG. 7 is a block diagram illustrating a logical configuration of a computing system (100) according to one embodiment of the present invention.

[0101] The computing system (100) may refer to a logical configuration equipped with hardware resources and / or software necessary to implement the technical idea of ​​the present invention, and does not necessarily refer to a single physical component or device. That is, the computing system (100) may refer to a logical combination of hardware and / or software provided to implement the technical idea of ​​the present invention, and, if necessary, may be implemented as a set of logical configurations for implementing the technical idea of ​​the present invention by being installed in devices spaced apart from each other and performing their respective functions. In addition, the computing system (100) may refer to a set of configurations that are separately implemented for each function or role for implementing the technical idea of ​​the present invention. Each configuration of the computing system (100) may be located on different physical devices or may be located on the same physical device. In addition, depending on the implementation example, the combination of software and / or hardware constituting each component of the computing system (100) may also be located on different physical devices, and the configurations located on different physical devices may be organically combined with each other to implement each of the modules.

[0102] Furthermore, the term "module" in this specification may refer to a functional and structural combination of hardware for implementing the technical concepts of the present invention and software for operating the hardware. For example, the module may refer to a logical unit of a given code and hardware resources for executing the given code, and it is readily apparent to an average expert in the technical field of the present invention that it does not necessarily refer to physically connected code or a single type of hardware.

[0103] Referring to FIG. 7, the computing system (100) may include a display module (110), a receiving module (120), a specific module (130), an annotation module (140), and a tissue mask image generation module (150). Depending on the embodiment of the present invention, some of the above-described components may not necessarily correspond to components essential to the implementation of the present invention, and, of course, depending on the embodiment, the computing system (100) may include more components than this. For example, the computing system (100) may further include a communication module (not shown) for communicating with an external device, a storage module (not shown) for storing data, a control module (not shown) for controlling components and resources of the computing system (100), etc.

[0104] The display module (110) can display a biological tissue image. In one embodiment, the display module (110) can display an annotation UI including the biological tissue image on a user terminal (20).

[0105] The receiving module (120) may receive a command to select an annotation type, which is one of a plurality of lesion types, and a command to designate a user-specified area, which is a portion of the displayed biological tissue image. In one embodiment, the receiving module (120) may receive a command to select an annotation type, which is a lesion type selected by a user through the annotation UI, and a command to designate a user-specified area designated by the user through the annotation UI.

[0106] The above specific module (130) can specify an annotation area that corresponds to a biological tissue among the user-specified areas.

[0107] In one embodiment, the tissue mask generation module (150) can generate a tissue mask image corresponding to the biological tissue image, and the specific module (130) can specify an annotation area corresponding to the biological tissue among the user-specified areas based on the tissue mask image. At this time, each pixel of the tissue mask image is assigned a binary value indicating whether the corresponding pixel of the biological tissue image corresponds to the biological tissue.

[0108] In one embodiment, the tissue mask generation module (150) may convert the biological tissue image into an HSV model, perform binarization on an S space of the biological tissue image converted into the HSV model to generate a first binarization result (wherein the S space is a saturation value space of the HSV model), perform binarization on a 1-V space of the biological tissue image converted into the HSV model to generate a second binarization result (wherein the V space is a brightness value space of the HSV model), and generate a tissue mask image corresponding to the biological tissue image based on the first binarization result and the second binarization result.

[0109] In one embodiment, the specific module (130) may generate a designated area mask corresponding to the user-specified area, and perform a logical AND operation on each pixel of the tissue mask image and the designated area mask image to specify an annotation area corresponding to a biological tissue among the user-specified area. At this time, each pixel of the designated area mask is assigned a binary value indicating whether the corresponding pixel of the biological tissue image is included in the user-specified area.

[0110] The above annotation module (140) can reflect the annotation type in a portion corresponding to the annotation area of ​​the pixel-by-pixel diagnosis result of the biological tissue image. At this time, the pixel-by-pixel diagnosis result of the biological tissue image can include the lesion type assigned to each pixel constituting the biological tissue image.

[0111] In one embodiment, the annotation module (140) may assign an annotation type to a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image to which no lesion type has been assigned yet, and may select and assign either the already assigned lesion type or the annotation type according to a predetermined selection criterion to a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image to which a lesion type different from the annotation type has already been assigned.

[0112] When the annotation type is reflected in a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image, the display module (110) can visualize the pixel-unit diagnosis result and display it together with the biological tissue image.

[0113] Meanwhile, depending on the implementation example, the computing system (100) may include a processor and a memory that stores a program executed by the processor. In this case, the computing system (100) may have a configuration as illustrated in FIG. 8. FIG. 9 is a diagram illustrating a schematic configuration of a computing system (100) according to an embodiment of the present invention.

[0114] The above computing system (100) may be equipped with a memory (102) in which a program for implementing the technical idea of ​​the present invention is stored, and a processor (101) for executing the program stored in the memory (102).

[0115] An average expert in the technical field of the present invention will be able to easily infer that the processor (101) may be named by various names such as CPU, APU, mobile processor, etc., depending on the implementation example of the computing system (100). The processor may include a single-core CPU or a multi-core CPU. In addition, the computing system (100) may be implemented by organically combining a plurality of physical devices, and in this case, an average expert in the technical field of the present invention will be able to easily infer that the computing system (100) of the present invention may be implemented by providing at least one processor (101) for each physical device.

[0116] The above memory (102) stores the program and may be implemented as any type of storage device that the processor can access to run the program. The memory (102) may include a high-speed random access memory and may also include non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory by the processor and other components may be controlled by a memory controller. In addition, depending on the hardware implementation example, the memory (102) may be implemented as a plurality of storage devices rather than a single storage device. In addition, the memory (102) may include a temporary storage device as well as a main memory device. In addition, it may be implemented as a volatile memory or a non-volatile memory, and may be defined to mean all types of information storage means that are implemented so that the program can be stored and run by the processor.

[0117] The above program, when executed by the processor (101), can cause the computing system (100) to perform the above-described annotation method.

[0118] Meanwhile, the method according to an embodiment of the present invention may be implemented in the form of computer-readable program commands and stored on a computer-readable recording medium, and the control program and target program according to an embodiment of the present invention may also be stored on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system.

[0119] The program commands recorded on the recording medium may be specially designed and configured for the present invention or may be known and available to those skilled in the software field.

[0120] Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Additionally, the computer-readable storage media can be distributed over network-connected computer systems so that the computer-readable code can be stored and executed in a distributed manner.

[0121] Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a device that processes information electronically, such as a computer, using an interpreter.

[0122] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0123] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

[0124] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

[0125]

[0126] The present invention can be used in a method for annotating a lesion type in a biological tissue image and a computing system for performing the same.

Claims

1. A step in which a computing system displays a biological tissue image; A step in which the computing system receives a command to select an annotation type that is one of a plurality of lesion types, wherein the lesion type means a grade according to a predetermined severity grading system or a histological classification of a predetermined lesion; A step in which the computing system receives a command to designate a user-designated area, which is a portion of the displayed biological tissue image; A step in which the computing system specifies an annotation area corresponding to a biological tissue among the user-specified areas; A step in which the computing system reflects the annotation type in a portion corresponding to the annotation area of ​​the pixel-by-pixel diagnosis result of the biological tissue image, wherein the pixel-by-pixel diagnosis result of the biological tissue image includes a lesion type assigned to each pixel constituting the biological tissue image; and A method comprising a step of the computing system visualizing the pixel-level diagnostic results and displaying them together with the biological tissue image.

2. In paragraph 1, The step of specifying the annotation area corresponding to the biological tissue among the above custom areas is: The computing system generates a tissue mask image of the biological tissue image, wherein each pixel of the tissue mask image is assigned a binary value indicating whether the corresponding pixel of the biological tissue image corresponds to biological tissue; and A method comprising the step of specifying an annotation area corresponding to a living tissue among the user-specified areas based on the tissue mask image.

3. In paragraph 2, The step of specifying an annotation area corresponding to a living tissue among the user-specified areas based on the tissue mask image is as follows: A step of generating a designated area mask corresponding to the user-specified area, wherein each pixel of the designated area mask is assigned a binary value indicating whether the corresponding pixel of the biological tissue image is included in the user-specified area; and A method comprising the step of performing a logical AND operation on each pixel of the tissue mask image and the designated area mask image.

4. In paragraph 1, The step of reflecting the annotation type in the part corresponding to the annotation area of ​​the pixel-level diagnosis result of the above biological tissue image is: A step of assigning the annotation type to a part corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image, to which no lesion type has yet been assigned; and A method comprising a step of selecting and assigning, according to a predetermined selection criterion, either the lesion type already assigned or the annotation type to a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image to which a lesion type different from the annotation type has already been assigned.

5. In paragraph 1, The above lesion types are, Gleason pattern according to the Gleason grading system for prostate cancer, Tumor grade according to the histologic grading system for breast cancer, Classification of invasive breast carcinoma and intraepithelial carcinoma lesions, and One of the methods for classifying invasive lesions and precancerous lesions of the lung.

6. A computer program recorded on a recording medium installed in a data processing device and for performing the method described in any one of claims 1 to 5.

7. A computer-readable recording medium having recorded thereon a computer program for performing the method described in any one of paragraphs 1 to 5.

8. As a computing system, a processor; and a memory for storing a computer program, The above computer program, A computing system that, when executed by the processor, causes the computing system to perform the method described in any one of claims 1 to 5.

9. A display module for displaying biological tissue images; Receives a selection command for an annotation type that is one of multiple lesion types, A receiving module for receiving a command to designate a user-specified area, which is a portion of the displayed biological tissue image, wherein the lesion type means a grade according to a predetermined severity grading system or a histological classification of a predetermined lesion; A specific module that specifies an annotation area corresponding to a biological tissue among the above user-specified areas; An annotation module that reflects the annotation type in a portion corresponding to the annotation area of ​​the pixel-by-pixel diagnosis result of the biological tissue image, wherein the pixel-by-pixel diagnosis result of the biological tissue image includes a lesion type assigned to each pixel constituting the biological tissue image. The above display module, A computing system that visualizes the pixel-unit diagnosis result and displays it together with the biological tissue image when the annotation type is reflected in a portion corresponding to the annotation area of ​​the pixel-unit diagnosis result of the biological tissue image.

10. In paragraph 9, the computing system, A tissue mask image generation module for generating a tissue mask image corresponding to the above biological tissue image, wherein each pixel of the tissue mask image is assigned a binary value indicating whether the corresponding pixel of the biological tissue image corresponds to biological tissue, The above specific module, A computing system for specifying an annotation area corresponding to a living tissue among the user-specified areas based on the tissue mask image.

11. In paragraph 10, The above specific module, Generating a designated region mask corresponding to the user-specified region, wherein each pixel of the designated region mask is assigned a binary value indicating whether the corresponding pixel of the biological tissue image is included in the user-specified region; A computing system that performs a logical product operation on each pixel of the tissue mask image and the designated area mask image to specify an annotation area corresponding to a living tissue among the user-designated areas.

12. In paragraph 9, The above annotation module, Among the parts corresponding to the annotation area of ​​the pixel-level diagnosis result of the above biological tissue image, the annotation type is assigned to a part to which no lesion type has been assigned yet, A computing system that selects and assigns, according to a predetermined selection criterion, either the lesion type already assigned or the annotation type, to a portion corresponding to the annotation area of ​​the pixel-level diagnostic result of the above biological tissue image, to which a lesion type different from the annotation type has already been assigned.

Images