Cell-specific staining intensity detection method based on staining intensity detection model

Abstract

To provide an improved method for analyzing tissue images by offering cell-specific staining intensities categorized into multiple classes. [Solution] The method for detecting cell-specific staining intensity from pathological images composed of tissue slide images according to the present disclosure may include the steps of: extracting bounding boxes containing target cells from the tissue slide images; inferring the staining intensity class of the bounding boxes based on a learned staining intensity detection model; and displaying the bounding boxes inferred as different staining intensity classes in different ways from each other.

Description

Technical Field

[0001] The technical idea of the present disclosure relates to a method for detecting staining intensity for each cell, and more particularly, to a method for detecting staining intensity for each cell based on a staining intensity detection model.

Background Art

[0002] Conventional pathological diagnosis methods use a method of placing body tissues or cell samples on a glass slide and observing them under a microscope. Since such a method requires each cell to be confirmed and analyzed by visual inspection, it takes a lot of time, and the process until the test results are obtained may be complicated and delayed. In particular, with the increase in the elderly population, the number of cancer patients has been rapidly increasing, and it is becoming difficult to efficiently meet the demand for pathological diagnosis with such a conventional method.

[0003] The introduction of digital pathology has emerged as an alternative to solve these problems. Digital pathology is a method of scanning a slide sample, converting it into a digital image, and managing and analyzing it via a computer. This method can greatly improve the efficiency of pathological diagnosis through automated analysis, and it is possible to minimize the delay in patient treatment by increasing the inspection speed. However, in order for digital pathology to function effectively, there is a practical need for a technical infrastructure that can efficiently process and transmit high-volume digital video data.

[0004] In response to these needs, recent research and technological developments have explored various methods for improving data processing and analysis in digital pathology. In particular, technologies have been developed that can provide the analysis results of digital videos in real time through efficient data communication between a server and a user terminal and can be easily visualized through a user interface. As a result, a foundation has been built that can further enhance the accuracy and speed of digital pathology and improve the overall quality of pathological diagnosis.

[0005] For example, in this type of digital pathology analysis, stained slides are used to confirm the expression of specific proteins such as HER2. HER2 is a protein that is overexpressed in certain types of cancer, such as breast cancer, and accurately determining the presence or absence and intensity of this protein plays a crucial role in determining the course of treatment.

[0006] In digital pathological analysis, techniques such as immunohistochemical staining (IHC) are used to determine the presence and intensity of the HER2 protein. For example, slides are stained with a specific antibody to confirm HER2 protein expression, and these stained samples are analyzed using a digital pathology system.

[0007] With samples stained in this manner, interpretations may differ among pathologists even for the same slide, and in particular, subtle differences in staining intensity can lead to inconsistencies in diagnostic results. Such subjectivity can hinder the establishment of accurate diagnoses and treatment plans. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] The technical concept of this disclosure aims to solve the problem of providing staining intensities divided into multiple classes for each cell, and a method for analyzing tissue images. [Means for solving the problem]

[0009] A method for detecting cell-specific staining intensity from pathological images composed of tissue slide images according to the embodiments of the present disclosure may include the steps of: extracting bounding boxes containing target cells from the tissue slide images; inferring the staining intensity class of the bounding boxes based on a learned staining intensity detection model; and displaying the bounding boxes inferred as different staining intensity classes in different ways.

[0010] According to one embodiment, the staining intensity classes are characterized by being divided into Negative Tumor Cell class, Weak Tumor Cell class, Moderate Tumor Cell class, and Strong Tumor Cell class.

[0011] According to one embodiment, the step of training the staining intensity detection model based on a training dataset in which bounding boxes are labeled with one of a plurality of staining intensity classes may be included.

[0012] According to one embodiment, the staining intensity detection model is characterized by being trained on a training dataset labeled with non-tumor cells whose bounding boxes do not fall within the staining intensity class.

[0013] According to one embodiment, the step of inferring the staining intensity class may include the steps of calculating a confidence value for each staining intensity class relative to the bounding box, and selecting one of a plurality of staining intensity classes based on the calculated confidence value.

[0014] According to one embodiment, the bias of the confidence value can be adjusted by the user for each pair of staining intensity classes, and when the bias is adjusted, the staining intensity class is re-selected.

[0015] According to one embodiment, the staining intensity detection model is characterized by utilizing a self-directed learning method feature extraction model as a backbone network. [Effects of the Invention]

[0016] The staining intensity detection method according to the embodiments of this disclosure can detect the staining intensity of stained tumor cells on a cell-by-cell basis within the entire slide image, enabling accurate detection of staining intensity. Furthermore, by visualizing the staining intensity in terms of multiple classes in addition to the presence or absence of staining, pathologists can easily identify the staining intensity.

[0017] The effects obtained from the exemplary embodiments of this disclosure are not limited to those described above, and other effects not mentioned can be clearly derived and understood by a person skilled in the art to which the exemplary embodiments of this disclosure belong, from the following description. That is, unintended effects of carrying out the exemplary embodiments of this disclosure can also be derived by a person skilled in the art to which the exemplary embodiments of this disclosure belong, from the exemplary embodiments of this disclosure. [Brief explanation of the drawing]

[0018] [Figure 1] Figure 1 shows the components of a staining intensity detection device according to an embodiment of the present disclosure. [Figure 2] Figure 2 shows the quantitative results calculated in one example. [Figure 3] Figure 3 shows an example in which cell-specific staining intensity was detected using a conventional method. [Figure 4] Figure 4 is a flowchart showing a method for detecting staining intensity according to one embodiment. [Figure 5a] Figure 5a shows the types of training datasets input to the stain intensity detection model according to one embodiment. [Figure 5b] Figure 5b shows the types of training datasets input to the staining intensity detection model according to one embodiment. [Figure 6] Figure 6 shows the structure of a staining intensity detection model according to one embodiment. [Figure 7] Figure 7 shows a UI / UX that allows for adjustment of confidence level bias using one embodiment. [Figure 8]FIG. 8 is a diagram showing an example in which the staining intensity for each cell is displayed according to the staining intensity class in one embodiment.

DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, various embodiments of the present disclosure will be described in relation to the accompanying drawings. Various embodiments of the present disclosure can be modified in various ways and can have several embodiments. Specific embodiments are illustrated in the drawings and detailed descriptions related thereto are provided. However, this is not intended to limit the various embodiments of the present disclosure to specific embodiments, but should be understood to include all modifications and / or equivalents or alternatives included in the spirit and technical scope of the various embodiments of the present disclosure. Regarding the description of the drawings, similar reference numerals are used for similar components.

[0020] In various embodiments of the present disclosure, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and are not to be construed as precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0021] In various embodiments of the present disclosure, expressions such as "or" include any and all combinations of the words in parallel. For example, "A or B" may include the case of including A, the case of including B, or the case of including both A and B.

[0022] In various embodiments of the present disclosure, expressions such as "first", "second", "primary" or "secondary" can modify the various components of the various embodiments, but do not limit the components. For example, the above expressions do not limit the order and / or importance of the components and can be used to distinguish one component from another.

[0023] When one component is referred to as being "linked" or "connected" to another component, it should be understood that the first component may be directly linked to or connected to the other component, but there may also be other components existing between the first component and the other component.

[0024] In the embodiments of this disclosure, terms such as “module,” “unit,” and “part” refer to components that perform at least one function or operation, and such components may be embodied in hardware or software, or in a combination of hardware and software. Furthermore, multiple “modules,” “units,” and “parts” may be integrated as at least one module or chip and embodied in at least one processor, unless each needs to be embodied in separate, specific hardware.

[0025] Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted as ideal or overly formal unless explicitly defined in the various embodiments of this disclosure.

[0026] The embodiments of this disclosure will be described in detail below with reference to the attached drawings.

[0027] Figure 1 shows the components of a staining intensity detection device according to an embodiment of the present disclosure.

[0028] Referring to Figure 1, the staining intensity detection device 10 can analyze the input image based on a neural network to detect the staining intensity of individual cells. In this case, the input image may be an image in which tissue cells have been stained, and may be called a pathology image, a whole slide image (WSI), or a stained tissue slide image. Images in which tissue cells have been stained may include images stained with HER2, ER (Estrogen Receptor), and PR (Progesterone Receptor) via immunochemical staining (IHC), H&E (Hematoxylin and Eosin) stained images, and may include all types of images in pathology images in which a specific type of cell is stained and identifiable.

[0029] The stain intensity detection device 10 may include any type of device capable of performing calculations by the processor 100, and may be applied to smart TVs, smartphones, mobile devices, video display devices, measuring devices, IoT (Internet of Things) devices, etc., or may be mounted on any of the various other types of electronic devices.

[0030] The stain intensity detection device 10 may include at least one IP block and a neural network processor 300. The stain intensity detection device 10 may include various types of IP blocks, for example, as shown in Figure 1, the stain intensity detection device 10 may include IP blocks such as a processor 100, RAM (200; Random Access Memory), an input / output device 400, and memory 500. In addition, the stain intensity detection device 10 may further include other general-purpose components such as an MFC (Multi-Format Codec), a video module (e.g., a camera interface, a JPEG (Joint Photographic Experts Group) processor, a video processor, or a mixer), a 3D graphics core, an audio system, a display driver, a GPU (Graphic Processing Unit), and a DSP (Digital Signal Processor).

[0031] The components of the stain intensity detection device 10, such as the processor 100, RAM 200, neural network processor 300, input / output device 400, and memory 500, can transmit and receive data via the system bus 600. For example, the system bus 600 may be configured with a standard bus protocol such as ARM's (Advanced RISC Machine) AMBA (Advanced Microcontroller Bus Architecture) protocol. However, it is not limited to this, and various types of protocols may be applied.

[0032] In the embodiment, the components of the stain intensity detection device 10, the processor 100, RAM 200, neural network processor 300, input / output device 400, and memory 500, may be implemented on a single semiconductor chip. For example, the stain intensity detection device 10 may be implemented on a system-on-a-chip (SoC). However, it is not limited thereto, and the stain intensity detection device 10 may be implemented using multiple semiconductor chips. In one embodiment, the stain intensity detection device 10 may be implemented on an application processor mounted on a mobile device.

[0033] The processor 100 can control the overall operation of the stain intensity detection device 10, and for example, the processor 100 may be a Central Processing Unit (CPU). The processor 100 may include a single core or multiple cores. The processor 100 can process or execute programs and / or data stored in the RAM 200 and memory 500. For example, the processor 100 can control various functions of the stain intensity detection device 10 by executing a program stored in memory 500.

[0034] RAM200 can temporarily store programs, data, or instructions. For example, programs and / or data stored in memory 500 may be temporarily loaded into RAM200 by the control or boot code of processor 100. RAM200 may be implemented using memory such as DRAM (Dynamic RAM) or SRAM (Static RAM).

[0035] The input / output device 400 can receive input data from a user or externally and output the data processing results of the stain intensity detection device 10. The input / output device 400 may be embodied using a touchscreen panel, a keyboard, or at least one of various types of sensors. In the embodiment, the input / output device 400 can collect information about the surroundings of the stain intensity detection device 10. For example, the input / output device 400 may include at least one of various types of sensing devices, such as a scanner, or may receive sensing signals from such devices. In the embodiment, the input / output device 400 may sense or receive image signals from outside the stain intensity detection device 10 and may convert the sensed or received image signals into image data, i.e., image frames. The input / output device 400 may store the image frames in memory 500 or provide them to the neural network processor 300.

[0036] Memory 500 serves as a storage location for data, and can store, for example, the OS (Operating System), various programs, and various data. Memory 500 may be, but is not limited to, DRAM. Memory 500 may include at least one of volatile memory or non-volatile memory. Non-volatile memory may include ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Electrically Programmable ROM), EEPROM (Electrically Erasable and Programmable ROM), flash memory, PRAM (Phase-change RAM), MRAM (Magnetic RAM), RRAM (Resistive RAM), FRAM (Ferroelectric RAM), etc. Volatile memory may include DRAM (Dynamic RAM), SRAM (Static RAM), SDRAM (Synchronous DRAM), etc. In one embodiment, the memory 150 may be implemented as a storage device such as an HDD (Hard Disk Drive), SSD (Solid-State Drive), CF (Compact Flash), SD (Secure Digital), Micro-SD (Micro Secure Digital), Mini-SD (Mini Secure Digital), xD (extreme digital), or Memory Stick.

[0037] The neural network processor 300 may generate a neural network model, train (or learn) a neural network, perform operations based on received input data and generate an information signal based on the results of the operations, or retrain a neural network. The neural network may include, but is not limited to, various types of neural network models such as CNN (Convolutional Neural Network), R-CNN (Region with Convolutional Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based Deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, and Classification Network.

[0038] According to embodiments of this disclosure, the neural network processor 300 of the stain intensity detection device 10 can train a stain intensity detection model based on input training data, and can store the trained stain intensity detection model in memory 500. In this case, the neural network model stored in memory 500 may be the weights and parameters between each layer, or it may be configured as a hierarchical layer structure.

[0039] Furthermore, the neural network processor 300 of the staining intensity detection device 10 can analyze the input data input as the target of analysis by loading a staining intensity detection model from the memory 500. The neural network processor 300 can output inference data by analyzing the input image based on the neural network model. According to one embodiment, the inference data may include the staining intensity class of each cell contained in at least a portion of the entire slide image and the confidence value corresponding to each class.

[0040] The processor 100 can identify which of several staining intensity classes each cell belongs to by calculating the inference data output from the neural network processor.

[0041] According to one embodiment, the processor 100 can adjust the bias of confidence values ​​for a pair of classes in response to user input, and can re-extract the staining intensity for each cell based on the adjusted confidence value bias.

[0042] In Figure 1, the configuration is described as distinguishing between the processor 100 and the neural network processor 300. However, the processor 100 and the neural network processor 300 may be composed of the same hardware and distinguished by software units that perform different calculations.

[0043] This specification will be described below with reference to Figure 1.

[0044] Figure 2 shows the quantitative results calculated in one example.

[0045] Referring to Figure 2, the image analysis device can analyze slide images stained with specific cell tissues through a neural network model, and can analyze the results to calculate quantitative results. For example, when the image analysis device analyzed slide images stained with ER / PR / HER2 by immunochemical staining, ER was evaluated as positive with 97% expression, and PR was evaluated as positive with 91% expression.

[0046] In contrast, HER2 was expressed in 44% of cells and evaluated as negative. However, the degree of HER2 expression falls within the margin of error when determining whether a result is positive or negative, and evaluation by a pathologist may be necessary in addition to the AI ​​analysis results. Specifically, a pathologist can directly analyze the staining intensity of cells in the slide images and determine whether the expression level of a particular protein is in a dangerous state based on the staining intensity.

[0047] If the expression level of a specific protein falls within the margin of error, and the AI ​​analysis results are inaccurate, the pathologist's judgment is also highly likely to be inaccurate. In such cases, it may be necessary to precisely observe specific areas within the slide image or analyze the staining intensity of each cell individually. When a pathologist analyzes stained slide images in this way without any auxiliary equipment, it not only makes accurate judgment difficult but also presents the problem of requiring a great deal of time for the analysis.

[0048] Figure 3 shows an example in which cell-specific staining intensity was detected using a conventional method.

[0049] Referring to Figure 3, the conventional example displays stained cells one by one as an auxiliary device to assist pathologists in their analysis, allowing them to identify the location of the cells to be analyzed.

[0050] In such conventional implementations, only whether or not a cell is stained can be detected using a neural network model. Even when cells detected based on staining intensity are classified into several groups, this is done solely by comparing the RGB values ​​of the stained areas. These methods of classifying cells into several groups by comparing the RGB values ​​of stained areas or comparing staining intensity with reference values ​​lack consistency and have the problem of not allowing for precise analysis.

[0051] In particular, cells that stain weakly or with intermediate staining intensity may not be clearly distinguishable, which could lead to the risk of missing important diagnostic information, especially in the case of tumor cells.

[0052] Figure 4 is a flowchart showing a method for detecting staining intensity according to one embodiment.

[0053] Referring to Figure 4, the stain intensity detection device 10 can classify cells with different stain intensity classes by extracting bounding boxes containing target cells and displaying them in different ways according to the stain intensity class of the bounding boxes. Target cells may refer to cells to be stained, such as tumor cells, and the stain intensity detection device 10 can analyze all target cells contained in the analysis area using bounding boxes. A bounding box may refer to an image of a specific shape and size containing a single target cell, with its boundary clearly distinguishable.

[0054] In step S110, the staining intensity detection device 10 can train a staining intensity detection model based on a bounding box containing the target cells and a training dataset in which the staining intensity class of the target cells is labeled.

[0055] The training dataset may consist of tissue slide images, labeled with a bounding box surrounding a single target cell and the staining intensity class of that target cell. The staining intensity classes may include Negative Tumor Cell, Weak Tumor Cell, Moderate Tumor Cell, and Strong Tumor Cell classes. The staining intensity classes may also be grades that classify how intensely cells targeted for staining, such as ER / PR / HER2, were stained.

[0056] According to one embodiment, bounding boxes containing substances other than the target cells may not be labeled with a staining intensity class and may be generated as a training dataset. In this case, the bounding boxes containing substances other than the target cells may be bounding boxes that do not contain any target cells at all.

[0057] Bounding boxes classified into staining intensity classes may all be displayed in the tissue slide image. Conversely, bounding boxes not classified into staining intensity classes may not be displayed in the tissue slide image.

[0058] Substances that are not the target cells may include extracellular matrix and may be classified as non-tumor cells. In addition, non-tumor cells may further include myoepithelial cells, inflammation cells, endothelial cells, perivascular cells, stromal cells, necrotic debris, and apoptotic bodies.

[0059] Inflammatory cells may include lymphocytes, neutrophils, and plasma cells. Interstitial cells may include inactive fibroblasts, fibroblasts, and myofibroblasts.

[0060] In step S120, the staining intensity detection device 10 can extract bounding boxes containing target cells from the entire slide image. The user can specify a region from the entire slide image from which they want to extract the staining intensity of target cells, and the staining intensity detection device 10 can extract bounding boxes containing target cells from that region.

[0061] According to one embodiment, the staining intensity detection device 10 can extract the bounding box of the target cell based on the cell shape, regardless of the staining intensity of the target cell.

[0062] In another embodiment, the stain intensity detection device 10 can extract bounding boxes classified into each of a plurality of stain intensity classes using a stain intensity detection model. For example, the stain intensity detection device 10 can extract bounding boxes classified into a first stain intensity class and bounding boxes classified into a second stain intensity class. In this case, the bounding boxes classified into the first and second stain intensity classes may be bounding boxes that the stain intensity detection model infers to belong to the first or second stain intensity class with a certain confidence level or higher. The bounding boxes corresponding to the first and second stain intensity classes may be extracted independently, and a bounding box classified into the first stain intensity class by the stain intensity detection model may be extracted as a bounding box classified into the second stain intensity class.

[0063] In step S130, the stain intensity detection device 10 may infer the stain intensity class of each bounding box based on the stain intensity detection model. The stain intensity detection model may calculate a confidence value for each of the multiple stain intensity classes for which the bounding box corresponds. The confidence value may represent a probability value for which the bounding box corresponds to that stain intensity class. Based on the calculated confidence value, the stain intensity detection device 10 may infer that the bounding box belongs to one of the multiple stain intensity classes.

[0064] For example, the stain intensity detection device 10 can calculate confidence values ​​corresponding to each of the first to fourth stain intensity classes for the bounding box. The stain intensity detection device 10 can infer the stain intensity class with the highest confidence value among the first to fourth stain intensity classes based on the stain intensity class of the bounding box.

[0065] In step S140, the stain intensity detection device 10 may display bounding boxes inferred as different stain intensity classes in different ways. That is, the stain intensity detection device 10 may display the bounding boxes in a way that distinguishes them from each other according to their stain intensity classes. Displaying them in different ways may mean displaying them in different colors.

[0066] The staining intensity detection device 10 displays bounding boxes corresponding to the staining intensity class according to each method, allowing pathologists analyzing tissue slide images to confirm which staining intensity class of cells has been identified in the image area they intend to analyze.

[0067] Figures 5a and 5b show the types of training datasets input to the stain intensity detection model according to one embodiment.

[0068] Referring to Figure 5a, the training dataset input to the staining intensity detection model may be a dataset in which each bounding box containing the target cells is labeled with one of the staining intensity classes. The staining intensity classes may include Negative Tumor Cell class, Weak Tumor Cell class, Moderate Tumor Cell class, and Strong Tumor Cell class. The staining intensity classes may also be grades that are divided according to the morphology, sharpness, RGB values, etc., of the target cells to be detected within the bounding box. The closer the staining intensity class is to a strong tumor cell, the more likely it is that the specific protein to be detected is expressed at a high level in that tissue or cell.

[0069] Referring to Figure 5b, bounding boxes that do not contain target cells can be used as training data. Bounding boxes containing substances other than target cells may not be labeled by staining intensity class, but rather by a type other than staining intensity class, and may be generated as training datasets. For example, a type other than staining intensity class may be called a non-tumor cell.

[0070] Non-tumor cells may further include myoepithelial cells, inflammation cells, endothelial cells, perivascular cells, stromal cells, necrotic debris, apoptotic bodies, and extracellular matrix.

[0071] Inflammatory cells may include lymphocytes, neutrophils, and plasma cells. Interstitial cells may include inactive fibroblasts, fibroblasts, and myofibroblasts.

[0072] Bounding boxes classified into staining intensity classes may all be displayed in the tissue slide image. Conversely, bounding boxes not classified into staining intensity classes may not be displayed in the tissue slide image.

[0073] The stain intensity detection model can learn not only from bounding boxes labeled by stain intensity class, but also from bounding boxes corresponding to non-tumor cells. While conventional neural network models for extracting stained cells learn only from bounding boxes containing stained cells, the stain intensity detection model of this disclosure learns from bounding boxes that do not contain the target cells, thereby enabling it to extract bounding boxes containing the target cells more accurately.

[0074] A trained stain intensity detection model can calculate confidence values ​​only for bounding boxes corresponding to stain intensity classes, and the bounding boxes can be displayed using a separate display method. In contrast, a stain intensity detection model can not only fail to calculate confidence values ​​for bounding boxes that classify cells as non-tumor cells, but can also maintain the original state without separately displaying the bounding boxes.

[0075] Specifically, the trained stain intensity detection model can calculate confidence values ​​corresponding to bounding box-negative tumor cell classes, low-intensity tumor cell classes, medium-intensity tumor cell classes, and high-intensity tumor cell classes, but it may not calculate confidence values ​​for bounding boxes that classify cells as non-tumor cells.

[0076] While HER2 biomarkers are described as being classified into four classes, the number and types of staining intensity classes in the examples of this disclosure are not limited thereto. For example, biomarkers such as Ki-67 can be classified into two classes (negative tumor cell class and positive tumor cell class), excluding non-tumor cells.

[0077] Figure 6 shows the structure of a staining intensity detection model according to one embodiment.

[0078] Referring to Figure 6, the stain intensity detection model can process dual label assignments to the input data and output inference results using a consistent match metric.

[0079] The backbone network into which the input data is fed may be a network that utilizes a DINO (Distillation with No Labels) based self-guided learning model. Through the self-guided learning model, the backbone network can learn on its own the hidden patterns and structures related to the target cells within the bounding box. The DINO model can be combined with a Vision Transformer architecture and used effectively to learn visual features in images.

[0080] Stain intensity detection models can utilize regression and classification models in the process of handling dual label assignment. A one-to-many head model can predict multiple stain intensity classes for a single input bounding box. In this case, the classification model of the stain intensity detection model can classify the stain intensity classes of the bounding boxes, and the regression model can calculate confidence values ​​for each stain intensity class. That is, a one-to-many head model can calculate confidence values ​​for each stain intensity class. In contrast, a one-to-one head model can predict only one stain intensity class for a single input bounding box.

[0081] The stain intensity detection model can compare the results of a one-to-many head model with those of a one-to-one head model through a consistent match metric to assess the degree of agreement. Through this agreement assessment, the confidence value corresponding to each stain intensity class can be accurately calculated.

[0082] Figure 7 shows a UI / UX that allows for adjustment of confidence level bias using one embodiment.

[0083] Referring to Figure 7, the stain intensity detection device 10 can adjust the bias of the confidence value for each pair of stain intensity classes. The stain intensity detection device 10 may have a user-operable bias adjustment GUI (Graphical User Interface), and by adjusting the bias adjustment GUI, the parameters of the stain intensity detection model may be adjusted, and the stain intensity class relative to the bounding box may be re-detected.

[0084] According to one embodiment, the confidence value for the stain intensity class to which the adjustment has been input can be adjusted by assigning weights or penalties to the loss function of the stain intensity detection model. For example, if a bias adjustment between a second stain intensity class and a third stain intensity class is input to the stain intensity detection model, the weights of the loss function corresponding to the stain intensity detection model can be adjusted.

[0085] If the bias adjustment GUI between low-intensity and medium-intensity tumor cell classes shifts to the right, the confidence values ​​may be adjusted to detect more bounding boxes corresponding to the medium-intensity tumor cell class. When the staining intensity detection model receives an input that shifts the bias adjustment GUI to the right, it can adjust the loss function value corresponding to the medium-intensity tumor cell class, assigning higher confidence values ​​even to bounding boxes with relatively low similarity. This can increase the probability that such bounding boxes are classified as intermediate-intensity tumor cells.

[0086] When the staining intensity detection model receives an input that shifts the bias adjustment GUI between the low-intensity and medium-intensity tumor cell classes to the left, it adjusts the value of the loss function corresponding to at least one of the low-intensity and medium-intensity tumor cell classes, and can assign a lower confidence value even to relatively similar bounding boxes. This can reduce the probability that the bounding box is classified as an intermediate-intensity tumor cell class.

[0087] Tissue slide images may exhibit varying staining intensity depending on the scanner's performance and staining conditions. Therefore, if a pathologist determines that it is necessary to adjust the confidence value bias for a particular staining intensity class for a stained tissue slide image, they can easily do so via the GUI provided in the embodiments of this disclosure.

[0088] The bias adjustment GUI according to the embodiments of this disclosure may be formed horizontally for each pair of staining intensity classes, and the bias may be adjusted by left-right scrolling, but is not limited thereto; it may also be formed vertically, and the bias may be adjusted by up-down scrolling. Furthermore, the bias adjustment GUI of this disclosure may also adjust the bias of staining intensity classes not only by scrolling in a specific direction, but also through any kind of GUI with adjustable steps.

[0089] Figure 8 shows an example in which the staining intensity of individual cells is displayed according to the staining intensity class.

[0090] Referring to Figure 8, the staining intensity detection device 10 according to the embodiment of this disclosure can identify a region for detecting the staining intensity of target cells by drag input from the user. The staining intensity detection device 10 can extract bounding boxes containing target cells in the identified region and classify a staining intensity class for each bounding box.

[0091] The stain intensity detection device 10 can count the number of bounding boxes for each stain intensity class within the identified area. The counted number of bounding boxes may be displayed on part of the display of the stain intensity detection device 10, or the ratio of the number of bounding boxes for each stain intensity class to the total number of bounding boxes may be displayed.

[0092] The staining intensity detection device 10 can calculate a final score based on the percentage calculated for each staining intensity class. The final score can be expressed as the staining intensity class of the identified region. The staining intensity class can indicate the degree of expression of a specific protein in the cell, and the final score can be calculated based on the extent to which the specific protein is expressed in that region.

[0093] According to the example in Figure 8, the staining intensity detection device 10 can detect the staining intensity for the HER2 protein. Of the region dragged by the user, the number of bounding boxes classified as negative tumor cells by the staining intensity detection model may be 1509, and the number of bounding boxes classified as low-intensity tumor cells may be 8946. Also, the number of bounding boxes classified as medium-intensity tumor cells may be 96, and the number of bounding boxes classified as high-intensity tumor cells may be 1.

[0094] The staining intensity detection device 10 can calculate the ratio of the number of bounding boxes for each staining intensity class to the total number of bounding boxes. The ratio for negative tumor cell classes can be classified as 14.3%, for low-intensity tumor cell classes as 84.8%, for medium-intensity tumor cell classes as 0.9%, and for high-intensity tumor cell classes as 0.0%.

[0095] The staining intensity detection device 10 can calculate the final score based on the staining intensity class with the highest ratio. Since the ratio to the low-intensity tumor cell class is 84.8%, the staining intensity detection device 10 can determine the final score of the staining intensity of the dragged area as low intensity (weak).

[0096] The stain intensity detection device 10 according to the embodiments of this disclosure can quickly and accurately provide a pathologist with information on the stain intensity class at which target cells are stained in the area they intend to observe. In addition, each stain intensity class can be displayed using different display methods, allowing the pathologist to intuitively identify the location of target cells classified as high-intensity staining classes. Conventionally, pathologists had to individually examine the location of high-intensity stained target cells, which was time-consuming or sometimes resulted in the inability to locate the target cells. In contrast, the stain intensity detection device 10 of this disclosure, through its intuitive display method (e.g., display in red), allows pathologists to immediately identify target cells that contain risk factors.

[0097] On the other hand, the methods according to the various embodiments of the present invention described above may be embodied in the form of an application or software program that can be installed on a conventional electronic device.

[0098] Furthermore, the method may consist of several software function modules and be embodied in an operating system (OS). Alternatively, each step may consist of one software function module, or each step may be combined to form one software function module and be embodied on an operating system. Thus, even if not all of the embodiments of the disclosure are embodied as a single software function module, if several software function modules embody each step of the disclosure, and several software function modules are embodied in a single operating system, this can be understood as embodying the method of the disclosure.

[0099] Furthermore, the methods described in the various embodiments of the present invention may also be implemented solely through software upgrades or hardware upgrades to conventional electronic devices. Moreover, the various embodiments of the present invention described above can also be implemented via an embedded server in the electronic device or an external server.

[0100] On the other hand, according to one embodiment of the present invention, the various embodiments described above may be embodied in software, hardware, or a combination thereof, including instructions stored on a computer-readable recording medium. In some cases, the embodiments described herein may be embodied in the processor itself. In a software embodiment, the embodiments such as the procedures and functions described herein may be embodied in separate software modules. Each software module may perform at least one of the functions and operations described herein.

[0101] On the other hand, a computer or similar device is a device capable of calling instructions stored from a storage medium and operating according to the called instructions, and may include the device according to the disclosed embodiment. When the instructions are executed by a processor, the processor may perform the function corresponding to the instructions directly or, under the control of the processor, using other components. The instructions may include code generated or executed by a compiler or interpreter.

[0102] Recording media that can be read by a device may be provided in the form of non-transitory computer-readable recording media. Here, "non-transitory" means that the storage medium does not contain signals and is tangible, and does not distinguish between data being stored on the storage medium semi-permanently or temporarily. In this case, a non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data only for a short time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USB memory sticks, memory cards, and ROMs.

[0103] As described above, exemplary embodiments have been disclosed in the drawings and specification. While specific terms have been used to describe the embodiments in this specification, these are used solely for the purpose of illustrating the technical idea of ​​the disclosure and not to limit its meaning or the scope of the claims. Therefore, a person with ordinary skill in the art will understand that a variety of variations and equivalent embodiments are possible. Accordingly, the true scope of technical protection of this disclosure should be determined by the technical idea of ​​the appended claims.

Claims

1. A method for detecting cell-specific staining intensity from pathological images composed of tissue slide images, The steps include extracting bounding boxes containing target cells from the aforementioned tissue slide image, The steps include inferring the staining intensity class of the bounding box based on a learned staining intensity detection model, The steps include displaying bounding boxes inferred as different staining intensity classes in different ways from each other, A method for detecting cell-specific staining intensity, including the above.

2. The aforementioned staining intensity class is, The cell-specific staining intensity detection method according to claim 1, characterized in that cells are classified into Negative Tumor Cell class, Weak Tumor Cell class, Moderate Tumor Cell class, and Strong Tumor Cell class.

3. The cell-specific staining intensity detection method according to claim 2, further comprising the step of training the staining intensity detection model based on a training dataset in which bounding boxes are labeled with one of a plurality of staining intensity classes.

4. The aforementioned staining intensity detection model is, The cell-specific staining intensity detection method according to claim 3, characterized in that the bounding box is learned based on a training dataset labeled with non-tumor cells that are not included in the staining intensity class.

5. The step of inferring the staining intensity class is: The steps include: calculating the confidence value for each staining intensity class relative to the bounding box; The steps include selecting one of several staining intensity classes based on the calculated confidence value, A method for detecting cell-specific staining intensity according to claim 1, characterized by including the following:

6. The cell-specific staining intensity detection method according to claim 5, characterized in that the bias of the confidence value can be adjusted by the user for each pair of staining intensity classes, and if at least one bias is adjusted, the staining intensity class is re-selected.

7. The aforementioned staining intensity detection model is, The cell-specific staining intensity detection method according to claim 1, characterized in that it is a model that utilizes a feature extraction model for self-directed learning methods as a backbone network.

Images