Apparatus, system and method using hierarchical deep learning medel

Abstract

The present invention relates to an apparatus, system, and method including a hierarchical deep learning model for analyzing large-scale pathology images in the form of Whole Slide Images (WSI) using a hierarchical deep learning model. More specifically, the invention relates to an apparatus, system, and method including a hierarchical deep learning model that divides a large-scale pathology image, which is a pyramid-shaped whole slide image composed of multiple magnification layers of high-magnification pathology images and low-magnification pathology images, into patches, thereby enabling more accurate analysis by applying the high-magnification pathology images to a segmentation model and generating faster analysis results by applying the low-magnification pathology images to a regression model.

Description

Technology Field

[0001] The present invention relates to an apparatus, system, and method including a hierarchical deep learning model for analyzing large volume pathology images in the form of Whole Slide Images (WSI) using a hierarchical deep learning model.

[0002] This invention is a research result obtained through support from the 2023 Initial Startup Package program of the Korea Institute of Startup & Entrepreneurship Promotion (Project No.: 20150406, Project Title: Artificial Intelligence Medical Platform for Prevention, Management, Intervention, and Treatment of Non-Alcoholic Fatty Liver Disease Applying Large-Scale Biomedical Imaging Data Processing and Analysis Technology). Background Technology

[0003] In the field of digital pathology, large-capacity pathology images in the form of Whole Slide Images (WSI) are being utilized. These images are created by scanning the entire area of ​​a glass slide with a digital slide scanner and aligning it to convert it into a single ultra-high-resolution digital image. These large-capacity pathology images are very large data, with a size ranging from hundreds of MB to several GB and an image resolution reaching gigapixels.

[0004] Therefore, it is very difficult to input these gigapixel-scale large-volume pathology images into a deep learning model or use them for training all at once.

[0005] To address these issues, attempts are being made to utilize large volume pathology images for training deep learning models by dividing them into small patch units. However, existing approaches have the problem that data processing and analysis are separated, resulting in reduced analysis efficiency.

[0006] Accordingly, there is a need to develop technology capable of analyzing large-scale pathology images in gigapixel units in real time more quickly and accurately. Prior art literature

[0007] Korean Patent Publication No. 10-2021-0145778, published December 02, 2021 (Title: Method for determining biomarkers from histopathology slide images) The problem to be solved

[0008] The present invention is proposed to solve the aforementioned conventional problems, and aims to provide an apparatus, system, and method including a hierarchical deep learning model capable of real-time analysis by dividing a large-capacity pathology image, which is a pyramid-shaped Whole Slide Image (WSI) composed of multiple magnification layers of high-magnification pathology images and low-magnification pathology images, into patches and applying it to a deep learning model simultaneously with loading.

[0009] In addition, the present invention aims to provide an apparatus, system, and method including a hierarchical deep learning model capable of generating a result image by dividing and loading a large-capacity pathology image in a pyramid shape into patches starting from an area corresponding to user input, applying a high-magnification pathology image corresponding to an area divided into patches to a segmentation model, and applying a low-magnification pathology image to a regression model.

[0010] However, the objectives of the present invention are not limited to the above objectives, and other unmentioned objectives will be clearly understood from the description below. means of solving the problem

[0011] An apparatus comprising a hierarchical deep learning model composed of a segmentation model and a regression model for analyzing a large volume pathology image, which is a Whole Slide Image (WSI) formed in a hierarchical form of a high-magnification pathology image and a low-magnification pathology image according to an embodiment of the present invention for achieving the purpose described above, wherein the apparatus may be configured to include: an image loading unit that divides and loads the large volume pathology image into patch units starting from an area corresponding to user input; and a deep learning unit that applies the high-magnification pathology image corresponding to the area divided into patch units through the image loading unit to the segmentation model and applies the low-magnification pathology image to the regression model to generate a result image.

[0012] Here, the user input includes a user field of view (FOV) and a magnification factor, and the image loading unit can be loaded by dividing the large-capacity pathology image into patches starting from the area corresponding to the user field of view and the magnification factor.

[0013] At this time, the above-mentioned segmentation model is a Convolutional Neural Network (CNN) having an encoder and decoder structure including a down-sampling layer, and the above-mentioned regression model may include a regression head connected to the encoder of the above-mentioned segmentation model, which performs regression learning on the extracted features when features are extracted through the encoder of the above-mentioned segmentation model.

[0014] At this time, the deep learning unit may apply the high-magnification pathology image to the segmentation model to generate a result image for the high-magnification pathology image, and while the result image for the high-magnification pathology image is being generated, apply the low-magnification pathology image to the regression model to extract features for the low-magnification pathology image through the encoder of the segmentation model, and perform regression learning on the extracted features through the regression head of the regression model to generate a result image for the low-magnification pathology image.

[0015] In addition, the deep learning unit may use an asymmetric loss function (Mean Asymmetric-Squared Error, MASE) according to the following mathematical formula as the loss function for the regression learning.

[0016] <Mathematical Formula 1>

[0017]

[0018] (Here, f is a function for the regression model, x is the low-magnification pathology image, y is the label, and ASE is the following asymmetric squared error) )

[0019] Additionally, the device may further include a result providing unit that stores the result image in a cache device corresponding to a patch, and provides the result image stored in the cache device to the user when the user requests a result image for the same patch.

[0020] A system comprising a hierarchical deep learning model for analyzing a large volume pathology image, which is a Whole Slide Image (WSI) formed in a hierarchical form of a high-magnification pathology image and a low-magnification pathology image according to an embodiment of the present invention for achieving the purpose described above, wherein the system may comprise: a front end that transmits a large volume pathology image including an area corresponding to a user input; and a back end that divides and loads the large volume pathology image transmitted from the front end into patch units starting from the area corresponding to the user input, applies the high-magnification pathology image corresponding to the area divided into patch units to the division model, applies the low-magnification pathology image to the regression model to generate a result image, and transmits the generated result image to the front end.

[0021] At this time, the front-end includes a slide viewer and checks the user input including the user viewpoint (FOV) and scale factor through the slide viewer, and when a result image is transmitted from the back-end, the transmitted result image can be output through the slide viewer.

[0022] At this time, the backend further includes a cache device that stores the generated result image corresponding to the patch, and when a request for a result image for the same patch is made from the frontend, the result image stored in the cache device can be delivered to the frontend.

[0023] A deep learning method in a device comprising a hierarchical deep learning model for analyzing a large volume pathology image, which is a Whole Slide Image (WSI) formed in a hierarchical form of a high-magnification pathology image and a low-magnification pathology image according to an embodiment of the present invention for achieving the purpose described above, wherein the device may perform the steps of: dividing and loading the large volume pathology image into patch units starting from an area corresponding to user input; and applying the high-magnification pathology image corresponding to the area divided into patch units to the dividing model and applying the low-magnification pathology image to the regression model to generate a result image.

[0024] At this time, the user input includes a user viewpoint (FOV) and a magnification factor, and the loading step can load by dividing the high-resolution pathology image into patches starting from the area corresponding to the user viewpoint and magnification factor.

[0025] Additionally, the above-mentioned segmentation model is a Convolutional Neural Network (CNN) having an encoder and decoder structure including a down-sampling layer, and the above-mentioned regression model may include a regression head connected to the encoder of the above-mentioned segmentation model, which performs regression learning on the extracted features when features are extracted through the encoder of the above-mentioned segmentation model.

[0026] At this time, the step of generating the result image may involve applying the high-magnification pathology image to the segmentation model to generate a result image for the high-magnification pathology image, while a low-magnification pathology image is being generated, applying the low-magnification pathology image to the regression model to extract features for the low-magnification pathology image through the encoder of the segmentation model, and performing regression learning on the extracted features through the regression head of the regression model to generate a result image for the low-resolution pathology image.

[0027] Additionally, the present invention may provide a computer-readable recording medium that records a program for executing the method described above. Effects of the invention

[0028] According to the apparatus, system, and method including the hierarchical deep learning model of the present invention, a large-capacity pathology image, which is a whole slide image, can be divided into patches and applied to a deep learning model simultaneously with loading, thereby enabling real-time analysis.

[0029] In addition, according to the present invention, a high-magnification pathology image, which is an image scanned at the highest magnification, can generate precise analysis results for a limited area through a segmentation model, and a low-magnification pathology image, which is scanned at a low magnification, can generate results by rapidly analyzing a wider area through a regression model.

[0030] In addition, according to the present invention, the regression model operates in a transfer learning manner by sharing the encoder of the partitioning model, thereby enabling precise learning even when generating rapid analysis results.

[0031] In addition, various effects other than those described above may be disclosed directly or implicitly in the detailed description according to the embodiments of the present invention to be described below. Brief explanation of the drawing

[0032] FIG. 1 is an illustrative diagram for explaining a system including a hierarchical deep learning model according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating the main configuration of a device including a hierarchical deep learning model according to an embodiment of the present invention. FIGS. 3 to 5 are illustrative diagrams for explaining a hierarchical deep learning model according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating a deep learning method in a device including a hierarchical deep learning model according to an embodiment of the present invention. Specific details for implementing the invention

[0033] In order to clarify the features and advantages of the means for solving the problem of the present invention, the present invention will be described in more detail with reference to specific embodiments of the present invention illustrated in the attached drawings.

[0034] However, detailed descriptions of known functions or configurations that may obscure the essence of the invention are omitted in the following description and the attached drawings. Additionally, it should be noted that identical components throughout the drawings are indicated by the same reference numerals whenever possible.

[0035] Terms and words used in the following description and drawings should not be interpreted as being limited to their ordinary or dictionary meanings, but should be interpreted in a meaning and concept consistent with the technical spirit of the invention, based on the principle that the inventor can appropriately define the concept of terms to best describe his invention. Accordingly, the embodiments described in this specification and the configurations illustrated in the drawings are merely the most preferred embodiments of the invention and do not represent all aspects of the technical spirit of the invention; therefore, it should be understood that various equivalents and modifications capable of replacing them may exist at the time of filing this application.

[0036] Furthermore, when it is stated that one component is "connected" or "joined" to another component, this implies that they may be connected or joined logically or physically. In other words, it should be understood that while a component may be directly connected or joined to another component, there may also be other components in between, or the connection may be indirect.

[0037] Furthermore, the terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Additionally, terms such as “comprising” or “having” described in this specification are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0038] An apparatus, system, and method including a hierarchical deep learning model according to an embodiment of the present invention will be described with reference to the drawings.

[0039] First, a system including a hierarchical deep learning model according to an embodiment of the present invention will be described.

[0040] FIG. 1 is an illustrative diagram for explaining a system including a hierarchical deep learning model according to an embodiment of the present invention.

[0041] Referring to FIG. 1, a system (500) including a hierarchical deep learning model may be configured to include a front end (100) and a back end (200).

[0042] First, the front end (100) is a user-side device capable of transmitting a large volume pathology image to the back end (200), wherein the large volume pathology image is an image of tissue extracted from the human body, etc. and scanned, and in particular, an image of a pathology slide that has been fixed and stained through a series of chemical processing steps on the removed tissue, etc., such as an H&E (Hematoxylin & Eosin) stained frozen section slide.

[0043] These pathological images may contain information about specific proteins, cells, tissues, and structures within the human body.

[0044] To this end, the front end (100) of the present invention may include an imaging device (not shown) and a digital scanner (not shown). For example, the front end (100) may include a device capable of capturing a pathology slide using an imaging device comprising a high-magnification objective lens of 20x, 40x, or higher, and scanning it through a digital scanner, and depending on the implementation method, the capturing and scanning may proceed simultaneously. Such imaging device and digital scanner may be implemented as independent devices or as a single device.

[0045] Additionally, the front end (100) can process user input including a field of view (FOV) and a magnification factor. Here, the field of view refers to the positional coordinates of the view image area currently displayed on the pathology slide, and the magnification factor refers to the magnification factor of the lens (e.g., 20x, 40x). These field of view and magnification can be input by the user adjusting the image device, and depending on the implementation method, they may be processed through a separate user interface. For example, if the front end (100) includes a user interface such as a slide viewer, the user can adjust the field of view and magnification factor through the slide viewer and may output the processed result image through the slide viewer.

[0046] Additionally, the user of the frontend (100) can pan the pathology image, zoom in / zoom out, scroll, etc. through the slide viewer user interface, and the slide viewer can immediately output the corresponding pathology image according to such user input.

[0047] As described above, the front end (100) sequentially scans the entire area of ​​the slide to generate a baseline full image, which is a full slide image, and transmits it to the back end (200). The baseline full image is a large-capacity pathology image with a resolution of, for example, 20,000 x 20,000 to 120,000 x 120,000 pixels or more. It may take time for the front end (100) to transmit the baseline full image to the back end (200), and even if the back end (200) intends to generate a result image by passing the large-capacity pathology image through a deep learning model, significant processing power is required for loading and analysis.

[0048] Accordingly, in the present invention, the area corresponding to the user viewpoint and scale factor is divided into patch units and transmitted. In other words, the front end (100) sequentially transmits a large-capacity pathology image corresponding to user input to the back end (200), and the back end (200) can divide the transmitted large-capacity pathology image into patch units, starting from the area corresponding to user input, for example, an area corresponding to user input such as 512x512 according to the user viewpoint and scale factor, and transmit it to a hierarchical deep learning model, and can quickly generate a result image and transmit it to the front end (100).

[0049] The backend (200) of the present invention includes a device (20) comprising a hierarchical deep learning model composed of a segmentation model and a regression model, wherein the device (20) comprising the hierarchical deep learning model loads a large volume pathology image transmitted from the frontend (100) by segmenting it into patch units starting from the area corresponding to user input, applies a high-magnification pathology image corresponding to the area segmented into patch units to the segmentation model, and applies a low-magnification pathology image to the regression model to generate a result image, and then immediately transmits the generated result image to the frontend (100), and the frontend (100) is able to immediately output the transmitted result image through a slide viewer.

[0050] In addition, the backend (200) of the present invention includes a cache device (30) and stores a generated result image. When the same request occurs from the frontend (100), instead of generating the result image again, the result image stored in the cache device (30) is transmitted to the frontend (100), thereby enabling the processing result of a large volume of pathology images to be provided more quickly and at high speed.

[0051] The operation of the device (20) including the hierarchical deep learning model of the present invention will be described in detail later. The system (500) including the front end (100) and back end (200) of the present invention may be a cloud-based distributed processing system, and the back end (200) may be a cloud computing system implemented in the form of a web server.

[0052] For example, the front-end (100) and the back-end (200) are connected via a wired / wireless communication network (not shown), and large-capacity pathology images scanned from the front-end (100) can be sequentially transmitted to the back-end (200) in real time via the wired / wireless communication network (not shown). The back-end (200), which is a cloud computing system implemented in the form of a web server, includes higher-spec hardware resources (CPU, disk, GPU, etc.) than the front-end (100), and a built hierarchical deep learning model is used, that is, high-magnification pathology images are applied to a segmentation model and low-magnification pathology images are applied to a regression model to generate result images, and the generated result images can be transmitted to the front-end (100) in real time via the wired / wireless communication network (not shown).

[0053] In addition, when the system (500) of the present invention is implemented in the form of a cloud-based distributed processing system, the cache device (30) of the present invention may be a storage space within the backend (200), and depending on the implementation method, it may be a separate storage server located adjacent to the frontend (100), but is not necessarily limited to the structure described above.

[0054] Hereinafter, the main configuration and operation of the device (20) including a hierarchical deep learning model according to an embodiment of the present invention will be described.

[0055] FIG. 2 is a block diagram illustrating the main configuration of a device including a hierarchical deep learning model according to an embodiment of the present invention, and FIGS. 3 to 5 are illustrative diagrams for explaining a hierarchical deep learning model according to an embodiment of the present invention.

[0056] First, referring to FIGS. 1 and 2, a device (20) including a hierarchical deep learning model according to an embodiment of the present invention may be configured to include an image loading unit (21), a deep learning unit (22), and a result providing unit (23).

[0057] The image loading unit (21) performs the role of loading a large volume pathology image in the form of a Whole Slide Image (WSI). Here, the large volume pathology image is a hierarchical large volume pathology image that is sequentially scanned and transmitted from the front end (100), and refers to a pyramid-shaped whole slide image consisting of several magnification levels, including a high-magnification pathology image (Level 0 image; L0 image) which corresponds to the entire baseline image and is the highest resolution image, and a low-magnification pathology image (Level 1 image ~ Level n image; L1 ~ Ln image) which is a downsampled, i.e., reduced image from the scanned high-magnification pathology image.

[0058] As described above, the low-magnification pathology image described in the present invention may refer to a low-resolution image downsampled from a high-magnification pathology image, but depending on the implementation method, the image may be a low-resolution image scanned through a low magnification setting of an imaging device (e.g., a microscope), and the difference in magnification between each level may vary depending on the settings of the imaging device. For example, if the high-magnification pathology image (L0) is an image scanned at 40x magnification, the low-magnification image may include L1 and L2 images, and the L1 image may be an image scanned at 10x magnification, and the L2 image may be an image scanned at 1x magnification.

[0059] The entire slide image, which consists of a hierarchical form of such high-magnification pathology images (L0) and low-magnification images (L1, L2), is a large-capacity pathology image. The large-capacity pathology image can be transmitted sequentially from the front end (100), but since the large-capacity pathology image transmitted sequentially also exceeds tens of thousands to hundreds of thousands of pixels, it must be divided into patches of a pre-set size for deep learning analysis, loaded, and diagnosed.

[0060] Accordingly, the image loading unit (21) according to an embodiment of the present invention divides the large-capacity pathology image into patch units starting from the area corresponding to the user input and loads it, and transmits it to the deep learning unit (22). Here, the user input includes the user's field of view (FOV) and a scaling factor, and the currently displayed view image area can be set into patch units and divided to load according to the scaling factor. However, the area corresponding to the user input is not necessarily set into patch units; depending on the implementation method, it may be divided into smaller units of a predetermined size, or it may be set differently depending on the specific protein, cell, tissue, structure, etc. to be extracted from the pathology image.

[0061] Additionally, if necessary, the image processing unit (22) may perform a preprocessing process on a large volume pathology image, for example, after performing image preprocessing such as contrast, brightness, saturation, and blur, it may divide the image into patches, or after dividing it into patches, it may perform the above-described process, or it may not perform a preprocessing process.

[0062] This image processing unit (22) transmits large-capacity pathology images divided into patches to the deep learning unit (22).

[0063] The deep learning unit (22) may be configured in a hierarchical form including a segmentation model (23a) and a regression model (23b) for large-scale pathology image analysis, as shown in FIG. 3.

[0064] At this time, the large-capacity pathology image corresponding to the area divided into patch units includes a hierarchical form of high-magnification pathology images and low-magnification pathology images, and the deep learning unit (22) applies the high-magnification pathology images to the segmentation model (23a) and the low-magnification pathology images to the regression model (23b) to generate a result image.

[0065] A hierarchical deep learning model including the segmentation model (23a) and regression model (23b) of the present invention will be described in more detail with reference to FIGS. 4 and FIGS. 5.

[0066] The segmentation model (23a) of the present invention may be composed of a Convolutional Neural Network (CNN) of an encoder and a decoder including a downsampling layer.

[0067] At this time, the encoder of the segmentation model (23a) is a convolutional neural network that extracts features from an input high-magnification pathology image and may include five convolutional layers as shown in FIG. 5. At this time, some of the three convolutional layers at the front may include downsampling layers, and the three convolutional layers at the front may be connected to the regression head of the regression model (23b).

[0068] Additionally, as shown in FIG. 4, the regression model (23b) is connected to the encoder of the segmentation model (23a), and may include a regression head that performs regression learning on the extracted features when features are extracted from the low-magnification pathology image through the encoder of the segmentation model (23a).

[0069] In this structure, while the deep learning unit (22) generates a result image by passing a high-magnification pathology image through the encoder and decoder of the segmentation model (23a), the deep learning unit (22) transmits a low-magnification pathology image to the regression model (23b), extracts features for the low-magnification pathology image through the encoder of the connected segmentation model (23a), and transmits the extracted features to the regression head to perform regression learning.

[0070] In this way, the regression model (23b) of the present invention does not include a separately trained encoder, but rather connects an encoder containing the trained parameters of the segmentation model (23a) by sharing and fine-tuning it in a transfer learning manner, thereby enabling more precise learning even when using the regression model. Furthermore, to this end, the segmentation model (23a) of the present invention can maintain a frozen state in which weights are not updated during regression learning, and the layer of the regression head can operate as a fixed feature extractor.

[0071] To explain the process of the deep learning unit (22) described above again with reference to FIG. 5, the L0 image is a high-magnification pathology image, which is a high-magnification pathology image divided into patch units (512x512x3) corresponding to the user viewpoint and magnification factor. The L1 image is a low-magnification pathology image corresponding to the area divided into patch units, and since the high-magnification pathology image and the low-magnification pathology image include a hierarchical image structure in the shape of a pyramid, it is a low-magnification pathology image divided into corresponding patch units (128x128x3).

[0072] Features of the L0 image can be extracted through the encoder of the segmentation model (23a), and a result image (A) for the high-magnification pathology image is generated through the decoder.

[0073] At this time, the encoder of the splitting model (23a) may include five convolution layers, some of the three convolution layers at the front may include downsampling layers, and the three convolution layers at the front may be connected to the regression head of the regression model (23b).

[0074] While the result image (A) for the high-magnification pathology image is being generated, the low-magnification pathology image may have features extracted and compressed through the encoder of the segmentation model (23a), and the extracted and compressed features are passed to the regression head of the regression model (23b). The regression head is also a convolutional neural network structure and can perform the role of converting the final output of the encoder into a semantic prediction and can generate the result image (B) for the low-magnification pathology image. Here, the result image (B) for the low-magnification pathology image is in a more compressed form than the low-magnification pathology image. Thus, the deep learning model of the present invention can quickly generate analysis results for a wide area with a smaller size for the low-magnification pathology image, and can perform precise analysis on a limited area unit for the high-magnification pathology image, which is the highest resolution, through the segmentation model.

[0075] In addition, when the deep learning unit (22) of the present invention performs regression learning through the regression model (23b), it may use an asymmetric loss function (Mean Asymmetric-Squared Error, MASE) according to the following mathematical formula as the loss function for regression learning.

[0076]

[0077] (Here, f is a function for the regression model, x is the low-magnification pathology image, y is the label, and ASE is the following asymmetric squared error) )

[0078] Afterwards, the generated result information is transmitted to the result providing unit (23), and the result providing unit (23) transmits the result image to the front end (100).

[0079] At this time, the result providing unit (23) may store the result image in the cache device (30) illustrated in FIG. 1 in correspondence with the patch, and when a user of the front end (100) requests the result image for the same patch, the result image stored in the cache device may be provided to the user of the front end (100).

[0080] For the time being, an apparatus including a hierarchical deep learning model according to an embodiment of the present invention has been described with reference to FIGS. 2 to 5.

[0081] Hereinafter, a deep learning method in a device including a hierarchical deep learning model according to an embodiment of the present invention will be described with reference to FIG. 6.

[0082] FIG. 6 is a flowchart illustrating a deep learning method in a device including a hierarchical deep learning model according to an embodiment of the present invention.

[0083] Referring to FIGS. 1 and 6, before describing a deep learning method in a device including a hierarchical deep learning model, first, a pathology slide, such as a frozen section slide stained with H&E (Hematoxylin & Eosin), which has been fixed and stained through a series of chemical processing steps on tissues removed from the human body, etc., may be prepared in the front end (100). Then, in the front end (100), the pathology slide may be photographed using an imaging device including a high-magnification objective lens of 20x, 40x, or higher, and scanned through a digital scanner to generate a large-capacity pathology image. At this time, the large-capacity pathology image is a pyramid-shaped Whole Slide Image (WSI) consisting of multiple magnification layers of high-magnification pathology images and low-magnification pathology images, and the pyramid-shaped hierarchical large-capacity pathology image may be sequentially transmitted to the back end (200).

[0084] A device (20) including a hierarchical deep learning model of a backend (200) can check a large volume of pathology images delivered sequentially (S100) and check user input including a user viewpoint (Field Of View, FOV) which is the location coordinate for the view image area currently displayed on the pathology slide of the frontend (100) and a scale factor (e.g., 40x) (S110).

[0085] And, the device (20) including a hierarchical deep learning model can load a large volume pathology image by dividing it into patches (e.g., 512x512) starting from the area corresponding to the user viewpoint and magnification factor (S120). Since the area divided into patches includes high-magnification pathology images and low-magnification pathology images, the high-magnification pathology images are applied to the segmentation model starting from the area divided into patches (S130), a segmentation branch process is performed, and a result image for the high-magnification pathology images is generated (S140).

[0086] At this time, the segmentation model of the present invention may use a Binary-Cross Entropy (BCE) loss function and may be trained for 60 epochs. The batch size of high-magnification pathology images segmented into 512x512 patch units may be set to, for example, 10, and parameter optimization may be performed by checking precise parameters at every epoch.

[0087] While a result image for a high-magnification pathology image is being generated using the segmentation model learned through this training process, a regression branch process (S150 ~ S160) can be performed through the encoder of the segmentation model.

[0088] That is, as described above, the regression model of the device (20) including the hierarchical deep learning model is connected to and shared with the encoder of the segmentation model, and the low-magnification pathology image is applied to the regression model in the area segmented by patch units, and features are extracted from the low-magnification pathology image through the encoder of the segmentation model, and the extracted features are applied to the regression head of the regression model to generate a result image for the low-magnification pathology image through regression learning (S160).

[0089] The regression model of the present invention is connected to the encoder of the segmentation model and can utilize low-resolution pathology images that are 16 times lower than high-resolution pathology images, and can generate labels with a resolution 1024 times lower. In the process of utilizing low-resolution images, the object size is also reduced, which may cause an under-fitting problem. To solve this, the regression model of the present invention uses an asymmetric loss function according to <Equation 1> described above when performing regression learning. Since the regression branch requires much less memory even when training with 300 epochs, which is higher than the segmentation branch due to the small size, the batch size can be 30, which is three times larger than the batch size of 10 of the segmentation model.

[0090] Thus, the present invention enables the generation of precise analysis results for a limited area of ​​high-magnification pathology images, which are the highest resolution images, through a segmentation model, and allows for the generation of analysis results more rapidly of reduced-magnification pathology images through a regression model, while enabling the analysis of a wider area without requiring additional memory. Furthermore, the regression model of the present invention operates in a transfer learning manner by sharing the encoder of the segmentation model, thereby enabling precise learning even when generating analysis results rapidly.

[0091] For the above, a deep learning method in a device including a hierarchical deep learning model according to an embodiment of the present invention has been described.

[0092] The deep learning method in a device including the hierarchical deep learning model of the present invention as described above may be provided in the form of a computer-readable medium suitable for storing computer program instructions and data.

[0093] In particular, the computer program of the present invention is a deep learning method in a device comprising a hierarchical deep learning model for analyzing a large volume pathology image, which is a Whole Slide Image (WSI) formed in a hierarchical form of a high-magnification pathology image and a low-magnification pathology image, wherein the device may execute steps such as: dividing and loading the large volume pathology image into patch units starting from an area corresponding to user input; applying the high-magnification pathology image corresponding to the area divided into patch units to the dividing model; and applying the low-magnification pathology image to the regression model to generate a result image.

[0094] Computer-readable media suitable for storing such computer program instructions and data include, for example, recording media such as magnetic media (e.g., hard disks, floppy disks, and magnetic tapes), optical recording media (e.g., CD-ROM, Digital Video Disk), and floptical disks (e.g., magneto-optical media), and semiconductor memories such as ROM (Read Only Memory), RAM (Random Access Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically Erasable Programmable ROM). Processors and memory may be supplemented by or integrated with special-purpose logic circuits.

[0095] Furthermore, computer-readable recording media may be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Additionally, the functional program for implementing the present invention, along with related code and code segments, may be easily inferred or modified by programmers skilled in the art to which the present invention pertains, taking into account the system environment of the computer that reads the recording media to execute the program.

[0096] In addition, a computer program recorded on a computer-readable recording medium as described above includes instructions that perform the functions described above, and can execute the aforementioned functions by being distributed and circulated through the recording medium, read, installed, and executed on a specific device or specific computer.

[0097] The present invention has been described with reference to embodiments illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical concept of the claims. Industrial applicability

[0098] The present invention relates to an apparatus, system, and method including a hierarchical deep learning model for analyzing pathology images in the form of a Whole Slide Image (WSI) using a hierarchical deep learning model.

[0099] According to the present invention, a large-capacity pathology image, which is a whole slide image, can be divided into patches and applied to a deep learning model simultaneously with loading, enabling real-time analysis. Furthermore, by utilizing a hierarchical deep learning model including a segmentation model and a regression model, it is possible to generate faster and more accurate analysis results, thereby contributing to the advancement of digital pathology and thus having sufficient potential for industrial application. Explanation of the symbols

[0100] 20: Device including a hierarchical deep learning model 21: Image Loading Section 22: Deep Learning Department 23: Result Provisioning Department 30: Cache device 100: Frontend 200: Backend 500: Systems including hierarchical deep learning models

Claims

Claim 1 An apparatus comprising a hierarchical deep learning model composed of a segmentation model and a regression model for analyzing a large volume pathology image, which is a Whole Slide Image (WSI) formed in a hierarchical form of high-magnification pathology images and low-magnification pathology images, wherein the apparatus comprises: an image loading unit that divides and loads the large volume pathology image into patch units starting from an area corresponding to user input; and a deep learning unit that applies the high-magnification pathology image corresponding to the area divided into patch units through the image loading unit to the segmentation model and applies the low-magnification pathology image to the regression model to generate a result image. Claim 2 The device according to claim 1, wherein the user input includes a user field of view (FOV) and a magnification factor, and the image loading unit loads the large-capacity pathology image by dividing it into patches starting from the area corresponding to the user field of view and the magnification factor. Claim 3 An apparatus according to claim 1, wherein the segmentation model is a Convolutional Neural Network (CNN) having an encoder and decoder structure including a down-sampling layer, and the regression model is connected to the encoder of the segmentation model and includes a regression head that performs regression learning on the extracted features when features are extracted through the encoder of the segmentation model. Claim 4 An apparatus according to claim 3, wherein the deep learning unit applies the high-magnification pathology image to the segmentation model to generate a result image for the high-magnification pathology image, and while the result image for the high-magnification pathology image is being generated, applies the low-magnification pathology image to the regression model to extract features for the low-magnification pathology image through the encoder of the segmentation model, and performs regression learning on the extracted features through the regression head of the regression model to generate a result image for the low-magnification pathology image. Claim 5 An apparatus according to claim 3, wherein the deep learning unit uses an asymmetric loss function (Mean Asymmetric-Squared Error, MASE) according to the following mathematical formula as the loss function for the regression learning. <Mathematical Formula 1> (Here, f is a function for the regression model, x is the low-magnification pathology image, y is the label, and ASE is the following asymmetric squared error) ) Claim 6 The device according to claim 1 further comprises a result providing unit that stores the result image in a cache device corresponding to a patch, and provides the result image stored in the cache device to the user when the user requests a result image for the same patch. Claim 7 A system comprising a hierarchical deep learning model for analyzing a large volume pathology image, wherein the whole slide image (WSI) is formed in a hierarchical form of high-magnification pathology images and low-magnification pathology images, the system comprises: a front end that transmits a large volume pathology image including an area corresponding to a user input; and a back end that divides and loads the large volume pathology image transmitted from the front end into patch units starting from the area corresponding to the user input, applies the high-magnification pathology image corresponding to the area divided into patch units to the division model, applies the low-magnification pathology image to the regression model to generate a result image, and transmits the generated result image to the front end. Claim 8 A system according to claim 7, wherein the front end includes a slide viewer, checks the user input including a user viewpoint (FOV) and a scale factor through the slide viewer, and outputs the transmitted result image through the slide viewer when a result image is transmitted from the back end. Claim 9 A system according to claim 7, wherein the backend further includes a cache device that stores the generated result image corresponding to a patch, and when a result image for the same patch is requested from the frontend, the result image stored in the cache device is transmitted to the frontend. Claim 10 A deep learning method in a device comprising a hierarchical deep learning model for analyzing a large volume pathology image, wherein the whole slide image (WSI) is formed in a hierarchical form of high-magnification pathology images and low-magnification pathology images, the device comprises the steps of: loading the large volume pathology image by dividing it into patch units starting from an area corresponding to user input; and applying the high-magnification pathology image corresponding to the area divided into patch units to the dividing model and applying the low-magnification pathology image to the regression model to generate a result image. Claim 11 A deep learning method according to claim 10, wherein the user input includes a user viewpoint (FOV) and a magnification factor, and the loading step is characterized by dividing and loading the high-magnification pathology image into patches starting from the area corresponding to the user viewpoint and the magnification factor. Claim 12 A deep learning method according to claim 10, wherein the segmentation model is a Convolutional Neural Network (CNN) having an encoder and decoder structure including a down-sampling layer, and the regression model is connected to the encoder of the segmentation model and includes a regression head that performs regression learning on the extracted features when features are extracted through the encoder of the segmentation model. Claim 13 A deep learning method according to claim 12, wherein the step of generating the result image is characterized by applying the high-magnification pathology image to the segmentation model to generate a result image for the high-magnification pathology image, applying the low-magnification pathology image to the regression model to extract features for the low-magnification pathology image through the encoder of the segmentation model, and performing regression learning on the extracted features through the regression head of the regression model to generate a result image for the low-magnification pathology image. Claim 14 A computer-readable recording medium storing a program for executing a deep learning method described in any one of paragraphs 11 through 13.

Images