System and method for classification of histopathologic images
By using CNN and LSTM networks in machine learning diagnostic systems to extract hierarchical relationship information of histopathological images, the problem of insufficient diagnostic accuracy in the prior art is solved, and higher classification accuracy and effective diagnosis of medical conditions are achieved.
Patent Information
- Application Number
- CN202380023322.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-02-28
- Filing Date
- 2023-02-27
- Publication Date
- 2024-11-08
Smart Images

Figure CN118922869A_ABST
Abstract
Description
Background Art
[0001] field
[0002] The described technology relates to classification of digital pathology images, and in particular, to a hierarchical artificial intelligence approach to classifying histopathology images.
[0003] Description of Related Technology
[0004] Machine learning diagnostic systems can be used to diagnose diseases or other conditions based on histopathological images. Typically, such systems employ artificial intelligence to identify patterns in the histopathological images that can be used to generate a diagnosis. In order to increase the accuracy of the diagnosis, there is a need to improve the deep learning architecture used in machine learning diagnostic systems. Summary of the invention
[0005] In one aspect, a device for detecting a medical condition in a histopathology image is provided, the device comprising: a hardware memory configured to store executable instructions; and a hardware processor in communication with the hardware memory, wherein the executable instructions, when executed by the processor, cause the processor to: obtain a plurality of patches at a plurality of magnification levels from the histopathology image; apply a deep learning algorithm to each of the patches; extract information representing a hierarchical relationship linking characteristics of the histopathology image existing at one level and another level in the plurality of magnification levels by applying the deep learning algorithm; and identify the medical condition based on the extracted information representing the hierarchical relationship of the characteristics existing at the one level and the another level in the plurality of magnification levels.
[0006] In some embodiments, the executable instructions, when executed by the processor, further cause the processor to: obtain the histopathology image; and crop the histopathology image to generate the plurality of tiles, wherein each of the tiles has a different magnification level than the other tiles.
[0007] In some embodiments, the deep learning algorithm includes a plurality of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, and wherein the executable instructions, when executed by the processor, further cause the processor to: provide each of the tiles to a corresponding one of the CNNs, and provide an output of each of the CNNs to the LSTM network, wherein the LSTM network is configured to extract the information representing the hierarchical relationship linking the features of the tissue pathology image present at the one level and the other level in the plurality of magnification levels.
[0008] In some embodiments, the executable instructions, when executed by the processor, further cause the processor to sequentially learn the hierarchical relationship linking the characteristics of the tissue pathology image at the one level and the other level among the multiple magnification levels by providing the LSTM network with a result of the LSTM network processing the output of a previous CNN each time the LSTM network is processing the output of a CNN other than an initial CNN.
[0009] In some embodiments, the initial CNN has the lowest magnification of any CNN from the plurality of CNNs.
[0010] In some embodiments, the deep learning algorithm also includes a Softmax activation function, and wherein the executable instructions, when executed by the processor, further cause the processor to: provide the output of the LSTM network to the Softmax activation function, and generate a tile level classification based on the output of the Softmax activation function.
[0011] In some embodiments, the Softmax activation function is configured to generate the output comprising a probability distribution over a set of predicted output classes.
[0012] In some embodiments, the hierarchical relationship linking the characteristics of the histopathology image present at the plurality of magnification levels represents a relationship between tissue morphology at a first one of the magnification levels and cellular structure at a second one of the magnification levels.
[0013] In some embodiments, the executable instructions, when executed by the processor, further cause the processor to: use an attention mechanism to identify a region of interest within the histopathology image; and crop the histopathology image based on the region of interest to generate the plurality of tiles.
[0014] In another aspect, a non-transitory computer-readable medium for detecting a medical condition in a histopathology image is provided, the computer-readable medium having program instructions for causing a hardware processor to: obtain a plurality of tiles at a plurality of magnification levels from the histopathology image; apply a deep learning algorithm to each of the tiles; extract, by applying the deep learning algorithm, information representing a hierarchical relationship linking characteristics of the histopathology image present at one level and another level in the plurality of magnification levels; and identify the medical condition based on the extracted information representing the hierarchical relationship of the characteristics present at the one level and the another level in the plurality of magnification levels.
[0015] In some embodiments, the instructions are further configured to cause the hardware processor to: obtain the histopathology image; and crop the histopathology image to generate the plurality of tiles; wherein each of the tiles has a different magnification level than the other tiles.
[0016] In some embodiments, wherein the deep learning algorithm comprises a plurality of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, and wherein the instructions are further configured to cause the hardware processor to: provide each of the tiles to a corresponding one of the CNNs; and provide an output of each of the CNNs to the LSTM network, wherein the LSTM network is configured to extract the information representing the hierarchical relationship linking the features of the tissue pathology image at the one level and the other level present in the plurality of magnification levels.
[0017] In some embodiments, the instructions are further configured to cause the hardware processor to sequentially learn the hierarchical relationship linking the features of the tissue pathology image at the one level and the other level among the multiple magnification levels by providing the LSTM network with a result of the LSTM network processing the output of a previous CNN each time the LSTM network is processing the output of a CNN other than an initial CNN.
[0018] In some embodiments, the initial CNN has the lowest magnification of any CNN from the plurality of CNNs.
[0019] In some embodiments, the deep learning algorithm also includes a Softmax activation function, and wherein the instructions are further configured to cause the hardware processor to: provide the output of the LSTM network to the Softmax activation function; and generate a tile level classification based on the output of the Softmax activation function.
[0020] In some embodiments, the Softmax activation function is configured to generate the output comprising a probability distribution over a set of predicted output classes.
[0021] In some embodiments, the hierarchical relationship linking the characteristics of the histopathology image present at the plurality of magnification levels represents a relationship between tissue morphology at a first one of the magnification levels and cellular structure at a second one of the magnification levels.
[0022] In some embodiments, the instructions are further configured to cause the hardware processor to: identify a region of interest within the histopathology image using an attention mechanism; and crop the histopathology image based on the region of interest to generate the plurality of tiles.
[0023] In yet another aspect, a method is provided, the method comprising: obtaining a plurality of tiles at a plurality of magnification levels from a histopathology image; applying a deep learning algorithm to each of the tiles; extracting, by applying the deep learning algorithm, information representing a hierarchical relationship linking characteristics of the histopathology image at one level and another level present in the plurality of magnification levels; and identifying the medical condition based on the extracted information representing the hierarchical relationship of the characteristics at the one level and the another level present in the plurality of magnification levels.
[0024] In some embodiments, the method further comprises: obtaining the histopathology image; and cropping the histopathology image to generate the plurality of tiles, wherein each of the tiles has a different magnification level than the other tiles.
[0025] In some embodiments, the deep learning algorithm includes a plurality of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, and wherein the method further comprises: providing each of the tiles to a corresponding one of the CNNs; and providing an output of each of the CNNs to the LSTM network, wherein the LSTM network is configured to extract the information representing the hierarchical relationship linking the features of the tissue pathology image at the one level and the other level present in the plurality of magnification levels.
[0026] In some embodiments, the method further includes sequentially learning the hierarchical relationship linking the features of the tissue pathology image at the one level and the other level among the multiple magnification levels by providing the LSTM network with a result of the LSTM network processing the output of a previous CNN each time the LSTM network is processing the output of a CNN other than an initial CNN.
[0027] In some embodiments, the initial CNN has the lowest magnification of any CNN from the plurality of CNNs. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Features and advantages of the devices, systems and methods described herein will become more apparent from the following description and the appended claims in conjunction with the accompanying drawings. These drawings depict several embodiments according to the present disclosure and should not be considered to limit the scope thereof. In the accompanying drawings, similar reference numerals or symbols typically identify similar components unless the context dictates otherwise. The accompanying drawings may not be drawn to scale.
[0029] Figure 1 An exemplary environment for a multispectral imaging system is shown.
[0030] Figure 2is an exemplary computing system that can implement any one or more of the imaging devices, image analysis systems, user computing devices, interface servers, machine learning servers, and other components described herein.
[0031] Figure 3 An exemplary training and / or inference platform for a machine learning system is shown.
[0032] FIG. 4A to FIG. 4C Exemplary histopathology images cropped from a histopathology whole slide image (WSI) at different magnification levels are shown.
[0033] Figure 5 is an exemplary block diagram illustrating a deep learning architecture that may be implemented by a machine learning diagnostic system and is configured to classify histopathology images.
[0034] Figure 6 is another exemplary block diagram illustrating a deep learning architecture configured to classify histopathology images.
[0035] Figure 7 It shows that Figure 6 An example block diagram of a convolutional neural network (CNN) used in a deep learning architecture.
[0036] Figure 8 It shows that Figure 6 An example block diagram of an LSTM used in a deep learning architecture.
[0037] Fig. 9 is an exemplary flow chart for classifying histopathology images. DETAILED DESCRIPTION
[0038] Features of systems and methods for classification of histopathology images will now be described in detail with reference to certain embodiments illustrated in the accompanying drawings. The illustrated embodiments described herein are provided by way of illustration and are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented. It will be readily appreciated that the aspects and features of the present disclosure described below and illustrated in the drawings may be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are a part of the present disclosure.
[0039] Multispectral Imaging System Overview
[0040] Figure 1An exemplary environment 100 (e.g., a multispectral imaging system) is shown in which a user and / or a multispectral imaging system can analyze a sample. The environment 100 includes an automated slide stainer that is controlled to produce consistently stained slides based on one or more protocols. The environment 100 may also include an imaging device 102 that generates a digital representation (e.g., an image) of the stained slide. The digital representation may be transmitted to a network 112 as a signal [C], and then transmitted to an image analysis system 108 for processing (e.g., feature detection, feature measurement, etc.). The image analysis system 108 may perform image analysis on the received image data. The image analysis system 108 may normalize the image data obtained using multispectral imaging for input to a machine learning algorithm and / or model that may determine characteristics of the image. The results from the image analysis system 108 may be transmitted to one or more display devices 110 (the one or more display devices may also be referred to herein as "display devices" or "client devices") as a signal [E].
[0041] In some implementations, the imaging device 102 includes a light source 104 configured to emit multispectral light onto a tissue sample and an imaging sensor 106 configured to detect the multispectral light emitted from the tissue sample. Multispectral imaging using the light source 104 may involve providing light to a tissue sample carried by a carrier within a range of frequencies. That is, the light source 104 may be configured to generate light across a frequency spectrum to provide multispectral imaging.
[0042] In certain embodiments, the tissue sample may reflect light received from the light source 104, which may then be detected at the imaging sensor 106. In these implementations, the light source 104 and the imaging sensor 106 may be located on substantially the same side of the tissue sample. In other implementations, the light source 104 and the imaging sensor 106 may be located on opposite sides of the tissue sample. The imaging sensor 106 may also be configured to generate image data based on multi-spectral light detected at the imaging sensor 106. In certain implementations, the imaging sensor 106 may include a high-resolution sensor configured to generate a high-resolution image of the tissue sample. A high-resolution image may be generated based on the excitation of the tissue sample in response to laser light emitted onto the sample at different frequencies (e.g., spectrums).
[0043] The imaging device 102 may capture and / or generate image data for analysis. The imaging device 102 may include one or more of a lens, an image sensor, a processor, or a memory. The imaging device 102 may receive user interactions. The user interaction may be a request to capture image data. Based on the user interaction, the imaging device 102 may capture image data. The imaging device may store image data and other information in the image and information database 113. In some embodiments, the imaging device 102 may periodically (e.g., every 10 minutes, 20 minutes, or 30 minutes) capture image data. In other embodiments, the imaging device 102 may determine that an object has been placed in the field of view of the imaging device 102 (e.g., a histological sample has been placed on a table and / or platform associated with the imaging device 102), and based on the determination, capture image data corresponding to the object. The imaging device 102 may also receive image data from additional imaging devices. For example, the imaging device 102 may be a node that routes image data from other imaging devices to the image analysis system 108. In some embodiments, the imaging device 102 may be located within the image analysis system 108. For example, the imaging device 102 may be a component of the image analysis system 108. In addition, the image analysis system 108 may perform imaging functions. In other embodiments, the imaging device 102 and the image analysis system 108 may be connected (e.g., wirelessly or wired). For example, the imaging device 102 and the image analysis system 108 may communicate via the network 112. In addition, the imaging device 102 and the image analysis system 108 may communicate via a wired connection. In one embodiment, the image analysis system 108 may include a docking station that enables the imaging device 102 to dock with the image analysis system 108. The electrical contacts of the image analysis system 108 may be connected to the electrical contacts of the imaging device 102. The image analysis system 108 may be configured to determine when the imaging device 102 has been connected to the image analysis system 108 based at least in part on the electrical contacts of the image analysis system 108. In some embodiments, the image analysis system 108 may use one or more other sensors (e.g., proximity sensors) to determine that the imaging device 102 has been connected to the image analysis system 108. In some embodiments, the image analysis system 108 may be connected (via wired or wireless connections) to multiple imaging devices.
[0044] The image analysis system 108 may include various components for providing the features described herein. In some embodiments, the image analysis system 108 may perform image analysis on image data received from the imaging device 102. The image analysis system 108 may use the image data to execute one or more imaging algorithms.
[0045] The image analysis system 108 may be connected to one or more display devices 110. The image analysis system 108 may be connected to (via a wireless or wired connection) a display device 110 to provide recommendations for a set of image data. The image analysis system 108 may transmit the recommendations to the display device 110 via a network 112. In some embodiments, the image analysis system 108 and the user computing device 110 may be configured for connection so that the user computing device 110 may engage and disengage with the image analysis system 108 in order to receive recommendations. For example, the display device 110 may engage with the image analysis system 108 upon determining that the image analysis system 108 has generated a recommendation for the display device 110. In addition, the display device 110 may be connected to the image analysis system 108 based on the image analysis system 108 performing image analysis on image data corresponding to a particular user computing device 110. For example, a user may be associated with multiple histological samples. Upon determining that a particular histological sample is associated with a particular user and a corresponding display device 110, the image analysis system 108 may transmit a recommendation for the histological sample to the particular display device 110. In some implementations, the display device 110 may interface with the image analysis system 108 in order to receive recommendations.
[0046] In some implementations, the imaging device 102, the image analysis system 108, and / or the display device 110 may communicate wirelessly. For example, the imaging device 102, the image analysis system 108, and / or the display device 110 may communicate via a network 112. The network 112 may include any feasible communication technology, such as wired and / or wireless modalities and / or technologies. The network may include a personal area network ("PAN"), a local area network ("LAN"), a campus network ("CAN"), a metropolitan area network ("MAN"), an extranet, an intranet, the Internet, a short-range wireless communication network (e.g., ZigBee, Bluetooth, etc.), a wide area network ("WAN") (both centralized and / or distributed), and / or any combination, arrangement, and / or aggregation thereof. The network 112 may include the Internet and / or may or may not be accessible to and / or from the Internet. The imaging device 102 and the image analysis system 108 may transmit image data. For example, the imaging device 102 may transmit image data associated with the histological sample to the image analysis system 108 via the network 112 for analysis. The image analysis system 108 and the display device 110 may transmit a recommendation corresponding to the image data. For example, the image analysis system 108 may transmit a diagnosis as to whether the image data indicates a disease present in the tissue sample. In some embodiments, the imaging device 102 and the image analysis system 108 may communicate via a first network, and the image analysis system 108 and the display device 110 may communicate via a second network. In other embodiments, the imaging device 102, the image analysis system 108, and the display device 110 may communicate via the same network.
[0047] One or more third-party computer systems 115 ("computer systems 115") may communicate with the imaging device 102, the image analysis system 108, and / or the display device 110. In some embodiments, the computer system 115 may communicate directly with the imaging device 102, the image analysis system 108, and / or the display device 110, either directly or via the network 112.
[0048] The computer system 115 can provide information to change the function on the imaging device 102, the image analysis system 108 and / or the display device 110 or even the network 112. For example, the information can be new software, software updates, new or revised lookup tables, or data or any other type of information used to generate, manipulate, transfer or render images in any way (for ease of reference, all are referred to as "updates" in this article). Updates can involve, for example, image compression, image transfer, image storage, image display, image rendering, etc. The computer system 115 can provide messages to the device or system to be updated, or can provide messages to users who interact with the system control of the update system. In some embodiments, the computer system 115 automatically provides updates, for example, periodically or as needed / when available. In some embodiments, the computer system 115 can provide updates in response to receiving an indication (for example, confirmation of the update or a request for the update) from a user who provides the update. Once the update has been performed, the system can perform a quality check to determine whether the update has changed the display mode of the image (for example, the color of the tissue sample). If the update has changed the way the image is displayed such that the change is greater than the quality threshold, the system can generate a message to warn the user to update when the image display quality degrades or changes.
[0049] Referring to the exemplary embodiment, at [A], the imaging device 102 may obtain block data. To obtain the block data, the imaging device 102 may image (e.g., scan, capture, record, etc.) a tissue block. The tissue block may be a histological sample. For example, the tissue block may be a biological tissue block that has been removed and prepared for analysis. As will be discussed in more detail below, various histological techniques may be performed on the tissue block in order to prepare the tissue block for analysis. The imaging device 102 may capture an image of the tissue block and store the corresponding block data in the imaging device 102. The imaging device 102 may obtain the block data based on user interaction. For example, a user may provide input through a user interface (e.g., a graphical user interface (“GUI”)) and request the imaging device 102 to image the tissue block. In addition, the user may interact with the imaging device 102 to cause the imaging device 102 to image the tissue block. For example, the user may flip a switch of the imaging device 102, press a button of the imaging device 102, provide a voice command to the imaging device 102, or otherwise interact with the imaging device 102 to cause the imaging device 102 to image the tissue block. In some embodiments, the imaging device 102 may image the tissue block based on the imaging device 102 detecting that the tissue block has been placed in the viewport of the imaging device 102. For example, the imaging device 102 may determine that the tissue block has been placed on the viewport of the imaging device 102 and determine to image the tissue block based on this.
[0050] At [B], the imaging device 102 may obtain slice data. In some embodiments, the imaging device 102 may obtain slice data and block data. In other embodiments, the first imaging device may obtain slices, and the second imaging device may obtain block data. In order to obtain slice data, the imaging device 102 may image (e.g., scan, capture, record, etc.) slices of the tissue block. The slices of the tissue block may be slices of histological samples. For example, the tissue block may be sliced (e.g., segmented) to generate one or more slices of the tissue block. In some embodiments, a portion of the tissue block may be sliced to generate slices of the tissue block, so that the first portion of the tissue block corresponds to the tissue block imaged to obtain block data, and the second portion of the tissue block corresponds to the slice of the tissue block imaged to obtain slice data. As will be discussed in more detail below, various histological techniques may be performed on the tissue block to generate slices of the tissue block. The imaging device 102 may capture images of the slices and store the corresponding slice data in the imaging device 102. The imaging device 102 may obtain slice data based on user interaction. For example, a user may provide input through a user interface and request the imaging device 102 to image the slices. In addition, the user may interact with the imaging device 102 to cause the imaging device 102 to image the slice. In some embodiments, the imaging device 102 may image the tissue block based on the imaging device 102 detecting that the tissue block has been sliced or a slice has been placed in the imaging device 102 viewport.
[0051] At [C], the imaging device 102 may transmit a signal representing the captured image data (e.g., block data and slice data) to the image analysis system 108. The imaging device 102 may send the captured image data as an electronic signal to the image analysis system 108 via the network 112. The signal may include and / or correspond to a pixel representation of the block data and / or slice data. It should be understood that the signal may include and / or correspond to more, less, or different image data. For example, the signal may correspond to multiple slices of a tissue block and may represent first slice data and second slice data. In addition, the signal may enable the image analysis system 108 to reconstruct the block data and / or slice data. In some embodiments, the imaging device 102 may transmit a first signal corresponding to the block data and a second signal corresponding to the slice data. In other embodiments, the first imaging device may transmit a signal corresponding to the block data, and the second imaging device may transmit a signal corresponding to the slice data.
[0052] At [D], the image analysis system 108 may perform image analysis on the block data and slice data provided by the imaging device 102. The image analysis system 108 may perform one or more image processing functions. For example, the image analysis system 108 may perform an imaging algorithm. The image analysis system 108 may also use a machine learning model (such as a convolutional neural network) to perform the image processing function. Based on performing the image processing function, the image analysis system 108 may determine the possibility that the block data and the slice data correspond to the same tissue block. For example, the image processing function may perform an edge analysis of the block data and the slice data, and based on the edge analysis, determine whether the block data and the slice data correspond to the same tissue block. The image analysis system 108 may obtain a confidence threshold from the display device 110, the imaging device 102, or any other device. In some embodiments, the image analysis system 108 may determine the confidence threshold based on the response of the display device 110 to a specific recommendation. In addition, the confidence threshold may be specific to a user, a user group, a type of tissue block, a location of a tissue block, or any other factor. The image analysis system 108 may compare the determined confidence threshold with the image analysis performed. Based on the comparison, the image analysis system 108 may generate a recommendation indicating a recommended action for the display device 110 based on the likelihood that the block data and the slice data correspond to the same tissue block. In other embodiments, the image analysis system 108 may provide a diagnosis as to whether the image data indicates a disease present in the tissue sample, for example based on the results of a machine learning algorithm.
[0053] At [E], the image analysis system 108 may transmit a signal to the display device 110. The image analysis system 108 may send the signal as an electrical signal to the display device 110 via the network 112. The signal may include and / or correspond to a representation of a diagnosis. Based on receiving the signal, the display device 110 may determine the diagnosis. In some embodiments, the image analysis system 108 may transmit a series of recommendations corresponding to a set of tissue blocks and / or a set of slices. The image analysis system 108 may include a recommended action for the user in the recommendation. For example, the recommendation may include a recommendation for the user to review the tissue blocks and slices. In addition, the recommendation may include a recommendation that the user does not need to review the tissue blocks and slices.
[0054] Computing system implementation details
[0055] Figure 2 In various embodiments, the devices described herein (such as Figure 1 1, 2, and 3. In various embodiments, the computing system 200 may implement the functionality of the interface server 304 and / or the machine learning server 316. Figure 2, the computing system 200 may include one or more hardware processors 202 (such as a physical central processing unit ("CPU")), one or more network interfaces 204 (such as a network interface card ("NIC")), and one or more computer-readable media 206. The computer-readable medium may be, for example, a high-density disk ("HDD"), a solid-state drive ("SDD"), a flash drive, and / or other persistent, non-transitory computer-readable media. The computing system 200 may also include an input / output device interface 208, such as an input / output ("IO") interface that communicates with one or more microphones, and one or more non-transitory computer-readable memories (or "media") 210, such as random access memory ("RAM") and / or other volatile, non-transitory computer-readable media.
[0056] The network interface 204 may provide a connection to one or more networks or computing systems. The hardware processor 202 may receive information and instructions from other computing systems or services via the network interface 204. The network interface 204 may also store data directly to the computer-readable memory 210. The hardware processor 202 may communicate to and from the computer-readable memory 210. The hardware processor 202 may execute instructions and process data in the computer-readable memory 210.
[0057] The computer readable memory 210 may include computer program instructions executed by the hardware processor 202 to implement one or more embodiments described herein. The computer readable memory 210 may store an operating system 212 that provides computer program instructions for use by the computer processor 202 in the general management and operation of the computing system 200. The computer readable memory 210 may also include program instructions and other information for implementing various aspects of the present disclosure. In one example, the computer readable memory 210 includes instructions for training the machine learning model 214. As another example, the computer readable memory 210 may include image data 216. In another example, the computer readable memory 210 includes instructions for classifying one or more images based on the trained machine learning model 214.
[0058] Overview of Machine Learning Systems
[0059] Figure 3An exemplary environment 311 in which the machine learning diagnostic system 300 can train and / or use a machine learning model is shown. The environment 311 may include one or more user computing devices 302 ("user computing devices 302") and a network 312. The machine learning diagnostic system 300 may include an interface server 304, a machine learning server 316, a training database 310, and a trained model database 314. Each of the interface server 304 and the machine learning server 316 may include at least a processor and a memory. Each of the interface server 304 and the machine learning server 316 may include additional hardware components, such as those described above with respect to Figure 2 The hardware components described.
[0060] In various embodiments, the exemplary environment 311 can be used to train one or more machine learning models. For example, the user computing device 302 can transmit image data (which may include annotated image data) to the interface server 304 (via the network 312) for training purposes. The interface server 304 can communicate with the machine learning server 316, such as by transmitting image data. The machine learning server 316 can store the image data and other training data (such as class label masks) in the training database 310. The machine learning server 316 can use the image data to train one or more machine learning models, and the image data may include annotated image data. Exemplary annotated image data may include annotated images in which a pathologist has drawn an outline or circle around a tumor area and marked the tumor area (e.g., with different colors) to depict different classes of tumors (e.g., benign, cancer, ductal carcinoma in situ (DCIS), uncertain and lobular carcinoma in situ (LCIS)). The trained machine learning model can be configured to classify the input image data. In other words, the trained machine learning model can be configured to output a predicted classification of new input data, such as by predicting whether a patch in an image corresponds to a class, such as whether an abnormal cell is present, and if so, the class corresponding to a cancer cell. The machine learning server 316 can store the machine learning model in the trained model database 314.
[0061] In various embodiments, the exemplary environment 311 can be used to apply one or more trained machine learning models. For example, the user computing device 302 can transmit image data to the interface server 304 via the network 312 for classification purposes. The interface server 304 can communicate with the machine learning server 316, such as by transmitting image data to be classified. The machine learning server 316 can retrieve the trained machine learning model from the trained model database 314. The machine learning server 316 applies one or more machine learning models to the input image data to receive a predicted classification. The interface server 304 can receive the predicted classification and can transmit the predicted classification to the user computing device 302 via the network 312. In various embodiments, the interface server 304 can present a user interface including the predicted classification to the user computing device 302.
[0062] Histopathology Image Classification Technology
[0063] Classification of histopathology images (eg, histopathology WSIs in general) can be used to diagnose a disease or condition. In particular, such classification is particularly useful for diagnosing cancer within histopathology WSIs.
[0064] Object classes in certain settings are related to each other by taxonomies. In digital pathology, histopathology images can be generated at multiple magnification levels, for example, by cropping a histopathology WSI. Using multiple magnifications can help diagnose a disease or condition. For example, at lower magnification levels, properties such as tissue morphology can be observed, while at relatively higher magnification levels, more detailed properties such as cellular structures and other microscopic components can be observed. Cellular structures and other microscopic components visible at higher magnification levels may be particularly helpful for cancer diagnosis.
[0065] Histopathology images taken at different magnification levels can be represented as a tree based on a hierarchy, where each child node represents more details about its parent node. FIG. 4A to FIG. 4C Exemplary histopathology images cropped from a histopathology whole slide image (WSI) at different magnification levels are shown. Specifically, Figure 4A An exemplary histopathology image 402 is shown at a first magnification level (eg, 10X magnification). Figure 4B An exemplary histopathology image 404 is shown at a second magnification level (eg, 20X magnification), and Figure 4CAn exemplary histopathology image 404 at a second magnification level (e.g., 40x magnification) is shown. In some embodiments, the machine learning diagnostic system 300 can generate tile images at higher magnification levels by center-cropping and resizing one or more histopathology whole slide images (WSIs) at the lowest magnification level. In other embodiments, the machine learning diagnostic system 300 can crop histopathology images 402-406 from different regions of the histopathology WSI or other source histopathology images. For example, as discussed herein, the machine learning diagnostic system 300 can use an attention mechanism to identify one or more regions of interest within the histopathology WSI and crop the histopathology WSI to include the one or more regions of interest.
[0066] Continue to refer FIG. 4A to FIG. 4C , the histopathology image 402 can be represented as a parent node that semantically contains a child node representing the histopathology image 404. Similarly, the histopathology image 404 can be represented as a parent node that semantically contains a child node representing the histopathology image 406. Therefore, there is a hierarchical relationship between the image blocks 402-406 at different magnification levels. Various aspects of the present disclosure are related to systems and techniques that can utilize the relationship between the cropped histopathology images 402-406 at different magnification levels to improve the performance of an artificial intelligence (AI) model in making more accurate diagnostic predictions.
[0067] Some AI models use multiple convolutional neural networks (CNNs) to make diagnostic predictions using histopathology images at different magnification levels. Typically, such diagnostic AI models do not use the semantic relationship between images taken at different magnification levels, and therefore cannot use any hierarchical relationship information contained in the histopathology images to make diagnostic predictions.
[0068] Therefore, aspects of the present disclosure relate to AI models that are capable of using hierarchical relationship information contained in histopathological images to make diagnostic predictions. In other words, aspects of the present disclosure relate to systems and techniques that can capture and / or extract information representing hierarchical relationships that link features of histopathological images that exist at multiple magnification levels. For example, the linked features may include tissue morphology that exists at lower magnifications and cellular structures that exist at higher magnifications. Aspects of the present disclosure also relate to using these hierarchical relationships when diagnosing diseases or conditions present in histological images.
[0069] Figure 5 is an exemplary block diagram illustrating a deep learning architecture 510 configured to classify a histopathology image (e.g., a histopathology WSI) that may be implemented by the machine learning diagnostic system 300. In some embodiments, Figure 5The deep learning architecture 510 is an end-to-end deep learning architecture 510 configured to classify the histopathology images 402-406 into one of a plurality of classes 520. In one specific implementation, the deep learning architecture 510 may be configured to classify the breast cancer histology images 402-406 into one of five classes 520 (e.g., benign, carcinoma, ductal carcinoma in situ (DCIS), normal, and lobular carcinoma in situ (LCIS)).
[0070] The deep learning architecture 510 is configured to receive a plurality of patch images 402-406 cropped at different magnification levels from one or more histopathology WSIs. Depending on the specific implementation, the size of the histopathology WSI may be approximately 100,000×100,000 pixels, which may occupy up to 3GB of data. Due to the large size of the histopathology WSI, it may not be practical to fit the entire histopathology WSI in memory. Therefore, aspects of the present disclosure use the tiles 402-406 cropped from the histopathology WSI to train the AI or deep learning model.
[0071] exist Figure 5 In the example shown in , the deep learning architecture 510 may receive three cropped tile images, where the first tile 402 is center-cropped at a 10x magnification, the second tile 404 is center-cropped at a 20x magnification, and the third tile 406 is center-cropped at a 40x magnification. However, in other embodiments, the deep learning architecture 510 may be configured to receive as few as two cropped tiles 402-406 or four or more cropped tiles 402-406 at different magnification levels. For example, the machine learning diagnostic system 300 may crop the histopathology WSI and resize the cropped image to generate a first tile 402, where subsequent tiles 404 and 406 are cropped and resized from previously generated tiles 402 and 404. In some embodiments, the learning diagnostic system 300 may resize each of the tiles 402-406 to substantially the same size.
[0072] Depending on the implementation, the machine learning diagnostic system 300 may crop different parts of the input or source histopathology image (e.g., histopathology WSI) to generate tiles 402-406 instead of using center cropping. For example, in some implementations, the machine learning diagnostic system 300 may implement an attention mechanism to identify regions of interest within the input histopathology image to generate each of tiles 402-406. As used herein, an attention mechanism generally refers to a category of a neural network configured to mimic cognitive attention in order to identify one or more regions of interest within an image. The machine learning diagnostic system 300 may implement one or more different types of attention mechanisms, such as: a transformer, a visual transformer, a recursive attention model, or other methods of reinforcement learning. In yet other implementations, the machine learning diagnostic system 300 may use other system cropping techniques (such as a random cropping technique, cropping the upper left corner of the image, or any other cropping technique) to crop different parts of the input histopathology image to generate tiles 402-406.
[0073] The deep learning architecture 510 is configured to extract at least some information about the relationship between the cropped histopathology images 402 - 406 at different magnification levels and use the extracted information in generating a classification 520 of the received tiles 402 - 406 . Figure 6 is another exemplary block diagram illustrating a deep learning architecture 510 configured to classify tissue pathology images. Specifically, Figure 6 An exemplary implementation of a deep learning architecture 510 is provided. In the illustrated embodiment, the deep learning architecture 510 includes a plurality of CNNs 602, 604, and 606, a long short-term memory (LSTM) network 610, and a Softmax activation function 618. As used herein, an LSTM network is a type of recurrent neural network (RNN), specifically a many-to-one RNN unit. It should be noted that although Figure 6 The LSTM network 610 is shown as including a plurality of LSTM cells, but the LSTM network 610 preferably does not include a plurality of discrete cells. Instead, individual cells are shown because, as described below, the LSTM network 610 will receive different inputs at each iteration in a series of iterations, and the individual cells are intended to clearly illustrate the different iterations processed by the LSTM network 610 (i.e., each cell represents the same LSTM network at a different iteration).
[0074] Each of the CNNs 602-606 is configured to receive a corresponding one of the image tiles 402-406. Thus, the number of CNNs 602-606 included in the deep learning architecture 510 may correspond to the number of input tiles 402-406 that the deep learning architecture 510 is configured to receive. Each of the CNNs 602-606 is configured (e.g., trained) to learn spatial information within a corresponding received tile image 402-406 at a particular magnification level of the tile image 402-406.
[0075] LSTM network 610 is configured (e.g., trained) to receive outputs from CNNs 602-606. LSTM network 610 is configured to process outputs from CNNs 602-606 in a sequential manner to learn semantic relationships (e.g., hierarchical representations) within tiles 402-406 at different magnification levels. For example, by processing outputs from CNNs 602-606 in a sequential manner, LSTM network 610 can learn semantic relationships linking tissue morphology from lower magnification levels with cellular structures from higher magnifications.
[0076] In order to process the outputs from CNNs 602-606 sequentially, LSTM network 610 may include a processing loop so that when LSTM network 610 is processing the outputs from a CNN, it may also receive the outputs from processing the outputs of the previous CNN. Except when it is processing the outputs of the first CNN 602, LSTM network 610 receives both the outputs from CNNs 604 and 606 and the outputs from processing the previous CNN.
[0077] The LSTM network 610 is configured to provide the output generated by processing the output of the final CNN to a Softmax activation function 618. The Softmax activation function 618 may be configured to classify the histopathology image based on the output from the LSTM network 610, thereby generating a classification 520 of the histopathology image. The classification 510 may include a probability that the histopathology image belongs to each of a plurality of classes (e.g., class 1, class 2, ... class N). For example, the Softmax activation function 618 may be configured to normalize the output of the LSTM network 610 to a probability distribution on the predicted output class.
[0078] Advantageously, by using the deep learning architecture 510, the machine learning diagnostic system 300 can not only learn spatial information from each of the tile images 402-406, but the machine learning diagnostic system 300 can also sequentially analyze the tile images 402-406 using the LSTM network 610 to extract hierarchical representations at different magnification levels. Therefore, because the deep learning architecture 510 is able to learn at least some of the semantic relationships between images taken at different magnification levels, the machine learning diagnostic system 300 can more accurately classify histopathology images compared to models that do not learn such semantic relationships.
[0079] Figure 7 is an exemplary block diagram showing a convolutional neural network (CNN) 700 that can be used in a deep learning architecture. In deep learning, CNN refers to a class of artificial neural networks that are often used to analyze image data. CNNs, also known as shift-invariant or spatially invariant artificial neural networks, are based on a shared weight architecture of convolutional kernels or filters that slide along input features and provide a translation-equivariant response called a feature map. CNNs exploit hierarchical patterns in the data and assemble patterns of increasing complexity using smaller and simpler patterns imprinted in their filters.
[0080] refer to Figure 7 , CNN 700 is configured to receive input 702, which is a tensor with the following shape: (number of inputs)×(input height)×(input width)×(input channels). Input 702 passes through a convolutional layer 704, which abstracts input 702 into a feature map 706, also known as an activation map, whose shape is: (number of inputs)×(feature map height)×(feature map width)×(feature map channels). CNN 700 also includes multiple pooling layers 708 and 716 and convolutional layers 704 and 712, which are configured to convolve the inputs received at the convolutional layers 704 and 712 and pass the results to the next layer. Each layer 704, 708, 712, 716 and 720 produces a collection of feature maps 706, 710, 714 and 718 of CNN 700, or a final classified output 722.
[0081] Figure 8 It shows that Figure 6800 is an exemplary block diagram of a long short-term memory (LSTM) network 800 used in a deep learning architecture of . LSTM network 800 is an artificial RNN architecture used in the field of deep learning. Unlike a standard feedforward neural network, LSTM network 800 has feedback connections. LSTM network 800 can process not only a single data point (such as an image), but also an entire data sequence. Public LSTM network 800 includes an input gate 802, an output gate 804, and a forget gate 806. LSTM network 800 is configured to remember values over arbitrary time intervals, and the input gate, output gate, and forget gates 802-806 are configured to regulate the flow of information into and out of LSTM network 800.
[0082] Fig. 9 is an exemplary flow chart for classifying histopathology images. Fig. 9 , one or more blocks of method 900 may be, for example, Figure 3 The method 900 starts at block 901. At block 902, the processor is configured to receive a tissue pathology image 902. The image may be obtained from, for example, Figure 1 The image may be received by an imaging device as shown in FIG. 904 or may be received as data via a communication path or retrieved from a memory. At block 904, the processor is configured to crop the histopathology image into tiles of equal size at multiple magnification levels (e.g., FIG. 4 to FIG. 5 ). Figure 6 402-406, which, as noted in the discussion of these figures, may have various magnification levels, such as predefined levels of 10x, 20x, and 40x). For example, the processor may center-crop the histopathology image to generate two or more tiles at different magnification levels. In other embodiments, the processor may apply an attention mechanism to identify a region of interest within the histopathology image and crop the histopathology image centered around the region of interest. Other methods of cropping the histopathology image may also be used, such as random cropping, cropping the upper left corner, or other cropping techniques.
[0083] At block 906, the processor is configured to provide each tile from a different magnification level to a corresponding CNN (e.g., Figure 6 At block 908, the processor is configured to provide the output of each of these CNNs to an LSTM network (e.g., Figure 6 The LSTM network 610 is configured to sequentially learn information representing a hierarchical relationship linking features of the histopathology images at multiple magnification levels using the LSTM network 610. For example, the features of the histopathology images at multiple magnification levels may represent a relationship between tissue morphology and cell structure at different magnification levels.
[0084] At block 910, the processor is configured to provide the processed output of the LSTM network to a function that generates a probability distribution over predicted output classes (e.g., Figure 6 Softmax activation function 618 shown). For example, when the machine learning diagnostic system 300 is configured to diagnose a medical condition such as breast cancer, the Softmax activation function (or other probability distribution generating function) can generate probabilities for each class (e.g., benign, cancer, DCIS, normal, and LCIS) to which a histopathology image may belong. That is, when the deep learning architecture 510 is trained using annotated images in which parts have been labeled as benign, cancer, DCIS, normal, and LCIS by a pathologist, the LSTM network can provide outputs having numerical values corresponding to each of these classes, and the Softmax activation function can convert these numerical values into probabilities for each class.
[0085] At block 912, the processor is configured to generate a tile level classification based on the output of the Softmax activation function (or other probability distribution generating function). For example, the classification may classify the tile images 402-406 as belonging to the class with the highest probability identified by the Softmax activation function. The method 900 ends at block 914.
[0086] in conclusion
[0087] The foregoing description details certain embodiments of the systems, devices, and methods disclosed herein. However, it will be appreciated that, no matter how detailed the foregoing is in the text, the systems, devices, and methods can be practiced in many ways. Also as described above, it should be noted that the use of specific terms in describing certain features or aspects of the present disclosure should not be taken as implying that the terms are redefined herein to include any specific characteristics associated with the terms that are associated with the features or aspects of the present technology.
[0088] Those skilled in the art will appreciate that various modifications and changes may be made without departing from the scope of the described technology. Such modifications and changes are intended to fall within the scope of the embodiments. Those skilled in the art will also appreciate that parts included in one embodiment may be interchangeable with other embodiments; one or more parts from a depicted embodiment may be included with other depicted embodiments in any combination. For example, any of the various components described herein and / or depicted in the figures may be combined, interchanged, or excluded from other embodiments.
[0089] With respect to the use of substantially any plural and / or singular terms herein, those skilled in the art may translate from the plural to the singular and / or from the singular to the plural as applicable to the context and / or application. The various singular / plural permutations may be expressly set forth herein for clarity.
[0090] Directional terms used herein (e.g., top, bottom, side, up, down, inward, outward, etc.) are generally used with reference to the orientation shown in the accompanying drawings and are not intended to be limiting. For example, the top surface described above may refer to the bottom surface or the side surface. Thus, features described on the top surface may be included on the bottom surface, the side surface, or any other surface.
[0091] Those skilled in the art will understand that, in general, the terms used herein are generally intended to be "open" terms (e.g., the term "including" should be interpreted as "including but not limited to", the term "having" should be interpreted as "having at least", the term "comprising" should be interpreted as "including but not limited to", etc.). Those skilled in the art will further understand that if a specific number of an introduced claim statement is intended, such intent will be expressly stated in the claim, and in the absence of such a statement, no such intent exists. For example, as an aid to understanding, the following claims may contain use of the introductory phrases "at least one" and "one or more" to introduce claim statements. However, the use of such phrases should not be interpreted as implying that a claim recitation introduced by the indefinite article "a" or "an" limits any particular claim containing such introduced claim recitation to embodiments containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should typically be interpreted as meaning "at least one" or "one or more"); the same applies to the use of definite articles used to introduce claim recitations. In addition, even if a specific number of introduced claim recitations is explicitly stated, those skilled in the art will recognize that such a statement typically should be interpreted as meaning at least the number of statements (e.g., merely stating "two statements" without other modifiers typically means at least two statements or two or more statements). Those skilled in the art will further understand that almost any transitional word or phrase presenting two or more alternative terms, whether in the specification, claims or drawings, should be understood to contemplate the possibility of including one of these terms, any of these terms, or both of these terms. For example, the phrase "A or B" will be understood to include the possibilities of "A" or "B" or "A and B."
[0092] As used herein, the term "comprising" is synonymous with "including," "containing," or "characterized by," and is inclusive or open-ended and does not exclude additional, unrecited elements or method steps.
[0093] The above description discloses several methods and materials of the present invention. The present invention is susceptible to modifications to methods and materials, as well as changes to manufacturing methods and equipment. Such modifications will become apparent to those skilled in the art by consideration of the present disclosure or practice of the invention disclosed herein. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but encompasses all modifications and alternatives that fall within the true scope and spirit of the present invention as embodied in the appended claims.
Claims
1. An apparatus for detecting a medical condition in a histopathological image, the apparatus comprising: a hardware memory configured to store executable instructions; as well as a hardware processor in communication with the hardware memory, wherein the executable instructions, when executed by the processor, cause the processor to: obtaining a plurality of image tiles at a plurality of magnification levels from the histopathology image, applying a deep learning algorithm to each of said tiles, extracting information representing a hierarchical relationship linking characteristics of the histopathology image existing at one level and another level in the plurality of magnification levels by applying the deep learning algorithm, and The medical condition is identified based on the extracted information representing the hierarchical relationship of characteristics existing in the one level and the other level among the plurality of magnification levels.
2. The apparatus of claim 1 , wherein the executable instructions, when executed by the processor, further cause the processor to: obtaining said histopathological image, and cropping the histopathology image to generate the plurality of tiles, Each of the tiles has a different magnification level than the other tiles.
3. The apparatus of claim 1 or claim 2, wherein the deep learning algorithm comprises a plurality of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, and wherein the executable instructions, when executed by the processor, further cause the processor to: providing each of the tiles to a corresponding one of the CNNs, and The output of each CNN in the CNN is provided to the LSTM network, wherein the LSTM network is configured to extract the information representing the hierarchical relationship linking the characteristics of the histopathology image existing at the one level and the other level among the plurality of magnification levels.
4. The apparatus of claim 3, wherein the executable instructions, when executed by the processor, further cause the processor to: The hierarchical relationship linking the characteristics of the tissue pathology image at the one level and the other level among the multiple magnification levels is sequentially learned by providing the LSTM network with the result of the LSTM network processing the output of the previous CNN each time the LSTM network is processing the output of a CNN other than the initial CNN.
5. The apparatus of claim 4, wherein the initial CNN has the lowest magnification of any CNN from the plurality of CNNs.
6. The apparatus of any one of claims 3 to 5, wherein the deep learning algorithm further comprises a Softmax activation function, and wherein the executable instructions, when executed by the processor, further cause the processor to: The output of the LSTM network is fed to the Softmax activation function, and A tile level classification is generated based on the output of the Softmax activation function.
7. The apparatus of claim 6, wherein the Softmax activation function is configured to generate the output comprising a probability distribution over a set of predicted output classes.
8. An apparatus as claimed in any preceding claim, wherein the hierarchical relationship linking the characteristics of the histopathology images present at the plurality of magnification levels represents a relationship between tissue morphology at a first one of the magnification levels and cellular structure at a second one of the magnification levels.
9. The apparatus of any preceding claim, wherein the executable instructions, when executed by the processor, further cause the processor to: using an attention mechanism to identify regions of interest within the histopathology image, and The histopathology image is cropped based on the region of interest to generate the plurality of tiles.
10. A non-transitory computer readable medium for detecting a medical condition in a histopathology image, the computer readable medium having program instructions for causing a hardware processor to: obtaining a plurality of image tiles at a plurality of magnification levels from the histopathology image; applying a deep learning algorithm to each of the tiles; extracting information representing a hierarchical relationship linking characteristics of the histopathology image existing at one level and another level of the plurality of magnification levels by applying the deep learning algorithm; and The medical condition is identified based on the extracted information representing the hierarchical relationship of characteristics existing in the one level and the other level among the plurality of magnification levels.
11. The non-transitory computer readable medium of claim 10, wherein the instructions are further configured to cause the hardware processor to: obtaining the histopathological image; and cropping the histopathology image to generate the plurality of image tiles; Each of the tiles has a different magnification level than the other tiles.
12. The non-transitory computer-readable medium of claim 10 or claim 11, wherein the deep learning algorithm comprises a plurality of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, and wherein the instructions are further configured to cause the hardware processor to: providing each of the tiles to a corresponding one of the CNNs; and The output of each CNN in the CNN is provided to the LSTM network, wherein the LSTM network is configured to extract the information representing the hierarchical relationship linking the characteristics of the histopathology image existing at the one level and the other level among the plurality of magnification levels.
13. The non-transitory computer readable medium of claim 12, wherein the instructions are further configured to cause the hardware processor to: Providing the output of each of the CNNs to a corresponding one of the LSTM units; and The hierarchical relationship linking the characteristics of the tissue pathology image at the one level and the other level among the multiple magnification levels is sequentially learned by providing the LSTM network with the result of the LSTM network processing the output of the previous CNN each time the LSTM network is processing the output of a CNN other than the initial CNN.
14. The non-transitory computer-readable medium of claim 13, wherein the initial CNN has a lowest magnification of any CNN from the plurality of CNNs.
15. The non-transitory computer-readable medium of any one of claims 12 to 14, wherein the deep learning algorithm further comprises a Softmax activation function, and wherein the instructions are further configured to cause the hardware processor to: Providing the output of the LSTM network to the Softmax activation function; and A tile level classification is generated based on the output of the Softmax activation function.
16. The non-transitory computer-readable medium of claim 15, wherein the Softmax activation function is configured to generate the output comprising a probability distribution over a set of predicted output classes.
17. The non-transitory computer-readable medium of any one of claims 10 to 16, wherein the hierarchical relationship linking the characteristics of the tissue pathology images present at the multiple magnification levels represents a relationship between a tissue morphology at a first one of the magnification levels and a cellular structure at a second one of the magnification levels.
18. The non-transitory computer-readable medium of any one of claims 10 to 17, wherein the instructions are further configured to cause the hardware processor to: identifying a region of interest within the histopathology image using an attention mechanism; and The histopathology image is cropped based on the region of interest to generate the plurality of tiles.
19. A method comprising: obtaining a plurality of tiles at a plurality of magnification levels from a histopathology image; applying a deep learning algorithm to each of the tiles; extracting, by applying the deep learning algorithm, information representing a hierarchical relationship linking characteristics of the histopathology image present at one level and another level of the plurality of magnification levels; as well as The medical condition is identified based on the extracted information representing the hierarchical relationship of characteristics existing in the one level and the other level among the plurality of magnification levels.
20. The method of claim 19, further comprising: obtaining the pathological image; as well as cropping the histopathology image to generate the plurality of tiles, Each of the tiles has a different magnification level than the other tiles.
21. The method of claim 19 or claim 20, wherein the deep learning algorithm comprises a plurality of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, and wherein the method further comprises: Providing each of the tiles to a corresponding one of the CNNs; as well as The output of each CNN in the CNN is provided to the LSTM network, wherein the LSTM network is configured to extract the information representing the hierarchical relationship linking the characteristics of the histopathology image existing at the one level and the other level among the plurality of magnification levels.
22. The method of claim 21, wherein the method further comprises: The hierarchical relationship linking the characteristics of the tissue pathology image at the one level and the other level among the multiple magnification levels is sequentially learned by providing the LSTM network with the result of the LSTM network processing the output of the previous CNN each time the LSTM network is processing the output of a CNN other than the initial CNN.
23. The method of claim 22, wherein the initial CNN has the lowest magnification of any CNN from the plurality of CNNs.
