Method and system for evaluating fibrosis in tissue samples

Abstract

The present invention provides a method for evaluating fibrosis of a tissue sample. The method comprises: obtaining a staining image of a stained tissue sample, wherein collagen in the stained tissue sample is stained; generating a plurality of different types of collagen images based on the staining image, wherein the plurality of different types of collagen images correspond to a plurality of different types of collagen respectively; and determining a plurality of different types of collagen parameters based on the plurality of different types of collagen images using a deep neural network. The plurality of different types of collagen include portal vein collagen, fibrillar collagen, and diaphragm collagen. The plurality of different types of collagen images are generated by determining whether the collagen meets a preset size condition or a preset distance condition relative to the portal duct and the central vein boundary.

Description

[0001] Cross-references

[0002] This application claims priority to Singapore Patent Application No. 10201806494Y, filed on July 30, 2018, which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present invention generally relates to methods and systems for evaluating fibrosis in tissue samples. Background Art

[0004] Fibrosis is the formation of excess fibrous connective tissue during the repair or reaction process of an organ or tissue. Since fibrosis is defined by the pathological accumulation of extracellular matrix (ECM) proteins, it leads to scarring and thickening of the affected tissue. In essence, this is an excessive wound healing response that interferes with normal organ function. If the tissue damage is severe or repeated, or the wound healing response itself is dysregulated, the initial process of tissue repair may lead to a progressive and irreversible fibrotic response. Therefore, it is necessary to evaluate the progression or regression of the degree of fibrosis in the affected tissue. For example, in the diseased liver, histopathological examination of tissue samples is the first choice for staging liver fibrosis and is still the "gold standard" for evaluating liver fibrosis. Histopathological features such as excessive accumulation and / or substantial disappearance of ECM (especially collagen) are incorporated into descriptive or semi-quantitative scoring systems for disease staging. Examples of scoring systems include Knodell histological activity index (HAI), Scheuer scoring system, Ishak scoring system, Metavir scoring system, and Ishak modified scoring system. These scores can be used to monitor progression or regression of diseased liver. However, these scoring systems are semiquantitative in nature and are subject to intra- and inter-observer variability. For example, the Metavir scoring system, which is commonly implemented in the clinical setting, is a classification task based primarily on observed architectural changes and fibrotic changes rather than a quantitative measure of fibrotic content. In addition, inter- and intra-observer variability as well as staining may introduce errors into disease staging.

[0005] With the development of mode-locked lasers and highly sensitive optical sensors, nonlinear optical microscopy (e.g., optical microscopy based on multiphoton excitation fluorescence and multiharmonic generation) has made dye-free imaging-based diagnosis a feasible approach. In addition, with the large amount of data generated by the use of these imaging modalities, several groups have explored the use of machine learning-based algorithms to score the stage of liver fibrosis. For example, systems using machine learning-based algorithms have been previously developed to build diagnostic tools to improve the quantification and scoring of liver fibrosis. These algorithms include artificial neural networks (ANNs), multinomial logistic regression (MLR), support vector machines (SVMs), and random forests (RFs). For example, machine learning can be used to predict, classify, and assess liver fibrosis / cirrhosis and treatment response in patients with hepatitis B and C. However, most previous machine learning-based methods have relied on multiphoton microscopy techniques to evaluate liver fibrosis / cirrhosis, such as second-harmonic generation (SHG) and two-photon-excited fluorescence (TPEF), which are not commonly used in clinical implementation. Furthermore, in addition to having a limited number of extracted features, the algorithms also require the use of domain knowledge of the images from a histopathological perspective for feature extraction prior to modeling and classification. In addition, the utility of various algorithms remains unclear as they are often case-dependent; and none of the existing algorithms are fully automated as they require the use of computer algorithms for image segmentation and feature extraction under the guidance of pathologists at some stage.

[0006] Therefore, it is necessary to provide a method / system for evaluating tissue fibrosis, which attempts to overcome or at least improve one or more deficiencies in conventional methods / systems for evaluating tissue fibrosis. The present invention was developed under this background. SUMMARY OF THE INVENTION

[0008] According to a first aspect of the present invention, there is provided a method for evaluating fibrosis of a tissue sample using at least one processor, the method comprising:

[0009] obtaining a staining image of a stained tissue sample, wherein collagen in the stained tissue sample is stained;

[0010] generating a plurality of different types of collagen images based on the stained image, wherein the plurality of different types of collagen images correspond to a plurality of different types of collagen respectively;

[0011] determining a plurality of different types of collagen parameters based on the plurality of different types of collagen images using a deep neural network; and

[0012] The degree of fibrosis of the stained tissue sample is determined based on the parameters of the plurality of different types of collagen.

[0013] According to a second aspect of the present invention, there is provided a system for evaluating fibrosis of a tissue sample, the system comprising:

[0014] at least one processor communicatively coupled to the memory and configured to:

[0015] obtaining a staining image of a stained tissue sample, wherein collagen in the stained tissue sample is stained;

[0016] generating a plurality of different types of collagen images based on the stained image, wherein the plurality of different types of collagen images correspond to a plurality of different types of collagen respectively;

[0017] determining a plurality of different types of collagen parameters based on the plurality of different types of collagen images using a deep neural network; and

[0018] The degree of fibrosis of the stained tissue sample is determined based on the parameters of the plurality of different types of collagen.

[0019] According to a third aspect of the present invention, there is provided a computer program product, contained in one or more non-transitory computer-readable storage media, the computer program product comprising instructions executed by at least one processor to implement a method for evaluating fibrosis of a tissue sample, the method comprising:

[0020] obtaining a staining image of a stained tissue sample, wherein collagen in the stained tissue sample is stained;

[0021] generating a plurality of different types of collagen images based on the stained image, wherein the plurality of different types of collagen images correspond to a plurality of different types of collagen respectively;

[0022] determining a plurality of different types of collagen parameters based on the plurality of different types of collagen images using a deep neural network; and

[0023] The degree of fibrosis of the stained tissue sample is determined based on the parameters of the plurality of different types of collagen. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Embodiments of the present invention will be better understood and will become apparent to those skilled in the art from the following written description, which is by way of example only, taken in conjunction with the accompanying drawings.

[0025] Figure 1depicts a schematic flow chart of a method for evaluating fibrosis in a tissue sample using at least one processor according to various embodiments of the present invention;

[0026] Figure 2 A schematic framework diagram of a system for evaluating tissue sample fibrosis in various embodiments of the present invention is depicted, for example corresponding to Figure 1 The method shown in ;

[0027] Figure 3 Depicting an exemplary computer system that may include the described system of various embodiments of the present invention;

[0028] Figure 4 An exemplary schematic diagram showing processing (or pre-processing) a stained image of a stained tissue sample to generate multiple different types of collagen images;

[0029] Figure 5 A diagram showing a framework for evaluating fibrosis in a tissue sample according to various exemplary embodiments of the present invention;

[0030] Figure 6 A diagram showing an exemplary architecture of a deep neural network for assessing fibrosis in a tissue sample according to various exemplary embodiments of the present invention;

[0031] Figure 7 Flowcharts showing various conventional techniques, and techniques or frameworks for evaluating tissue sample fibrosis in various exemplary embodiments of the present invention;

[0032] Figures 8A-8C An exemplary comparison of changes in collagen patterns in stained tissue sample images using sqFibrosis and collagen proportional area (CPA) to evaluate various stages of liver fibrosis is shown;

[0033] Fig. 9 A graph showing the performance evaluation of the deep learning model using unprocessed stained images, processed collagen images that were not classified into different types, and processed collagen images that were classified into different types and the sqFibrosis index of CPA.

[0034] Figures 10A-10B The stained images of stained tissue samples of various sample sizes from animal studies and the AUROC values ​​of sqFibrosis and CPA of various sample sizes evaluated based on different stages of fibrosis in tissue samples are shown respectively;

[0035] Figures 11A-11Bstained images of stained tissue samples at various image resolutions from animal studies, and AUROC values ​​of sqFibrosis and CPA at various image resolutions evaluated based on different stages of fibrosis in tissue samples; and

[0036] Figures 12A-12B Shown are stained images of stained tissue samples at various image intensities from animal studies, and AUROC values ​​of sqFibrosis and CPA at various image intensities evaluated based on different stages of fibrosis in the tissue samples, respectively. DETAILED DESCRIPTION OF THE INVENTION

[0038] Various embodiments of the present invention provide a method (computer-implemented method) and system (including a memory and at least one processor coupled to the memory) for evaluating fibrosis of a tissue sample (e.g., determining the extent or level of fibrosis). In various embodiments, a fully automatic technique for evaluating (e.g., staging) fibrosis based on an image of a stained tissue sample (which may be referred to herein as a "stained image") is provided. For example, tissue samples can be extracted from various anatomical locations of an animal or human body by biopsy. For example, tissue samples can be extracted from an anatomical structure (e.g., liver) in which fibrosis is to be evaluated or determined. Those skilled in the art will appreciate that tissue samples extracted from other anatomical structures or locations of an animal or human body by biopsy are within the scope of the present invention, such as, but not limited to, lungs, skin, stomach, small intestine, and large intestine. Therefore, fibrosis at an anatomical location can be evaluated, such as to determine the extent (e.g., extent, level, or stage) of fibrosis in an anatomical structure. For example, the determined extent of fibrosis can be used to predict, classify, diagnose, and evaluate the progression or regression of a diseased anatomical structure in response to treatment of fibrosis in the tissue. In various embodiments, fully automated techniques for staging fibrosis in an anatomical structure or location may use images of existing stained tissue samples (eg, tissue samples that have been stained with a dye to provide detail of tissue structure / morphology).

[0039] Figure 1A schematic flow chart of a method 100 (computer-implemented method) for evaluating fibrosis of a tissue sample using at least one processor in various embodiments of the present invention is depicted. The method 100 includes obtaining (at 102) a stained image of a stained tissue sample, wherein collagen in the stained tissue sample is stained; generating (at 104) a plurality of different types of collagen images based on the stained image, wherein the plurality of different types of collagen images correspond to a plurality of different types of collagen, respectively; determining (at 106) a plurality of different types of collagen parameters based on the plurality of different types of collagen images using a deep neural network; and determining (at 108) a degree of fibrosis of the stained tissue sample based on the plurality of different types of collagen parameters.

[0040] For example, with respect to 102, the tissue sample can be extracted from the anatomical position of the animal or human body to be evaluated. In various embodiments, the collagen in the tissue sample can be stained. For example, the tissue sample can be stained using various existing staining techniques to highlight the relevant tissue morphology or features including collagen in the tissue to evaluate the fibrosis in the tissue. An image (stained image) of the stained tissue sample can be obtained and processed to determine the degree of fibrosis of the stained tissue sample. The stained image as described herein refers to an image of a stained tissue sample that can be obtained from an image capture device. In various embodiments, the image capture device can generate a stained image of the stained tissue sample, and then at least one processor can acquire (or receive) the stained image for processing to evaluate the fibrosis of the stained tissue sample. As a non-limiting example, the image capture device can be, or, include: a microscope, a tissue scanner, or a combination thereof.

[0041] For example, with respect to 104, the stained image may be processed (or preprocessed) to identify and distinguish (or separate or segment) a plurality of different types of collagen captured in the stained image. For example, the stained image may be processed and the collagen in the stained image may be classified into different types (e.g., different categories or classifications) according to histopathological features. For example, three main different types of collagen may be identified to evaluate fibrosis in an anatomical structure or location, such as evaluating liver fibrosis. In various embodiments, as will be described later in various exemplary embodiments of the present invention, the plurality of different types of collagen may include portal collagen, fibrillar collagen, and septal collagen. For example, liver fibrosis may be evaluated by identifying or utilizing the portal collagen, fibrillar collagen, and septal collagen. It should be understood that the present invention is not limited to the above three different types of collagen and the evaluation of liver fibrosis. For example, other types of collagen and / or other quantities (e.g., more than three) of collagen may be obtained or identified as needed or appropriate. For example, other types and quantities of pathology-related features, such as other collagen regions in the tissue based on morphological or structural and / or spatially related features, can also be used to classify collagen types to evaluate the fibrosis of the anatomical structure. In another exemplary embodiment, different types of collagen content can be identified from the stained image in the detection of cirrhosis, for example, to distinguish the progressive / regressive distribution pattern of collagen fibers to further define the subclassification of cirrhosis.

[0042] In various embodiments, the generation of multiple different types of collagen images includes generating a first image corresponding to (e.g., relative to or representative of) the tissue in the stained image, and generating a second image corresponding to (e.g., relative to or representative of) the collagen in the stained image. In various embodiments, the first image and the second image can be generated based on a clustering technique such as K-means clustering. For example, the stained image can be processed based on the pixel distribution of pixels in Lab / RGB / HSV space according to K-means clustering to separate collagen from tissue (and / or background).

[0043] In various embodiments, the above-mentioned generation of multiple different types of collagen images further includes: generating a third image corresponding to (e.g., relative to or representing) the portal duct and central vein in the first image. For example, when the first image records the signal of a tissue (e.g., a hepatocyte), the portal duct and central vein can be represented by a blank space. In various embodiments, the pixels corresponding to the portal duct and central vein can be separated (or identified) based on the intensity value of the pixel. The third image can be generated based on the identified pixels corresponding to the portal duct and central vein.

[0044] In various embodiments, the generation of multiple different types of collagen images further includes: generating a first binary image based on the second image; generating a second binary image based on the third image; and generating multiple different types of collagen images based on the first and second binary images. For generating the first binary image based on the second image, the second image can be processed using enhanced contrast to increase the collagen intensity, and can be converted to the first binary image using an adaptive threshold. In addition, in some cases, morphological closure can be used to connect broken collagen fibers and / or a despeckle filter can be used to remove noise. With respect to generating the second binary image based on the third image, various operations can be performed, including, for example, but not limited to, Otsu thresholding, morphological opening and closing, and hole-filing operations to smooth the space. In addition, spaces smaller than a preset size (e.g., the size of a hepatocyte) and spaces greater than the length of the major axis to the length of the minor axis (e.g., assumed to be sinusoidal spaces) can be removed.

[0045] In various embodiments, the plurality of different types of collagen images include a first type of collagen image, a second type of collagen image, and a third type of collagen image, and the generation of the plurality of different types of collagen images based on the first binarized image and the second binarized image includes: generating the first type of collagen image based on collagen that satisfies a preset size condition in the first binarized image; generating the second type of collagen image based on collagen that satisfies a preset distance condition relative to the boundary of the portal duct and the central vein in the second binarized image in the first binarized image; and generating the third type of collagen image based on collagen that does not satisfy the preset size condition and the preset distance condition in the first binarized image. In a non-limiting example, the preset size condition may be a collagen image having a size smaller than a preset value (e.g., 10 um) in the first binarized image. 2) area of ​​collagen fibers. For example, collagen that meets this preset size condition can represent broken collagen distributed in the tissue. In a non-limiting example, the distance condition preset relative to the boundary of the portal duct and the central vein in the second binarized image can be a preset distance threshold, for example, a cutoff distance determined relative to the portal duct and the central vein. For example, pixels from the boundary of the portal duct and the central vein can be first selected, and a cutoff distance can be determined (or selected) to expand the selected pixels. The second type of collagen image can be generated based on the collagen located within the determined cutoff distance in the first binarized image. The first type of collagen image can correspond to a first type of collagen, the second type of collagen image can correspond to a second type of collagen, and the third type of collagen image can correspond to a third type of collagen. In various embodiments, the first type of collagen can be fibrillar collagen, the second type of collagen can be portal vein collagen, and the third type of collagen can be diaphragm collagen.

[0046] Regarding 106, in various embodiments, the plurality of different types of collagen parameters each include a quantitative fibrosis value associated with a corresponding type of collagen in the plurality of different types of collagen in the stained tissue sample. In various embodiments, the quantitative fibrosis value of the corresponding type of collagen can be a fibrosis index of the type of collagen in the stained tissue sample. For example, for each type of collagen image in the plurality of different types of collagen images, the deep neural network can be used to extract collagen features of the corresponding type of collagen in the collagen image to determine the respective collagen parameters. The respective collagen parameters can be determined based on the type of collagen.

[0047] In various embodiments, the deep neural network may include multiple deep neural sub-networks, such as multiple deep convolutional neural networks (DCNNs), to determine the multiple different types of collagen parameters based on multiple different types of collagen images. In various embodiments, the deep neural network includes multiple deep neural sub-networks corresponding to multiple different types of collagen, respectively. Each deep neural sub-network used may be a pre-trained DCNN. Multiple deep neural sub-networks may be trained for the corresponding types of collagen, respectively, to generate or determine the corresponding collagen parameters of the corresponding types of collagen. In other words, each deep neural sub-network in multiple deep neural sub-networks may be trained for a specific type of collagen, and based on the collagen image corresponding to the type of collagen, to determine the collagen parameters of the type of collagen. In various embodiments, each deep neural sub-network may be pre-trained based on a learning transfer technique. In various embodiments, each deep neural sub-network may be a modified AlexNet-CNN. In various embodiments, each deep neural sub-network in the multiple deep neural sub-networks may be configured to output a quantitative fibrosis value related to the corresponding collagen type based on the collagen image corresponding to the type of collagen. For example, three CNNs may be trained to generate a first quantitative fibrosis value corresponding to the first type of collagen, a second quantitative fibrosis value corresponding to the second type of collagen, and a third quantitative fibrosis value corresponding to the third type of collagen, respectively.

[0048] About 108, in each embodiment, can be based on different types of collagen parameters, use polynomial logistic regression to determine the degree of fibrosis of the tissue sample.For example, can be based on the corresponding portal collagen, fibrillar collagen and diaphragm collagen determined collagen parameters to determine the degree of fibrosis of the tissue sample.In each embodiment, the degree of fibrosis determined in the stained tissue sample can be a final fibrosis index (which is also referred to as sqFibrosis index in this article).For example, can be based on a variety of different types of collagen parameters to determine a single evaluation index to evaluate the fibrosis of the tissue sample.For example, can be based on the first quantitative fibrosis value, the second quantitative fibrosis value and the third quantitative fibrosis value, use logistic regression to determine a single evaluation metric.The final fibrosis index combines the prediction results (a variety of different types of collagen parameters) from a variety of different types of collagen.For example, can be performed corresponding types of collagen features of statistical analysis to determine the final fibrosis index.The final fibrosis index can be set according to the Metavir scoring system F0-F4, and can be used as a final stage indicator of fibrosis progression. In other words, a single index (sqFibrosis) can be generated at the tissue level for evaluating fibrosis at F0-F4. The final fibrosis index can be generated from a set of predefined final fibrosis indices according to the Metavir score, which can indicate the stage of fibrosis in the tissue (tissue sample).

[0049] Thus, embodiments of the present invention advantageously provide a fully automated technique for staging or scoring tissue sample fibrosis. In various exemplary embodiments, existing stained tissue samples can be used to assess their fibrosis. Furthermore, embodiments of the present invention enable disease diagnosis even with limited amounts of biopsy samples or clinical information.

[0050] Figure 2 A schematic block diagram of a system 200 for evaluating fibrosis in a tissue sample according to various embodiments of the present invention is depicted, such as a method 100 for evaluating fibrosis in the tissue sample according to various embodiments of the present invention described herein.

[0051] The system 200 includes a memory 204, at least one processor 206 communicatively coupled to the memory 204, and is configured to: obtain a staining image of a stained tissue sample, wherein collagen in the stained tissue sample is stained; generate a plurality of different types of collagen images based on the staining image, wherein the plurality of different types of collagen images correspond to a plurality of different types of collagen, respectively; determine a plurality of different types of collagen parameters based on the plurality of different types of collagen images using a deep neural network; and determine the degree of fibrosis of the stained tissue sample based on the plurality of different types of collagen parameters.

[0052] Those skilled in the art will appreciate that at least one processor 206 may be configured to perform the desired functions or operations through a set (or sets) of instructions (eg, software modules) executable by at least one processor 206. Figure 2 As shown, the system 200 may further include an image acquisition module (or circuit) 208, which is configured to acquire a stained image of a stained tissue sample; an image processing module (or circuit) 210, which is configured to generate a plurality of different types of collagen images based on the stained image; a collagen parameter determination module (or circuit) 212, which is configured to use a deep neural network to determine a plurality of different types of collagen parameters based on the plurality of different types of collagen images; and a fibrosis level determination module (or circuit) 214, which is configured to determine the degree of fibrosis of the stained tissue sample based on the plurality of different types of collagen parameters.

[0053] Those skilled in the art will appreciate that the above modules (or circuits) are not necessarily separate modules, and two or more modules may be implemented or implemented as needed or appropriate by one functional module (e.g., circuit or software program) without departing from the scope of the present invention. For example, the image acquisition module 208, the image processing module 210, the collagen parameter determination module 212, and / or the fibrosis level determination module 214 may be implemented (e.g., compiled together) as an executable software program (e.g., software application, or, for example, simply referred to as "app"), which may be stored in the memory 204 and may be executed by at least one processor 206 to perform the functions / operations as described herein according to various embodiments.

[0054] In various embodiments, Figure 1As shown, the system 200 corresponds to the method 100 described above. Therefore, according to various embodiments, various functions / operations configured to be executed by at least one processor 206 may correspond to various steps or operations in the method 100 described above, and for the sake of clarity and conciseness, there is no need to repeat them in the system 200. In other words, the various embodiments described in the method are similarly effective for the corresponding system (for example, it can also be embodied as a device).

[0055] For example, in various embodiments, the memory 204 may store therein the image acquisition module 208, the image processing module 210, the collagen parameter determination module 212 and / or the fibrosis level determination module 214, respectively corresponding to the various steps or operations of the method 100 described above, which may be executed by at least one processor 206 to perform the corresponding functions / operations as described herein.

[0056] According to the various embodiments in the present disclosure, a computing system, a controller, a microcontroller or any other system providing processing power can be provided. Such a system can be considered to include one or more processors and one or more computer-readable storage media. For example, the aforementioned system 200 may include a processor (or controller) 206 and a computer-readable storage medium (or memory) 204, which is used, for example, in the various processes performed as described herein. The memory or computer-readable storage medium used in each embodiment can be a volatile memory, such as a DRAM (Dynamic Random Access Memory) or a non-volatile memory, such as a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM, an erasable PROM), an EEPROM (Electrically Erasable PROM) or a flash memory, such as a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory, a magnetoresistive random access memory) or a PCRAM (Phase Change Random Access Memory, a phase change random access memory).

[0057] In various embodiments, a "circuit" may be understood as any kind of logic implementation entity, which may be a dedicated circuit or a processor that executes software stored in a memory, firmware, or any combination thereof. Therefore, in one embodiment, a "circuit" may be a hard-wired logic circuit or a programmable logic circuit, such as a programmable processor, such as a microprocessor (e.g., a complex instruction set computer (CISC) processor or a reduced instruction set (RISC) processor). A "circuit" may also be a processor that executes software, such as any kind of computer program, for example, a computer program using a virtual machine code (e.g., Java). According to various optional embodiments, any other kind of implementation of each function described in more detail below may also be understood as a "circuit". Similarly, a "module" may be part of a system of various embodiments of the present invention, and may include a "circuit" as described above, or may be understood as any kind of logic implementation entity therein.

[0058] Certain portions of the present disclosure are presented explicitly or implicitly through algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means by which those skilled in the art of data processing most effectively convey the substance of their work to others skilled in the art. An algorithm is generally considered herein to be a self-consistent sequence of steps leading to a desired result. These steps are steps requiring physical manipulation of physical quantities, such as electrical, magnetic or optical signals capable of being stored, transmitted, combined, compared and otherwise manipulated.

[0059] Unless otherwise specifically noted, and as will become apparent from the following, it should be understood that throughout this specification, discussions utilizing terms such as "determine," "obtain," "generate," and the like refer to the actions and processes of a computer system or similar electronic device that convert data represented by physical quantities within the computer system into other data represented by similar physical quantities within a computer system or other information storage, transmission, or display device.

[0060] This specification also discloses a system (also embodied as a device or apparatus) for performing the operations / functions of the methods described herein. Such systems may be specially constructed for the desired purpose, or may include a general-purpose computer or other devices selectively activated or reconfigured by a computer program stored in the computer. The algorithm proposed herein is not inherently related to any particular computer or other device. According to the teachings of this article, various general-purpose machines may be used together with computer programs. Optionally, it may be appropriate to construct a more specialized device to perform the desired method steps.

[0061] In addition, this specification also discloses at least a computer program or software / functional module implicitly, because it is obvious to those skilled in the art that the various steps or operations of the method described herein can be performed by computer code. The computer program is not limited to any specific programming language and its implementation. It will be understood that various programming languages ​​and their encodings can be used to implement the teachings of the present disclosure contained herein. In addition, the computer program is not limited to any specific control flow. There are many other variants of the computer program, and they can use different control flows without departing from the scope of the present invention. Those skilled in the art will recognize that the various modules described herein (such as image acquisition module 208, image processing module 210, collagen parameter determination module 212 and / or fibrosis level determination module 214) can be software modules implemented by computer programs or instruction sets that can be executed by computer processors to perform desired functions, or can be hardware modules designed to be functional hardware units that perform desired functions. It should also be understood that a combination of hardware and software modules can be implemented.

[0062] In addition, one or more steps or operations of the computer program / module or method described herein can be performed in parallel rather than sequentially. Such a computer program can be stored on any computer-readable medium. The computer-readable medium may include other storage devices such as a disk or optical disk, a memory chip, or that are suitable for interfacing with a general-purpose computer. When the computer program is loaded and executed on such a general-purpose computer, the step or operation of the method described herein is finally produced to effectively realize.

[0063] In various embodiments, a computer program product is provided, which is contained in one or more computer-readable storage media (non-transitory computer-readable storage media), and includes instructions executable by one or more computer processors (e.g., image acquisition module 208, image processing module 210, collagen parameter determination module 212 and / or fibrosis level determination module 214) to perform the above-mentioned Figure 1 The method 100 for evaluating fibrosis in a tissue sample is shown. Therefore, various computer programs or modules described herein may be stored in a computer program product that can be received by a system (eg, a computer system or an electronic device), such as Figure 2 The system 200 is shown to be operable to be executed by at least one processor 206 of the system 200 to perform any needed or desired functions.

[0064] The software or functional modules described herein may also be implemented as hardware modules. More specifically, in a hardware sense, a module is a functional hardware unit designed to be used with other components or modules. For example, a module may be implemented using separate electronic components, or a module may form part of an entire electronic circuit (e.g., an application specific integrated circuit (ASIC)). There are many other possibilities. Those skilled in the art will appreciate that the software or functional modules described herein may also be implemented in a combination of hardware and software modules.

[0065] In various embodiments, the above-described computer system can be implemented by any computer system (e.g., a portable or desktop computer system), such as Figure 3 The computer system 300 schematically shown in the figure is only used as an example and is not limited. Various methods / operations or functional modules (such as image acquisition module 208, image processing module 210, collagen parameter determination module 212 and / or fibrosis level determination module 214) can be implemented by software, such as a computer program executed in the computer system 300, and instruct the computer system 300 (specifically one or more processors) to perform the methods / functions of each embodiment described herein. The computer system 300 may include a computer module 302, an input module (such as a keyboard 304 and a mouse 306), and multiple output devices (such as a display 308) and a printer 310. The computer module 302 can be connected to a computer network 312 by a suitable transceiver device 314 to enable access to, for example, the Internet or other network systems, such as a local area network (LAN) or a wide area network (WAN). The computer module 302 in this example may include a processor 318 for executing various instructions, a random access memory (RAM) 320, and a read only memory (ROM) 322. The computer module 302 may also include a plurality of input / output (I / O) interfaces, such as an I / O interface 324 for the display 308, and an I / O interface 326 for the keyboard 304. The components of the computer module 302 typically communicate via an interconnected bus 328 and in a manner known to those skilled in the relevant art.

[0066] Those skilled in the art will appreciate that the terms used herein are only for the purpose of describing each embodiment and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when used in this specification, terms such as "include" and / or "comprise" specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0067] In order to make the present invention easy to understand and put into practice, various exemplary embodiments of the present invention will be described below only by way of example and not limitation. However, it will be appreciated by those skilled in the art that the present invention can be implemented in various forms or configurations and should not be construed as being limited to the exemplary embodiments set forth below. Rather, these exemplary embodiments are provided to make the present disclosure thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0068] Specifically, in order to better understand the present invention and without limitation or loss of generality, various exemplary embodiments of the present invention will now be described with respect to the evaluation of fibrosis of liver tissue extracted from rats. For example, a stained image of a stained tissue sample can be obtained from a collagen-stained liver tissue sample extracted from a rat fibrosis-induced liver. The stained image of the stained biopsy tissue sample can be evaluated to determine the degree of fibrosis in a rat fibrosis-induced liver. However, those skilled in the art will appreciate that the present invention is not limited to evaluating the fibrosis of a tissue sample extracted from the liver, and the method for evaluating the fibrosis of a tissue sample disclosed herein according to each embodiment can be used to evaluate the fibrosis of tissue samples from various other types of anatomical locations of an animal or human body by pathological staining, such as, but not limited to, the lungs, stomach, small intestine, and large intestine, etc.

[0069] In various exemplary embodiments, the stained images of the stained tissue samples can be processed to identify and classify multiple different types of collagen based on the histopathological characteristics in the tissue samples. In each exemplary embodiment, the multiple different types of collagen can include portal collagen, fibrillar collagen, and diaphragm collagen. For example, the stained images can be processed to classify collagen fibers into three different types. Other types of collagen and / or other types of collagen can be classified according to the type of tissue fibrosis to be evaluated.

[0070] Figure 4An exemplary schematic diagram of processing (or pre-processing) a stained image 400 of a stained tissue sample to generate multiple different types of collagen images is shown. In various exemplary embodiments, collagen in the stained tissue sample may be stained.

[0071] In various exemplary embodiments, a stained image of the stained tissue sample 410 can be obtained from an image capture device. In various exemplary embodiments, the image capture device can be or include a bright field microscope for capturing images of stained tissue samples on a slide. Other image capture devices, such as a tissue scanner, can be used to obtain stained images.

[0072] In each exemplary embodiment, the stained image can be divided into different categories of stained images including background, tissue and collagen fibers to generate a first image 420 corresponding to the tissue in the stained image and a second image 430 corresponding to the collagen in the stained image. For example, in a non-limiting example, collagen and tissue in the stained image can be identified and extracted based on clustering techniques (e.g., K-means clustering). For example, Chen, CW, Luo, J. & Parker, KJ Image segmentation via adaptive K-mean clustering and knowledge-based morphological operations with biomedical applications. IEEE transactions on image processing: a publication of the IEEE Signal Processing Soceity 7, 1673-1683, doi: 10.1109 / 83.730379 (1998) discloses an image segmentation technique based on K-means clustering, which can be used to identify and extract collagen and tissue in stained images, and its contents are incorporated herein by reference in their entirety for all purposes. For example, collagen and tissue in the stained image may be identified and extracted by K-means clustering based on the distribution of pixels representing tissue and collagen fibers in Lab / RGB / HSV space.

[0073] For example, clustering techniques such as K-means clustering can be used to determine the natural spectral groupings present in the data set. In such a process, a predefined number of clusters can be set (for example, a desired number of clusters located in the data acceptable to the analyst). The technique can be started with arbitrarily positioned seeds corresponding to multiple cluster averages of the predefined number of clusters, respectively. Subsequently, in a first operation, each pixel in the stained image can be assigned to the closest cluster average. In a second operation, a revised mean vector can be calculated for each cluster. The first and second operations can be iterated. Iterations in the first and second operations can be continued until a predefined criterion is met. In a non-limiting example, the predefined criterion can be where multiple cluster averages do not change in the iteration. The efficiency of clustering can be determined, for example, by a clustering algorithm for unsupervised classification of remote sensing data.

[0074] In each exemplary embodiment, the K-means clustering technique based on color can be performed as follows. First, the color band in the stained image can be extracted as a separate two-dimensional (2D) array, and each color component (e.g., red, blue and green channels) uses a 2D array (or color channel). Next, a histogram can be calculated for each color channel (e.g., red, green and blue channels). Through the realization of K-means clustering, the pixels in the stained image can be automatically clustered under the situation of a given predefined number of clusters. For example, the clustering of a predefined number can correspond to cytoplasm, nucleus, collagen fiber and background noise. This process can be repeated repeatedly until the separation result is optimized with the minimum distance in each settlement.

[0075] In various exemplary embodiments, a third image corresponding to the portal duct and central vein in the first image 420 can be generated, and a second binary image (or a processed image corresponding to tissue with identified portal duct and central vein) 422 can be generated based on the third image. Since the portal duct and central vein are just blood vessels (e.g., blood flow components), the space inside the portal duct and central vein will not be stained and can be displayed as a white area using a clustering technique (K-means separation). For example, since the first image 420 records the signal of liver tissue 421 (e.g., the signal of liver cells), the portal duct 424 and the central vein 425 will appear as blank areas, so the segmentation process can select and separate those pixels that represent or belong to the portal duct and central vein based on their intensity. In various exemplary embodiments, image segmentation based on thresholds (e.g., Otsu thresholds) and image preprocessing based on morphological operations (e.g., morphological opening and closing) can be applied to the stained image 410 and / or the first image 420 to identify blank areas representing the portal duct and central vein, thereby generating a second binary image 422. In addition, a punching operation can be performed to smooth the space. The portal duct and central vein can be further distinguished by their different sizes and shapes (e.g., the portal duct has larger irregular holes, while the central vein has smaller round holes). In addition, before further processing, the empty space, which is considered to be a sinusoidal space with a relatively small area and an elliptical shape, can be removed. For example, spaces smaller than a preset size (e.g., 100 μm) can be removed. 2 area) (e.g., the size of a liver cell) and the space from the major axis to the minor axis (assumed to be a sinusoidal space).

[0076] In various exemplary embodiments, the second image 430 corresponding to collagen can be processed using contrast enhancement to enhance the signal, and further converted into a first binary image 432 by, for example, an adaptive threshold. A first binarized image 432 corresponding to collagen is generated. In order to generate the first binarized image 432 from the second image 430 as a color image corresponding to collagen, the second image 430 (e.g., an RGB image) can be converted into a grayscale image using a weighted averaging technique, and the grayscale image can be converted into the first binarized image 432 using adaptive thresholding. In some cases, morphological operations such as morphological closing can be used to connect broken collagen fibers and / or filters such as despeckle filters can be used to remove noise.

[0077] In various exemplary embodiments, the process 400 further includes a segmentation process to classify the collagen fibers into various different types of collagen. The segmentation process to classify the collagen fibers into different types may be performed based on the first and second binarized images 432 and 422. In various exemplary embodiments, various types of collagen including portal vein collagen, septal collagen, and fibrillar collagen in the first binarized image 432 are classified based on histopathological information related to the relative positions of the portal duct and the central vein from the second binarized image 422. In various exemplary embodiments, the portal vein collagen, septal collagen, and fibrillar collagen may be classified in sequence based on a plurality of preset conditions.

[0078] like Figure 4 As shown, a plurality of different types of collagen images (e.g., a portal collagen image 453 corresponding to portal collagen, a fibril collagen image 455 corresponding to fibril collagen, and a diaphragm collagen image 457 corresponding to diaphragm collagen) can be generated based on the first and second binarized images 432 and 422. In each exemplary embodiment, fibril collagen can be first selected because its area is less than 10 μm. 2 , represents the broken collagen distributed in the tissue (generating the first type of collagen image based on the collagen that meets the preset size condition in the first binary image). For example, based on tissue pathology knowledge, the area is less than 10um 2The collagen fibers can be used as (or regarded as) fibrillar collagen. In addition to the smaller area size, for example, the fibrillar collagen is usually broken, separated collagen, which is distributed in the pericellular or sinusoidal pericellular space and can be identified or selected based on its distribution. Next, for portal vein collagen, pixels from the boundary of the portal duct and the central vein can be first selected, and a cutoff distance can be selected to expand those selected pixels, wherein the pixels representing all collagen within the cutoff distance in the first binarized image can be considered as pixels representing portal vein collagen (to generate a second type of collagen image based on the collagen in the first binarized image that meets the preset boundary distance condition relative to the portal duct and the central vein in the second binarized image). Next, all remaining collagen can be regarded as diaphragm collagen (to generate a third type of collagen image based on the collagen in the first binarized image that does not meet the preset size condition and the preset distance condition). Multiple different types of collagen images corresponding to the corresponding types of collagen can contain signals of this type of collagen. It should be understood that in some cases, the signal of a particular type of collagen (e.g., fibrillar collagen) may be much weaker than other types of collagen (e.g., portal collagen and septal collagen) and cannot be observed by direct visualization from the collagen image (e.g., fibrillar collagen image 455). Figure 5 A framework diagram 500 for evaluating fibrosis of a tissue sample according to various exemplary embodiments of the present invention is shown. Multiple different types of collagen parameters can be determined based on multiple different types of collagen images by a deep neural network 510. The degree of fibrosis of a stained tissue sample can be determined based on multiple different types of collagen parameters.

[0079] like Figure 5 As shown, multiple different types of collagen images (e.g., portal vein collagen image 453, fibril collagen image 455, and septum collagen image 457) can be evaluated by a deep learning-based classification model or deep neural network 510 to obtain prediction results of multiple different types of collagen (multiple different types of collagen parameters). The multiple different types of collagen parameters each include a quantitative fibrosis value related to the type of collagen corresponding to the multiple different types of collagen in the stained tissue sample. For example, the quantitative fibrosis value can be a collagen fibrosis index. For example, multiple different types of collagen parameters (e.g., in Figure 5523, 525 and 527) may include a first quantitative fibrosis value corresponding to a first type of collagen (e.g., portal collagen), a second quantitative fibrosis value corresponding to a second type of collagen (e.g., fibrillar collagen) and a third quantitative fibrosis value corresponding to a third type of collagen (e.g., septal collagen).

[0080] In various exemplary embodiments, the deep neural network 510 may include multiple deep neural sub-networks to determine multiple different types of collagen parameters based on multiple different types of collagen images. Multiple deep neural sub-networks can be trained for various types of collagen to generate multiple different types of collagen parameters. In other words, each of the multiple deep neural sub-networks can be trained for a specific type of collagen, and the collagen parameters of the collagen can be determined based on the collagen image corresponding to the type of collagen. In various exemplary embodiments, each deep neural sub-network can be a DCNN. For example, three DCNNs can be trained separately to generate a first quantitative fibrosis value corresponding to the first type of collagen, a second quantitative fibrosis value corresponding to the second type of collagen, and a third quantitative fibrosis value corresponding to the third type of collagen. The architecture of each CNN may include an input layer and an output layer, as well as multiple hidden layers consisting of convolutional layers (applying convolution operations to the input and passing the results to the next layer), activation functions (defining the output of a node for a given set of inputs), pooling layers (combining the outputs of a cluster of neurons in one layer into a single neuron in the next layer), fully connected layers (connecting each neuron in one layer to each neuron in another layer), and classification layers (converting back to data labels). The processed different types of collagen images of various types of collagen fibers may be fed into respective DCNNs and analyzed using the above layers, thereby generating final class labels representing the degree of fibrosis of each type of collagen from the respective collagen images.

[0081] In various exemplary embodiments, multiple different types of collagen parameters can be combined to generate a final quantitative fibrosis value or final fibrosis index (referred to herein as sqFibrosis index) 550 for a tissue sample to determine the degree of fibrosis in the tissue sample. In various exemplary embodiments, in a non-limiting example, multiple different types of collagen parameters can be combined using logistic regression 530 (e.g., multinomial logistic regression).

[0082] Figure 6A diagram showing an exemplary architecture of a deep neural subnetwork 515 for evaluating tissue sample fibrosis according to various exemplary embodiments of the present invention is shown. As described, according to various exemplary embodiments, a plurality of deep neural subnetworks may be provided, wherein each of the plurality of deep neural subnetworks may be used for a particular type of collagen, and the collagen parameters of the type of collagen are determined based on a collagen image corresponding to the type of collagen. In various exemplary embodiments, the deep neural subnetwork 515 may be a DCNN. The deep neural subnetwork may include a plurality of convolutional layers, a maximum pooling layer, and a fully connected layer. In various exemplary embodiments, for each type of collagen, the deep neural subnetwork 515 may be a pre-trained AlexNet-CNN for training and testing purposes. In various exemplary embodiments, the deep neural subnetwork may be a 7-layer AlexNet-CNN. For example, the deep neural subnetwork may be a modified 7-layer AlexNet algorithm for computer-assisted liver fibrosis scoring. For example, Hoo-Chang, S. et al. Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning. IEEE transactions on medical imaging 35, 1285-1298, doi: 10.1109 / TMI.2016.2528162 (2016) discloses an exemplary CNN that can be modified to determine collagen parameters of a type of collagen based on the type of collagen corresponding to a collagen image, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0083] In each exemplary embodiment, each CNN may include an input layer, seven hidden layers (five convolutional layers C1-C5 and two fully connected layers FC6-FC7) and an output layer using batch stochastic gradient descent, with specific values ​​of momentum and weight decay. In an exemplary embodiment related to liver fibrosis evaluation, the input layer and output layer of the original AlexNet-CNN network can be replaced accordingly in the improved deep neural subnetwork to evaluate liver fibrosis. For example, in the improved deep neural subnetwork, the basic structure of the deep learning neuronal network can be maintained while replacing the input and output. For example, the input layer can be used to provide different types of collagen images to the deep neural subnetwork, and the output layer (e.g., the last few convolutional layers and fully connected layers) is used to generate collagen parameters for the corresponding type of collagen image.

[0084] In various exemplary embodiments, a deep neural network (e.g., each of a plurality of deep neural subnetworks) can be trained based on transfer learning techniques. For example, Desautels, T. et al. Using Transfer Learning for Improved Mortality Prediction in a Data-Scarce Hospital Setting. Biomedical informatics insights 9, 1178222617712994, doi: 10.1177 / 1178222617712994 (2017); Sharma, H., Zerbe, N., Klempert, I., Hellwich, O. & Hufnagl, P. Deep convolutional neural networks for automatic classification of gastric carcinoma using whole slide images in digital histopathology. Computerized medical imaging and graphics: the official journal of the Computerized Medical Imaging Society 61, 2-13, doi: 10.1016 / j.compmedimag.2017.06.001 (2017); and Krizhevsky, A., Sutskever, I. & Hinton, GE in Proceedings of the 25th International Conference on Neural Information Processing Systems-Volume 11097-1105 (Curran Associates Inc., Lake Tahoe, Nevada, 2012), describe the use of transfer learning methods in deep neural networks, the contents of which are incorporated herein by reference in their entirety for all purposes. For example, transfer learning is a machine learning method in which a model developed for a task is reused as the starting point of a model for a second task. In various exemplary embodiments, the deep neural network can be pre-trained using a weakly related or even unrelated image source using a transfer learning technique using a large number of training data sets worldwide. In a transfer learning technique using a pre-trained model, a source model can first be selected. With respect to selecting a source model, a pre-trained source model can be selected from the available models.In an exemplary embodiment, the AlexNet model previously trained with a large and challenging data set can be included in the candidate model pool available for selection. Next, the selected source model can be used again. For example, the selected pre-trained source model can be used as the starting point for a model of a second task of interest (e.g., liver fibrosis classification). Depending on the modeling technique used, this may involve using all or part of the model. Optionally, the selected source model can be adjusted. For example, it may be necessary to improve or perfect the selected source model based on the input-output pair data that can be used for the task of interest. It has been found that the pre-trained deep neural network by the transfer learning method can automatically and accurately perform liver fibrosis in stages without feature extraction.

[0085] The multiple different types of collagen images (e.g., processed portal vein collagen image 453, fibrillar collagen image 455, and septum collagen image 457) can be resized and copied to fit as input images into the corresponding CNNs trained for each portal vein collagen, fibrillar collagen, and septum collagen. In each exemplary embodiment, the processed portal vein collagen, fibrillar collagen, and septum collagen images are resized and copied to 224×224×3 pixels to fit the corresponding CNNs as input images. In each exemplary embodiment, each CNN can further include two maximum pooling layers of size 2×2 pixels after the first and second convolutional layers, wherein the size and number of filters of the first and second convolutional layers are 11×11×3 pixels, 96 and 5×5×96 pixels, 256, respectively. Each CNN may include three additional convolutional layers (e.g., the third, fourth, and fifth convolutional layers), wherein the size and number of filters may be 3×3×256 pixels, 384 and 3×3×384 pixels, 384 and 3×3×384 pixels, 256, respectively. The convolutional layer may be implemented before the third maximum pooling layer. Each CNN may further include the sixth and seventh hidden layers, which may be 4096-dimensional fully connected layers, wherein the input matrix is ​​converted to a vector for the Softmax activation function by a general matrix vector multiplication (GEMV) method. In each exemplary embodiment, the morphological and / or texture features extracted from the deep neural network may be used as input to an algorithm that is not based on deep learning. In the feature extraction process, disease patterns are extracted at the cell or tissue level to measure changes in morphological features or to classify collagen images to grade diseases. Cellular-level features focus on quantifying the properties of individual cells without considering any potential spatial relationships that may exist between cells. For individual cells, features based on morphology, texture, fractal, and / or intensity may be extracted. Tissue-level features quantify the distribution of cells throughout the tissue; these features capture spatial information or the dependence of the gray levels of pixels. For tissues, texture, fractal, and / or topological features can be extracted. Non-deep learning algorithms are more computationally efficient and less time consuming. And because they are related to the pathology examination process, they are more relevant to pathology interpretation.

[0086] In various exemplary embodiments, a final output layer of each deep neural sub-network in the plurality of deep neural sub-networks may be configured to output a quantitative fibrosis value associated with a corresponding type of collagen based on a collagen image of the corresponding type of collagen.

[0087] Various non-limiting examples of framework diagrams for evaluating fibrosis in tissue samples according to various exemplary embodiments of the present invention are as follows:

[0088] Building the sqFibrosis Index

[0089] In order to merge all collagen indices (portal vein, fibril and diaphragm index) into the final sqFibrosis index, multinomial logistic regression (MLR) can be used. For example, in Lee, K., Ahn, H., Moon, H., Kodell, RL & Chen, JJ Multinomial logistic regression ensembles. Journal of biopharmaceutical statistics 23, 681-694, doi: 10.1080 / 10543406.2012.756500 (2013), MLR is described, and its content is incorporated herein by reference as a whole for all purposes. MLR may be a generalization of normal logistic regression, and multiple discrete results (for various fibrosis scoring systems) can be generated. In order to calculate the fibrosis index of a tissue sample, all other tissue samples in the same sample can be used as a training set. The probability that the sample belongs to one of 0 to 4 stages can be predicted by the trained MLR model. In various exemplary embodiments, fibrosis indices from various types of collagen can be used as new feature inputs for MLR, and a matrix p of coefficient estimates for the multinomial logistic regression of the nominal response of the fibrosis category of the predictor in feature space can be obtained. The coefficients represent both the effect of the predictor variable on the relative risk and the log odds of being in a certain category relative to the reference category. Once the probabilities of a given image corresponding to different categories of fibrosis are obtained from multiple different types of collagen images corresponding to multiple types of collagen, respectively, the final index (fibrosis index) can be determined (or calculated) using Equation 1 as shown below:

[0090] index=α∑p i *E i , i = 0, 1, 2, 3, 4, ——(Equation 1)

[0091] Among them, p i is the predicted probability of fibrosis stage i, E i is the expected value at each stage, and α is a scaling factor that normalizes the index to a certain range. In a non-limiting example, α can be set to 1 / 4 so that the index is a continuous variable ranging from 0 to 1.

[0092] Tissue samples and image acquisition of tissue samples

[0093] In each exemplary embodiment, as described above, a stained tissue sample can be obtained for evaluation. In addition, the stained tissue sample can be used for training a deep neural network. In each exemplary embodiment, tissue samples can be obtained according to an animal model for evaluating rat liver fibrosis induced by, for example, thioacetamide (TAA). For example, all protocols for evaluating or studying the rat model of liver fibrosis induced by TAA have been reviewed and approved by the Institutional Animal Care and Use Committee (IACUC) of the Biological Resource Center (BIRC, BRC). In each exemplary embodiment, 20 rats were randomly divided into 6 groups, representing six time points related to treating rats with TAA. The 6 groups representing the six time points can include a group without drug treatment and a group receiving TAA treatment for 4, 6, 7, 8 and 12 weeks, respectively. For example, a liver sample or sample of the left lateral liver of each animal can be fixed with formalin, paraffin-coated, and then cut into 5 μm sections. Liver samples can be stained with Masson Trichrome (MT) to obtain stained tissue samples. In addition, MT staining can be performed to obtain stained tissue samples for histological examination. The stained images of the stained tissue samples can be further obtained by imaging techniques such as direct brightfield imaging. For example, the samples used for training and testing the sqFibrosis index can be imaged by brightfield microscopy. Image acquisition for brightfield microscopy can be performed on stained sections of tissue samples using a 4× objective. In an exemplary embodiment, in order to cover most areas of the sample, four 12 mm images can be acquired for each sample by brightfield microscopy. 2 (4 x 3 mm) image size. The four images can be used as separate samples.

[0094] In addition, the score can be performed by an experienced pathologist using the Metavir fibrosis staging system to compare with the prediction of fibrosis in the tissue sample using the framework diagram 500 as described above. Evaluated by the Metavir fibrosis staging system, sqFibrosis showed similar trends in the regression and progression patient groups, highly correlated with the Metavir fibrosis score. Therefore, the framework diagram 500 including the deep learning-based algorithm can automatically quantify the progression of liver fibrosis in TAA-induced fibrotic rat livers using collagen staining of tissue samples, which are commonly available in clinical settings. And different stages of liver fibrosis can be scored with high sensitivity and specificity.

[0095] Figure 7A flow chart 700 of different conventional techniques 710 and 720 and a technique or framework diagram 500 for evaluating tissue sample fibrosis according to various exemplary embodiments of the present invention are shown. Conventional technique 720 is a qFibrosis method based on a stain-free multiphoton microscope. For example, collagen features can be extracted from unstained sections of tissue samples using second harmonic generation / two-photon excitation fluorescence (SHG / TPEF) microscopy. However, the multiphoton microscope is not commonly used in clinics or laboratories, which makes the clinical application of these techniques difficult in addition to changes caused by the imaging process and the use of multiphoton microscopes. Compared with previous techniques, the sqFibrosis index obtained by the framework diagram 500 provides a fully quantitative computer-assisted liver fibrosis staging for evaluating fibrosis of biopsy-based tissue samples, especially using stained tissue samples. In addition, since a standard optical microscope can be used, the framework diagram 500 can have improved clinical applicability compared to conventional techniques.

[0096] Statistical analysis

[0097] In various exemplary embodiments, the Kruskal-Wallis (KW) test can be used to estimate the statistical differences in the sqFibrosis index and collagen ratio area (CPA) between Metavir fibrosis stages. A two-tailed Wilcoxon rank sum test can be performed to estimate the statistical differences in the sqFibrosis index and CPA between different Metavir stages. For example, the statistical significance level can be set to p<0.05.

[0098] Figures 8A-8C An exemplary comparison of changes in collagen patterns in stained tissue sample images using sqFibrosis and CPA to evaluate various stages of liver fibrosis is shown. Fig. 8A Stained images 810 and processed images 820 of the stained images (e.g., processed portal collagen image 820a, processed fibrillar collagen image 820b, and processed septal collagen image 820c) of changes in collagen patterns in animals at various stages of fibrosis (e.g., F0, F1, F2, F3, F4) in chronic liver disease based on the Metavir staging system are shown. In a non-limiting example illustration, portal, fibrillar, and septal collagen (not shown) may be represented by colors such as blue, red, and green, respectively. For example, at different stages of fibrosis, the processed images may include different patterns of collagen fibers visualized in blue, red, and / or green. Figure 8B and 8CThe changes in sqFibrosis index and CPA as fibrosis progresses between stages of the animals are shown, respectively. The sqFibrosis index can reliably summarize the various fibrosis staging systems, as it can identify differences between animal samples at various stages (p<0.001), for example, with better performance compared to CPA (p=0.152). The sqFibrosis values ​​increase as fibrosis progresses and show significant differences between all stages of liver fibrosis. CPA shows strong changes only in the later stages and cannot distinguish between the early stages (stages 1 and 2).

[0099] Fig. 9 A graph is shown illustrating the performance evaluation of the deep learning model and collagen proportion area (CPA) of the sqFibrosis index using unprocessed stained images, processed collagen images that were not classified into different types, and processed collagen images that were classified into different types of collagen (e.g., portal, fibrillar, and septal collagen). In the case of the deep learning model using processed collagen images that were classified into different types of collagen, the sqFibrosis index was more reliable, with an area under the ROC (AUROC) value of 0.94 for fibrosis detection; an AUROC value of 0.87 for significant fibrosis detection, an AUROC value of 0.93 for advanced fibrosis detection, and an AUROC value of 0.94 for cirrhosis detection. On the other hand, in the case of determining the sqFibrosis index using unprocessed stained images, the AUROC value for fibrosis detection was 0.75, the AUROC value for significant fibrosis detection was 0.69, the AUROC value for advanced fibrosis detection was 0.71, and the AUROC value for cirrhosis detection was 0.78. As for the case where the processed collagen images that were not classified into different types were used to determine the sqFibrosis index, the AUROC values ​​for fibrosis detection were 0.84, 0.81 for significant fibrosis detection, 0.92 for advanced fibrosis detection, and 0.83 for cirrhosis detection. Therefore, the sqFibrosis index determined using a deep learning model of processed collagen images that were classified into different types of collagen allowed for sensitivity grading of fibrosis scores in animal samples. The sqFibrosis index also showed superior performance to CPA (AUROC value for fibrosis detection, 0.89; AUROC value for significant fibrosis detection, 0.88; AUROC value for advanced fibrosis detection, 0.91; AUROC value for cirrhosis detection, 0.87).

[0100] Sampling error is a major limitation when applying quantitative methods such as CPA. To evaluate the sensitivity of the sqFibrosis index to sampling error, a proof-of-concept was performed on animal samples. Various image sizes, image resolutions, and image intensities were generated and examined to simulate samples of varying quality. Figures 10A-10B Shown are the stained images of various sample sizes of stained tissue samples from animal studies, and the AUROC values ​​of sqFibrosis and CPA for various sample sizes evaluated based on different stages of fibrosis in tissue samples, respectively. Figures 11A-11B Shown are stained images of stained tissue samples at various image resolutions from animal studies, and AUROC values ​​of sqFibrosis and CPA at various image resolutions evaluated based on different stages of fibrosis in tissue samples, respectively. FIG. 12A to FIG. 12B Stained images of stained tissue samples from animal studies at various image intensities and AUROC values ​​of sqFibrosis and CPA at various image intensities evaluated based on different stages of fibrosis in tissue samples are shown, respectively. The AUROC value of the sqFibrosis index slightly decreased with decreasing sample size, resolution, and intensity. Under standard conditions, CPA achieved similar AUROC values ​​to the sqFibrosis index; however, the AUROC value of CPA dropped dramatically when sample size, resolution, and intensity were reduced.

[0101] It has been found that although conventional semi-automatic learning algorithms can be used for liver fibrosis scoring, deep learning-based algorithms are more promising because they automatically discover features and calculate the weight of each feature by its contribution, which will be more time-saving and cost-effective. Inevitably, deep learning-based algorithms require large data sets and complex training neural networks, which can be compensated by using transfer learning from weakly related or even unrelated image sources. The sqFibrosis index according to an embodiment of the present invention is capable of evaluating important features. For example, image preprocessing of stained images can classify various types of collagen, so that further feature calculation and classification can be performed using deep learning / transfer learning methods. This determines the minimum feature set required for classification, thereby reducing the number of parameters in the deep learning model. The sqFibrosis index can be further evaluated using other scoring systems (such as Ishak staging and Beijing PIR classification) to analyze intrahepatic cirrhosis detection and monitoring before and after treatment from a clinical perspective.

[0102] In various exemplary embodiments, the sqFibrosis index is a deep learning-based algorithm using a pre-trained deep neural network (e.g., AlexNet-CNN) that automatically improves the accuracy and throughput of staging fibrosis, allowing for instant quantitative staging of liver fibrosis using existing stained liver samples of collagen in less complex clinical or laboratory settings and hospitals. In addition, stained tissue samples are inexpensive, easy to understand, and well documented. Transfer learning methods using weakly correlated or even unrelated image sources may help address the need for large data sets for deep learning-based algorithms. This advantageously addresses problems in technologies such as magnetic resonance imaging (MRI) and computed tomography (CT) images of liver fibrosis, which require the use of large training data sets, which biopsy sample images generally cannot meet. Since there are already a large number of repositories / databases of such histological staining samples worldwide, this computer-assisted and automatic quantification of liver fibrosis scoring stages can be extended to the design of high-performance classification systems for other medical imaging tasks.

[0103] Although the embodiments of the present invention have been specifically shown and described with reference to specific embodiments, it will be understood by those skilled in the art that various changes may be made in form and detail without departing from the scope of the present invention, without departing from the scope of the present invention as defined in the appended claims. The scope of the present invention is shown by the appended claims, and includes all changes within the equivalent meaning and range of the claims.

Claims

1. A computer-implemented method for evaluating fibrosis in a tissue sample, using at least one processor, the method comprising: obtaining a staining image of a stained tissue sample, wherein collagen in the stained tissue sample is stained; generating a plurality of different types of collagen images based on the stained image, wherein the plurality of different types of collagen images correspond to a plurality of different types of collagen respectively; Determining parameters of a plurality of different types of collagen based on the plurality of different types of collagen images using a deep neural network, wherein the deep neural network is trained based on a transfer learning technique; and The degree of fibrosis of the stained tissue sample is determined based on the multiple different types of collagen parameters; wherein the multiple different types of collagen parameters are combined to generate a final quantitative fibrosis value or a final fibrosis index of the stained tissue sample to determine the degree of fibrosis of the stained tissue sample.

2. The method according to claim 1, characterized in that The generating of multiple different types of collagen images includes: generating a first image corresponding to the tissue in the stained image; and A second image corresponding to the collagen in the stained image is generated.

3. The method according to claim 2, characterized in that The generating of multiple different types of collagen images further comprises: A third image is generated corresponding to the portal duct and central vein in the first image.

4. The method according to claim 3, characterized in that The generating of multiple different types of collagen images further comprises: generating a first binarized image based on the second image; generating a second binarized image based on the third image; and The plurality of different types of collagen images are generated based on the first binarized image and the second binarized image.

5. The method according to claim 4, characterized in that The first image and the second image are generated based on K-means clustering.

6. The method according to claim 4, characterized in that The plurality of different types of collagen images include a first type of collagen image, a second type of collagen image, and a third type of collagen image, and generating the plurality of different types of collagen images based on the first binarized image and the second binarized image includes: Generate the first type of collagen image based on the collagen that meets the preset size condition in the first binary image; generating the second type of collagen image based on the collagen in the first binarized image that satisfies a preset distance condition relative to the boundaries of the portal duct and the central vein in the second binarized image; and The third type of collagen image is generated based on the collagen in the first binarized image that does not meet the preset size condition and the preset distance condition.

7. The method according to claim 1, characterized in that The multiple different types of collagen include portal collagen, fibrillar collagen and septal collagen.

8. The method according to claim 1, characterized in that The plurality of different types of collagen parameters each include a quantitative fibrosis value associated with a corresponding type of collagen among the plurality of different types of collagen in the stained tissue sample.

9. The method according to claim 1, characterized in that The degree of fibrosis of the tissue samples was determined based on the different types of collagen parameters using multinomial logistic regression.

10. The method according to claim 1, characterized in that The deep neural network includes a plurality of deep neural sub-networks corresponding to the plurality of different types of collagen, respectively, wherein each of the plurality of deep neural sub-networks is configured to output a quantitative fibrosis value regarding a corresponding type of collagen.

11. A system for evaluating fibrosis in a tissue sample, the system comprising: Memory; and at least one processor communicatively coupled to the memory and configured to: obtaining a staining image of a stained tissue sample, wherein collagen in the stained tissue sample is stained; generating a plurality of different types of collagen images based on the stained image, wherein the plurality of different types of collagen images correspond to a plurality of different types of collagen respectively; Determining parameters of a plurality of different types of collagen based on the plurality of different types of collagen images using a deep neural network, wherein the deep neural network is trained based on a transfer learning technique; and determining the degree of fibrosis of the stained tissue sample based on the plurality of different types of collagen parameters; Therein, a plurality of different types of collagen parameters are combined to generate a final quantitative fibrosis value or a final fibrosis index for the stained tissue sample to determine the degree of fibrosis of the stained tissue sample.

12. The system according to claim 11, characterized in that The generating of multiple different types of collagen images includes: generating a first image corresponding to the tissue in the stained image; and A second image corresponding to the collagen in the stained image is generated.

13. The system of claim 12, wherein: The generating of multiple different types of collagen images further comprises: A third image is generated corresponding to the portal duct and central vein in the first image.

14. The system of claim 13, wherein: The generating of multiple different types of collagen images further comprises: generating a first binarized image based on the second image; generating a second binarized image based on the third image; and The plurality of different types of collagen images are generated based on the first and second binarized images.

15. The system of claim 14, wherein: The first image and the second image are generated based on K-means clustering.

16. The system of claim 14, wherein: The plurality of different types of collagen images include a first type of collagen image, a second type of collagen image, and a third type of collagen image, and generating the plurality of different types of collagen images based on the first binarized image and the second binarized image includes: Generate the first type of collagen image based on the collagen that meets the preset size condition in the first binary image; generating the second type of collagen image based on the collagen in the first binarized image that satisfies a preset distance condition relative to the boundaries of the portal duct and the central vein in the second binarized image; and The third type of collagen image is generated based on the collagen in the first binarized image that does not meet the preset size condition and the preset distance condition.

17. The system of claim 11, wherein: The multiple different types of collagen include portal collagen, fibrillar collagen and septal collagen.

18. The system of claim 11, wherein: The plurality of different types of collagen parameters each include a quantitative fibrosis value associated with a corresponding type of collagen among the plurality of different types of collagen in the stained tissue sample.

19. The system of claim 11, wherein: The degree of fibrosis of the tissue samples was determined based on the different types of collagen parameters using multinomial logistic regression.

20. A computer program product, embodied in one or more non-transitory computer-readable storage media, the computer program product comprising instructions executed by at least one processor to implement a method of evaluating fibrosis in a tissue sample, the method comprising: obtaining a staining image of a stained tissue sample, wherein collagen in the stained tissue sample is stained; generating a plurality of different types of collagen images based on the stained image, wherein the plurality of different types of collagen images correspond to a plurality of different types of collagen respectively; Determining parameters of a plurality of different types of collagen based on the plurality of different types of collagen images using a deep neural network, wherein the deep neural network is trained based on a transfer learning technique; and The degree of fibrosis of the stained tissue sample is determined based on the multiple different types of collagen parameters; wherein the multiple different types of collagen parameters are combined to generate a final quantitative fibrosis value or a final fibrosis index of the stained tissue sample to determine the degree of fibrosis of the stained tissue sample.

Images