A liver biopsy slice basic lesion evaluation method based on multi-scale feature extraction

Abstract

A liver biopsy slice basic lesion evaluation method based on multi-scale feature extraction, the method comprising constructing a stained slice image dataset, randomly intercepting a high-resolution patch; labeling the pathological features, constructing a pathological prior knowledge base; training a multi-scale local lesion grading model based on ResNet; using image scaling technology to obtain a low-resolution patch from the high-resolution patch, extracting and fusing multi-scale deep features, realizing patch-level local lesion grading; constructing a global lesion evaluation model, aggregating local grading results, realizing whole slice-level lesion grading. Based on multi-scale feature extraction and fusion, the method realizes efficient extraction of the pathological features of the patch, thereby realizing accurate classification of the patch-level local lesion; on the other hand, the method can accurately locate the relevant area associated with the lesion, thereby providing the model with interpretability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of histopathological image processing, in particular to a liver biopsy slice basic lesion evaluation method based on multi-scale feature extraction. BACKGROUND

[0002] Chronic liver disease refers to a liver disease lasting more than six months. Its severe cases can lead to cirrhosis. According to relevant research, the main causes of chronic liver disease include alcoholic liver disease caused by alcoholism, non-alcoholic fatty liver disease caused by obesity, viral hepatitis caused by viruses, and autoimmune hepatitis. With the development of society, chronic liver disease has become a major global health threat. Cirrhosis and even liver cancer caused by it have become one of the common causes of death. Death is mainly due to the complications of cirrhosis and hepatocellular carcinoma, and the proportion of death due to acute hepatitis is small. For the evaluation of the progress of chronic liver disease, liver biopsy pathological diagnosis is the gold standard. However, the shortage of pathologists is the most serious problem in the field of pathological diagnosis.

[0003] Many studies use deep learning to realize intelligent liver biopsy pathological diagnosis, but there are still some problems. On the one hand, due to the lack of liver biopsy pathological samples, the current classification work at the slice level is still relatively small, which leads to the fact that the research on liver biopsy basic indicators cannot meet the requirements of pathological practice. Roy et al. proposed a segmentation model to accurately quantify the problem of overlapping fat droplets. However, their research dataset only includes the pathological sections of 36 patients, so more data are needed to verify and promote this method. Heinemann et al. designed an automatic diagnosis model, but it is only limited to four basic indicators related to non-alcoholic fatty liver diagnosis, and the number of cases they used reached 467. On the other hand, the current deep neural network has the problem of insufficient explainability, which limits its application in the clinic. Arjmand et al. trained two convolutional neural networks with different optimization methods to detect the main differences between enlarged liver cells and fat droplets, combined with the visualization of the first two layers of feature maps, to improve the accuracy and explainability of liver biopsy classification results. These studies have solved some problems of liver biopsy to a certain extent. However, it can be seen that in many studies, the research on liver biopsy is hindered by the small number of biopsy samples and the size of the slice, resulting in a small number of researches. In addition, the existing researches are all slice-level classification studies on part of the liver biopsy indicators, that is, the current block-level classification research is insufficient, and the liver biopsy indicators are not comprehensive. On the other hand, the existing researches have the problem of insufficient explainability, and the model is difficult to explain its decision-making process, which cannot provide enough credibility to doctors and patients, so the clinical application of these researches is limited.

[0004] It is to be understood that the information disclosed in the Background section is merely for the purpose of enhancing the understanding of the present application, and thus can include information that is not prior art to those skilled in the art. SUMMARY

[0005] The main purpose of the present application is to overcome the defects of the background art, and provide a liver biopsy section basic lesion evaluation method based on multi-scale feature extraction.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] A liver biopsy section basic lesion evaluation method based on multi-scale feature extraction, characterized by the following steps performed on a computer:

[0008] S0, construct a stained section image dataset, and randomly cut patches from the whole slide images of the dataset to obtain high-resolution patches;

[0009] S1, label the pathological features of the patches to construct a pathological prior knowledge base;

[0010] S2, construct and use a multi-scale local lesion grading model based on ResNet to train the stained section image and the corresponding labeled data; use image scaling technology to obtain corresponding low-resolution patches from the high-resolution patches, use the multi-scale local lesion grading model to extract the multi-scale deep features of the high-resolution patches and the low-resolution patches, then classify the multi-scale fusion features, thereby realizing local lesion grading at the patch level;

[0011] S3, construct a global lesion evaluation model based on prior guidance, aggregate the local lesion grading results, thereby realizing lesion grading at the whole section level.

[0012] Further:

[0013] Step S0 specifically includes:

[0014] S0-0, scan the H&E section, Masson trichrome section and HBsAg immunohistochemical section collected from the liver biopsy case to obtain whole slide images;

[0015] S0-1, use a random cutting algorithm to select patches from the whole slide images obtained by scanning the sections for training of the local lesion grading model of ballooning, inflammation activity, steatosis, fibrosis and HBsAg positive lesion.

[0016] In step S1, the degree of ballooning, inflammation activity and steatosis of the H&E tile is labeled, the degree of fibrosis of the Masson's trichrome tile is labeled, and the degree of HBsAg positivity of the HBsAg immunohistochemistry image is labeled.

[0017] In step S2, the image scaling technique includes a bilinear interpolation method.

[0018] Step S2 specifically includes:

[0019] S2-0, using a pre-trained ResNet-34 to extract multi-scale deep features of the high-resolution tile and the low-magnification image;

[0020] S2-1, using adaptive pooling to down-sample the high-resolution scale features to the same scale as the low-resolution scale features, and adopting a channel maximum difference degree to fuse the features of the two scales extracted from the high-resolution tile and the low-resolution tile.

[0021] S2-2, using a pooling layer and a fully connected layer to classify the multi-scale fused features, thereby realizing tile-based local lesion grading.

[0022] In step S2, the loss function of the training process includes two parts of classification loss and similarity loss, wherein the classification loss adopts a weighted cross-entropy loss function

[0023]

[0024] wherein N is the total number of the input high-resolution tiles, y i is the true label of the i-th tile, is the result of direct classification by the fully connected layer after the feature extractor of the high-resolution image,

[0025] For the multi-scale similarity loss the L1-Norm of the corresponding two tiles is calculated,

[0026]

[0027] wherein w and h are the feature map sizes of a single low-resolution image, C is the channel of the feature map (j, k) position, X is the feature map of the high-resolution tile after adaptive pooling, and Y is the feature map of the low-resolution tile.

[0028] The total loss function is

[0029]

[0030] In step S3, based on the local lesion grading results, a predicted mean aggregation strategy is used to achieve lesion grading at the whole slice level.

[0031] In step S3, the aggregation rules include determining the optimal patch ratio thresholds between different levels of the underlying lesion at the WSI level based on an adaptive thresholding method; preferably, this includes determining the patch ratio thresholds for ballooning degeneration, inflammatory activity, fatty degeneration, fibrosis, and HBsAg-positive lesions.

[0032] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for assessing basic lesions in liver biopsy sections based on multi-scale feature extraction.

[0033] A computer program product, wherein the computer program product is implemented by a processor at runtime, the method for assessing basic lesions in liver biopsy sections based on multi-scale feature extraction.

[0034] The present invention has the following beneficial effects:

[0035] This invention provides a method for assessing underlying lesions in liver biopsy sections based on multi-scale feature extraction. Applied to the histopathological evaluation process of liver biopsy sections, it achieves end-to-end, automated, AI-assisted assessment of underlying lesions at the whole-section level. Based on multi-scale feature extraction and fusion models, this invention achieves accurate patch-level grading of underlying lesions, providing a prerequisite for accurate whole-section-level diagnosis. Through a multi-scale local lesion grading model and a priori-guided global lesion assessment model, AI-assisted assessment of underlying lesions such as ballooning degeneration, fibrosis, HBsAg positivity, inflammatory activity, and fibrosis can be achieved.

[0036] The significant advantages of this method are as follows: Firstly, based on multi-scale feature extraction and fusion, it achieves efficient extraction of pathological features from map patches, thereby enabling accurate classification of local lesions at the patch level. Secondly, it can accurately locate relevant regions associated with lesions, thus providing interpretability for the model. Furthermore, based on predictive aggregation, it achieves accurate basic lesion grading at the whole-slice level.

[0037] In the preferred scheme, by constructing a pathological prior knowledge base, five basic lesion indicators on liver biopsy sections stained with three types of staining, namely H&E, Massen trichrome, and HBsAg immunohistochemistry, are labeled, and local lesion classification and whole-section level lesion grading are realized, thus effectively realizing the construction of a basic lesion assessment system for liver biopsy sections.

[0038] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0039] Figure 1 Flow chart of the liver biopsy slice basic lesion evaluation method based on multi-scale feature extraction of the embodiment of the present application.

[0040] Figure 2 Basic lesion example graph of the embodiment of the present application.

[0041] Figure 3 Multi-scale local lesion grading model detail graph of the embodiment of the present application.

[0042] Figure 4A Balloon-like change example graph of the embodiment of the present application.

[0043] Figure 4B Fibrosis example graph of the embodiment of the present application.

[0044] Figure 4C HBsAg positive example graph of the embodiment of the present application.

[0045] Figure 4D Inflammation activity degree example graph of the embodiment of the present application.

[0046] Figure 4E Fatty degeneration example graph of the embodiment of the present application.

[0047] Figure 5 Grading result display graph of the embodiment of the present application.

[0048] Figure 6 Overall framework schematic diagram of the algorithm of the embodiment of the present application.

[0049] Figure 7 Engineering software interface graph of the embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described in detail below. It should be emphasized that the following description is merely exemplary, and is not intended to limit the scope of the present application and its applications.

[0051] Referring to Figure 1 The embodiment of the present application provides a liver biopsy slice basic lesion evaluation method based on multi-scale feature extraction, characterized in that the following steps are performed on a computer:

[0052] S0, constructing a stained slice image dataset, randomly cutting a graph block from a whole slide image of the dataset to obtain a high-resolution graph block;

[0053] S1, labeling the pathological features of the graph block to construct a pathological prior knowledge base;

[0054] S2, a multi-scale local lesion grading model based on ResNet is trained and used by constructing and using a stained section image and corresponding labeled data; an image scaling technique is used to obtain a corresponding low-resolution image block from the high-resolution image block, multi-scale deep features of the high-resolution image block and the low-resolution image block are extracted and fused by using the multi-scale local lesion grading model, and then the multi-scale fused features are classified, so that local lesion grading at the image block level is realized;

[0055] S3, a global lesion assessment model based on prior guidance is constructed, and local lesion grading results are aggregated, so that lesion grading at the whole section level is realized.

[0056] The liver biopsy section basic lesion assessment method based on multi-scale feature extraction of the embodiment can realize end-to-end and automated artificial intelligence assisted assessment of basic lesions at the whole section level. Based on the multi-scale feature extraction and fusion model, accurate basic lesion grading at the image block level is realized, which provides a premise for accurate basic lesion diagnosis at the whole section level.

[0057] In some embodiments, by constructing a pathological prior knowledge base, annotation of five basic lesion indicators on liver biopsy sections stained by H&E, Masson trichrome, HBsAg immunohistochemistry and the like is realized, local lesion classification and lesion grading at the whole section level are realized, and the construction of the liver biopsy section basic lesion assessment system is realized. By using the multi-scale local lesion grading model and the global lesion assessment model based on prior guidance, artificial intelligence assisted assessment of basic lesions such as ballooning, fibrosis, HBsAg positivity, inflammation activity and fibrosis can be realized.

[0058] The method has the following advantages: on the one hand, the method realizes efficient extraction of pathological features of the image block based on multi-scale feature extraction and fusion, thereby realizing accurate classification of local lesions at the image block level; on the other hand, the method can accurately locate the relevant area associated with the lesion, thereby providing the model with interpretability. Further, based on prediction aggregation, accurate basic lesion grading at the whole section level is realized.

[0059] The specific embodiments of the present application are further described below.

[0060] As an example, in the following specific embodiments, first, a data set composed of images stained by H&E, Masson trichrome and HBsAg immunohistochemistry and the like is constructed, and image blocks are randomly cut from the data set; second, a pathological prior knowledge base is constructed by annotating lesions such as ballooning, inflammation activity, steatosis, fibrosis degree and HBsAg positivity; then, a local lesion grading model based on multi-scale feature extraction is used to realize lesion grading at the image block level; finally, a prediction mean aggregation strategy is used to realize lesion grading at the whole section level.

[0061] The overall framework of the algorithm of the embodiment of the present application is as shown in the figure. Figure 6

[0062] The specific steps of the liver biopsy section basic lesion evaluation system based on multi-scale feature extraction include:

[0063] Step S0 constructs a data set containing H&E images, Masson trichrome images and HBsAg immunohistochemical images, and randomly cuts patches from the whole slide images of the data set, and finally performs preprocessing.

[0064] Step S0-0 uses H&E sections, Masson trichrome sections and HBsAg immunohistochemical sections collected from liver biopsy cases to scan the sections using a resolution of 0.105um / pixel to obtain whole slide images (WSIs).

[0065] Step S0-1 uses a random cutting algorithm to select patches from H&E sections, Masson trichrome sections and HBsAg immunohistochemical sections for training of local lesion grading models for ballooning, inflammation activity, steatosis, fibrosis and HBsAg positive lesions, with a patch resolution of 0.42um / pixel and a size of 224*224 pixels.

[0066] Step S1 constructs a pathological prior knowledge base, annotates the degrees of ballooning, inflammation activity and steatosis of H&E patches, annotates the degree of fibrosis of Masson trichrome patches, and annotates the degree of HBsAg positivity of HBsAg immunohistochemical images.

[0067] Step S1-0 constructs a pathological prior knowledge base for the patches used for training of each model, and annotates the patches with ballooning (normal, water degeneration, ballooning), inflammation activity (none, yes), steatosis (none, yes), fibrosis (none, yes) and HBsAg positive lesions (none, yes), respectively.

[0068] Step S2 constructs a multi-scale local lesion grading model based on ResNet, and uses images of various stains and corresponding annotation data for model training.

[0069] Step S2-0 uses a pre-trained ResNet-34 to extract multi-scale deep features of high-magnification images and low-magnification images. For a high-resolution patch of 224*224 pixels, a low-magnification version is obtained using bilinear interpolation, with a size of 112*112 pixels. This ensures the heterogeneity of the information contained in the two scale patches. A pre-trained ResNet-34 on ImageNet is used to extract deep features of the two resolution patches.

[0070] ​Step S2-1 fuses the features of two scales extracted from the high-resolution patches and the low-resolution patches. The high-resolution scale features are down-sampled to the same scale as the low-resolution scale features using adaptive pooling, and the feature fusion is performed in the form of channel maximum difference.

[0071] Step S2-2 classifies the multi-scale fused features. The multi-scale fused features are classified using a pooling layer and a fully connected layer, so as to realize local lesion grading based on patches. The loss function of the training process includes two parts of classification loss and similarity loss. The classification loss adopts a weighted cross-entropy loss function

[0072]

[0073] wherein N is the total number of 224x224 resolution patches of 20 times input. y i is the true label of the i-th patch, is the result of direct classification of the fully connected layer after the feature extractor of the high-resolution image.

[0074] For the multi-scale similarity loss We use the calculation of the L1-Norm of the corresponding two patches.

[0075]

[0076] wherein N is the total number of 224x224 resolution patches of 20 times input. w and h are the feature map size of a single picture of 112x112 resolution, C is the channel of the feature map (j, k) position, X is the feature map of the 224x224 resolution patch of 20 times after adaptive pooling, and Y is the feature map of the 112x112 resolution patch of 10 times.

[0077] Finally, the total loss function is

[0078]

[0079] Step S3 constructs a global lesion evaluation model based on prior guidance, aggregates the local lesion grading results, and realizes lesion grading at the whole slice level.

[0080] Step S3-0 uses a predicted mean aggregation strategy to realize lesion grading at the whole slice level based on the local lesion grading results.

[0081] The aggregation strategy is based on clinical practice. In actual clinical evaluation, the proportion of lesion area is usually an important reference index for doctors to diagnose the progression of the disease. In the training set, the optimal tile proportion threshold between each level of WSI level-based lesions is determined based on the adaptive threshold method.

[0082] In one specific example, the aggregation rule is:

[0083] For balloon-like changes, the tile proportion threshold of WSI level with or without water-like changes is 18.852%, and the tile proportion threshold of WSI level with water-like changes and balloon-like changes is 50.164%.

[0084] For inflammation activity, the tile proportion threshold of WSI level is 48.411%.

[0085] For fatty degeneration, the tile proportion threshold of WSI level is 57.262%.

[0086] For fibrosis, the tile proportion threshold of WSI level is 28.443% 50.502%.

[0087] For HBsAg positive, the tile proportion threshold of WSI level without, mild and severe positive is 51.023% 58.216%.

[0088] This step models the pathological diagnosis process by obtaining adaptive tile proportion thresholds on the training set, efficiently introduces pathological priori into the whole slice level lesion grading model, and realizes accurate whole slice level grading.

[0089] In some specific embodiments, the step S0 comprises:

[0090] Step S0-0 uses 1278 H&E sections, 1133 Masson trichrome sections and 1217 HBsAg immunohistochemical sections collected from 1322 liver biopsy cases, scans the sections using a resolution of 0.105 um / pixel to obtain whole slide images. This step has high resolution for digital imaging and large image data. Step S0-1 uses tiles cut from the data set for training of the local lesion classification model. The constructed data set is as follows Table 1:

[0091] Table 1

[0092]

[0093]

[0094] Step S1 comprises:

[0095] The constructed dataset is locally annotated to construct a pathological prior knowledge base. The selected lesions include ballooning, inflammation activity and steatosis lesions in H&E images, fibrosis in Masson's trichrome images, and HBsAg positive in HBsAg immunohistochemical images. Ballooning annotation includes normal, hydropic degeneration, and ballooning. Hydropic degeneration refers to the presence of severe pathological changes of cells with intracellular edema in the patch. Ballooning refers to the presence of hepatocytes that are swollen like a ball, cytoplasm is loose, and cytoplasm is almost transparent due to the increase of water in hepatocytes. Inflammation activity includes two levels, with and without. Inflammation activity indicates that there are a large number of lymphocytes and plasma cells in the patch. Steatosis index is divided into two categories, with and without steatosis. The specific characteristics of the patch with steatosis are the presence of obvious fat droplets in the patch. Fibrosis includes two categories, with and without. Fibrosis refers to the presence of fibrosis area in the patch, and the color is blue. HBsAg positive includes two categories, with and without. HBsAg positive refers to the presence of brown color in the patch under DAB-H staining agent. The lesions annotated in this step are the basic lesions for diagnosing chronic liver disease. These lesions are of great significance for judging the occurrence and development of chronic liver disease. Figure 2 are examples of various basic lesions.

[0096] Step S2 includes:

[0097] Step S2-0 extracts multi-scale deep features of the high-magnification image and the low-magnification image using a pre-trained ResNet-34. Since the pathological image has two characteristics of extremely large size and multi-level pyramid structure, if the pathological image is only cut into several equal-resolution patches, the multi-level characteristics thereof are ignored. Conversely, if a single level is directly input into the model, the local information of the low-resolution picture located at the high level is seriously lost, and the size of the high-resolution picture located at the low level is too large to be accommodated by the model. Based on this, the application uses a multi-scale classification model. This model takes into account the structural characteristics of the pathological image and does not cause the size to be too large to be trained. In this way, it is ensured that the information contained in the images at the corresponding positions of the two scales of WSI is consistent. Then, they are respectively input into two feature extractors composed of the same backbone network for feature extraction. The backbone network constituting the feature extractor selected in this paper is ResNet34. Because it has a moderate model depth, it is neither too shallow to learn key features for patch-level classification problems, nor too deep to cause overfitting. In addition, ResNet also has a residual module to further avoid the risk of overfitting and enhance the generalization of the network. Finally, compared with the Transformer, ResNet also has the advantages of small parameter quantity, which is helpful for capturing local features of the image. Compared with other convolutional neural networks, the residual connection module of ResNet allows information to be re-injected into the network after skipping some layers, which helps to better reuse and transfer features and improves the representation ability of the network.

[0098] Step S2-1 fuses the features of two scales extracted from the high-resolution patch and the low-resolution patch. The application first changes the high-magnification features into the same size as the low-magnification features through the operation of adaptive pooling, and then adopts the channel maximum difference degree to fuse the features. This fusion method can efficiently integrate the detailed information from the high-resolution and the macro information from the low-resolution, thereby strengthening the recognition ability of the model for the pathological image features associated with the basic lesions.

[0099] Step S2-2 classifies the multi-scale fused features. This step fuses the traditional cross-entropy classification loss and the multi-scale consistency loss to achieve accurate diagnosis of multiple basic lesions of multiple staining sections.

[0100] The model details of step S2 are shown in Figure 3 .

[0101] The step S3 uses a prediction mean aggregation strategy to realize lesion grading at the whole slice level based on the local lesion grading result. Through the trained multi-scale tile-level grading model, a feature vector of the classification prediction of each tile can be obtained. However, from the perspective of pathology, whether a small local problem exists does not mean that the whole problem exists. Whether the whole pathological section has a problem needs to pay attention to the overall information, and the proportion of the lesion part in the whole section is usually an important consideration factor. Therefore, in the process of aggregating the results from the tile level to the slice level, the information between the contexts needs to be concerned. The present application uses an aggregation strategy based on prediction mean aggregation. The aggregation strategy is based on clinical practice. In actual clinical evaluation, the positive proportion is usually an important reference index for doctors to diagnose the progress of the disease. Therefore, in this part, the slice is first extracted from the foreground area, and the foreground area is cut into several tiles. Then the tiles are predicted by the multi-scale grading model. Finally, the proportion of each class of tile is calculated, and the mean value is taken as the prediction mean value of the slice. Through the determination of the adaptive tile proportion threshold, the present application introduces the pathological priori, realizes the aggregation from the local lesion classification result at the tile level to the lesion grading result at the whole slice level.

[0102] Effects and advantages of the above-mentioned embodiments of the present application:

[0103] 1. Through the construction of the pathological priori knowledge base, the labeling of five basic lesion indicators on the liver biopsy sections of H&E, Masson trichrome and HBsAg immunohistochemistry is realized, and the local lesion classification and the whole slice level lesion grading are realized, and the construction of the liver biopsy section basic lesion evaluation system is realized.

[0104] 2. Based on the multi-scale feature extraction and fusion model, accurate tile-level basic lesion grading is realized, which provides a premise for accurate whole slice level basic lesion diagnosis.

[0105] In order to verify the superiority of the method proposed in the present application, objective evaluation and subjective evaluation are used to highlight the performance. The objective evaluation is carried out on the private constructed data set and a public data set. The tile-level classification results on the private data set and the public data set are as follows. From the accuracy and F1 score, the method of the present application is superior to all other methods in most indicators.

[0106] In order to verify the superiority of the method proposed in the present application, objective evaluation and subjective evaluation are used to highlight the performance. The objective evaluation is carried out on the private constructed data set and a public data set. The tile-level classification results on the private data set and the public data set are as follows. From the accuracy and F1 score, the method of the present application is superior to all other methods in most indicators.

[0107] Table 2

[0108]

[0109] Table 3

[0110]

[0111]

[0112] Subjective evaluation is also conducted on private and public datasets, using Grad-CAM-based visualization method to highlight the lesion-related areas that the model focuses on, and the results are as follows Figures 4A to 4E .

[0113] Combining objective evaluation and subjective evaluation, the advantages of the method are: on the one hand, the method is based on multi-scale feature extraction and fusion, which realizes efficient extraction of pathological features of the graph block, thereby realizing accurate classification of local lesions at the graph block level; on the other hand, the method can accurately locate the relevant areas related to the lesion, thereby providing the model with explainability.

[0114] 3. Based on prediction aggregation, accurate basic lesion grading at the whole slice level is realized. The grading results are as shown in Table 4 and Figure 5 :

[0115] Table 4

[0116]

[0117]

[0118] Table 4 and Figure 5 The grading results shown in Table 4 and Table 5 show that the method (MAPred) proposed in the application is superior to other methods in most indicators. The advantages of the method are: on the one hand, based on graph block level prediction for whole slice level aggregation, the steps are simple and the computational complexity is low; on the other hand, the aggregation method models the diagnosis process of the pathologist, realizing the balance between generalization performance and explainability.

[0119] The embodiment of the application also provides a storage medium for storing a computer program, which is executed to at least perform the method described above.

[0120] The embodiment of the application also provides a control device, which includes a processor and a storage medium for storing a computer program; wherein the processor is used to execute the computer program to at least perform the method described above.

[0121] The embodiment of the application also provides a processor, which executes a computer program to at least perform the method described above.

[0122] The storage medium can be implemented by any type of nonvolatile storage device, or a combination thereof. The nonvolatile memory can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Ferromagnetic Random Access Memory (FRAM), a Flash Memory, a magnetic surface storage, an optical disc, or a Compact Disc Read-Only Memory (CD-ROM). The magnetic surface storage can be a magnetic disc memory or a magnetic tape memory. The storage medium described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable type of memory.

[0123] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. The described device embodiments are merely schematic, and the division of the units is merely a logical function division. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or in other forms.

[0124] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0125] In addition, each functional unit in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate physical unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0126] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program performs the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0127] Alternatively, the integrated unit of the present application can be stored in a computer readable storage medium if it is realized in the form of a software function module and sold or used as an independent product. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes a mobile storage device, a ROM, a RAM, a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0128] The methods disclosed in the several method embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0129] The features disclosed in the several product embodiments of the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0130] The features disclosed in the several method or device embodiments of the present application can be combined arbitrarily without conflict to obtain new method or device embodiments.

[0131] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A liver biopsy slide ground truth assessment method based on multi-scale feature extraction, characterized in that, The following steps are included in the processing on the computer: S0, constructing a stained section image dataset, randomly intercepting patches from whole slide images of the dataset to obtain high-resolution patches; S1, labeling the pathological features of the patches to construct a pathological prior knowledge base; S2, constructing and training a multi-scale local lesion grading model based on ResNet using stained section images and corresponding labeled data; using image scaling technology to obtain corresponding low-resolution patches from the high-resolution patches, using the multi-scale local lesion grading model to extract multi-scale deep features of the high-resolution patches and the low-resolution patches for fusion, and then classifying the multi-scale fusion features to realize local lesion grading at the patch level; Step S2 specifically includes: S2-0, using a pre-trained ResNet-34 to extract multi-scale deep features of the high-resolution patches and the low-resolution patches; S2-1, using adaptive pooling to downsample high-resolution scale features to the same scale as low-resolution scale features, and adopting a channel maximum difference method to fuse features of two scales extracted from the high-resolution patches and the low-resolution patches; S2-2, using a pooling layer and a fully connected layer to classify multi-scale fusion features to realize local lesion grading based on patches; S3, constructing a global lesion assessment model based on prior guidance, aggregating local lesion grading results to realize whole section level lesion grading.

2. The method for liver biopsy slide ground-truth assessment based on multi-scale feature extraction as claimed in claim 1, wherein, Step S0 specifically includes: S0-0, scanning H&E sections, Masson trichrome sections, and HBsAg immunohistochemical sections collected from liver biopsy cases to obtain whole slide images; S0-1, using a random intercept algorithm to select patches from whole slide images obtained by scanning the sections for training of local lesion grading models for balloon-like changes, inflammation activity, steatosis, fibrosis, and HBsAg positive lesions.

3. The method for liver biopsy slide ground-truth assessment based on multi-scale feature extraction as claimed in claim 2, wherein, In step S1, the degrees of balloon-like changes, inflammation activity, and steatosis of H&E patches are labeled, the degree of fibrosis of Masson trichrome patches is labeled, and the degree of HBsAg positivity of HBsAg immunohistochemical images is labeled.

4. The method of claim 1 to 3, wherein the method of liver biopsy slide ground-truth assessment based on multi-scale feature extraction is characterized by, In step S2, the image scaling technology includes a bilinear interpolation method.

5. The method of claim 1 to 3, wherein the method of liver biopsy slide ground truth assessment based on multi-scale feature extraction, further comprises: In step S2, the loss function of the training process includes two parts of classification loss and similarity loss, wherein the classification loss adopts a weighted cross-entropy loss function , ； wherein, is the total number of high resolution patches inputted, is the true label of the th patch, is the result of direct classification by the fully connected layer after the feature extractor of the high resolution image, For multi-scale similarity loss L1-Norm of the corresponding two patches is calculated ； wherein, and is a feature map size of a low resolution single picture, is a feature map of a channel of a position, is a feature map of the high resolution patch after adaptive pooling, is a feature map of the low resolution patch, The total loss function is ; 。 6. The method of claim 1 to 3, wherein the method of liver biopsy slide ground-truth assessment based on multi-scale feature extraction, further comprises: In step S3, based on local lesion grading results, a prediction mean aggregation strategy is used to realize whole section level lesion grading.

7. The method of claim 1 to 3, wherein the method of liver biopsy slide ground-truth assessment based on multi-scale feature extraction, further comprises: In step S3, the aggregation rules include determining the optimal patch proportion threshold between each level of WSI level basic lesions based on an adaptive threshold method.

8. The method for liver biopsy slide ground-truth assessment based on multiscale feature extraction as claimed in claim 7, wherein, In step S3, determining the optimal patch proportion threshold between each level of WSI level basic lesions includes determining patch proportion thresholds for balloon-like changes, inflammation activity, steatosis, fibrosis, and HBsAg positive lesions.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the liver biopsy section basic lesion assessment method based on multi-scale feature extraction as claimed in any one of claims 1-8.

10. A computer program product, characterised in that, The computer program product, when executed by a processor, implements the liver biopsy section basic lesion assessment method based on multi-scale feature extraction as claimed in any one of claims 1-8.

Images