Full-slide liver pathologic image fatty degeneration evaluation method based on deep learning

Abstract

The invention relates to the technical field of deep learning, in particular to a full-slide liver pathology image fatty degeneration evaluation method based on deep learning, which comprises the following steps of: firstly, obtaining a tile screening score by combining various morphological characteristics and color characteristics, and automatically and quickly screening tiles of a full-slide liver pathology image through the score to obtain a full-slide liver pathology image fatty degeneration evaluation result; and then segmenting the interested tiles by using a deep learning technology to quickly obtain an accurate fat score of each tile, and further obtaining a fat score of the full-slide liver pathology image. By quickly generating the standardized input data, the deep learning model is helped to efficiently process massive full-slide liver pathologic image resources in a medical scene, and the fatty degeneration degree in the full-slide liver pathologic image can be objectively, accurately and quickly quantified, so that the fat quantification precision and efficiency of the full-slide liver pathologic image are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and particularly to a method for evaluating steatosis of whole-slide liver pathological images based on deep learning. Background Art

[0002] Fatty liver is the abnormal accumulation of triglycerides in hepatocytes and is the most common chronic liver disease, affecting approximately 25% of the world's population. According to the severity of the lesion, fatty liver can progress from simple steatosis to steatohepatitis and may ultimately progress to cirrhosis. In clinical practice, accurately evaluating the degree of liver steatosis using whole-slide liver pathological images is crucial for disease diagnosis and treatment plan formulation. Especially during the evaluation process of donor liver transplantation, the quantitative analysis of steatosis directly determines the availability of the donor liver. Therefore, developing an efficient and accurate method for quantifying steatosis has become an important research direction in the field of pathology.

[0003] Deep learning techniques developed in recent years have shown excellent performance in biomedical image analysis, especially in small-scale region-of-interest segmentation tasks. However, there are still many challenges when directly applying these techniques to the quantitative analysis of steatosis in whole-slide liver pathological images: 1. High computational resource consumption: Classical segmentation models such as U-Net perform well in processing small-scale regions of interest, but in whole-slide level analysis, due to the need to process ultra-high-resolution images, the computational resource consumption is huge, and the GPU video memory is insufficient to support the inference of the entire whole-slide liver pathological image; 2. High training cost: The cost of high-quality pathological-level pixel annotation is extremely high, and existing publicly available fatty liver pathological datasets are scarce. For example, there is less annotation data for steatosis in the TCGA database. Compared with the traditional method of visual assessment by pathologists, deep learning techniques can provide more objective, accurate, and quantitative assessment results, but the segmentation operation of whole-slide images is complex, time-consuming, and laborious, and full automation has not been achieved yet. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a method for evaluating steatosis of whole-slide liver pathological images based on deep learning. The fractional pre-screening method is used to quickly exclude the tissue parts that the model is not interested in, and the remaining regions of interest are input into the deep learning model to calculate the fat fraction (FF), thereby improving the accuracy and efficiency of fat quantification in whole-slide liver pathological images.

[0005] The technical problems to be solved by the present invention are realized by the following technical solutions:

[0006] A method for evaluating steatosis of whole-slide liver pathological images based on deep learning, comprising the following steps:

[0007] Step (1): Collect whole-slide liver pathology images, construct a tile image dataset, and then train an optimal segmentation model UNet++ to obtain the best segmentation effect;

[0008] Step (2): Calculate the tile image dataset obtained in step (1), and extract the tissue proportion p, morphological factor f mor , color factor f col , saturation factor f sat and tissue proportion factor f qua ;

[0009] Step (3): Reasonably weight the various factors obtained in step (2) to obtain a tile score formula, and calculate the screening score score of each tile;

[0010] Step (4): Use the optimal segmentation model UNet++ obtained in step (1) and the tile score formula obtained in step (3) to process the whole-slide liver pathology image data, automatically extract the region of interest, segment the steatosis region, and calculate the fat score of the whole-slide liver pathology image in combination with the tile score.

[0011] As a further improvement of the present invention, step (1) specifically includes the following process:

[0012] Step (11): Cut the whole-slide liver pathology image into non-overlapping tile images with a pixel size of 2048×2048, and perform fat droplet instance annotation on the tile images to obtain a tile and mask image dataset;

[0013] Step (12): Divide the tile image dataset into a training set, a validation set, and a test set according to a specific ratio, and then use a parameter tuning method based on Bayesian optimization to train the optimal segmentation model UNet++ to obtain the best segmentation effect.

[0014] As a further improvement of the present invention, the tile tissue proportion p in step (2) is obtained from the ratio of the tissue in each tile to the number of pixel points contained in the tile, and is calculated by formula (1):

[0015]

[0016] In formula (1), n1 is the number of pixel points of the tissue in the mask image, and n2 is the total number of tile pixel points.

[0017] As a further improvement of the present invention, the morphological factor f mor in step (2) is obtained from the number of circles detected in each tile, and is calculated by formula (2):

[0018] f mor = 1 - e -k·circle, circle ≥ 0 (2)

[0019] In formula (2), circle is the number of circles detected by the Hough gradient method. Since fat droplets are approximately circular, the number of fat droplets is roughly estimated by detecting the number of circles in the slide. k is the growth rate coefficient, and the larger the value, the faster the growth.

[0020] As a further improvement of the present invention, in step (2), the color factor f col is obtained by converting the proportion of lipid droplet pixels in each slide, and is calculated by formulas (3), (4), and (5):

[0021]

[0022] C adjusted = C raw ·(1 - e -m·p ) (4)

[0023]

[0024] In formula (3), C raw is the initial color factor, n2 is the total number of pixel points in the slide, n3 is the number of white pixel points in the slide. Formula (4) indicates dynamically adjusting the weight of effective white pixels according to the tissue proportion. C adjusted is the adjusted color factor, m is the tissue proportion sensitivity coefficient. The larger the value of m, the stronger the suppression of low tissue proportion. p is the tissue proportion. Formula (5) indicates that when the proportion of white pixels exceeds the theoretical maximum value of the tissue area, non-linear penalty is activated.

[0025] As a further improvement of the present invention, in step (2), the saturation factor f sat is obtained by calculating the standard deviation of the saturation s and lightness v of all pixels after converting the real image from the RGB color space to the HSV color space, and is calculated by formula (6):

[0026]

[0027] In formula (6), s' represents the standard deviation of the saturation of the slide, v' represents the standard deviation of the lightness of the slide, T s and T v are the dynamic thresholds of the whole slide.

[0028] As a further improvement of the present invention, T s and T v are the dynamic thresholds of the whole slide, and are calculated by formulas (7) and (8):

[0029] T s = μ s - 2σ s (7)

[0030] T v = μ v - 2σ v (8)

[0031] In formulas (7) and (8), μ s and μ v respectively represent the mean saturation and mean brightness of the whole-slide liver pathological image, and σ s and σ v respectively represent the standard deviation of saturation and the standard deviation of brightness of the whole-slide liver pathological image.

[0032] As a further improvement of the present invention, the tissue proportion factor f qua in step (2) is the result of further distributing weights for the tissue proportion, and is calculated by formula (9):

[0033]

[0034] As a further improvement of the present invention, step (3) specifically includes the following process:

[0035] Step (31), using the objective weighting method to process the morphological features (f mor , f qua ) and color features (f col , f sat ) extracted in step (2);

[0036] Step (32), automatically calculating the standard deviation and correlation coefficient between various factors of multiple groups of tiles, and obtaining the weight w1 of the morphological features and the weight w2 of the color features by the objective weighting method. The weight w1 of the morphological features and the weight w2 of the color features obtained from the same whole-slide image are a group;

[0037] Step (33), obtaining the tile score formula (10), and the screening score score of each tile is calculated by formula (10):

[0038]

[0039] In formula (10), the value of score ranges from 0 to 1, and ∈ = 10 -6 is a smoothing term to prevent division by zero.

[0040] As a further improvement of the present invention, step (4) specifically includes the following process:

[0041] Input the tiles of interest obtained in step (3) into the optimal segmentation model UNet++, use this model to automatically classify and label each pixel in each tile image of interest, obtain the segmentation image and fat fraction of a single tile, and then integrate them by area weighted average to obtain the fat fraction of the whole-slide liver pathological image.

[0042] The beneficial effects of the present invention are:

[0043] The present invention provides a method for evaluating steatosis of whole-slide liver pathological images based on deep learning. By quickly generating standardized input data, it helps the deep learning model to efficiently process a large number of whole-slide liver pathological image resources in the medical scenario, and can objectively, accurately and quickly quantify the degree of steatosis in the whole-slide liver pathological image, thereby improving the accuracy and efficiency of fat quantification of the whole-slide liver pathological image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention will be further described below with reference to the drawings and embodiments:

[0045] Figure 1 is a flow chart of the present invention;

[0046] Figure 2 is a heat map of a whole-slide image after being screened and processed by the present invention;

[0047] Figure 3 is the consistency analysis of the FF predicted by the model and the true FF. Left figure: fitting analysis, right figure: Bland-Altman analysis. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] In order to make the technical means, creative features, achieved purposes and effects of the present invention easy to understand, the present invention will be further described below with reference to the drawings and embodiments.

[0049] Embodiment 1

[0050] As Figure 1 shown, a method for evaluating steatosis of whole-slide liver pathological images based on deep learning includes the following steps:

[0051] Step (1), collect whole-slide liver pathological images, construct a tile image dataset, and then train an optimal segmentation model UNet++ to obtain the best segmentation effect.

[0052] Step (1) specifically includes the following processes:

[0053] Step (11): Use 30 whole-slide liver pathological images of 30 objects, cut them into non-overlapping tile images with a pixel size of 2048×2048, and perform instance annotation of fat droplets on the tile images to obtain a tile and mask image dataset. It should be noted that in this embodiment, the instance annotation of fatty degeneration on the tile images is performed by three pathologists, and a two-level review mechanism (double-blind annotation, differential arbitration) is adopted to ensure the annotation quality.

[0054] Step (12): Divide the tile image dataset into a training set, a validation set, and a test set according to the ratio of 8:1:1, and then use the parameter tuning method based on Bayesian optimization to train the optimal segmentation model UNet++, so as to obtain the best segmentation effect. It should be noted that the Bayesian optimization method is used to synchronously optimize the Dice coefficient, Hausdorff distance, and inference speed to realize the optimization of non-parametric models.

[0055] Step (2): Calculate based on the tile image dataset generated in step (1), and extract the tissue ratio p, morphological factor f mor , color factor f col , saturation factor f sat and tissue ratio factor f qua .

[0056] In step (2), the tissue ratio p of the tile is obtained by the ratio of the tissue in each tile to the number of pixel points contained in the tile, and is calculated by formula (1):

[0057]

[0058] In formula (1), n1 is the number of pixel points of the tissue in the mask image, and n2 is the total number of pixel points of the tile.

[0059] In step (2), the morphological factor f mor is obtained from the number of circles detected in each tile, and is calculated by formula (2):

[0060] f mor = 1 - e -k·circle , circle≥0 (2)

[0061] In formula (2), circle is the number of circles detected by the Hough gradient method. Since fat droplets are approximately circular, the number of fat droplets is roughly estimated by detecting the number of circles in the tile. k is the growth rate coefficient, and the larger the value, the faster the growth. In this embodiment, preferably k = 0.4.

[0062] In step (2), the color factor f col is obtained by converting the ratio of the pixels of fat droplets in each tile, and is calculated by formulas (3), (4), and (5):

[0063]

[0064] C adjusted = C raw ·(1 - e -m·p ) (4)

[0065]

[0066] In Equation (3), C raw is the initial color factor, n2 is the total number of tile pixels, n3 is the number of white tile pixels. Equation (4) represents dynamically adjusting the weight of effective white pixels according to the tissue proportion. C adjusted is the adjusted color factor, m is the tissue proportion sensitivity coefficient. The larger the value of m, the stronger the inhibition of low tissue proportion. p is the tissue proportion. In this embodiment, preferably m = 0.7. Equation (5) represents starting non - linear penalty when the proportion of white pixels exceeds the theoretical maximum value of the tissue area.

[0067] In step (2), the saturation factor f sat is obtained by calculating the standard deviations of the saturation s and lightness c of all pixels after converting the real - image from the RGB color space to the HSV color space, and is calculated by Equation (6):

[0068]

[0069] In Equation (6), s' represents the standard deviation of tile saturation, v' represents the standard deviation of tile lightness, T s and T v are the whole - slide dynamic thresholds. T s and T v are the whole - slide dynamic thresholds, and are calculated by Equations (7) and (8):

[0070] T s = μ s - 2σ s (7)

[0071] T v = μ v - 2σ v (8)

[0072] In Equations (7) and (8), μ s and μ v respectively represent the mean saturation and mean lightness of the whole - slide liver pathological image, and σ s and σ v respectively represent the standard deviation of saturation and standard deviation of lightness of the whole - slide liver pathological image.

[0073] In step (2), the tissue proportion factor f quaIt is the result of further assigning weights to the tissue proportion, and is calculated by Equation (9):

[0074]

[0075] Step (3): Reasonably assign weights to each factor calculated in Step (2) to obtain the tile score formula, and calculate the screening score score of each tile.

[0076] Step (3) specifically includes the following process:

[0077] Step (31): Process the morphological features and color features extracted in Step (2) using the objective weighting method.

[0078] Step (32): Automatically calculate the standard deviation and correlation coefficient between each factor of multiple groups of tiles, and obtain the weight w1 of the morphological features and the weight w2 of the color features through the objective weighting method. The weight w1 of the morphological features and the weight w2 of the color features obtained from the same whole slide image are a group. It should be noted that since the overall brightness and color distribution of the same whole slide image are relatively consistent, and the tissue features of different tiles in the image are similar (for example, they are all liver sections), for a single whole slide image, only one calculation of the weights is required. The weights in the present invention are calculated by the objective weighting method, which is one of the existing technologies. Those skilled in the art should be able to calculate the weight w1 of the morphological features and the weight w2 of the color features using this method according to the existing description, so it will not be elaborated in detail here.

[0079] Step (33): Obtain the tile score formula (10), and the screening score score of each tile is calculated by Equation (10):

[0080]

[0081] In Equation (10), the value of score ranges from 0 to 1, and ∈ = 10 -6 It is a smoothing term to prevent division by zero.

[0082] When the value of score is 0, the tile image completely does not contain fat droplets, and the model is completely not interested; when the value of score is 1, the tile image contains more fat droplets, and the model is most interested.

[0083] Step (4): Process the whole slide image data using the optimal segmentation model UNet++ obtained in Step (1) and the tile score formula obtained in Step (3), automatically extract the region of interest, segment the fatty degeneration region, and calculate the fat fraction of the whole slide liver pathological image in combination with the tile score.

[0084] Step (4) specifically includes the following process:

[0085] Input the tiles of interest obtained in step (3) into the optimal segmentation model UNet++. The optimal segmentation model UNet++ automatically classifies and labels each pixel in each tile image of interest, obtains the segmentation image and fat fraction of a single tile, and then integrates them through the area weighted average method to obtain the fat fraction of the whole-slide liver pathological image.

[0086] Example Two

[0087] The difference from Example One is as follows:

[0088] In step (13), a UNet++ model is trained using a dataset of 900 tile images. Before training, the tile image data is preprocessed. All tile images are denoised by median filtering, and histogram equalization is used to enhance image details to improve image quality. The dataset is randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1, and the datasets do not cross each other. The Bayesian method is used to optimize and automatically adjust the hyperparameters to obtain the optimal segmentation model.

[0089] In steps (2) and (3), the tile image dataset obtained in step (1) is used for calculation to obtain the tissue proportion, morphological factors, color factors, saturation factors, and tissue proportion factors of each tile, and a tile screening score formula is combined. A screening score is assigned to each tile to obtain a heat map of the degree of interest of the model, as Figure 2 shown. The darker the color, the higher the degree of interest. In addition, in step (3), the weight w1 of the morphological features obtained is 0.1, and the weight w2 of the color features is 0.52.

[0090] In step (4), the inference time is shortened from 123 seconds to 30 seconds. The final fat fraction of the whole slide is 0.4%, and the fatty degeneration grade is grade 0, which is consistent with the diagnosis result of the pathologist. In addition, as Figure 3 shown, the FF value predicted by the model has a high consistency with the true FF value (Dice = 0.97), indicating that the model has high accuracy in the task of whole-slide liver pathological image segmentation.

[0091] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art of this industry should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification is only the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning, characterized by: The following steps are involved: Step (1), collect whole slide liver pathology images, construct a tile image dataset, and then train an optimal segmentation model UNet++; Step (2), calculate the tile image data set obtained in step (1) to extract the tissue proportion p and morphological factor f of each tile mor , color factor f col , saturation factor f sat and tissue proportion factor f qua ; Step (3), weighting each factor obtained in step (2), obtaining a tile score formula, and calculating the screening score of each tile; Step (4), using the optimal segmentation model UNet++ obtained in step (1) and the tile score formula obtained in step (3) to process the whole slide liver pathology image data, automatically extract the region of interest, segment the fatty degeneration area, and calculate the fat score of the whole slide liver pathology image in combination with the tile score.

2. The method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning according to claim 1, characterized in that: Step (1) specifically includes the following process: Step (11), dividing the whole slide liver pathology image data into non-overlapping tile images with a pixel size of 2048×2048, annotating the tile images with fat droplet instances, and obtaining a tile and mask image dataset; Step (12), divide the tile image dataset into a training set, a validation set, and a test set in a ratio of 8:1:1, and then use the Bayesian optimization-based parameter adjustment method to train the optimal segmentation model UNet++ to obtain the best segmentation effect.

3. The method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning according to claim 2, characterized in that: In step (2), the tile tissue ratio p is obtained by the ratio of the tissue in each tile to the number of pixels contained in the tile, which is calculated by formula (1): In formula (1), n1 is the number of pixels of the tissue in the mask image, and n2 is the total number of tile pixels.

4. The method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning according to claim 2, characterized in that: The morphological factor f in step (2) mor The number of circles detected in each tile is obtained by calculating the formula (2): f mor =1-e -k·circle ,circle≥0 (2) In formula (2), circle is the number of circles detected by the Hough gradient method, and k is the growth rate coefficient. The larger the value, the faster the growth.

5. The method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning according to claim 2, characterized in that: The color factor f in step (2) col It is calculated by the proportion of fat droplet pixels in each tile and obtained through equations (3), (4) and (5): C adjusted =C raw ·(1-e -m·p ) (4) In formula (3), C raw is the initial color factor, n2 is the total number of tile pixels, n3 is the number of white pixels in the tile, and formula (4) represents the dynamic adjustment of the effective white pixel weight by tissue proportion, C adjusted is the adjusted color factor, m is the tissue ratio sensitivity coefficient, the larger the value of m is, the stronger the suppression of low tissue ratio is, p is the tissue ratio, and formula (5) indicates that when the proportion of white pixels exceeds the theoretical maximum value of the tissue area, the nonlinear penalty is activated.

6. The method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning according to claim 2, characterized in that: Saturation factor f in step (2) sat It is obtained by converting the real image from RGB color space to HSV color space and calculating the standard deviation of saturation s and brightness v of all pixels, which can be calculated by formula (6): In formula (6), s' represents the standard deviation of tile saturation, v' represents the standard deviation of tile brightness, T s and T v is the dynamic threshold of the whole slide.

7. The method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning according to claim 6, characterized in that: T s and T v is the dynamic threshold of the whole slide, which is calculated by equations (7) and (8): T s =μ s -2s s (7) T v =μ v -2σ v (8) In formula (7) and formula (8), μ s and μ v They represent the saturation mean and brightness mean of the whole slide liver pathology image, σ s and σ v They represent the saturation standard deviation and brightness standard deviation of whole-slide liver pathology images, respectively.

8. The method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning according to claim 2, characterized in that: The tissue ratio factor f in step (2) qua is the result of further weighting of the organizational proportion, calculated by formula (9):

9. The method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning according to claim 1, characterized in that: Step (3) specifically includes the following process: Step (31), using the objective weighting method to weight the morphological features (f mor 、f qua ) and color characteristics (f col 、f sat ) for processing; Step (32), automatically calculating the standard deviation and correlation coefficient between the factors of multiple groups of tiles, obtaining the weight w1 of the morphological feature and the weight w2 of the color feature by the objective weighting method, and the morphological feature weight w1 and the color feature weight w2 obtained from the same whole slide liver pathology image are grouped together; Step (33), obtain the tile score formula (10), and calculate the screening score of each tile: In formula (10), the value of score is between 0 and 1, ∈ = 10 -6 Smoothing term to prevent division by zero.

10. The method for assessing fatty degeneration in whole-slide liver pathological images based on deep learning according to claim 1, characterized in that: Step (4) specifically includes the following process: The tiles of interest obtained in step (3) are input into the optimal segmentation model UNet++, and the model is used to automatically classify and label each pixel in each tile image of interest to obtain the segmented image and fat score of a single tile, which are then integrated through the area weighted average method to obtain the fat score of the whole slide liver pathology image.

Images