Lipid droplet segmentation methods and electronic devices for pathological slide images

Abstract

This invention proposes a method and electronic device for lipid droplet segmentation in pathological slide images. The lipid droplet segmentation method includes: inputting the pathological slide image into a machine learning model to output a region map and an edge map, wherein the region map includes multiple region probability values ​​and the edge map includes multiple edge probability values; and segmenting multiple lipid droplets in the pathological slide image based on the region probability values ​​and edge probability values. This allows for the objective identification of lipid droplets.

Description

Technical Field

[0001] This disclosure relates to a method and electronic device for segmenting lipid droplets in pathological tissue images. Background Technology

[0002] In clinical practice, pathologists use hematoxylin and eosin (H&E) staining to observe different structures in tissue sections. Several indicators have been proposed for diagnosing non-alcoholic fatty liver disease (NAFLD), with steatosis being one of the most important. When steatosis occurs, excess fat is stored in hepatocytes as lipid droplets, which appear as white bubbles in stained images. In current diagnostic methods, doctors analyze the degree or type of steatosis by observing the density and type of lipid droplets under a microscope. This process is not only quite crude and time-consuming, but the diagnostic values ​​are also subjective and prone to error. Therefore, there is an urgent need for an automated lipid droplet quantification system to assist doctors in the diagnostic process. Summary of the Invention

[0003] The embodiments disclosed herein propose a lipid droplet segmentation method for pathological slide images, applicable to computer systems. This segmentation method includes: inputting the pathological slide image into a machine learning model to output a region map and an edge map, the region map including multiple region probability values, and the edge map including multiple edge probability values; segmenting multiple lipid droplets in the pathological slide image based on the region probability values ​​and edge probability values.

[0004] In some embodiments, the machine learning model is a convolutional neural network (CNN) with a U-network architecture. This U-network architecture includes multiple downsampling and multiple upsampling operations. At the bottom layer of the U-network architecture, the CNN has receptive field squares, each containing multiple convolutional kernels, at least one of which is a dilation convolutional kernel.

[0005] In some embodiments, the output of the machine learning model includes a first branch and a second branch, the first branch being used to output a region map and the second branch being used to output an edge map. The lipid droplet segmentation method further includes: training the first branch using labeled pathological slide images during machine learning model training; and training the second branch by subtracting the result of morphological erosion processing from the result of morphological dilation processing of the labeled pathological slide images.

[0006] In some embodiments, the step of segmenting lipid droplets based on region probability values ​​and edge probabilities includes: determining whether the probability value of each region is greater than or equal to a first threshold to generate a mask; using the region probability values ​​and the edge probability values ​​corresponding to the same location to calculate multiple center scores and gradients; generating multiple markers based on the center scores; and performing a watershed algorithm based on the mask, gradients, and markers.

[0007] In some embodiments, the method further includes: calculating the average edge probability value of each lipid droplet as the edge score of the corresponding lipid droplet; and deleting the corresponding lipid droplet if the edge score is less than a third threshold value.

[0008] From another perspective, the embodiments disclosed herein propose an electronic device comprising multiple modules to perform the above-described lipid droplet segmentation method.

[0009] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a method for lipid droplet segmentation of pathological slide images according to an embodiment;

[0011] Figure 2 This is a schematic diagram illustrating the architecture of a convolutional neural network according to an embodiment;

[0012] Figure 3 This is a schematic diagram illustrating the architecture of a sensing field block according to an embodiment;

[0013] Figure 4 This is a flowchart illustrating the watershed algorithm according to one embodiment;

[0014] Figure 5 This is a schematic diagram illustrating the masking, gradient, and markers in the watershed algorithm according to one embodiment;

[0015] Figure 6 This is a schematic diagram illustrating the removal of lipid droplets according to one embodiment;

[0016] Figure 7 This is a schematic diagram illustrating an electronic device according to one embodiment.

[0017] [Symbol Explanation]

[0018] 100: Lipid Droplet Segmentation Method

[0019] 110: Pathological slide images

[0020] 120: Result

[0021] 101-103: Steps

[0022] 210, 220: Feature Map

[0023] 231: Regional Map

[0024] 232: Edge Graph

[0025] 240: Experience Wild Blocks

[0026] 301~310: Convolutional kernel

[0027] 401-404: Steps

[0028] 501: Mask

[0029] 502: Gradient

[0030] 503, 504: Marks

[0031] 505: Result

[0032] 610, 620, 630: Context

[0033] 611, 621, 631, 632: Lipid droplet regions

[0034] 612,622,633: Area

[0035] AA',BB',CC',DD',EE',FF': Tangents

[0036] 700: Electronic Devices

[0037] 710: Processor

[0038] 720: Memory Detailed Implementation

[0039] The terms "first," "second," etc., used in this article do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms.

[0040] In this embodiment, a Leica scanner is used to scan H&E-stained liver tissue at 40x magnification to obtain a whole slide image (WSI). Each WSI image is segmented into tens of thousands of partially overlapping images, each 512 pixels x 512 pixels in size. These segmented images are referred to below as pathological slide images. However, in other embodiments, any suitable device can be used to obtain slide images of any organ or tissue. This disclosure does not limit the size of these pathological slide images or whether the pathological slide images overlap.

[0041] Figure 1 This is a flowchart illustrating a method for lipid droplet segmentation in pathological slide images, based on an embodiment. Please refer to... Figure 1 The lipid droplet segmentation method 100 disclosed herein is used to segment lipid droplets from a pathological slide image 110 and output a result 120, in which the green lines represent the identified lipid droplets. The lipid droplet segmentation method 100 can be executed by any suitable computer system and includes steps 101 to 103, which will be described in detail below.

[0042] First, in step 101, the pathological slide image 110 is input into a machine learning model to output a region map and an edge map. In this embodiment, the machine learning model is a convolutional neural network with a U-network architecture, for example... Figure 2 As shown in the diagram, the squares represent feature maps, and the numbers on the feature maps represent length x width x depth. For example, feature map 210 is the input, and feature map 220 is 512x512x16 in size, and so on. The output of this network is region map 231 and edge map 232, both of which are 512x512x1 in size. Region map 231 includes multiple region probability values ​​to indicate the probability that each pixel is classified as a lipid droplet region. Edge map 232 includes multiple edge probability values ​​to indicate the probability that each pixel is classified as a lipid droplet edge. Here, the arrows from the left to the right of region map 231 and edge map 232 represent convolutional layers, containing 1x1 convolutional kernels and a sigmoid function. The remaining arrows to the right also represent convolutional layers, but they contain 3x3 convolutional kernels with a stride of 1, and also include batch normalization (BN) and activation (AC). In this embodiment, the Mish function is used as the activation function. The arrow in the lower right corner indicates downsampling, which in practice can include a 3x3 convolutional kernel with a stride of 2 to achieve downsampling. It also includes batch normalization and activation functions. The arrow pointing upwards indicates upsampling, which can be implemented using linear functions or any other suitable algorithm. This indicates a connection of feature maps, meaning that two feature maps are joined together to form a feature map with greater depth. In other words, the network architecture used here includes 4 downsampling and 4 upsampling operations, but this architecture is only an example, and other network architectures can be used in other embodiments.

[0043] Notably, at the very bottom layer of this network architecture is a Receptive Field Block (RFB) 240, the details of which are as follows: Figure 3As shown, the receptive field block 240 includes convolutional kernels 301-310. The input from the previous layer is first processed by convolutional kernel 301, which has a kernel size of 1x1, and then processed by convolutional kernels 302-305. Convolutional kernel 302 has a size of 1x1, convolutional kernel 303 has a size of 3x3, convolutional kernel 304 has a size of 5x5, and convolutional kernel 305 has a size of 7x7. The outputs of convolutional kernels 302-305 are then fed into convolutional kernels 306-309 for further processing. Notably, convolutional kernels 306-309 are dilation convolutional kernels, with convolutional kernel 306 having a dilation rate of 1x, convolutional kernel 307 having a dilation rate of 3x, convolutional kernel 308 having a dilation rate of 5x, and convolutional kernel 309 having a dilation rate of 7x. All of these convolutional kernels 306-309 have a size of 3x3. For details regarding the expansion of the convolutional kernels, please refer to the paper "Receptive field block net for accurate and fast object detection," Proceedings of the European Conference on Computer Vision (ECCV), 2018. Details will not be elaborated here. Finally, the inputs of convolutional kernels 306-309 are concatenated, then processed by convolutional kernel 310, and finally concatenated with the inputs to form the output.

[0044] exist Figure 2 In the network architecture, the final output consists of two branches: the first branch (also called the region branch) outputs region map 231, and the second branch (also called the edge branch) outputs edge map 232. Different labels are used to train these two branches when training this network. Specifically, lipid droplet regions in pathological slide images are manually labeled by professionals, and these labeled pathological slide images are used to train the region branch. For the edge branch, morphological processes are used to generate the required labels, and the ground truth output of this branch is shown in Equation 1 below.

[0045] [Mathematical Expression 1]

[0046] GT B =Dilate(GT) R )-Erode(GT R )

[0047] Among them GT RThis represents the manually labeled lipid droplet region, which is the actual output of the first branch. `Dilate(·)` indicates two dilation processes, using a core size of, for example, 3x3. `Erode(·)` indicates two erosion processes, also using a core size of, for example, 3x3. The difference between the results of these two morphological processes is the actual output GT of the second branch. B From another perspective, since the edges of lipid droplets are not marked in typical manual labeling, this embodiment uses morphological methods to generate the realistic output needed for edge recognition. Expansion processing increases the size of the lipid droplet region (expanding outwards), while erosion processing causes the lipid droplet region to shrink inwards; therefore, the difference between these two can be considered the edge. In some embodiments, the lipid droplet region GT can also be directly... R The edge is used as the actual output of the second branch.

[0048] The loss function of the entire network consists of two components: region loss and edge loss, which can be expressed as the following mathematical formula 2.

[0049] [Mathematical Expression 2]

[0050] L = L R (p R ,y R )+L B (p B ,y B )

[0051] Where L represents the loss function of the entire network, L R The loss function representing the region branch, L B The loss function representing the marginal branches. p R The value predicted by the representative region branch, y R The actual output representing the branch in the region. B The value predicted by the marginal branch, y B This represents the actual output of the edge branch. In some embodiments, the loss function for the region branch can also be written as the following mathematical expressions 3-8.

[0052] [Mathematical Expression 3]

[0053] L R (p,y)=λ a L BCE (p,y)+λ b L tversky(0.5,0.5) (p,y)+λ c L activeContour (p,y)

[0054] [Mathematical Expression 4]

[0055] L BCE (p,y)=-(ylog(p)+(1-y)log(1-p))

[0056] [Mathematical Expression 5]

[0057]

[0058] [Mathematical Expression 6]

[0059] L activeContour (p,y)=L leng (p)+λL region (p,y)

[0060] [Mathematical Expression 7]

[0061]

[0062] [Mathematical Expression 8]

[0063] L region (p,y)=μp(y-1) 2 +(1-p)y 2

[0064] Where p is the value output by the corresponding branch, and y is the corresponding actual output. λ a , λ b , λ c The value is a real number and can be determined experimentally. Formula 4 represents cross-entropy, but the problem with this loss function is the imbalance between the foreground and background.

[0065] In mathematical formula 5, the loss function L tversky(α,β) Two parameters, α and β, can be used to adjust false positives (FP) and false negatives (FN). During training region branching, both parameters α and β are set to 0.5, meaning that false positives and false negatives are given the same weight. ∈ is a very small value to avoid division by zero.

[0066] In mathematical equations 6-8, a loss function L is added. activeContour While (p,y) doesn't show better performance in terms of data, it has a better visual effect on the human eye. This represents the derivative of the output value p in the x and y directions, respectively. λ and μ are real numbers, and their appropriate values ​​can be determined experimentally. The loss function L in mathematical formula 7... leng (p) can be seen as a penalty related to the length of the outline, thereby producing fewer broken complete objects.

[0067] On the other hand, the loss function for the marginal branches is shown in Equation 9 below. Where λ d , λ e It is a real number, and the appropriate value can be determined based on experiments.

[0068] [Mathematical Expression 9]

[0069] L B (p,y)=λ d L BCE (p,y)+λ e L tversky(0.1,0.9) (p,y)

[0070] Unlike region branches, edge branches require greater attention to false negatives for two reasons. First, the data characteristics are related to the sparser distribution and fewer numbers of edges compared to regions, leading to more conservative predictions from the model. This means the model needs additional motivation to predict edges. Second, the method described in this paper is crucial. Too many false negatives might prevent the separation of interconnected lipid droplets. Conversely, too many false positives, while important in this method, won't significantly negatively impact the final result. Therefore, when calculating the edge loss, the loss function L... tversky(α,β) In this setting, α is set to 0.1 and β is set to 0.9 to reduce false negatives.

[0071] The following will explain... Figure 1 Step 102 involves segmenting multiple lipid droplets in the pathological slide image based on the region probability value in the region map and the edge probability value in the edge map. In some embodiments, step 102 can employ the watershed algorithm. Roughly speaking, the watershed algorithm treats the entire image as a terrain surface, with different types of water injected into different locations and flowing towards low-lying areas. These water levels gradually rise until the entire area is filled, thus segmenting the original image into different types of water bodies, representing the segmentation results. In this way, each segmented object (i.e., lipid droplet) will have a precisely defined segmentation line with a pixel width. The detailed process of the watershed algorithm is as follows... Figure 4 As shown.

[0072] First, a mask needs to be calculated to indicate the location of lipid droplets (which may be two droplets connected together). Simultaneously, the water level in the watershed algorithm will only rise within the masked region. In step 401, it is determined whether the probability value of each region is greater than or equal to the first critical value. If it is, it represents a "1" for the mask; otherwise, it is a "0", as shown in the following mathematical formula 10.

[0073] [Mathematical Expression 10]

[0074]

[0075] Where τ is the first critical value, and Mask represents the masking effect. Figure 5 This is a schematic diagram illustrating masking, gradient, and labeling in the watershed algorithm according to one embodiment. Please refer to... Figure 4 and Figure 5 In this example, the calculated mask 501 contains two connected lipid droplets, and the graph on the left shows the mask value (1 or 0) along the tangent AA'. Alternatively, mask 501 represents the result of a typical segmentation algorithm that uses a fixed threshold to generate a binary map of foreground and background, with the foreground being lipid droplets. However, this foreground may contain multiple distinct but connected lipid droplets, thus requiring further segmentation using a watershed algorithm.

[0076] In the watershed algorithm, the dividing line between different objects always appears at a relatively high gradient (representing elevation). The next step is to calculate a suitable gradient. In step 402, the region probability value p is... r Subtract the marginal probability value p corresponding to the same position respectively b To calculate the center score p c The center score represents the probability that the corresponding pixel belongs to the center of the lipid droplet. When the model predicts a large region probability value and a small edge probability value, it means that the corresponding pixel is likely far from the edge of the lipid droplet. Conversely, if both the region and edge probabilities are large, the corresponding pixel may be in the middle of two lipid droplets. The negative value of the center score can be regarded as the gradient, that is, the center of the lipid droplet will have a lower gradient. The detailed calculation is shown in the following mathematical formulas 11 and 12, where Gradient represents the gradient.

[0077] [Mathematical Expression 11]

[0078] Gradient = -p c

[0079] [Mathematical Expression 12]

[0080] p c =p r -p b

[0081] exist Figure 5 The graph in the middle shows gradient 502, and the graph on the left shows the gradient along the tangent BB'. In this example, the gradient gradually decreases towards the center of the lipid droplet, forming two depressions.

[0082] In step 403, it is determined whether the center score is greater than or equal to the second critical value to generate multiple markers, as shown in the following mathematical formula 13.

[0083] [Mathematical Expression 13]

[0084]

[0085] Where τ c The second threshold is used, and the marker represents the starting point where the water level begins to rise. A higher center fraction indicates a pixel is more likely to be at the center of the lipid droplet. As the water level rises from the center of the droplet, it's more likely to form a dividing line between two droplets. Mathematical equation 13 yields many pixels. Within the same lipid droplet, multiple pixels may have center fractions greater than the second threshold. These pixels may be connected, and therefore, connected pixels can be considered the same type of water, or the same marker. Figure 5 In this example, two labels, 503 and 504, will be generated. In other embodiments, arbitrary detection algorithms can be performed on the center score to obtain the label. In summary, the label is generated based on the center score.

[0086] Next, in step 404, the watershed algorithm can be executed based on the mask, gradient, and markers calculated above. That is, within the masked area, different types of water rise from different markers, and at slightly higher gradients, different types of water will meet, and the point of meeting is the dividing line. Figure 5 The result 505 after segmentation is shown, which can separate two originally connected lipid droplets.

[0087] In principle, as long as the regional probability value p r With marginal probability value p b If both are large, it is more likely to be the middle of two connected lipid droplets. In some embodiments, step 102 can also be based on the center fraction p described above. c This is used to separate lipid droplets. For example, the center fraction p can be determined. c If the value is greater than a critical value, it is considered the foreground of the lipid droplet; otherwise, it is considered the background. In some embodiments, a region probability value p can also be used. r With marginal probability value p b The difference of any kind between them is taken as the center score p c For example, the difference of squares, the difference of absolute values, and the regional probability value p. r The square minus the marginal probability value p b The square of, etc. In some embodiments, the regional probability value p can also be calculated. r With marginal probability value p b The product or sum of these factors is then used to determine the foreground of the lipid droplet, and whether it is less than a critical value is used to identify the foreground. In other words, the region probability value p is used. r The marginal probability value p corresponding to the same position bThe center score can be calculated. Since the gradient is calculated based on the center score, it can also be said that the region probability value p is used. r The marginal probability value p corresponding to the same position b The gradient can be calculated. Those skilled in the art can design other methods based on the above, and the content of step 102 is not limited here.

[0088] Relying on deep learning models to segment lipid droplets can still lead to errors, for three reasons. First, some regions may not be lipid droplets but have a similar shape. Second, physical damage may occur during the slice image extraction process, making lipid droplets unrecognizable to the human eye for training. Third, some inputs may appear too infrequently in the training set, generating noise in the prediction; these situations can all result in fragmented masks. This embodiment proposes an edge score to quantify these uncertainties; please refer to [reference needed]. Figure 1 Step 103 involves calculating the edge score for each lipid droplet based on its edge probability value, and then deleting some lipid droplets based on their edge scores. The edge score is calculated as defined in the following mathematical formula 14.

[0089] [Mathematical Expression 14]

[0090]

[0091] BMS stands for Boundary Score. Let C represent the edge probability value corresponding to the pixel at position (i,j), and C be the set of pixels in a lipid droplet segmented by the watershed algorithm. Alternatively, Equation 14 calculates the average edge probability value for each lipid droplet as the corresponding edge score. If the edge score is less than the third threshold, the corresponding lipid droplet can be deleted. Figure 6 This is a schematic diagram illustrating the removal of lipid droplets according to one embodiment. Figure 6 The diagram illustrates three scenarios 610, 620, and 630. The top shows the detected lipid droplet regions 611, 621, 631, and 632, as well as regions with higher edge probability values ​​612, 622, and 633. The bottom shows the edge probability values ​​p along the tangents DD', EE', and FF', respectively. bScenario 610 illustrates a correct lipid droplet detection, where the detected lipid droplet range 611 is surrounded by the region 612 with a higher edge probability value. Scenarios 620 and 630 represent incorrect detections. In scenario 620, the detected lipid droplet range 621 is smaller, and the region 622 with a higher edge probability value is incomplete. In scenario 630, the detected lipid droplet regions 631 and 632 only overlap a small portion of the region 633 with a higher edge probability value. Scenario 610 calculates a larger edge score, while scenarios 620 and 630 calculate smaller edge scores. This method can be used to remove incorrect detections in scenarios 620 and 630.

[0092] Figure 7 This is a schematic diagram illustrating an electronic device according to one embodiment. Please refer to... Figure 7 The electronic device 700 can be a smartphone, tablet computer, personal computer, notebook computer, server, distributed computer, cloud server, industrial computer, or various electronic devices with computing capabilities, etc., and this invention is not limited thereto. The electronic device 700 includes a processor 710 and a memory 720. The processor 710 is communicatively connected to the memory 720, and this communication connection can be achieved through any wired or wireless communication means. The processor 710 can be a central processing unit, microprocessor, microcontroller, image processing chip, special application integrated circuit, etc. The memory 720 can be random access memory, read-only memory, flash memory, floppy disk, hard disk, optical disk, USB flash drive, magnetic tape, or a database accessible via the Internet, storing multiple modules. These modules are program modules, and the processor 710 executes these modules to complete the aforementioned droplet segmentation method.

[0093] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for lipid droplet segmentation of pathological slide images, applicable to a computer system, characterized in that, The segmentation method includes: The pathological slide image is input into a machine learning model to output a region map and an edge map, wherein the region map includes multiple region probability values ​​and the edge map includes multiple edge probability values; and Multiple lipid droplets in the pathological slide image are segmented based on the region probability value and the edge probability value; The step of segmenting the lipid droplets in the pathological slide image based on the region probability value and the edge probability value includes: Determine whether the probability value of each region is greater than or equal to a first threshold value to generate a mask; The region probability value and the edge probability value corresponding to the same location are used to calculate multiple center scores and gradients; Multiple labels are generated based on the central score; and A watershed algorithm is executed based on the mask, the gradient, and the marker.

2. The lipid droplet segmentation method as described in claim 1, characterized in that, The machine learning model is a convolutional neural network with a U-network architecture. The U-network architecture includes multiple downsampling and multiple upsampling. The convolutional neural network has a receptive field square at the bottom layer of the U-network architecture. The receptive field square includes multiple convolutional kernels, at least one of which is a dilation convolutional kernel.

3. The lipid droplet segmentation method as described in claim 2, characterized in that, The output of the machine learning model includes a first branch and a second branch. The first branch outputs the region map, and the second branch outputs the edge map. The lipid droplet segmentation method further includes: When training the machine learning model, the first branch is trained using labeled pathological slide images; and The second branch is trained by subtracting the result of morphological erosion processing from the result of morphological expansion processing of the marked pathological slide image.

4. The lipid droplet segmentation method as described in claim 1, characterized in that, Also includes: The average of the edge probability values ​​for each lipid droplet is calculated as an edge score for the corresponding lipid droplet; as well as If the edge score is less than a third critical value, the corresponding lipid droplet is deleted.

5. An electronic device, characterized in that, include: A module for inputting a pathological slide image into a machine learning model to output a region map and an edge map, wherein the region map includes multiple region probability values ​​and the edge map includes multiple edge probability values. as well as A module for segmenting multiple lipid droplets in the pathological slide image based on the region probability value and the edge probability value; The module used to segment the lipid droplets in the pathological slide image based on the region probability value and the edge probability value includes: A module used to determine whether the probability value of each region is greater than or equal to a first threshold value in order to generate a mask; A module for calculating multiple center scores and gradients using the region probability value and the edge probability value corresponding to the same location; A module for generating multiple tags based on the center score; and A module for performing a watershed algorithm based on the mask, the gradient, and the marker.

6. The electronic device as claimed in claim 5, characterized in that, The machine learning model is a convolutional neural network with a U-network architecture. The U-network architecture includes multiple downsampling and multiple upsampling. The convolutional neural network has a receptive field square at the bottom layer of the U-network architecture. The receptive field square includes multiple convolutional kernels, at least one of which is a dilation convolutional kernel.

7. The electronic device as claimed in claim 6, characterized in that, The output of the machine learning model includes a first branch and a second branch, the first branch being used to output the region map and the second branch being used to output the edge map. The electronic device further includes: A module used to train the first branch with labeled pathological slide images during the training of the machine learning model; and The module of the second branch is trained by subtracting the result of the morphological erosion processing from the result of the morphological expansion processing of the marked pathological slide image.

8. The electronic device as claimed in claim 5, characterized in that, Also includes: A module for calculating the average of the edge probability values ​​of each lipid droplet as an edge fraction for the corresponding lipid droplet; as well as If the edge score is less than a third threshold value, the module used to delete the corresponding lipid droplet is used.

Images