A pathological image automatic segmentation method and device for liver cancer diagnosis and prognosis prediction, electronic equipment and medium

Abstract

The present disclosure relates to the technical field of medical image processing, in particular to a pathological image automatic segmentation method and device, electronic equipment and medium. The method comprises: obtaining a plurality of first pathological image blocks with a first resolution and a plurality of second pathological image blocks with a second resolution from a pathological image, the pathological image comprising a plurality of image regions, the first resolution being lower than the second resolution; performing image segmentation for each image region, comprising: performing tissue level ROI segmentation on the corresponding first pathological image block, performing cell level ROI segmentation on the corresponding plurality of second pathological image blocks, and obtaining the image segmentation result of the image region based on the tissue level ROI segmentation result and the plurality of first cell level ROI segmentation results. The present disclosure not only pays attention to the details of the image, but also pays attention to the global spatial relationship between the image blocks, not only the ROI region information of the pathological image is utilized, but also the accuracy of the ROI segmentation is higher.

Description

Technical Field

[0001] This disclosure relates to the field of medical image processing technology, specifically to an automatic segmentation method and apparatus for pathological images, electronic equipment, and media. Background Technology

[0002] Hepatocellular carcinoma (HCC) is the most common malignant tumor of the liver and the fourth leading cause of cancer-related deaths worldwide. Pathological diagnosis of HCC is crucial for prognostic management and treatment planning. Recently, researchers have discovered a link between clinical outcomes and the tumor microenvironment (TME), which comprises the tumor, immune system, stromal cells, fibroblasts, myofibroblasts, vascular tissue, and the surrounding extracellular matrix. Our further research revealed a significant association between the adjacent normal tissue microenvironment (PTME) and HCC prognosis. However, due to the complexity of the tumor and adjacent normal tissue microenvironments and the shortage of pathologists, they are difficult to identify manually, resulting in these two types of information being rarely used in the diagnosis and prognostic prediction of HCC. Summary of the Invention

[0003] To address the problems in the related technologies, this disclosure provides an automatic segmentation method and apparatus for pathological images, as well as an electronic device and medium.

[0004] In a first aspect, this disclosure provides an automatic segmentation method for pathological images, comprising the following steps:

[0005] From the pathological image, a plurality of first pathological image blocks with a first resolution and a plurality of second pathological image blocks with a second resolution are obtained. The pathological image includes a plurality of image regions, each image region corresponding to at least one first pathological image block and a plurality of second pathological image blocks, wherein the first resolution is lower than the second resolution. Image segmentation is performed for each image region, including: performing tissue-level ROI (Region of Interest) segmentation on the corresponding first pathological image block to obtain tissue-level ROI segmentation results; and performing cell-level ROI segmentation on the corresponding multiple second pathological image blocks to obtain multiple first cell-level ROI segmentation results. Based on the tissue-level ROI segmentation results and the multiple first-cell-level ROI segmentation results, the image segmentation result of the image region is obtained; The segmentation result of the pathological image is obtained based on the image segmentation result of each image region.

[0006] According to embodiments of this disclosure, obtaining a plurality of first pathological image blocks with a first resolution and a plurality of second pathological image blocks with a second resolution from the pathological image includes the following steps: The pathological image is cut into the plurality of first pathological image blocks at the first resolution; The pathological image is cut into the plurality of second pathological image blocks at the second resolution.

[0007] According to embodiments of this disclosure, Each image region corresponds to a first pathological image block, and the first pathological image block and the second pathological image block have the same number of pixel rows and the same number of pixel columns; The image segmentation for each image region includes using a multi-level segmentation network model to segment the image for each image region; The step of obtaining the image segmentation result of the image region based on the tissue-level ROI segmentation result and the plurality of first cell-level ROI segmentation results includes using a multi-level segmentation network model to obtain the image segmentation result of the image region based on the tissue-level ROI segmentation result and the plurality of first cell-level ROI segmentation results.

[0008] According to embodiments of this disclosure, the multi-level segmentation network model includes: The tissue-level segmentation module includes a first Dense-U-net network for performing tissue-level ROI segmentation on the first pathological image block; The cell-level segmentation module includes multiple second Dense-U-net networks for cell-level ROI segmentation of corresponding second pathological image blocks, wherein each second Dense-U-net network corresponds to one second pathological image block; The aggregation module is used to aggregate multiple first-cell-level feature maps obtained during cell-level ROI segmentation to obtain second-cell-level feature maps. A multi-level attention module is used to fuse the tissue-level feature map and the second cell-level feature map obtained during the tissue-level ROI segmentation process to obtain a multi-level spatial attention map, which includes a tissue-level spatial attention map and a cell-level spatial attention map. The multi-level attention fusion module is used to perform multi-level attention fusion on the tissue-level ROI segmentation results and the multiple first-cell-level ROI segmentation results using a multi-level spatial attention map, and to process the multi-level attention fusion results using the Argmax function to obtain the image segmentation results of the image region.

[0009] According to embodiments of this disclosure, The step of performing tissue-level ROI segmentation on the corresponding first pathological image block includes: inputting the first pathological image block into the first Dense-U-net network and outputting the tissue-level ROI segmentation result; The step of performing cell-level ROI segmentation on the corresponding multiple second pathological image blocks includes: inputting the multiple second pathological image blocks into the corresponding second Dense-U-net network respectively, and outputting multiple corresponding first cell-level ROI segmentation results, wherein each second Dense-U-net network corresponds to one second pathological image block.

[0010] According to embodiments of this disclosure, obtaining the image segmentation result of the image region based on the tissue-level ROI segmentation result and the plurality of first cell-level ROI segmentation results includes the following steps: The multiple first-cell-level ROI segmentation results are aggregated to obtain second-cell-level ROI segmentation results; The tissue-level ROI segmentation result is fused with the second cell-level ROI segmentation result, and the Argmax function is used to process the multi-level attention fusion result to obtain the image segmentation result of the image region.

[0011] According to embodiments of this disclosure, fusing the tissue-level ROI segmentation result with the second cell-level ROI segmentation result includes the following steps: During the organization-level ROI segmentation process, organization-level feature maps are obtained; During cell-level ROI segmentation, multiple first-cell-level feature maps are obtained; The multiple first-cell-level feature maps are aggregated to obtain a second-cell-level feature map. The tissue-level feature map and the second cell-level feature map are fused to obtain a multi-level spatial attention map, which includes a tissue-level spatial attention map and a cell-level spatial attention map. Multilevel attention fusion is performed on the tissue-level ROI segmentation results and the second-cell-level ROI segmentation results using a multilevel spatial attention map to obtain the multilevel attention fusion result.

[0012] According to embodiments of this disclosure, The organization-level feature map is the output of the penultimate layer in the Dense-U-net network during the organization-level ROI segmentation process. The first cell-level feature map is the output of the penultimate layer in the Dense-U-net network during cell-level ROI segmentation.

[0013] According to embodiments of this disclosure, the multi-level attention fusion of tissue-level ROI segmentation results and second-cell-level ROI segmentation results using multi-level spatial attention maps includes fusion using the following formula:

[0014] in, Image segmentation results for the image region; Organizational-level spatial attention map; This is a cellular-level spatial attention map. The result of upsampling the organization-level ROI segmentation result n times, where n is the value obtained by dividing the second resolution by the first resolution; This is the result of second-level ROI segmentation.

[0015] According to embodiments of this disclosure, the ROI region segmented at the tissue level includes at least one or more of the following: tumor tissue, low-elasticity dilated blood vessels, and microvascular invasion; The ROI regions segmented at the cell level include at least one or more of the following: tumor tissue, low-elasticity dilated blood vessels, microvascular infiltration, immune cell infiltration, connective tissue hyperplasia response, and hepatic steatosis.

[0016] Secondly, this disclosure provides an automatic pathological image segmentation device, which performs automatic segmentation using the method described in any of the preceding claims, and the device includes: a sampling module, an image segmentation module, a fusion module, and an acquisition module; The sampling module is configured to obtain a plurality of first pathological image blocks with a first resolution and a plurality of second pathological image blocks with a second resolution from the pathological image. The pathological image includes a plurality of image regions, each image region corresponding to at least one first pathological image block and a plurality of second pathological image blocks, wherein the first resolution is lower than the second resolution. The image segmentation module is configured to perform image segmentation for each image region, including: performing tissue-level ROI segmentation on the corresponding first pathological image block to obtain tissue-level ROI segmentation results, and performing cell-level ROI segmentation on the corresponding multiple second pathological image blocks to obtain multiple first cell-level ROI segmentation results. The fusion module is configured to obtain the image segmentation result of the image region based on the tissue-level ROI segmentation result and the multiple first cell-level ROI segmentation results; The acquisition module is configured to obtain the segmentation result of the pathological image based on the image segmentation result of each image region.

[0017] Thirdly, embodiments of this disclosure provide an electronic device including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method as described in any of the first aspects.

[0018] Fourthly, this disclosure provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the method described in the first aspect.

[0019] According to the technical solution provided in this disclosure, a plurality of first pathological image blocks with a first resolution and a plurality of second pathological image blocks with a second resolution are obtained from a pathological image. The pathological image includes a plurality of image regions, each image region corresponding to at least one first pathological image block and a plurality of second pathological image blocks, wherein the first resolution is lower than the second resolution. Image segmentation is performed for each image region, including: performing tissue-level ROI segmentation on the corresponding first pathological image block to obtain tissue-level ROI segmentation results; performing cell-level ROI segmentation on the corresponding plurality of second pathological image blocks to obtain a plurality of first cell-level ROI segmentation results; and obtaining the image segmentation result of the image region based on the tissue-level ROI segmentation results and the plurality of first cell-level ROI segmentation results, thereby obtaining the segmentation result of the pathological image. This disclosure can intelligently divide the ROI region of a pathological image. Furthermore, this method employs tissue-level and cell-level ROI segmentation, focusing on both image details and the global spatial relationship between image blocks. This not only utilizes the ROI region information of the pathological image but also improves the accuracy of ROI segmentation.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings: Figure 1 A flowchart illustrating an automatic segmentation method for pathological images according to an embodiment of the present disclosure is shown. Figure 2 This diagram illustrates the structure of a multi-level segmentation network model according to an embodiment of the present disclosure. Figure 3 This diagram illustrates image segmentation based on a Dense-U-net network in an automatic pathological image segmentation method according to an embodiment of the present disclosure. Figure 4 This diagram illustrates the fusion of tissue-level feature maps and second-cell-level feature maps in an automatic pathological image segmentation method according to an embodiment of the present disclosure. Figure 5 The diagram shows a visualization of the results of testing an automatic pathological image segmentation method according to an embodiment of the present disclosure on a segmentation test set. Figure 6 A structural block diagram of an automatic pathological image segmentation apparatus according to an embodiment of the present disclosure is shown; Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown; Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the method according to embodiments of the present disclosure is shown. Detailed Implementation

[0022] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of exemplary embodiments have been omitted from the drawings.

[0023] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, parts or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other features, figures, steps, behaviors, components, parts or combinations thereof.

[0024] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] In this disclosure, any operation involving the acquisition of user information or user data, or the display of user information or user data to others, is an operation authorized or confirmed by the user, or actively selected by the user.

[0026] Pathological diagnosis of liver cancer is of great significance for the management of liver cancer prognosis and the formulation of treatment plans. Further research by the inventors revealed a significant link between the microenvironment of adjacent normal tissue and the prognosis of liver cancer. However, due to the complexity of the tumor tissue and adjacent normal tissue microenvironment and the shortage of pathologists, it is difficult to identify them manually, resulting in these two types of information being rarely used in the diagnosis and prognosis prediction of liver cancer. This disclosure provides an automatic pathological image segmentation method, comprising: obtaining multiple first pathological image blocks with a first resolution and multiple second pathological image blocks with a second resolution from a pathological image, wherein the pathological image includes multiple image regions, each image region corresponding to at least one first pathological image block and multiple corresponding second pathological image blocks, the first resolution being lower than the second resolution; performing image segmentation for each image region, including: performing tissue-level ROI segmentation on the corresponding first pathological image block to obtain tissue-level ROI segmentation results, performing cell-level ROI segmentation on the corresponding multiple second pathological image blocks to obtain multiple first cell-level ROI segmentation results; and obtaining the image segmentation result of the image region based on the tissue-level ROI segmentation results and the multiple first cell-level ROI segmentation results, thereby obtaining the segmentation result of the pathological image. According to embodiments of this disclosure, the ROI region of a pathological image can be intelligently divided, and the image feature quantification value of the pathological image can be given to generate a pathological diagnosis report, thereby enabling subsequent management of liver cancer prognosis and formulation of treatment plans. At the same time, the method adopts tissue-level and cell-level ROI segmentation, which focuses on both image details and the global spatial relationship between pathological image blocks, thus improving the accuracy of ROI segmentation.

[0027] Figure 1 A flowchart illustrating an automatic segmentation method for pathological images according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps S101 to S104.

[0028] In step S101, a plurality of first pathological image blocks with a first resolution and a plurality of second pathological image blocks with a second resolution are obtained from the pathological image. The pathological image includes a plurality of image regions, each image region corresponding to at least one first pathological image block and a plurality of second pathological image blocks. The first resolution is lower than the second resolution.

[0029] According to embodiments of this disclosure, the pathological images are pathological images of liver cancer patients, but are not limited thereto.

[0030] According to embodiments of this disclosure, obtaining a plurality of first pathological image blocks with a first resolution and a plurality of second pathological image blocks with a second resolution from the pathological image includes the following steps: The pathological image is cut into the plurality of first pathological image blocks at the first resolution; The pathological image is cut into the plurality of second pathological image blocks at the second resolution.

[0031] According to embodiments of this disclosure, the second resolution is a multiple of the first resolution. Assuming the first resolution is X, then the second resolution is tX, where t is an integer greater than or equal to 2. The pathological image is segmented into multiple first pathological image blocks at resolution X, and into multiple second pathological image blocks at resolution tX. Assuming each image region corresponds to T first pathological image blocks, then it corresponds to the corresponding T... t 2 A second pathological image block.

[0032] In this disclosure, for ease of explanation, the example given is that each image region corresponds to one first pathological image block and a plurality of corresponding second pathological image blocks. For instance, in a specific embodiment, the pathological image is segmented into a plurality of first pathological image blocks at five times resolution and into a plurality of second pathological image blocks at twenty times resolution, thus each image region corresponds to one first pathological image block and sixteen second pathological image blocks. Those skilled in the art will understand that the principles and solutions of this disclosure are also applicable to situations where each image region corresponds to more than one first pathological image block and a plurality of corresponding second pathological image blocks.

[0033] Given the current limitations of GPU memory, directly inputting pathological images with hundreds of millions of pixels into deep semantic segmentation models such as FCN or U-Net is not feasible. One existing solution is to crop the pathological image into multiple pathological image patches for analysis, and then aggregate the predictions from these patch-level analyses to obtain the final segmentation result. However, this method only divides the pathological image at the same level, ignoring the spatial features and global information between the patches. For some cellular-level ROIs within the pathological image patches, such as inflammatory cells and collagen fibers, detailed information is lost at lower resolutions, causing the segmentation model to fail to segment accurately.

[0034] In step S102, image segmentation is performed for each image region, including: performing tissue-level ROI segmentation on the corresponding first pathological image block to obtain tissue-level ROI segmentation results, and performing cell-level ROI segmentation on the corresponding multiple second pathological image blocks to obtain multiple first cell-level ROI segmentation results.

[0035] In step S103, the image segmentation result of the image region is obtained based on the tissue-level ROI segmentation result and the plurality of first cell-level ROI segmentation results.

[0036] To accurately quantify the image features of pathological images, the inventors proposed a multi-level segmentation network model (MSegNet).

[0037] According to embodiments of this disclosure, each image region corresponds to a first pathological image block, and the first pathological image block and the second pathological image block have the same number of pixel rows and the same number of pixel columns; The image segmentation for each image region includes using a multi-level segmentation network model (MSegNet) to segment the image for each image region; The step of obtaining the image segmentation result of the image region based on the tissue-level ROI segmentation result and the multiple first-cell-level ROI segmentation results includes using a multi-level segmentation network model (MSegNet) to obtain the image segmentation result of the image region based on the tissue-level ROI segmentation result and the multiple first-cell-level ROI segmentation results.

[0038] Figure 2 A schematic diagram of the structure of a multi-level segmentation network model (MSegNet) according to an embodiment of the present disclosure is shown.

[0039] like Figure 2 As shown, the multi-level segmentation network model (MSegNet) includes: The tissue-level segmentation module includes a first Dense-U-net network for performing tissue-level ROI segmentation on the first pathological image block; The cell-level segmentation module includes multiple second Dense-U-net networks for cell-level ROI segmentation of corresponding second pathological image blocks, wherein each second Dense-U-net network corresponds to one second pathological image block; The aggregation module is used to aggregate multiple first-cell-level feature maps obtained during cell-level ROI segmentation to obtain second-cell-level feature maps. A multi-level attention module is used to fuse the tissue-level feature map and the second cell-level feature map obtained during the tissue-level ROI segmentation process to obtain a multi-level spatial attention map, which includes a tissue-level spatial attention map and a cell-level spatial attention map. The multi-level attention fusion module is used to perform multi-level attention fusion on the tissue-level ROI segmentation results and the multiple first-cell-level ROI segmentation results using a multi-level spatial attention map, and to process the multi-level attention fusion results using the Argmax function to obtain the image segmentation results of the image region.

[0040] Specifically, the result of multi-level attention fusion is used as the input of the Argmax function, and the ROI category with the highest predicted probability for each pixel in the image of the multi-level attention fusion result is output, thereby obtaining the image segmentation result of the image region.

[0041] According to embodiments of this disclosure, the aggregation module is used to aggregate multiple first cell-level feature maps obtained during cell-level ROI segmentation, including: splicing the multiple first cell-level feature maps in a spatial dimension, but not limited thereto.

[0042] According to embodiments of this disclosure, the ROI region segmented at the tissue level includes at least one or more of the following: tumor tissue, low-elasticity dilated blood vessels, and microvascular invasion; The ROI regions segmented at the cell level include at least one or more of the following: tumor tissue, low-elasticity dilated blood vessels, microvascular infiltration, immune cell infiltration, connective tissue hyperplasia response, and hepatic steatosis.

[0043] According to embodiments of this disclosure, the tissue-level segmentation module includes a first Dense-U-net network, and the cell-level segmentation module includes sixteen second Dense-U-net networks; the first Dense-U-net network and the second Dense-U-net network have the same network structure but do not share network parameters.

[0044] According to an embodiment of this disclosure, the step of performing tissue-level ROI segmentation on the corresponding first pathological image block includes: inputting the first pathological image block into a first Dense-U-net network and outputting tissue-level ROI segmentation results; The step of performing cell-level ROI segmentation on the corresponding multiple second pathological image blocks includes: inputting the multiple second pathological image blocks into the corresponding second Dense-U-net network respectively, and outputting multiple corresponding first cell-level ROI segmentation results, wherein each second Dense-U-net network corresponds to one second pathological image block.

[0045] Figure 3 This diagram illustrates image segmentation based on a Dense-U-net network in an automatic pathological image segmentation method according to an embodiment of the present disclosure.

[0046] like Figure 3As shown, the Dense-U-net network consists of an encoder and a decoder. It integrates U-Net and the DenseBlock from DenseNet. U-Net is a commonly used network model in automatic image segmentation methods; however, it skips many concrete connections, making it unable to meet the requirements for accurate segmentation of images within a Region of Interest (ROI). Therefore, U-Net is combined with the DenseBlock from DenseNet. In each DenseBlock, there are direct connections from any layer to all subsequent layers. One advantage of dense connections between layers is that it has fewer output dimensions than traditional networks, avoiding the learning of redundant features. The U-Net architecture, on the other hand, preserves low-level image information through long-distance connections (such as skip connections or concatenation connections) between the encoder and decoder. Therefore, the Dense-U-net network combines the advantages of both U-Net and DenseBlock.

[0047] Furthermore, the feature extraction part of the Dense-U-net network consists of densely connected blocks with different output sizes. Each layer generates k feature maps, where k is the growth rate. Dense connections ensure maximum information flow between layers, thereby improving gradient flow and reducing the burden of finding the optimal solution in deep neural networks, thus improving the segmentation performance of U-Net.

[0048] According to embodiments of this disclosure, the step of inputting the first pathological image block into a first Dense-U-net network and outputting tissue-level ROI segmentation results, and inputting the plurality of second pathological image blocks into corresponding second Dense-U-net networks and outputting plurality of corresponding first cell-level ROI segmentation results, includes: After inputting a first pathological image patch into a first Dense-U-net network and inputting a second pathological image patch into a second Dense-U-net network; for each first or second pathological image patch, in the encoder part, the pathological image patch is downsampled through a convolutional layer to reduce the size of the pathological image patch and increase the number of channels, high-scale features are downsampled through a pooling layer, and high-scale features are extracted through dense blocks. After repeating three times, three different high-scale features are extracted. Specifically, the number of times of extracting said different scale features is determined according to the model architecture of the Dense-U-Net network; in the decoder part, low-scale features are upsampled through a deconvolutional layer, so that the high-scale features from the encoder part can be fused respectively through long-range connections. After fusing with the three different high-scale features from the encoder part, output is performed through convolution and an activation function to obtain a tissue-level ROI segmentation result or a first cell-level ROI segmentation result; wherein, a conv+BN+ReLu structure is used in the convolutional layer, a deconv+BN+ReLu structure is used in the deconvolutional layer, and a Sigmoid function is used as the activation function.

[0049] Both the tissue-level ROI segmentation result and the first cell-level ROI segmentation result of the present disclosure fuse high-scale features and low-scale features. Low-scale features contain more spatial information, while high-scale features contain more semantic information, enabling the present disclosure to pay attention to both the spatial relationship between various pathological image patches and the detailed features contained in the pathological image patches.

[0050] Specifically, assuming that the first pathological image patch downsampled at 5× resolution is an image patch of 224×224×3 pixels , which is used as the input for tissue-level segmentation and input into the first Dense-U-net network, and a tissue-level segmentation result of 224×224×C1 pixels is obtained; assuming that a third pathological image patch downsampled at 20× resolution is an image patch of 896×896×3 pixels which contains 16 second pathological image patches of 224×224×3 pixels , which is used as the input for cell-level segmentation and input into the second Dense-U-net network, and a first cell-level segmentation result of 224×224×C2 pixels is obtained; wherein, C1 and C2 are the number of types of ROI regions, and C1<C2; in a specific embodiment, C1=3 and C2=6.

[0051] According to an embodiment of the present disclosure, the tissue-level ROI segmentation result and a plurality of first cell-level ROI segmentation results are RGB images.

[0052] According to embodiments of this disclosure, obtaining the image segmentation result of the image region based on the tissue-level ROI segmentation result and the plurality of first cell-level ROI segmentation results includes the following steps: The multiple first-cell-level ROI segmentation results are aggregated to obtain second-cell-level ROI segmentation results; The tissue-level ROI segmentation result is fused with the second cell-level ROI segmentation result, and the Argmax function is used to process the multi-level attention fusion result to obtain the image segmentation result of the image region.

[0053] Considering the varying recognition capabilities of pathological image blocks for different types of tissue regions under different resolution inputs, the inventors proposed a more flexible fusion method based on multi-level spatial attention maps to fuse the tissue-level ROI segmentation results with the second cell-level ROI segmentation results. This is because, during manual ROI segmentation, low-resolution samples are observed first, followed by zooming in for close inspection. Since scaling decisions are highly correlated with the tissue segmentation task, fusion using multi-level spatial attention maps can share image feature maps at different resolutions.

[0054] According to embodiments of this disclosure, fusing the tissue-level ROI segmentation result with the second cell-level ROI segmentation result includes the following steps: During the organization-level ROI segmentation process, organization-level feature maps are obtained; During cell-level ROI segmentation, multiple first-cell-level feature maps are obtained; The multiple first-cell-level feature maps are aggregated to obtain a second-cell-level feature map. The tissue-level feature map and the second cell-level feature map are fused to obtain a multi-level spatial attention map, which includes a tissue-level spatial attention map and a cell-level spatial attention map. Multilevel attention fusion is performed on the tissue-level ROI segmentation results and the second-cell-level ROI segmentation results using a multilevel spatial attention map to obtain the multilevel attention fusion result.

[0055] According to embodiments of this disclosure, the aggregation of the plurality of first cell-level feature maps includes: splicing the plurality of first cell-level feature maps in a spatial dimension, but is not limited thereto.

[0056] Figure 4 This diagram illustrates the fusion of tissue-level feature maps and second-cell-level feature maps in an automatic segmentation method for pathological images according to an embodiment of the present disclosure.

[0057] like Figure 4 As shown, the fusion of tissue-level feature maps and second-cell-level feature maps to obtain a multi-level spatial attention map includes: Assume the organizational level feature map is as follows: The second-level cell-level feature map is as follows: R represents the set of real numbers, because the feature map is a feature map with feature dimensions of H×W×C composed of multiple real numbers, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map; and As input, firstly Perform 4 upsampling operations, so that and The feature maps are of the same size; then the feature maps obtained from the two resolutions are fused (e.g., by concatenating them using the concat function), and the fused result is convolved, for example, by performing two convolution operations, and then an activation function (e.g., the sigmoid function) is used to obtain the output. ,in, The feature maps include two channels, each corresponding to an organization-level spatial attention map. and cellular-level spatial attention maps .

[0058] According to an embodiment of this disclosure, the organization-level feature map is the output of the penultimate layer in the Dense-U-net network during the organization-level ROI segmentation process; The first cell-level feature map is the output of the penultimate layer in the Dense-U-net network during cell-level ROI segmentation.

[0059] Since the feature map of the penultimate layer in the Dense-U-net network is a fusion of multi-scale features and contains the most semantic and spatial information, the inventors used the tissue-level feature map of the penultimate layer in the tissue-level ROI segmentation process and the cell-level feature map of the penultimate layer in the Dense-U-net network in the cell-level ROI segmentation process as inputs to obtain the most semantic and spatial information, making the image segmentation results of multi-level attention fusion using multi-level spatial attention maps more accurate.

[0060] According to embodiments of this disclosure, the multi-level attention fusion of tissue-level ROI segmentation results and second-cell-level ROI segmentation results using multi-level spatial attention maps includes fusion using the following formula:

[0061] in, Image segmentation results for the image region; Organizational-level spatial attention map; This is a cellular-level spatial attention map. The result of upsampling the organization-level ROI segmentation result n times, where n is the value obtained by dividing the second resolution by the first resolution; This is the result of second-level ROI segmentation.

[0062] In step S104, the segmentation result of the pathological image is obtained based on the image segmentation result of each image region.

[0063] According to embodiments of this disclosure, the image segmentation results of each image region in a pathological image are fused to obtain the segmentation result of the pathological image.

[0064] To train the multi-level segmentation network model (MSegNet) and test its accuracy, the inventors verified the prognostic prediction capability of pathological image features on the publicly available external test set TCGA-LIHC. Pathological image blocks and corresponding labeled masks were randomly extracted from 104 pathological images in the training set and 42 pathological images in the internal test set to construct automatic segmentation training and test sets, respectively.

[0065] The automatic segmentation training set consists of 9.3k pathological image blocks extracted from 104 pathological images in the training set, and 3.2k pathological image blocks obtained through data augmentation (image rotation, translation, or flipping), for a total of 12.5k pathological image blocks. Among them, tumor tissue, dilated blood vessels, vascular tumor emboli, inflammatory tissue, collagen fibers, and fatty degeneration contain 2.1k, 2.4k, 2.0k, 2.0k, 1.7k, and 2.3k images, respectively.

[0066] The automatic segmentation test set consists of 3.2k pathological image blocks extracted from 42 pathological images in the internal test set, including 803 images of tumor tissue, 574 images of dilated blood vessels, 322 images of vascular tumor emboli, 839 images of inflammatory tissue, 501 images of collagen fibers, and 501 images of fatty degeneration.

[0067] All pathological image blocks are 224×224 pixels; the labeled mask is obtained by determining the image binarization segmentation threshold using the OTSU algorithm. The OTSU algorithm divides the image into background and foreground based on its grayscale characteristics. Since variance is a measure of the uniformity of grayscale distribution, the larger the inter-class variance between the background and foreground, the greater the difference between the two parts of the image. Misclassifying part of the foreground as background or part of the background as foreground will reduce the difference between the two parts. The OTSU algorithm maximizes the inter-class variance by iterating through images with two peaks in their histograms and finding the trough between those two peaks, ultimately generating a mask image with the same resolution as the pathological image.

[0068] The results of various ROIs of the multi-level segmentation network model (MSegNet) on the automatic segmentation test set are shown in Table 1: Table 1

[0069] Therefore, this disclosure can automatically segment the six microenvironments of tumor tissue and adjacent tissue in liver cancer patients with high segmentation accuracy. Based on the high-accuracy segmentation results, a pathological diagnosis report can be generated to provide auxiliary judgment in subsequent diagnosis.

[0070] Figure 5 The diagram shows a visualization of the results of an automatic pathological image segmentation method according to an embodiment of the present disclosure being tested on a segmentation test set.

[0071] like Figure 5 As shown, three pathological image samples were randomly selected from the internal test set, and the results of the multi-level segmentation network model (MSegNet) on the segmentation test set were visualized. Each column represents a sample, the first row is the original pathological image, the second row is the manually labeled result, and the third row is the segmentation result using the multi-level segmentation network model (MSegNet).

[0072] The results above demonstrate that the multi-level segmentation network model (MSegNet) can reduce the time cost of manual annotation to a certain extent and achieve automatic segmentation of ROIs in pathological images.

[0073] According to embodiments of this disclosure, the loss function of the multi-level segmentation network model (MSegNet) is as follows: ; in, This represents the segmentation results of the multi-level segmentation network model (MSegNet). The true mask representing the image annotation. This represents the cross-entropy loss (as shown in 3-31). This represents the Dice loss (as shown in 3-32). The parameters of the loss function represent the cross-entropy loss. The parameters of the loss function representing the Dice loss. Representing the attention matrix of Norm, Representing the attention matrix of Norm, express The regularization parameter, express The regularization parameter.

[0074] ; ; in, Indicates the pixel position in a pathological image. Indicates the number of types of ROI regions. It is a pixel The tag, if the category is ,but =1, otherwise equals 0; and These represent the height and width of the pathological image, respectively.

[0075] According to embodiments of this disclosure, Dice loss and CE loss are used in combination for model evaluation. While Dice loss can mitigate the negative impact of foreground-background (area) imbalance in samples, Dice loss training focuses more on foreground region extraction, ensuring a low FN (Failure Number), but it suffers from loss saturation. CE loss, on the other hand, calculates the loss of each pixel equally, with the loss of the current point only related to the distance between the current predicted value and the true label value. Therefore, Dice loss and CE loss are used in combination.

[0076] Figure 6 A structural block diagram of an automatic pathological image segmentation device according to an embodiment of the present disclosure is shown. This device can be implemented as part or all of an electronic device through software, hardware, or a combination of both.

[0077] like Figure 6 As shown, an automatic pathological image segmentation device 6 includes: a sampling module 601, an image segmentation module 602, a fusion module 603, and an acquisition module 604; The sampling module 601 is configured to obtain a plurality of first pathological image blocks with a first resolution and a plurality of second pathological image blocks with a second resolution from the pathological image. The pathological image includes a plurality of image regions, each image region corresponding to at least one first pathological image block and a plurality of second pathological image blocks, wherein the first resolution is lower than the second resolution. The image segmentation module 602 is configured to perform image segmentation for each image region, including: performing tissue-level ROI segmentation on the corresponding first pathological image block to obtain tissue-level ROI segmentation results, and performing cell-level ROI segmentation on the corresponding multiple second pathological image blocks to obtain multiple first cell-level ROI segmentation results. The fusion module 603 is configured to obtain the image segmentation result of the image region based on the tissue-level ROI segmentation result and the multiple first cell-level ROI segmentation results; The acquisition module 604 is configured to obtain the segmentation result of the pathological image based on the image segmentation result of each image region.

[0078] This disclosure also discloses an electronic device. Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0079] like Figure 7 As shown, the electronic device includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to embodiments of the present disclosure.

[0080] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the method according to embodiments of the present disclosure is shown.

[0081] like Figure 8 As shown, the computer system includes a processing unit that can execute various methods described above based on a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer system. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0082] The following components are connected to the I / O interface: input sections including keyboards, mice, etc.; output sections including cathode ray tubes (CRTs), liquid crystal displays (LCDs), and speakers; storage sections including hard disks; and communication sections including network interface cards such as LAN cards and modems. The communication section performs communication processes via a network such as the Internet. Drives are also connected to the I / O interface as needed. Removable media, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive as needed so that computer programs read from them can be installed into the storage section as required. The processing unit can be implemented as a CPU, GPU, TPU, FPGA, NPU, etc.

[0083] In particular, according to embodiments of this disclosure, the methods described above can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the methods described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium.

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0085] The units or modules described in the embodiments of this disclosure can be implemented in software or programmable hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0086] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. An automatic segmentation method for pathological images used in the diagnosis and prognosis prediction of liver cancer, characterized in that, Includes the following steps: From the pathological image, a plurality of first pathological image blocks with a first resolution and a plurality of second pathological image blocks with a second resolution are obtained. The pathological image includes a plurality of image regions, each image region corresponding to at least one first pathological image block and a plurality of second pathological image blocks. The first resolution is lower than the second resolution, and the second resolution is an integer multiple of the first resolution. Image segmentation is performed on each image region using a multi-level segmentation network model. This model includes a tissue-level segmentation module, a cell-level segmentation module, an aggregation module, a multi-level attention module, and a multi-level attention fusion module. The image segmentation for each region includes: inputting a corresponding first pathological image block into the tissue-level segmentation module for tissue-level ROI segmentation to obtain tissue-level ROI segmentation results, and extracting tissue-level feature maps during the segmentation process; inputting multiple corresponding second pathological image blocks into the cell-level segmentation module for cell-level ROI segmentation to obtain multiple first-cell-level ROI segmentation results, and extracting multiple first-cell-level feature maps during the segmentation process. The ROI regions segmented at the tissue level include: tumor tissue, low-elasticity dilated blood vessels, and microvascular invasion. The ROI regions segmented at the cell level include: tumor tissue, low-elasticity dilated blood vessels, microvascular invasion, immune cell invasion, connective tissue hyperplasia response, and hepatic steatosis. Based on the tissue-level ROI segmentation results and the multiple first-cell-level ROI segmentation results, the image segmentation result of the image region is obtained, including: inputting the multiple first-cell-level feature maps into the aggregation module for aggregation to obtain a second-cell-level feature map; inputting the tissue-level feature map and the second-cell-level feature map into the multi-level attention module for fusion to obtain a multi-level spatial attention map, wherein the multi-level spatial attention map includes a tissue-level spatial attention map and a cell-level spatial attention map; inputting the tissue-level ROI segmentation results, the multiple first-cell-level ROI segmentation results, and the multi-level spatial attention map into the multi-level attention fusion module, using the multi-level spatial attention map to perform multi-level attention fusion on the tissue-level ROI segmentation results and the multiple first-cell-level ROI segmentation results, and processing the result of the multi-level attention fusion to obtain the image segmentation result of the image region; The segmentation result of the pathological image is obtained based on the image segmentation result of each image region.

2. The method according to claim 1, characterized in that, Obtaining multiple first pathological image blocks with a first resolution and multiple second pathological image blocks with a second resolution from the pathological image includes the following steps: The pathological image is cut into the plurality of first pathological image blocks at the first resolution; The pathological image is cut into the plurality of second pathological image blocks at the second resolution.

3. The method according to claim 1, characterized in that, Each image region corresponds to a first pathological image block, which has the same number of pixel rows and the same number of pixel columns as the second pathological image block.

4. The method according to claim 1, characterized in that: The tissue-level segmentation module includes a first Dense-U-net network for performing tissue-level ROI segmentation on the first pathological image block; The cell-level segmentation module includes multiple second Dense-U-net networks for cell-level ROI segmentation of corresponding second pathological image blocks, wherein each second Dense-U-net network corresponds to one second pathological image block; The aggregation module is used to aggregate multiple first-cell-level feature maps obtained during cell-level ROI segmentation to obtain a second-cell-level feature map. The multi-level attention module is used to fuse the tissue-level feature map and the second cell-level feature map obtained during the tissue-level ROI segmentation process to obtain a multi-level spatial attention map, which includes a tissue-level spatial attention map and a cell-level spatial attention map. The multi-level attention fusion module is used to perform multi-level attention fusion on the tissue-level ROI segmentation results and the multiple first-cell-level ROI segmentation results using a multi-level spatial attention map, and to process the multi-level attention fusion results using the Argmax function to obtain the image segmentation results of the image region.

5. The method according to claim 1, characterized in that, The step of performing tissue-level ROI segmentation on the corresponding first pathological image block includes: inputting the first pathological image block into the first Dense-U-net network and outputting the tissue-level ROI segmentation result; The step of performing cell-level ROI segmentation on the corresponding multiple second pathological image blocks includes: inputting the multiple second pathological image blocks into the corresponding second Dense-U-net network respectively, and outputting multiple corresponding first cell-level ROI segmentation results, wherein each second Dense-U-net network corresponds to one second pathological image block.

6. The method according to claim 5, characterized in that, The process of obtaining the image segmentation result of the image region based on the tissue-level ROI segmentation result and the multiple first cell-level ROI segmentation results includes the following steps: The multiple first-cell-level ROI segmentation results are aggregated to obtain second-cell-level ROI segmentation results; The tissue-level ROI segmentation result is fused with the second cell-level ROI segmentation result, and the Argmax function is used to process the multi-level attention fusion result to obtain the image segmentation result of the image region.

7. The method according to claim 1, characterized in that, The organization-level feature map is the output of the penultimate layer in the Dense-U-net network during the organization-level ROI segmentation process. The first cell-level feature map is the output of the penultimate layer in the Dense-U-net network during cell-level ROI segmentation.

8. The method according to claim 1, characterized in that, The multi-level attention fusion of tissue-level ROI segmentation results and second-cell-level ROI segmentation results using multi-level spatial attention maps includes the following fusion formula: in, This is the result of multi-level attention fusion; Organizational-level spatial attention map; This is a cellular-level spatial attention map. The result of upsampling the organization-level ROI segmentation result n times, where n is the value obtained by dividing the second resolution by the first resolution; This is the result of second-level ROI segmentation.

9. An automatic segmentation device for pathological images used in the diagnosis and prognosis prediction of liver cancer, characterized in that, The apparatus performs automatic segmentation using the method described in any one of claims 1 to 8, and the apparatus includes: a sampling module, an image segmentation module, a fusion module, and an acquisition module; The sampling module is configured to obtain multiple first pathological image blocks with a first resolution and multiple second pathological image blocks with a second resolution from the pathological image. The pathological image includes multiple image regions, each image region corresponding to at least one first pathological image block and multiple second pathological image blocks. The first resolution is lower than the second resolution, and the second resolution is an integer multiple of the first resolution. The image segmentation module is configured to perform image segmentation for each image region using a multi-level segmentation network model. The multi-level segmentation network model includes: a tissue-level segmentation module, a cell-level segmentation module, an aggregation module, a multi-level attention module, and a multi-level attention fusion module. The image segmentation for each image region includes: inputting the corresponding first pathological image block into the tissue-level segmentation module for tissue-level ROI segmentation to obtain tissue-level ROI segmentation results, and extracting tissue-level feature maps during the tissue-level ROI segmentation process; inputting the corresponding multiple second pathological image blocks into the cell-level segmentation module for cell-level ROI segmentation to obtain multiple first cell-level ROI segmentation results, and extracting multiple first cell-level feature maps during the cell-level ROI segmentation process. The ROI regions of the tissue-level ROI segmentation include: tumor tissue, low-elasticity dilated blood vessels, and microvascular invasion. The ROI regions of the cell-level ROI segmentation include: tumor tissue, low-elasticity dilated blood vessels, microvascular invasion, immune cell invasion, connective tissue hyperplasia response, and hepatic steatosis. The fusion module is configured to obtain an image segmentation result of the image region based on the tissue-level ROI segmentation result and the plurality of first cell-level ROI segmentation results, including: inputting the plurality of first cell-level feature maps into the aggregation module for aggregation to obtain a second cell-level feature map; inputting the tissue-level feature map and the second cell-level feature map into the multi-level attention module for fusion to obtain a multi-level spatial attention map, wherein the multi-level spatial attention map includes a tissue-level spatial attention map and a cell-level spatial attention map; inputting the tissue-level ROI segmentation result, the plurality of first cell-level ROI segmentation results, and the multi-level spatial attention map into the multi-level attention fusion module, using the multi-level spatial attention map to perform multi-level attention fusion on the tissue-level ROI segmentation result and the plurality of first cell-level ROI segmentation results, and processing the result of the multi-level attention fusion to obtain the image segmentation result of the image region; The acquisition module is configured to obtain the segmentation result of the pathological image based on the image segmentation result of each image region.

10. An electronic device, characterized in that, It includes a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method of any one of claims 1 to 8.

11. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1 to 8.

Images