A frame-assisted liver cancer tissue whole-slide tumor microenvironment analysis method

Abstract

The application discloses a kind of framework auxiliary liver cancer tissue whole slide tumor microenvironment analysis methods in the field of artificial intelligence and bioinformatics, which includes the following steps: data collection and pretreatment are carried out to the whole slide image of liver cancer tissue;Using multi-label diagnostic framework assists whole slide image to complete classification task;Using multi-label diagnostic framework assists whole slide image to complete segmentation task.The framework auxiliary liver cancer tissue whole slide tumor microenvironment analysis method is through the multi-label diagnostic framework of deep learning technology to the whole slide image of liver cancer tissue.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and bioinformatics, specifically to a framework-assisted method for analyzing the tumor microenvironment of a full slide of liver cancer tissue. Background Technology

[0002] Liver cancer is a common disease, and early diagnosis and screening are extremely important for the prognosis of liver cancer patients.

[0003] Histopathological images reflect the survival status and differentiation degree of cancer cells at a microscopic level. Therefore, histological examination of histopathological sections is the gold standard for diagnosing and grading liver cancer. Traditionally, pathologists typically use microscopes to observe histopathological images. Whole slide images (WSIs) are histopathological images taken from liver cancer patients, used to analyze large tissue samples containing signs of cancer and other diseases. These images can be billions of pixels in size. With the development of digital imaging technology, pathologists can remotely observe histopathological images on a monitor to determine sensitive areas and the degree of malignancy. Currently, deep learning technology has been widely used in the segmentation and classification of histopathological images, demonstrating superior performance compared to existing methods; it can be used to construct whole slide images (WSIs). The Multi-Label Diagnostic Framework-TR (MLDF-TR) is a deep learning network model framework that includes a classification network and a segmentation network. The classification network is used to complete the classification task, and the segmentation network is used to complete the segmentation task. However, the subjective judgment of pathologists in existing technologies is sometimes influenced by experience, and local vision often interferes with the diagnosis of whole slide images (WSIs). In order to achieve objective diagnosis, it is crucial to use computational pathology methods for quantitative analysis of images. The classification and segmentation of tissue images is the first step in analyzing the liver cancer microenvironment. For example, the classification task can obtain relevant quantitative information such as the tumor stroma ratio, and the segmentation task can obtain the distribution of sensitive areas in the tissue image. This helps to construct the prognostic histological features of tumor subtypes and patient survival. Therefore, histological image classification and segmentation techniques play an important role in computer-aided diagnosis and grading systems for cancer. Existing methods for analyzing the tumor microenvironment of whole-slide liver cancer tissue using other techniques have various limitations, including poor feature extraction, low training efficiency and degradation, poor accuracy, sensitivity, and specificity in analysis and prediction, and poor classification and segmentation performance on datasets. Therefore, there is an urgent need to design a framework-assisted method for analyzing the tumor microenvironment of whole-slide liver cancer tissue. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide a framework-assisted method for analyzing the tumor microenvironment of a full slide of liver cancer tissue. This method overcomes the various limitations of existing techniques that rely on other technologies to assist in the analysis of the tumor microenvironment of a full slide of liver cancer tissue, including poor feature extraction, low training efficiency and degradation, poor accuracy, sensitivity, and specificity in analysis and prediction, and poor classification and segmentation of the dataset. This provides a benchmark for future segmentation work.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] According to one aspect of the present invention, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue is provided, comprising the following steps:

[0007] S1. Data collection and preprocessing of full-slide images of liver cancer tissue;

[0008] S2. Use a multi-label diagnostic framework to assist in classifying full-slide images;

[0009] S3. Use a multi-label diagnostic framework to assist in the segmentation of full-slide images.

[0010] According to the above-described aspects of the present invention, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue is provided, wherein S1 includes the following steps:

[0011] S11. Take five full-slide images from five liver cancer patients and construct a classification dataset and a segmentation dataset;

[0012] S12. Use the flip and translation preprocessing methods to preprocess the classification dataset and the segmentation dataset.

[0013] According to the above-described aspects of the present invention, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue is provided, wherein S2 includes the following steps:

[0014] S21. Slice the full slide image from step S11, divide it into regions, and organize it into a multi-label dataset;

[0015] S22. Use the SLSR network to extract features and classify slices of the full slide image;

[0016] S23. Use the SLSR model to count the number of different labels in the slices, and then calculate the tumor stroma ratio.

[0017] According to the above-described aspects of the present invention, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue is provided, wherein S21 includes the following steps:

[0018] S211. Divide the five full-slide images into small slices of size 224×224, resulting in a total of 68,175 small slices;

[0019] S212. Label the small slices involving two regions in step S211 with two labels;

[0020] S213. Control the ratio of slice data of each category in the small slice to within 1:3;

[0021] S214. Organize the different types of slices from the small slices into a multi-label dataset.

[0022] According to the above-described aspects of the present invention, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue is provided, wherein S22 includes the following steps:

[0023] S221. The classification network that integrates the swin transformer module, local depth convolution module, SE module and residual neural network is named SLSR network;

[0024] S222. The slicing first generates non-overlapping patches through the patch segmentation module. Each patch is regarded as a token. All tokens are linearly mapped to transform the feature dimension of the input patch into an arbitrary dimension C.

[0025] S223. Perform Phase 1: Feed any dimension C into the Swin transformer module, and the local depth convolution module further learns the patch feature dimension, passing the multi-channel feature map to the SE layer to generate a new feature map.

[0026] S223 includes the following steps:

[0027] The S2231.swin transformer module includes shifted non-overlapping windows, which restrict attention to each window and reduce the computational cost generated by the attention mechanism.

[0028] S2232. The local depth convolution module reconstructs the image based on the features extracted by the swin transformer and performs convolution operations in multiple channels. The generated feature map is then passed to the next layer.

[0029] The S2333.SE module then uses the corresponding weight coefficient vector obtained from the channel to perform a dot product with the original feature map vector matrix, and generates a representative feature map.

[0030] S224. Proceed with stages 2, 3, and 4 in sequence. The main structure of stages 2, 3, and 4 is similar to that of stage 1.

[0031] S225. Proceed to stage 5, where the residual neural network performs the classification function of the sliced ​​images.

[0032] According to the above-described aspects of the present invention, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue, wherein step S223 includes the following steps:

[0033] The S2231.swin transformer module includes shifted non-overlapping windows, which restrict attention to each window and reduce the computational cost generated by the attention mechanism.

[0034] S2232. The local depth convolution module reconstructs the image based on the features extracted by the swin transformer and performs convolution operations in multiple channels. The generated feature map is then passed to the next layer.

[0035] The S2333.SE module then uses the corresponding weight coefficient vector obtained from the channel to perform a dot product with the original feature map vector matrix, and generates a representative feature map.

[0036] According to the above-described aspects of the present invention, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue is provided, wherein S23 includes the following steps:

[0037] S231. Accurately classify liver cancer tissue pathology images using a trained SLSR model, including counting the number of different labels in the full slide image;

[0038] S232. The tumor-to-stromal ratio is calculated based on the number of tags. The formula for calculating the tumor-to-stromal ratio is as follows:

[0039]

[0040] In the formula, C t It is a class with only a single label and is a tumor; C′ t There are two labels in one category, one of which is tumor; C i It is a class containing only a single tag and is interstitial fluid; C′ t There are two labels in one category, one of which is interstitial fluid.

[0041] According to the above-mentioned aspects of the present invention, a framework-assisted method for tumor microenvironment analysis of whole slides of liver cancer tissue is provided, wherein S3 includes the following steps:

[0042] S31. Segment and slice the full slide image from step S11;

[0043] S32. A CNN-based model and a transformer model are used to process the slices using a segmentation network;

[0044] S33. The segmentation network is tested on the dataset and predicts segmentation of the slices.

[0045] According to the above-described aspects of the present invention, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue is provided, wherein S31 includes the following steps:

[0046] S311. Randomly select four full-slide images from step S11 as training images and one as a test image;

[0047] S312. Downsample and scale the entire slide image to one-quarter of its original size;

[0048] S313. Use a sliding window to divide the entire slide image into small slices of 224×224, with 13322 slices used for the training set and 3916 slices used for the test set;

[0049] S314. During the testing phase, the predicted small slice labels are merged according to their original positions, and then upsampled and interpolated to form the shape of the initial full slide image before calculating its evaluation index.

[0050] According to the above-described aspects of the present invention, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue is provided, wherein S32 includes the following steps:

[0051] S321. Fully convolutional networks based on CNN models, such as FCN, UNet series, Res-UNet series, and ConvNeXt series, are used to segment the slices;

[0052] S322. Transunet, Swin-unet, Transfuse, and Medical, based on the transformer model, are used to further extract and segment the sliced ​​images.

[0053] By adopting the above technical solution, the present invention has the following advantages:

[0054] 1. This invention provides a framework-assisted method for analyzing the tumor microenvironment of a full slide of liver cancer tissue. It uses deep learning technology to classify and segment the full slide image of liver cancer tissue, and uses a multi-label diagnostic framework (MLD-TR) to preliminarily analyze the liver tumor microenvironment, providing a benchmark for future segmentation work.

[0055] 2. The classification network in this invention uses a Swing transformer module, a local depthwise convolution module, an SE module, and a residual neural network to construct the classification network SLSR. The local depthwise convolution module reconstructs the image using the features extracted by the Swing transformer module. The local depthwise convolution module efficiently passes the feature map to the SE module and generates a new feature map with the original image. This process is repeated in stages 1, 2, 3, and 4. Finally, the residual neural network performs the classification function of the sliced ​​image. In the embodiments, extensive experiments were conducted to compare the classification performance of models such as DieT, T2T-ViT, PVT, TNT, Swing transformer, and Crossover on multi-label datasets. Ablation experiments were also conducted to demonstrate the role of the depthwise convolution module and the SE module in the classification network, providing a benchmark for future segmentation work.

[0056] 3. This invention segments full-slide images using segmentation networks based on CNN and transformer models. Through extensive experiments, it was found that ResUNet can achieve the best segmentation results to date, providing a benchmark for future segmentation work. Attached Figure Description

[0057] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0058] Figure 1 This is a flowchart of the tumor microenvironment analysis method for full slides of liver cancer tissue assisted by the framework of this invention;

[0059] Figure 2 This is the classification network framework of the present invention;

[0060] Figure 3 This is the segmentation network framework of the present invention;

[0061] Figure 4 This is a visualization of the results of segmenting WSIs on a test set based on CNN and transformer networks, according to the present invention. Detailed Implementation

[0062] The technical solution of the present invention will be specifically described below with reference to the embodiments. The detailed features and advantages of the present invention are described in detail in the specific embodiments. The content is sufficient to enable any person skilled in the art to understand the technical content of the present invention and implement it accordingly. Moreover, based on the specification, claims and drawings disclosed in this specification, those skilled in the art can easily understand the related objects and advantages of the present invention.

[0063] like Figure 1 As shown, a framework-assisted method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue includes the following steps:

[0064] S1. Data collection and preprocessing of full-slide images of liver cancer tissue;

[0065] S11. The experiment constructed a classification dataset and a segmentation dataset by taking 5 whole-slide images (WSIs) from 5 liver cancer patients. The classification dataset consists of multi-label images, while the segmentation dataset consists of single-label images.

[0066] S12. The experiment uses the flip and translation method to preprocess the classification dataset and the segmentation dataset.

[0067] S2. Using a multi-label diagnostic framework to assist in the classification task of full-slide images, as detailed below. Figure 2 As shown;

[0068] S2 includes the following steps:

[0069] S21. Slice the full slide image from step S11, divide it into regions, and organize it into a multi-label dataset;

[0070] S21 includes the following steps:

[0071] S211. In the classification task of step S21, the five full slide images are divided into small slices of size 224×224, resulting in a total of 68175 small slices;

[0072] S212. Since some slice images involve two regions, we will label the slices that involve the two regions in step S211 with two labels;

[0073] S213. Due to the small number of necrotic slices, in order to avoid the impact of imbalance in the dataset, we control the ratio of slice data of each class within 1:3 to prevent the diagnostic model generated by training from overfitting. We organize the slices of different types into a multi-label dataset.

[0074] S22. Use the SLSR network to extract features and classify slices of the full slide image;

[0075] S22 includes the following steps:

[0076] S221. A classification network integrating the swin transformer module, local depthwise convolutional module (LDC), SE module, and residual neural network is named SLSR network. A classification network combining the swin transformer module, local depthwise convolutional module (LDC), and SE module is proposed and named SLS network.

[0077] The pathological image of S222.RGB is first divided into non-overlapping patches by a patch segmentation module. Each patch is treated as a token, and its features are set as the concatenation of the original pixel RGB. In the classification task experiment, the patch size is 4×4, the feature dimension of each patch is 4×4×3=48, and the number of patches is H / 4×W / 4. All tokens are projected to an arbitrary dimension (denoted as C) through convolution of the linear embedding layer. All tokens are linearly mapped to transform the feature dimension of the input patch into an arbitrary dimension C.

[0078] S223. Perform Stage 1: After linear mapping, the feature dimension of the patch after input partitioning is transformed into C, and then fed into the Swin transformer module. The local depth convolution module further learns the features and passes the multi-channel feature map to the SE module to generate a new feature map. As the number of network layers increases, the slice fusion layer between Stage 1 and Stage 2 is used to reduce the number of tokens. The first slice fusion layer splices the features of each adjacent 2×2 slice, and the number of slice blocks becomes H / 8x W / 8. The spliced ​​4c-dimensional features are then fed into the linear layer for processing to generate 2c-dimensional features.

[0079] The Swin transformer module includes a standard multi-head self-attention (MSA) module based on a moving window, followed by a two-layer MLP with GELU nonlinearity in the middle. An LN (LayerNorm) layer is added before each MSA module and each MLP module, and residual connections are added after each module. A key design feature of the Swin transformer is its shifted non-overlapping windows, which significantly reduces the algorithm's complexity, allowing the algorithm to grow linearly with the input size. Furthermore, due to the different sliding windows, the non-overlapping window design is more hardware-friendly, resulting in faster actual running speeds. Assuming the computational complexity of a window based on h×w patch images is:

[0080] Φ=4hwC 2 +2M 2 hwC

[0081] The former is a quadratic function of patch hw, while the latter is a linear function when M is fixed (default is 7). To address the impact of the lack of cross-window connectivity, the shifted window partitioning method alternates between adjacent Swin transformer modules. In the self-attention mechanism, we introduce an existing method to calculate similarity, where each head contains a relative positional deviation B:

[0082]

[0083] In the formula It consists of a query matrix, a key matrix, and a value matrix; D represents the query / key dimension, and M represents the value matrix. 2 This represents the number of patches in a single window.

[0084] However, the attention mechanism of the swin transformer only captures global dependencies between tokens. The swin transformer does not model and analyze local dependencies between adjacent pixels. Local depth convolution is an effective way to introduce locality into the network. The local depth convolution module consists of convolutional layers, batch normalization layers, and h_swish activation layers. The 8C-dimensional label sequence is first reshaped into a feature map that is rearranged on a 2D lattice. Then, it is input into a 2D convolutional layer. The k×k (k>1) convolutional kernel aggregates the features in multiple channels to compute a new feature to learn a richer feature representation and pass the generated feature map to the next layer.

[0085] The feature maps resulting from local depthwise convolutions are then passed to the Squeeze-and-Excitation Networks for crucial operations. First, the Squeeze operation compresses features along the spatial dimension, transforming each two-dimensional feature channel into a real number. This real number possesses a global receptive field to some extent, and the output dimension matches the number of input feature channels. It represents the global distribution of responses across the feature channels and allows layers closer to the input to also acquire a global receptive field. This can be represented as:

[0086]

[0087] The primary function is to transform the H×W×C features into a 1×1×1 output. The second step is an extraction operation, which uses a mechanism similar to gates in a recurrent neural network. Weights are generated for each feature channel using the parameter w, which is learned to explicitly model the correlations between feature channels. Finally, a weight adjustment operation treats the weights of the extracted output as important features for each feature channel after feature selection, and then multiplies these weights channel-by-channel to recalibrate the original features along the channel dimension.

[0088] S224. Proceed with stages 2, 3, and 4 in sequence. The main structure of stages 2, 3, and 4 is similar to that of stage 1. The number of slices processed is H / 8 x W / 8, H / 16 x W / 16, and H / 32 x W / 32, respectively. The dimensions of these slices in different stages are 2C, 4C, and 8C, respectively.

[0089] S225. Proceed to stage 5, where the residual neural network undertakes the classification function of the sliced ​​images. As the classification module of the entire classification network, the residual neural network can effectively solve the problem of network degradation and accelerate the convergence of results, making the classification results faster. Secondly, the residual network also prevents overfitting by randomly discarding the role of some neurons, which to a certain extent solves the overfitting problem that occurs when the dataset is small and the model is large.

[0090] S23. Use the SLSR model to count the number of different labels in the slices, and then calculate the tumor stroma ratio.

[0091] S23 includes the following steps:

[0092] S231. Benefiting from the SLSR network in the MLDF-TR framework, the accuracy rates of 0.9460, 0.9960, 0.9808, and 0.9847 were achieved on the interstitial tissue, necrosis, non-tumor, and tumor, respectively. Therefore, we can use the trained SLSR model to accurately classify liver cancer histopathological images, including counting the number of different labels in WSIs.

[0093] S232. Since our network can perform multi-label classification, the tumor-to-stromal ratio is calculated based on the number of labels. The formula for calculating the tumor-to-stromal ratio is as follows:

[0094]

[0095] In the above formula, C t It is a class with only a single label and is a tumor; C′ t There are two labels in one category, one of which is tumor; C i It is a class containing only a single tag and is interstitial fluid; C′ t There are two labels in one category, one of which is interstitial fluid.

[0096] Therefore, this classification method provides doctors with a reference quantitative indicator for analyzing the tumor survival microenvironment, while also reducing the suffering of patients in biological testing. The multi-label diagnostic framework can help doctors intuitively observe sensitive areas in histopathological images and use different colors to distinguish non-cancer, cancer, necrosis, and interstitial fluid areas. Therefore, through the tumor stroma ratio and WSIs segmentation effect diagram, we can conduct a preliminary analysis of the liver cancer tumor microenvironment.

[0097] S3. Using a multi-label diagnostic framework to assist in the segmentation task of the full slide image is shown in Figure 3, where S3 includes the following steps:

[0098] S31. Segment and slice the full slide image from step S11;

[0099] S31 includes the following steps:

[0100] S311. Randomly select 4 WSIs from the full slide images in step S11 as training images and 1 WSI as testing images.

[0101] S312. Since only one of the five WSIs contains a necrotic region, the segmentation effect of the necrotic region cannot be calculated normally. To improve computational efficiency, the WSIs are downsampled and scaled to one-quarter of their original size.

[0102] S313. Use a sliding window to divide the entire slide image into small slices of 224×224. 13322 images are used for the training set and 3916 images are used for the test set. During the testing phase, the predicted slice labels are merged according to their original positions, and then upsampled and interpolated to form the shape of the initial WSIs before calculating their evaluation index.

[0103] S314. During the testing phase, the predicted small slice labels are merged according to their original positions, and then upsampled and interpolated to form the shape of the initial full slide image before calculating its evaluation index.

[0104] S32. A CNN-based model and a transformer model are used to process the slices using a segmentation network;

[0105] S32 includes the following steps:

[0106] S321. Fully convolutional networks (FCN), UNet series, Res-UNet series, and ConvNeXt series based on CNN models are used for slice segmentation. Among them, the FCN network is the basic framework for semantic segmentation, and subsequent algorithms are improvements based on this framework. U-Net is a special type of fully convolutional network, consisting of two parts: the first part is feature extraction; the second part is an upsampling part. Each upsampling step fuses the feature extraction part with the same number of channels, but the feature extraction part is cropped before fusion. Because the network structure resembles a U-shape, it is called the U-Net network. The Res-UNet model consists of symmetric residual modules, and the ConvNeXt series is a pure convolutional neural network.

[0107] S322. The core architecture of the transformer-based segmentation model is still the Encoder-Decoder structure. However, compared with the traditional CNN-based encoder structure, it is entirely replaced by transformers. The four main transformer-based models are: Transunet, Swin-unet, Transfuse, and Medical, which further extract and segment sliced ​​images.

[0108] S33. The segmentation network is tested on the dataset and predicts segmentation of the slices.

[0109] In a specific embodiment, for the classification task, the experiment selected precision, recall, F1 score, accuracy, confusion matrix, and ROC curve to comprehensively evaluate the model's performance. Precision represents the percentage of correctly predicted positive samples out of the predicted positive samples; recall represents the percentage of correctly predicted positive samples out of the predicted correct samples; F1 score is a comprehensive evaluation of precision and recall; accuracy represents the percentage of correctly predicted samples out of the predicted samples; the confusion matrix represents the probability of correct and incorrect results for each label; the true positive rate (TPR) and false positive rate (FPR) in the ROC curve reflect the model's predictive ability. Given that the SLSR network is a multi-label classification network, the embodiment carefully analyzes and discusses the SLSR network for each label. In the segmentation task, the experiment used the Dice-Similarity coefficient (Dice) and accuracy to evaluate the model's generalization ability. Dice is the most frequently used metric in medical segmentation tasks and is a set similarity measure, usually used to calculate the similarity between two samples; the closer to 1, the better the effect. The precision, recall, F1 score, and accuracy values ​​of 11 classification networks tested on the dataset are shown in Table 1 below.

[0110] Table 1. Test results of 11 classification networks on the test set for labeled intercellular matrix.

[0111]

[0112] As shown in Table 1, Swin ViT achieved recall of 0.9515, F1 score of 0.9471, and accuracy of 0.9487, surpassing the performance of all other classification networks. Swin+LS achieved an accuracy of 0.9443, outperforming other classification networks. Therefore, in terms of classification intercellular matrix, the Swin transformer module model plays an important role in improving classification accuracy, sensitivity, and specificity.

[0113] Table 2 presents the precision, recall, F1 score, and accuracy of 11 classification networks tested on the dataset.

[0114] Table 2. Test results of 11 classification networks exhibiting label death on the test set.

[0115]

[0116] As shown in Table 2, the CrossViT classification network achieved an accuracy of 0.9895, likely due to the learning of cross-attention, which improved the model's classification performance. SLSR achieved a recall of 0.9963; Swin ViT achieved an F1 score of 0.9896; and T2T+LS achieved an accuracy of 0.9973. This is because SLSR and Swin ViT contain a large number of Swin transformer modules, which can help improve the model's performance. T2T's classification performance ranked last among all models. However, T2T+LS performed well, possibly because the LS module played a key role in helping the T2T model obtain crucial feature information during feature extraction.

[0117] Table 3 shows the precision, recall, F1 score, and accuracy of 11 classification networks tested on the dataset for normal organization.

[0118] Table 3. Test results of 11 classification networks on the test set with labels indicating normal organization.

[0119]

[0120] As shown in Table 3, SLSR outperforms the other 10 classification networks in precision, recall, F1 score, and accuracy (0.9776, 0.9807, 0.9791, and 0.9808 respectively). This is likely because the shifted non-overlapping window in the Swin transformer module facilitates feature extraction, and the Local Depth Convolutional (LDC) module focuses more on the relationship between adjacent pixels and passes features in parallel, multi-channel manner to the SE module. The SE module then extracts effective information from the feature map and combines it with the original image to generate a new feature map. Secondly, DeiT, TNT, T2T, Swin ViT, and CrossViT have lower prediction accuracy and sensitivity, possibly because these models do not achieve optimal performance in feature extraction.

[0121] Table 4 shows the precision, recall, F1 score, and accuracy of 11 classification networks tested on the dataset.

[0122] Table 4. Test results of 11 classification networks for labeled tumors on the test set.

[0123]

[0124]

[0125] As shown in Table 4, CrossViT+LS achieved a precision of 0.9827. However, CrossViT+LS was slightly higher than CrossViT in terms of precision, recall, F1 score, and accuracy. This is because the LS module played a crucial role in feature extraction. SLSR achieved recall of 0.9838, F1 score of 0.9832, and accuracy of 0.9847. This is likely due to the combined effect of the Swin transformer module and the LS module, which enabled SLSR to achieve the best performance among classification networks.

[0126] In one specific embodiment, Table 5 describes the accuracy and Dice obtained by eight segmentation networks on the test data.

[0127] Table 5. Statistics of test results of the segmentation model on the dataset.

[0128]

[0129] As shown in Table 5, CNN-based segmentation networks outperform transformer-based models in almost every aspect. Res-UNet achieves a Dice score of 0.86, surpassing the test results of other segmentation networks. SwinUNet achieves an accuracy of 0.9525, also outperforming other segmentation networks. Res-UNet improves feature extraction and image restoration by replacing each sub-module of UNet with residual and dense connections. The Swin transformer module uses a hierarchical and symmetrical construction method to build SwinUNet. Furthermore, it introduces the concept of local attention, performing self-attention calculations within non-overlapping window regions. Therefore, Res-UNet and SwinUNet are better methods for segmenting WSIs.

[0130] Figure 4 The visualization results of eight segmentation networks on the test set are presented. The visualizations more intuitively reflect the positions of different labels on WSIs, distinguishing between cancer regions, tissue regions, and interstitial fluid. Since necrotic tissue is absent in this WSI, the segmentation model did not predict its segmentation. Through eight predicted segmented WSIs, we found that FCN, ConvNeXt-UNet, TransUnet, SwinUnet, TransFuse, and Medical Transformer exhibited local blurring, uneven edges, and segmentation errors after segmenting WSIs. However, Res-UNet, based on U-Net, performed better in segmenting WSIs. Among the eight models selected in the experiment, Res-UNet achieved the best segmentation performance. In conclusion, Res-UNet can be used as a standard segmentation network for WSIs.

[0131] In one specific embodiment, to evaluate the impact of the deep convolutional module on the SLSR network of the model, we removed the deep convolutional module from the SLSR network, trained a new model, and obtained the prediction results using the new model, as shown in Table 6 below:

[0132] Table 6. Impact of Deep Convolutional Modules on SLSR Networks

[0133]

[0134] As shown in Table 6, the new model without the LCD module only achieved higher accuracy than the SLSR model on the intercellular matrix. The SLSR model's prediction accuracy on necrotic, normal, and tumor tissues exceeded that of the new model by 0.0015, 0.0049, and 0.0024, respectively. This may be because the LCD module plays an important role in feature extraction. Although the SLSR model's prediction accuracy on the intercellular matrix decreased, its prediction accuracy on other labels increased. Therefore, we believe that the LCD module is a core component of the SLSR network.

[0135] To evaluate the impact of deep convolutional modules on the SLSR network of the model, we removed the SE module from the SLSR network, trained a new model, and obtained the prediction results using the new model, as shown in Table 7.

[0136] Table 7. Evaluation of the impact of the SE module on the SLSR network

[0137]

[0138] As shown in Table 7, the new model, with the SE module removed, achieved higher accuracy than the SLSR model only on the interstitial tissue. The SLSR model's prediction accuracy on necrotic, normal, and tumor tissues exceeded that of the new model by 0.0006, 0.0009, and 0.0006, respectively. This is likely because the SE module plays a crucial role in feature extraction. Although the SLSR model's prediction accuracy for the interstitial tissue decreased, its prediction accuracy for other labels increased. Therefore, we consider the SE module to be a core component of the SLSR network.

[0139] Finally, it should be noted that although the present invention has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, any changes or modifications to the above embodiments within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A method for analyzing the tumor microenvironment of a whole slide of liver cancer tissue with the aid of a frame, characterized by, The method comprises the following steps: S1. Data collection and preprocessing of whole slide images of liver cancer tissues; The S1 comprises the following steps: S11. Five whole slide images are taken from five liver cancer patients to construct a classification data set and a segmentation data set; S12. The classification data set and the segmentation data set are preprocessed by using a flipping and translation preprocessing method; S2. The whole slide image is classified by using a multi-label diagnosis framework; The S2 comprises the following steps: S21. The whole slide image of step S11 is sliced, divided into regions and arranged into a multi-label data set; S22. The slices of the whole slide image are feature-extracted and classified by using an SLSR network; The S22 comprises the following steps: S221. A classification network fused with a swin transformer module, a local deep convolution module, an SE module and a residual neural network is named as an SLSR network; S222. The slices are first generated into non-overlapping patches by a patch segmentation module, each patch is regarded as a token, and all tokens are subjected to linear mapping to change the patch feature dimension after slicing into an arbitrary dimension C; S223. Phase 1: The arbitrary dimension C is sent into the swin transformer module, the local deep convolution module further learns the patch feature dimension, the multi-channel feature map is transmitted to the SE layer to generate a new feature map; The S223 comprises the following steps: S2231. The swin transformer module comprises a shifted non-overlapping window, which limits the attention within each window and reduces the calculation amount generated by the attention mechanism; S2232. The local deep convolution module reconstructs the features extracted by the swin transformer and performs convolution operation in the multi-channel to generate a feature map transmitted to the next layer; S2333. The SE module performs dot product between the corresponding weight coefficient vector obtained from the channel and the original feature vector matrix, and generates a representative feature map; S224. Phases 2, 3 and 4 are sequentially performed, and the main structure of phases 2, 3 and 4 is similar to that of phase 1; S225. Phase 5 is performed, and the classification function of the slice image is undertaken by the residual neural network; S23. The number of different labels in the slice is counted by using the SLSR model, and then the tumor stroma ratio is calculated; The S23 comprises the following steps: S231. The liver cancer histopathology image is accurately classified by using the trained SLSR model, including counting the number of different labels in the whole slide image; S232. The tumor stroma ratio is calculated by the number of labels, and the tumor stroma ratio calculation formula is as follows: wherein is a class with only a single label and is tumor; is a class with two labels, one of which is tumor; is a class with only a single label and is interstitial fluid; is a class with two labels, one of which is interstitial fluid; S3. The whole slide image is segmented by using a multi-label diagnosis framework; The S3 comprises the following steps: S31. The whole slide image of step S11 is segmented and sliced; S32. The slice is processed by a segmentation network based on a CNN-based model and a transformer model; The S32 comprises the following steps: S321. The full convolutional network FCN, UNet series, Res-UNet series and ConvNeXt series based on CNN are used to segment the slices; S322. The Transunet, Swin-unet, Transfuse and Medical based on transformer model are used to further extract and segment the slice images; S33. The segmentation network is tested on the data set to predict the segmentation of the slices. 2.The method according to claim 1, wherein, The S21 includes the following steps: S211. Five full slide images are cut into small slices with a size of 224x224, and a total of 68175 small slices are obtained; S212. The small slices in step S211 involving two regions are labeled with two labels; S213. The proportion of each type of slice data in the small slices is controlled within the range of 1:3; S214. The different types of slices in the small slices are arranged into a multi-label data set.

3. The method for analyzing the tumor microenvironment of liver cancer tissue using a frame-assisted full slide as described in claim 1, characterized in that, The S31 includes the following steps: S311. Four full slide images of step S11 are randomly selected as training, and one is used as testing; S312. The full slide image is down-sampled to one-fourth of the original size; S313. The full slide image is cut into small slices of 224x224 using a sliding window, 13322 for the training set and 3916 for the test set; S314. In the test stage, the predicted small slice labels are merged according to the original position, interpolated into the shape of the initial full slide image by up-sampling, and then the evaluation index is calculated.

Images