A WSI image classification system based on deep learning

Abstract

The present invention belongs to the field of image data processing technology, and specifically relates to a WSI image classification system based on deep learning, including an image collection and preprocessing module for acquiring and preprocessing WSI images; a preliminary feature extraction module for obtaining preliminary extracted features of WSI images of different scales; a multi-scale feature fusion module for performing multi-scale feature fusion on the extracted preliminary extracted features; a Gaussian contextual Transformer attention gating module for acquiring the importance of the corresponding features of each channel and assigning weight values ​​based on the fused features, and performing dimensionality reduction on the multi-scale features; a two-branch attention classification module for learning, training and predicting the processed features, summarizing the patch-level features into a WSI image-level representation for final diagnosis prediction. The present invention realizes the rapid classification of multi-scale WSI images and improves the efficiency and accuracy of WSI image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a WSI image classification system based on deep learning. Background Art

[0002] Deep learning (DL) is a subfield of machine learning. It is a learning algorithm based on artificial neural networks. Deep learning models simulate the way the human brain processes information and use a multi-layer (deep) neural network structure to learn the complex patterns and features of data. It can automatically extract imperceptible hidden features from images and is widely used in image classification.

[0003] In recent years, Transformer and graph convolution have shown excellent performance in medical image classification. Transformer is a neural network architecture based on the self-attention mechanism. By processing all elements of the input sequence in parallel, it can capture long-distance dependencies and perform well in tasks such as machine translation and text summarization. Graph convolutional network (GCN) is a deep learning model for graph-structured data. It learns the embedded representation of nodes by applying convolution operations on the nodes of the graph, and can effectively capture the topological structure and local connection patterns between nodes.

[0004] In addition, the attention mechanism has been proven to enable deep learning networks to pay more attention to useful information for classification and ignore useless information during the learning process, effectively improving the classification performance of the model. Information of different scales will provide different image feature information. High-scale images will reflect global features, while low-scale images will highlight detail information. Networks based on multi-scale images often learn more features than single-scale networks, and can achieve mutual complementation between features.

[0005] The existing WSI image classification system does not take into account the features of different granularities contained in images of different scales. Specifically, high-scale WSI, i.e., fine-grained images, can reflect detailed cellular features, while low-scale WSI, i.e., coarse-grained images, can reflect macroscopic tumor microenvironmental features. In addition, previous studies only considered contextual features or graph features obtained by interactions between cells, without combining the two. Furthermore, classification was performed only through Transformer or GCN networks, without combining the two at the same time. Summary of the invention

[0006] In view of the above deficiencies in the prior art, the present invention provides a WSI image classification system based on deep learning. Based on the deep learning method, it realizes the rapid classification of multi-scale WSI images based on Transformer and graph convolution, thereby improving the efficiency and accuracy of WSI image classification.

[0007] To achieve the above objectives, the present invention provides a WSI image classification system based on deep learning, comprising the following steps:

[0008] The image collection and preprocessing module is used to acquire WSI images and perform preprocessing to obtain preprocessed sample images. The preprocessing process includes grayscale, denoising, binarization, cropping and segmentation.

[0009] The preliminary feature extraction module is used to input the preprocessed sample images into the Swin-Transformer network and the graph convolutional network GCN for preliminary feature extraction, and obtain the preliminary extracted features of WSI images of different scales;

[0010] Multi-scale feature fusion module, which performs multi-scale feature fusion on the preliminary extracted features extracted from the Swin-Transformer network and the graph convolutional network GCN;

[0011] Gaussian contextual Transformer attention gating, based on the fused features, obtains the importance of the features corresponding to each channel and assigns a weight value to enhance the channels that are useful for the current task and reduce the dimensionality of multi-scale features;

[0012] The two-branch attention classification module learns, trains and predicts the features after the Gaussian contextual Transformer attention gating processing, and summarizes the patch-level features into WSI image-level representations for the final diagnosis prediction.

[0013] In the image collection and preprocessing module, the acquired WSI images are 10×, 20×, and 40× magnification images of the original WSI images, and are divided into patches of 512×512 pixels after preprocessing, namely, the preprocessed sample images.

[0014] The image collection and preprocessing module includes an image graying unit, a median filter, a binarization unit, a cropping unit and a segmentation unit, which are used to implement graying, denoising, binarization, cropping and segmentation in the preprocessing process, specifically:

[0015] The image grayscale unit uses the cvtColor function of the image processing package Opencv to convert the three-channel RGB image into a single-channel HSV grayscale image.

[0016] Median filter, removes noise from WSI images and enhances the tumor tissue area to obtain a median filtered image: select a window with an odd number of length greater than 1 point to scan the WSI image, arrange the pixels contained in the window in ascending or descending order of grayscale, and take the grayscale value in the middle to replace the grayscale value of the pixel in the selected area of ​​the window, thereby eliminating isolated noise points;

[0017] The binarization unit performs binarization processing on the denoised WSI image to obtain a binarized image: a threshold is determined using the threshold function of the Opencv package, the pixel values ​​of the WSI image are divided into two groups and the inter-class variance of the two groups of pixel values ​​is maximized, and then the values ​​of the pixels greater than the threshold are set to 255, and the values ​​of the pixels less than the threshold are set to 0, thereby binarizing the WSI image to separate the tumor tissue area and the background area, and obtaining a binarized WSI image corresponding to the original WSI image;

[0018] The cropping unit is used to crop the tumor tissue area from the binary WSI image: the findContours function of the Opencv package is used to establish two levels of contours for the binary WSI image. The outer contour is the outer boundary, and the inner contour is the boundary information of the inner hole. The contour is filtered by the _filter_contours function to obtain the contour of the tumor tissue area.

[0019] The segmentation unit is used to divide the WSI image with the tumor tissue area cropped out into patches of 512×512 pixels according to a length and width of 512 pixels and a coordinate interval of 512 pixels.

[0020] In the preliminary feature extraction module, the Swin-Transformer network includes an image segmentation module, a linear embedding layer, a Swin-Transformer module and a patch merging layer, and the Swin-Transformer network includes four stages, wherein stage 1 includes 2 linear embedding layers and 2 Swin-Transformer modules, stage 2 includes 2 patch merging layers and 2 Swin-Transformer modules, stage 3 includes 6 patch merging layers and 6 Swin-Transformer modules, stage 4 includes 2 patch merging layers and 2 Swin-Transformer modules, and the image segmentation module is located before stage 1.

[0021] In the Swin-Transformer network:

[0022] The image splitting module splits the input 512×512×3 sample image into non-overlapping M×(P 2×3) sub-patch, we get Dimensional features, M is the effective input sequence length of Transformer, and P represents pixels;

[0023] The linear embedding layer The feature features of the dimension are projected onto any dimension C, and we get Characteristics of dimensions;

[0024] Swin-Transformer module, used to extract patch features, including normalization layer, MLP layer and window-based self-attention module:

[0025] The normalization layer is used to normalize the input features to obtain normalized feature data;

[0026] The MLP layer is a multi-layer perceptron, which is used for feature representation learning and includes an input layer, a hidden layer, and an output layer;

[0027] The window-based self-attention module evenly divides the input feature map into 4 patches, and then cyclically shifts it to the upper left using the window configuration shifted from the previous layer. Pixels;

[0028] The patch merging layer concatenates each group of 256×256 adjacent patches, and the number of patch tokens becomes 1 / 4 of the original, while the patch token dimension is expanded by 4 times; then, a linear embedding layer is used for the patch concatenation features with a dimension expanded by 4 times to reduce the output dimension by half.

[0029] In the Swin-Transformer network, in stage 1, the 512×512 feature map is divided into 4 256×256 patches through the Swin-Transformer module, then merged and split again in stage 2, merged and split again in stage 3, and merged and split for the last time in stage 4.

[0030] The GCN network includes a patch embedding module based on contrastive learning, a graph construction module and two graph convolution layers, where:

[0031] The patch embedding module based on contrastive learning embeds the input patch into a feature vector: a representative embedding vector is extracted from each patch through the ResNet50 network, and then the projection head maps the embedding vector to the latent space where contrastive learning loss is applied to obtain the embedded patch feature vector;

[0032] The graph construction module uses the patch feature vector as the node feature, expressed as F=[f1, f2, ..., f N ],F∈R N ×D , where f i (i=1, 2, …, N) is the D-dimensional embedding vector obtained using ResNet50 with a patch embedding module based on contrastive learning, N is the total number of patches from a WSI image, and up to 8 nodes are selected by the K-nearest neighbor method according to the spatial position of the corresponding patch on the WSI image to connect them into edges to form a graph;

[0033] The graph convolution layer extracts graph features by propagating and aggregating information.

[0034] In the multi-scale feature fusion module, for the preliminary extracted features extracted from the Swin-Transformer network and the graph convolutional network GCN, each network first fuses the features of the received patches at three magnifications, and then the features of the two networks are input into the multi-scale feature fusion module for the final multi-scale feature fusion.

[0035] The Gaussian contextual Transformer attention gating includes a global context embedding module, a channel normalization module, and a gating adaptation module, wherein:

[0036] The global context embedding unit obtains input channel statistics and aggregates global information in the space of multi-scale features;

[0037] Channel normalization unit, which performs channel normalization by embedding information and trainable parameters to establish a competitive relationship between neurons;

[0038] The gated adaptive unit forms competition and cooperation relationships between features. When the gating weight of a channel is actively activated, it promotes the channel to compete with other channels, and when the gating weight is negatively activated, it encourages the channel to cooperate with other channels.

[0039] The two-branch attention classification module includes a training clustering layer and an attention pooling layer, wherein:

[0040] The clustering layer is trained to cluster patches with high attention scores and patches with low attention scores, so that the clustering layer learns a rich patch-level feature space that is separable between features of different categories: the first K patches with high attention scores and the last K patches with low attention scores in the training clustering layer are used as representative samples for training the clustering layer, and the clustering layer learns a rich patch-level feature space that is separable between positive and negative labels of different categories;

[0041] Attention pooling layer aggregates patch-level predictions to obtain WSI image-level predictions: patches are weighted according to their respective attention scores, and patch-level features are summarized into WSI image-level representations for the final diagnosis prediction.

[0042] The modules and algorithms of the present invention can be executed by an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The modules and algorithms are implemented by executing the program on the processor.

[0043] The beneficial effects of the present invention are:

[0044] The present invention realizes the rapid classification of multi-scale WSI images based on Transformer and graph convolution, and improves the efficiency and accuracy of WSI image classification. Based on the representation at the WSI image level, it can deeply mine WSI classification features from the perspective of the context structure of cells and patches, and improve the efficiency and accuracy of image classification. It can extract features that are not observable to the naked eye, thereby assisting doctors in clinical diagnosis. It has a certain degree of interpretability, where the weight of each patch calculated in the attention gating stage of the Gaussian context Transformer can be mapped to the WSI as an attention score, which helps doctors understand the classification performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a principle block diagram of the system of the present invention;

[0046] Figure 2 It is a schematic diagram of the structure of the Swin-Transformer network in the system of the present invention;

[0047] Figure 3 It is a flow chart of a method for classifying WSI images using the system of the present invention. DETAILED DESCRIPTION

[0048] The embodiments of the present invention are further described below in conjunction with the accompanying drawings:

[0049] like Figure 1 As shown, a WSI image classification system based on deep learning includes the following steps:

[0050] The image collection and preprocessing module is used to acquire WSI images and perform preprocessing to obtain preprocessed sample images. The preprocessing process includes grayscale, denoising, binarization, cropping and segmentation.

[0051] The preliminary feature extraction module is used to input the preprocessed sample images into the Swin-Transformer network and the graph convolutional network GCN for preliminary feature extraction, and obtain the preliminary extracted features of WSI images of different scales;

[0052] Multi-scale feature fusion module, which performs multi-scale feature fusion on the preliminary extracted features extracted from the Swin-Transformer network and the graph convolutional network GCN;

[0053] Gaussian contextual Transformer attention gating, based on the fused features, obtains the importance of the features corresponding to each channel and assigns a weight value to enhance the channels that are useful for the current task, suppress the channels that are not very useful for the current task, and reduce the dimensionality of multi-scale features;

[0054] The two-branch attention classification module learns, trains and predicts the features after the Gaussian contextual Transformer attention gating processing, and summarizes the patch-level features into WSI image-level representations for the final diagnosis prediction.

[0055] In the image collection and preprocessing module, the acquired WSI images are 10×, 20×, and 40× magnification images of the original WSI images, and are divided into patches of 512×512 pixels after preprocessing, which are the preprocessed sample images.

[0056] The image collection and preprocessing module includes an image graying unit, a median filter, a binarization unit, a cropping unit, and a segmentation unit, which are used to implement graying, denoising, binarization, cropping, and segmentation in the preprocessing process, specifically:

[0057] The image grayscale unit uses the cvtColor function of the image processing package Opencv to convert the three-channel RGB image into a single-channel HSV grayscale image.

[0058] Median filter, removes noise from WSI images and enhances the tumor tissue area to obtain a median filtered image: select a window with an odd number of length greater than 1 point to scan the WSI image, arrange the pixels contained in the window in ascending or descending order of grayscale, and take the grayscale value in the middle to replace the grayscale value of the pixel in the selected area of ​​the window, thereby eliminating isolated noise points;

[0059] The binarization unit performs binarization on the denoised WSI image to obtain a binarized image: a threshold is determined using the threshold function of the Opencv package, the pixel values ​​of the WSI image are divided into two groups and the inter-class variance of the two groups of pixel values ​​is maximized, and then the values ​​of the pixels greater than the threshold are set to 255 (tumor tissue area), and the values ​​of the pixels less than the threshold are set to 0 (background area), thereby binarizing the WSI image to separate the tumor tissue area and the background area, and obtaining a binarized WSI image corresponding to the original WSI image;

[0060] The cropping unit is used to crop the tumor tissue area from the binary WSI image: the findContours function of the Opencv package is used to establish two levels of contours for the binary WSI image. The outer contour is the outer boundary, and the inner contour is the boundary information of the inner hole. The contour is filtered by the _filter_contours function to obtain the contour of the tumor tissue area.

[0061] The segmentation unit is used to divide the WSI image with the tumor tissue area cropped out into patches of 512×512 pixels according to a length and width of 512 pixels and a coordinate interval (step length) of 512 pixels.

[0062] In the preliminary feature extraction module, Figure 2 As shown, the Swin-Transformer network includes an image splitting module, a linear embedding layer, a Swin-Transformer module ( Figure 2 The Swin-Transformer network consists of four stages, where stage 1 consists of 2 linear embedding layers and 2 Swin-Transformer modules, stage 2 consists of 2 patch merging layers and 2 Swin-Transformer modules, stage 3 consists of 6 patch merging layers and 6 Swin-Transformer modules, stage 4 consists of 2 patch merging layers and 2 Swin-Transformer modules, and the image splitting module is located before stage 1.

[0063] In the Swin-Transformer network:

[0064] The image splitting module splits the input 512×512×3 sample image into non-overlapping M×(P 2 ×3) sub-patch, we get Dimensional features, M is the effective input sequence length of Transformer, and P represents pixels;

[0065] The linear embedding layer The feature features of the dimension are projected onto any dimension C, and we get Characteristics of dimensions;

[0066] Swin-Transformer module, used to extract patch features, including normalization layer, MLP layer and window-based self-attention module:

[0067] The normalization layer is used to normalize the input features to obtain normalized feature data;

[0068] The MLP layer is a multi-layer perceptron, which is used for feature representation learning and includes an input layer, a hidden layer, and an output layer;

[0069] The window-based self-attention module evenly divides the input feature map into 4 patches, and then cyclically shifts it to the upper left using the window configuration shifted from the previous layer. Pixels;

[0070] The patch merging layer concatenates each group of 256×256 adjacent patches, and the number of patch tokens becomes 1 / 4 of the original, while the patch token dimension is expanded by 4 times; then, a linear embedding layer is used for the patch concatenation features with a dimension expanded by 4 times to reduce the output dimension by half.

[0071] In the Swin-Transformer network, in stage 1, the 512×512 feature map is divided into 4 256×256 patches through the Swin-Transformer module, then merged and split again in stage 2, merged and split again in stage 3, and merged and split for the last time in stage 4.

[0072] Specifically, in stage 1, the linear embedding layer obtains The dimensional features are divided into four 256×256 patches by the Swin-Transformer module, and then merged and split again in stage 2 to become The features of the dimensions are merged and split again in stage 3, becoming The characteristics of the dimension are finally merged and split for the last time in stage 4, becoming Characteristics of dimensions.

[0073] The GCN network consists of a patch embedding module based on contrastive learning, a graph construction module, and two graph convolutional layers, where:

[0074] The patch embedding module based on contrastive learning embeds the input patch into a feature vector: a representative embedding vector is extracted from each patch through the ResNet50 network, and then the projection head maps the embedding vector to the latent space where contrastive learning loss is applied to obtain the embedded patch feature vector;

[0075] The graph construction module uses the patch feature vector as the node feature, expressed as F=[f1, f2, ..., f N ],F∈R N ×D , where f i (i=1, 2, …, N) is the D-dimensional embedding vector obtained using ResNet50 with a patch embedding module based on contrastive learning, N is the total number of patches from a WSI image, and up to 8 nodes are selected by the K-nearest neighbor method according to the spatial position of the corresponding patch on the WSI image to connect them into edges to form a graph;

[0076] The graph convolution layer extracts graph features by propagating and aggregating information.

[0077] In the multi-scale feature fusion module, for the preliminary extracted features extracted from the Swin-Transformer network and the graph convolutional network GCN, each network first fuses the features of the received patches at three magnifications, and then the features of the two networks are input into the multi-scale feature fusion module for the final multi-scale feature fusion.

[0078] Gaussian contextual Transformer attention gating includes a global context embedding module, a channel normalization module, and a gating adaptation module, where:

[0079] The global context embedding unit obtains input channel statistics and aggregates global information in the space of multi-scale features;

[0080] Channel normalization unit, which performs channel normalization by embedding information and trainable parameters to establish a competitive relationship between neurons;

[0081] The gated adaptive unit forms competition and cooperation relationships between features. When the gating weight of a channel is actively activated, it promotes the channel to compete with other channels, and when the gating weight is negatively activated, it encourages the channel to cooperate with other channels.

[0082] The two-branch attention classification module includes a training clustering layer and an attention pooling layer, where:

[0083] The clustering layer is trained to cluster patches with high attention scores and patches with low attention scores, so that the clustering layer learns a rich patch-level feature space that is separable between features of different categories: the first K patches with high attention scores and the last K patches with low attention scores in the training clustering layer are used as representative samples for training the clustering layer, and the clustering layer learns a rich patch-level feature space that is separable between positive and negative labels of different categories;

[0084] Attention pooling layer aggregates patch-level predictions to obtain WSI image-level predictions: patches are weighted according to their respective attention scores, and patch-level features are summarized into WSI image-level representations for the final diagnosis prediction.

[0085] like Figure 3 As shown, the method for classifying WSI images using the system of the present invention is:

[0086] S1. Based on the image collection and preprocessing module, the WSI image is acquired and preprocessed to obtain a preprocessed sample image. The preprocessing process includes grayscale, denoising, binarization, cropping and segmentation.

[0087] S2. Based on the preliminary feature extraction module, the preprocessed sample images are respectively input into the Swin-Transformer network and the graph convolutional network GCN for preliminary feature extraction to obtain preliminary extracted features of WSI images of different scales;

[0088] S3, based on the multi-scale feature fusion module, multi-scale feature fusion is performed on the preliminary extracted features extracted from the Swin-Transformer network and the graph convolutional network GCN;

[0089] S4, based on Gaussian contextual Transformer attention gating, for the fused features, obtain the importance of the corresponding features of each channel and assign weight values ​​to enhance the channels that are useful for the current task, suppress the channels that are not very useful for the current task, and reduce the dimensionality of multi-scale features;

[0090] S5. Based on the two-branch attention classification module, the features after Gaussian context Transformer attention gating processing are learned, trained and predicted, and the patch-level features are summarized into WSI image-level representations for the final diagnosis prediction.

Claims

1. A WSI image classification system based on deep learning, characterized by Includes the following modules: The image collection and preprocessing module is used to acquire WSI images and perform preprocessing to obtain preprocessed sample images. The preprocessing process includes grayscale, denoising, binarization, cropping and segmentation. The preliminary feature extraction module is used to input the preprocessed sample images into the Swin-Transformer network and the graph convolutional network GCN for preliminary feature extraction, and obtain the preliminary extracted features of WSI images of different scales; Multi-scale feature fusion module, which performs multi-scale feature fusion on the preliminary extracted features extracted from the Swin-Transformer network and the graph convolutional network GCN; Gaussian contextual Transformer attention gating, based on the fused features, obtains the importance of the features corresponding to each channel and assigns a weight value to enhance the channels that are useful for the current task and reduce the dimensionality of multi-scale features; The two-branch attention classification module learns, trains, and predicts the features after the Gaussian contextual Transformer attention gating process, and summarizes the patch-level features into WSI image-level representations for the final diagnosis prediction; In the image collection and preprocessing module, the acquired WSI images are 10×, 20×, and 40× magnification images of the original WSI images, and are divided into patches of 512×512 pixels after preprocessing, which are the preprocessed sample images; The graph convolutional network GCN includes a patch embedding module based on contrastive learning, a graph construction module and two graph convolution layers, where: The patch embedding module based on contrastive learning embeds the input patch into a feature vector: a representative embedding vector is extracted from each patch through the ResNet50 network, and then the projection head maps the embedding vector to the latent space where contrastive learning loss is applied to obtain the embedded patch feature vector; The graph construction module uses the patch feature vector as the node feature, expressed as F=[f1, f2, ..., f N ],F∈R N×D , where f i is the D-dimensional embedding vector obtained using ResNet50 with a patch embedding module based on contrastive learning. N is the total number of patches from a WSI image. According to the spatial position of the corresponding patch on the WSI image, up to 8 nodes are selected by the K-nearest neighbor method to connect them into edges to form a graph. The graph convolution layer extracts graph features by propagating and aggregating information.

2. The WSI image classification system based on deep learning according to claim 1, characterized in that: The image collection and preprocessing module includes an image graying unit, a median filter, a binarization unit, a cropping unit and a segmentation unit, which are used to implement graying, denoising, binarization, cropping and segmentation in the preprocessing process, specifically: The image grayscale unit uses the cvtColor function of the image processing package Opencv to convert the three-channel RGB image into a single-channel HSV grayscale image. Median filter, removes noise from WSI images and enhances the tumor tissue area to obtain a median filtered image: select a window with an odd number of length greater than 1 point to scan the WSI image, arrange the pixels contained in the window in ascending or descending order of grayscale, and take the grayscale value in the middle to replace the grayscale value of the pixel in the selected area of ​​the window, thereby eliminating isolated noise points; The binarization unit performs binarization processing on the denoised WSI image to obtain a binarized image: a threshold is determined using the threshold function of the Opencv package, the pixel values ​​of the WSI image are divided into two groups and the inter-class variance of the two groups of pixel values ​​is maximized, and then the values ​​of the pixels greater than the threshold are set to 255, and the values ​​of the pixels less than the threshold are set to 0, thereby binarizing the WSI image to separate the tumor tissue area and the background area, and obtaining a binarized WSI image corresponding to the original WSI image; The cropping unit is used to crop the tumor tissue area from the binary WSI image: the findContours function of the Opencv package is used to establish two levels of contours for the binary WSI image. The outer contour is the outer boundary, and the inner contour is the boundary information of the inner hole. The contour is filtered by the _filter_contours function to obtain the contour of the tumor tissue area. The segmentation unit is used to divide the WSI image with the tumor tissue area cropped out into patches of 512×512 pixels according to a length and width of 512 pixels and a coordinate interval of 512 pixels.

3. The WSI image classification system based on deep learning according to claim 1, characterized in that: In the preliminary feature extraction module, the Swin-Transformer network includes an image segmentation module, a linear embedding layer, a Swin-Transformer module and a patch merging layer, and the Swin-Transformer network includes four stages, wherein stage 1 includes 2 linear embedding layers and 2 Swin-Transformer modules, stage 2 includes 2 patch merging layers and 2 Swin-Transformer modules, stage 3 includes 6 patch merging layers and 6 Swin-Transformer modules, stage 4 includes 2 patch merging layers and 2 Swin-Transformer modules, and the image segmentation module is located before stage 1.

4. The WSI image classification system based on deep learning according to claim 3, characterized in that: In the Swin-Transformer network: The image splitting module splits the input 512×512×3 sample image into non-overlapping M×(P 2 ×3) sub-patch, we get Dimensional features, M is the effective input sequence length of Transformer, and P represents pixels; The linear embedding layer The feature features of the dimension are projected onto any dimension C, and we get Characteristics of dimensions; Swin-Transformer module, used to extract patch features, including normalization layer, MLP layer and window-based self-attention module: The normalization layer is used to normalize the input features to obtain normalized feature data; The MLP layer is a multi-layer perceptron, which is used for feature representation learning and includes an input layer, a hidden layer, and an output layer; The window-based self-attention module evenly divides the input feature map into 4 patches, and then cyclically shifts it to the upper left using the window configuration shifted from the previous layer. Pixels; The patch merging layer concatenates each group of 256×256 adjacent patches, and the number of patch tokens becomes 1 / 4 of the original, while the patch token dimension is expanded by 4 times; then, a linear embedding layer is used for the patch concatenation features with a dimension expanded by 4 times to reduce the output dimension by half.

5. The WSI image classification system based on deep learning according to claim 4, characterized in that: In the Swin-Transformer network, in stage 1, the 512×512 feature map is divided into 4 256×256 patches through the Swin-Transformer module, then merged and split again in stage 2, merged and split again in stage 3, and merged and split for the last time in stage 4.

6. The WSI image classification system based on deep learning according to claim 1, characterized in that: In the multi-scale feature fusion module, for the preliminary extracted features extracted from the Swin-Transformer network and the graph convolutional network GCN, each network first fuses the features of the received patches at three magnifications, and then the features of the two networks are input into the multi-scale feature fusion module for the final multi-scale feature fusion.

7. The WSI image classification system based on deep learning according to claim 1, characterized in that: The Gaussian contextual Transformer attention gating includes a global context embedding module, a channel normalization module, and a gating adaptation module, wherein: The global context embedding unit obtains input channel statistics and aggregates global information in the space of multi-scale features; Channel normalization unit, which performs channel normalization by embedding information and trainable parameters to establish a competitive relationship between neurons; The gated adaptive unit forms competition and cooperation relationships between features. When the gating weight of a channel is actively activated, it promotes the channel to compete with other channels, and when the gating weight is negatively activated, it encourages the channel to cooperate with other channels.

8. The WSI image classification system based on deep learning according to claim 1, characterized in that: The two-branch attention classification module includes a training clustering layer and an attention pooling layer, wherein: The clustering layer is trained to cluster patches with high attention scores and patches with low attention scores, so that the clustering layer learns a rich patch-level feature space that is separable between features of different categories: the first K patches with high attention scores and the last K patches with low attention scores in the training clustering layer are used as representative samples for training the clustering layer, and the clustering layer learns a rich patch-level feature space that is separable between positive and negative labels of different categories; Attention pooling layer aggregates patch-level predictions to obtain WSI image-level predictions: patches are weighted according to their respective attention scores, and patch-level features are summarized into WSI image-level representations for the final diagnosis prediction.

Images