Gastric cancer pathological whole-slide t staging category prediction method based on swin transformer and weak supervision

Abstract

The application discloses a gastric cancer pathological whole section T stage class prediction method based on SwinTransformer and weak supervision, and comprises the following steps: 1, image acquisition and pretreatment; 2, a first SwinTransformer model capable of predicting positive and negative classes of image blocks is established; 3, the first SwinTransformer model is trained offline; 4, the trained first SwinTransformer model is used to remove negative image blocks, and the remaining positive image blocks are used to generate a T stage image block dataset; 5, a second SwinTransformer model capable of predicting the T stage class of a whole section is established; 6, the second SwinTransformer model is trained offline; and 7, the two trained SwinTransformer models are used to predict the T stage class of a gastric cancer tissue pathological whole section.

Description

Technical Field

[0001] This invention relates to the field of T-staging in gastric cancer histopathology, specifically to a method for predicting the T-staging category of whole gastric cancer histopathological sections based on Swing Transformer and weakly supervised learning. Background Technology

[0002] Traditional pathological analysis and diagnosis require professional pathologists to observe tens of thousands of cells and tissues under a microscope. However, the target tissue area and cells related to the disease only account for a very small part. A large amount of redundant information will increase the workload of pathologists, leading to various serious situations such as missed diagnosis and misdiagnosis. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention proposes a method for predicting the T-stage category of gastric cancer pathological whole slides based on Swin Transformer and weak supervision. The aim is to accurately and efficiently predict the T-stage category of gastric cancer tissue pathology, thereby helping pathologists to make quick judgments and screenings and reducing their workload.

[0004] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0005] The present invention provides a method for predicting the T-staging category of gastric cancer pathological whole slides based on Swing Transformer and weak supervision, characterized by the following steps:

[0006] Step 1: Image Acquisition and Preprocessing

[0007] Step 1.1: Obtain a dataset of whole-slice images of gastric cancer tissue pathology with category labels and perform preprocessing to obtain the preprocessed whole-slice image dataset X = {X1, X2, ..., X...} n ,...,X N} and the corresponding label Y = {Y1,Y2,...,Y} n ,...,Y N}, where X n Let Y represent the nth full slice image. n X represents the nth full slice image. n Corresponding category label; Y m ∈{1,2,…,α}; α represents the category rank; when Y m When = 1, it means X n For negative whole-section images, when Y m ≠1, indicating X n This is a positive whole-section image;

[0008] Step 1.2: Remove the nth full-slice image X. nAfter removing the blank background, the image is divided into blocks to obtain S image blocks, denoted as Sblock. in, X represents the nth full slice image. n The i-th image block, c represents the number of channels in the image block, and p×p represents the width and height of each image block; Represents the i-th image patch Corresponding category tags; when express A negative image patch, i.e. when When, it means A positive image patch, i.e. i = 1, 2, ..., S; S represents the number of image patches;

[0009] Step 2: Build a first Swin Transformer model that can predict the positive or negative category of an image patch, and use it to obtain the input image patch. Classification results

[0010] Step 3: Train the first Swing Transformer model offline:

[0011] The cross-entropy loss function L is constructed using equation (11), and the first Swing Transformer model is trained using the gradient descent algorithm, so that the cross-entropy loss function L converges, thereby obtaining the trained first Swing Transformer model.

[0012]

[0013] In equation (11), represents the positive and negative pseudo-labels corresponding to the image patches, and S is the total number of image patches;

[0014] Step 4: Use the trained first Swin Transformer model to remove negative image patches, and use the remaining positive image patches to generate a T-stage image patch dataset:

[0015] Step 4.1: Extract full-slice images with a class label other than "1" from the preprocessed full-slice image dataset X, thus obtaining the filtered full-slice image dataset X′={X1′,X2′,...,X′ m ,...,X′ M} and its corresponding label Y′={Y1′,Y2′,...,Y m ′,...,Y M ′};where X′ m Y represents the m-th filtered full slice image. m′ represents X′ m Corresponding category tags;

[0016] Step 4.2: Remove the m-th filtered full-slice image X′ m After removing the blank background, the image is divided into blocks to obtain Q image blocks and their category labels, denoted as . in, X′ represents the m-th filtered full-slice image. m The q-th image patch, Represents the q-th image patch Corresponding category tags, q = 1, 2, ..., Q; Q represents the number of image patches;

[0017] Step 4.3, the m-th filtered full-slice image X′ m Q image blocks The input is processed by a pre-trained Swin Transformer model that can predict the positive or negative category of image patches to remove negative image patches, thus obtaining a T-stage image patch dataset consisting of positive image patches, denoted as . in, X′ represents the m-th filtered full-slice image. m The k-th image block after removing negative image blocks Represents the k-th image patch Corresponding category tags, k = 1, 2, ..., K; K represents the number of image patches;

[0018] Step 5: Build a second Swin Transformer model that can predict the T-stage category of the whole slice, and use it to obtain the m-th whole slice X′. m The output result p m ∈{2,…,α};

[0019] Step 6: Train the second Swing Transformer model offline:

[0020] The cross-entropy loss function L is constructed using equation (20), and the second Swing Transformer model is trained using the gradient descent algorithm, so that the cross-entropy loss function L converges, thereby obtaining the trained second Swing Transformer model.

[0021]

[0022] In equation (20), y m For the m-th full slice X′ m The corresponding T-installment category label;

[0023] Step 7: Predict the T-stage category of gastric cancer tissue pathology whole slides using two trained Swin Transformer models:

[0024] Step 7.1: Obtain a whole-section image of the histopathology tissue to be predicted, remove the blank background area, and then process it into blocks to obtain multiple image block sequences (x1, x2, ..., x...). j ,…,x m ); where x j This represents the j-th image patch in the whole-section image of the histopathology to be predicted;

[0025] Step 7.2: Divide the image patch sequence (x1, x2, ..., x...) into... j ,…,x m The images are fed into the first Swin Transformer model, which is trained to predict the positive or negative categories of image patches, to predict the category of the image patches and remove negative image patches, thus obtaining the sequence of positive image patches (x1′,x′2,…,x′). j ,…,x′ n ), where x′ j Let j represent the j-th image patch in the sequence of whole-section histopathological images after removing negative image patches; calculate the proportion of positive image patches. And compare it with the set classification threshold t, if t pos If ≥t, it means the whole tissue pathology slide image to be predicted is positive, and step 7.3 is executed; otherwise, it means the whole tissue pathology slide image to be predicted is negative, that is, the whole slide category is "1";

[0026] Step 7.3: Separate the positive image patch sequence (x1′, x′2, ..., x′) j ,…,x′ n The T stage of the whole histopathological slice image is obtained by processing the trained second Swin Transformer model that can predict the T stage category of the whole histopathological slice image.

[0027] The gastric cancer pathological whole slice T-stage category prediction method based on Swin Transformer and weak supervision described in this invention is characterized in that the first Swin Transformer model in step 2 is composed of a first block layer, a first stage module, a second stage module, a third stage module, a fourth stage module, a first normalization layer, a first pooling layer, and a first classification layer. The first stage module includes a linear embedding layer and a Swin Transformer block, and the other three stage modules each include an image block fusion layer and a Swin Transformer block.

[0028] Step 2.1: The first block layer processes the input image blocks. After block processing, the transformed feature map is flattened along the channel dimension.

[0029] Step 2.2: The linear embedding layer in the first Stage module modulates the feature map. Linear feature embedding is performed to obtain the feature map of the first Stage module. C represents the size of the feature map;

[0030] Step 2.3, the Swing Transformer block in the first Stage module The process is performed to obtain the output feature map, denoted as...

[0031] Step 2.4: The image patch fusion layer in the second Stage module affects the feature map. Slice fusion processing is performed to obtain the feature map of the second Stage module.

[0032] Step 2.5: The Swing Transformer block in the second Stage module follows the process in step 2.3. The process is performed to obtain the output feature map of the second Stage module.

[0033] Step 2.6, Image patch fusion layer in the third Stage module Slice fusion is performed to obtain the feature map of the third stage module.

[0034] Step 2.7: The Swing Transformer block in the third Stage module follows the process in step 2.3. The process is performed to obtain the output feature map of the third stage module.

[0035] Step 2.8, Image patch fusion layer in the fourth Stage module Slice fusion is performed to obtain the feature map of the fourth stage module.

[0036] Step 2.9: The Swing Transformer block in the fourth Stage module follows the process in step 2.3. The process is performed to obtain the output feature map of the fourth stage module.

[0037] Step 2.10, the normalization layer... After normalization, the normalized feature map is obtained.

[0038] Step 2.11, the pooling layer for features Perform global pooling to obtain the pooled vector.

[0039] Step 2.12, the classification layer uses formula (5) to... Perform a linear transformation to obtain the input image patch x i n Classification results

[0040]

[0041] In equation (10), Linear(·) represents a linear classification function;

[0042] Step 2.3 is performed as follows:

[0043] Step 2.3.1: Use equation (1) to obtain the output z of the l-th multi-head self-attention layer based on the window. l ′;

[0044] z l ′=W_MSA(LN(z l-1 ))+z l-1 (1)

[0045] In equation (1), W_MSA(·) represents the processing of the window-based multi-head self-attention mechanism layer; LN(·) represents the processing of the normalization layer; z l-1 This represents the output feature map of the (l-1)th layer multilayer perceptron. When l = 1, let...

[0046] Step 2.3.2: Use equation (2) to obtain the output feature map z of the l-th layer multilayer perceptron. l ;

[0047] z l =MLP(LN(z) l ′))+z l (2)

[0048] In equation (2), MLP(·) represents the processing of the multilayer perceptron;

[0049] Step 2.3.3: Use equation (3) to obtain the output z of the l-th multi-head self-attention layer based on the moving window. l ′ +1 ;

[0050] zl ′ +1 =SW_MSA(LN(z l ))+z l (3)

[0051] In equation (3), SW_MSA(·) represents the processing of the multi-head self-attention layer based on the moving window;

[0052] Step 2.3.4: Use equation (4) to obtain the output feature map z of the l-th layer multilayer perceptron. l+1 ;

[0053] z l+1 =MLP(LN(z) l ′ +1 ))+z l ′ +1 (4)

[0054] In equation (4), MLP(·) represents the processing of the multilayer perceptron;

[0055] Step 2.3.5: After performing L-layer processing according to steps 2.3.1-2.3.4, the final output feature map of the L-th multilayer perceptron is used as the output feature map of the first Stage module.

[0056] The second Swin Transformer model in step 5 consists of a second block layer, a fifth stage module, a sixth stage module, a seventh stage module, an eighth stage module, a second normalization layer, a second pooling layer, a feature encoding layer, an attention mechanism layer, a feature aggregation layer, and a second classification layer. The fifth stage module includes a linear embedding layer and a Swin Transformer block, while the other three stage modules each include an image patch fusion layer and a Swin Transformer block.

[0057] Step 5.1: The second block layer processes the input image blocks. After block processing, the transformed feature map is flattened along the channel dimension.

[0058] Step 5.2: The linear embedding layer in the fifth stage module pairs the feature map. Linear feature embedding is performed to obtain the feature map of the fifth stage module.

[0059] Step 5.3: The Swing Transformer block in the fifth Stage module performs the process described in step 2.3. After processing, the output feature map of the fifth stage module is obtained, denoted as...

[0060] Step 5.4, the image patch fusion layer in the sixth stage module for feature maps Slice fusion processing is performed to obtain the feature map of the sixth stage module.

[0061] Step 5.5: In the sixth stage module, the Swing Transformer block follows the process in step 5.3. The process is performed to obtain the output feature map of the sixth stage module.

[0062] Step 5.6, Image patch fusion layer in the seventh stage module Slice fusion is performed to obtain the feature map of the seventh stage module.

[0063] Step 5.7: The Swing Transformer block in the seventh Stage module follows the process in step 5.3. The process is performed to obtain the output feature map of the seventh stage module.

[0064] Step 5.8, Image patch fusion layer in the eighth stage module Slice fusion is performed to obtain the feature map of the eighth stage module.

[0065] Step 5.9: The Swing Transformer block in the eighth Stage module follows the process in step 5.3. The process is performed to obtain the output feature map of the eighth stage module.

[0066] Step 5.10, the second normalization layer... After normalization, the normalized feature map is obtained.

[0067] Step 5.11, the second pooling layer on features Perform global pooling to obtain the pooled vector.

[0068] Step 5.12, the feature coding layer uses equation (16) to... Perform a linear transformation to obtain the eigenvectors after the linear transformation.

[0069]

[0070] In equation (16), c represents the dimension of the feature vector;

[0071] Step 5.13: The attention mechanism layer uses equation (17) to calculate the m-th full slice X′. m The k-th image patch Corresponding Gating Attention Weights

[0072]

[0073] In equation (17), and All are parameters of a linear transformation; tanh(·) and sigm(·) are linear activation functions; T denotes transpose. Let X′ represent the m-th full slice. m The feature encoding vector of the j-th image block, where j = 1, 2, ..., K;

[0074] Step 5.14: The feature aggregation layer uses equation (18) to obtain the m-th full slice X′. m Features after aggregation

[0075]

[0076] Step 5.15: The second classification layer aggregates the features H. m Perform a linear transformation to obtain the m-th full slice X′. m The output result p m ∈{2,…,α}.

[0077] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing any of the aforementioned methods for predicting the T-staging category of gastric cancer pathological whole slides based on Swin Transformer and weak supervision, and the processor is configured to execute the program stored in the memory.

[0078] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs any of the steps of the method for predicting the T-staging category of gastric cancer pathological whole slides based on Swin Transformer and weak supervision.

[0079] Compared with existing technologies, the present invention has the following advantages:

[0080] 1. This invention uses artificial intelligence technologies such as the Swin Transformer model to achieve accurate and efficient prediction of T stage of gastric cancer tissue pathology whole slides. Furthermore, by using a weakly supervised learning mechanism to perform classification modeling using only whole slide level labels, it overcomes the problem of high cost of fine annotation of pathology whole slide images, which can reduce the burden of pathologists in reading slides and has certain clinical application value.

[0081] 2. This invention utilizes the Swin Transformer model to perform feature learning on whole-slice images of gastric cancer tissue pathology. It not only possesses the ability of global dynamic adaptive modeling, but also achieves the ability to focus on local features through a shift window mechanism, thereby improving the feature representation ability of whole-slice lesion tissue and improving the T-staging prediction effect.

[0082] 3. This invention uses a two-stage Swing Transformer model to form the backbone network. The first visual transformer completes the positive and negative classification of the whole slice image of gastric cancer tissue pathology and extracts the positive image patch. The second visual transformer only extracts features from the positive image patch and performs T-stage prediction, which effectively reduces the error rate of false labels, improves the robustness of the model, and improves the accuracy of prediction. Attached Figure Description

[0083] Figure 1 This is a model structure diagram of the method of the present invention;

[0084] Figure 2 This is a flowchart illustrating the overall process of the method of the present invention. Detailed Implementation

[0085] In this embodiment, a method for predicting the T-staging category of gastric cancer pathological whole slides based on Swing Transformer and weak supervision is described, such as... Figure 1 As shown, the overall training and prediction process is as follows: Figure 2 As shown:

[0086] Step 1: Image Acquisition and Preprocessing

[0087] Step 1.1: Obtain a dataset of whole-slice images of gastric cancer tissue pathology with category labels and perform preprocessing to obtain the preprocessed whole-slice image dataset X = {X1, X2, ..., X...} n ,...,X N} and the corresponding label Y = {Y1,Y2,...,Y} n ,...,Y N}, where X n Let Y represent the nth full slice image. n X represents the nth full slice image. n Corresponding category label; Ym ∈{1,2,…,α}; α represents the category rank; when Y m When = 1, it means X n For negative whole-section images, when Y m ≠1, indicating X n The data used in this embodiment includes five categories: T0, T1, T2, T3, and Tis, where 1 represents T0 and the rest represent T1 through Tis. 70% of the data is used as the training set, and 30% is used for testing.

[0088] Step 1.2: Remove the nth full-slice image X. n After removing the blank background, the image is divided into blocks to obtain S image blocks, denoted as Sblock. in, X represents the nth full slice image. n The i-th image block, c represents the number of channels in the image block, and p×p represents the width and height of each image block; Represents the i-th image patch Corresponding category tags; when When, it means A negative image patch, i.e. when When, it means A positive image patch, i.e. i = 1, 2, ..., S; S represents the number of image blocks; in this embodiment, the input image block size is 224 × 224, therefore p = 224 and c = 3. S = 500.

[0089] Step 2: Establish a first Swin Transformer model capable of predicting the positive and negative categories of image patches. The first Swin Transformer model consists of a first block layer, a first Stage module, a second Stage module, a third Stage module, a fourth Stage module, a first normalization layer, a first pooling layer, and a first classification layer. The first Stage module includes a linear embedding layer and a Swin Transformer block, while the other three Stage modules each include an image patch fusion layer and a Swin Transformer block.

[0090] Step 2.1: The first block layer processes the input image blocks. After block processing, the transformed feature map is flattened along the channel dimension.

[0091] Step 2.2: The linear embedding layer in the first Stage module modulates the feature map. Linear feature embedding is performed to obtain the feature map of the first Stage module. C represents the size of the feature map; in this embodiment, P = 16 and C = 128.

[0092] Step 2.3, Processing of the Swing Transformer block in the first Stage module:

[0093] Step 2.3.1: Use equation (1) to obtain the output z of the l-th multi-head self-attention layer based on the window. l ′;

[0094] z l ′=W_MSA(LN(z l-1 ))+z l-1 (1)

[0095] In equation (1), W_MSA(·) represents the processing of the window-based multi-head self-attention mechanism layer; LN(·) represents the processing of the normalization layer; z l-1 This represents the output feature map of the (l-1)th layer multilayer perceptron. When l = 1, let...

[0096] Step 2.3.2: Use equation (2) to obtain the output feature map z of the l-th layer multilayer perceptron. l ;

[0097] z l =MLP(LN(z) l ′))+z l (2)

[0098] In equation (2), MLP(·) represents the processing of the multilayer perceptron;

[0099] Step 2.3.3: Use equation (3) to obtain the output z of the l-th multi-head self-attention layer based on the moving window. l ′ +1 ;

[0100] z l ′ +1 =SW_MSA(LN(z l ))+z l (3)

[0101] In equation (3), SW_MSA(·) represents the processing of the multi-head self-attention layer based on the moving window;

[0102] Step 2.3.4: Use equation (4) to obtain the output feature map z of the l-th layer multilayer perceptron. l+1 ;

[0103] z l+1 =MLP(LN(z) l ′ +1 ))+zl ′ +1 (4)

[0104] In equation (4), MLP(·) represents the processing of the multilayer perceptron;

[0105] Step 2.3.5: After performing L-layer processing according to steps 2.3.1-2.3.4, the final output feature map of the L-th multilayer perceptron is used as the output feature map of the first Stage module, denoted as... In this embodiment, L = 2 in the Swing Transformer block of the first Stage module.

[0106] Step 2.4: The image patch fusion layer in the second Stage module affects the feature map. Slice fusion processing is performed to obtain the feature map of the second Stage module.

[0107] Step 2.5: The Swing Transformer block in the second Stage module follows the process in step 2.3. The process is performed to obtain the output feature map of the second Stage module. In this embodiment, L = 2 in the SwinTransformer block of the second Stage module.

[0108] Step 2.6, Image patch fusion layer in the third Stage module Slice fusion is performed to obtain the feature map of the third stage module.

[0109] Step 2.7: The Swing Transformer block in the third Stage module follows the process in step 2.3. The process is performed to obtain the output feature map of the third stage module. In this embodiment, L = 6 in the Swing Transformer block of the third Stage module.

[0110] Step 2.8, Image patch fusion layer in the fourth Stage module Slice fusion is performed to obtain the feature map of the fourth stage module.

[0111] Step 2.9: The Swing Transformer block in the fourth Stage module follows the process in step 2.3. The process is performed to obtain the output feature map of the fourth stage module. In this embodiment, L = 2 in the Swing Transformer block of the fourth Stage module.

[0112] Step 2.10, Normalization layer pair After normalization, the normalized feature map is obtained.

[0113] Step 2.11: Pooling layer for features Perform global pooling to obtain the pooled vector.

[0114] Step 2.12, Classification layer uses formula (5) to... Perform a linear transformation to obtain the input image patch. Classification results

[0115]

[0116] In equation (10), Linear(·) represents a linear classification function;

[0117] Step 3: Train the first Swing Transformer model offline:

[0118] The cross-entropy loss function L is constructed using equation (11), and the first Swing Transformer model is trained using the gradient descent algorithm, so that the cross-entropy loss function L converges, thereby obtaining the trained first Swing Transformer model.

[0119]

[0120] In equation (11), Let L be the pseudo-label for the image patch, and S be the total number of image patches. In this embodiment, the Adam optimizer is used to optimize the loss value L, thereby adjusting all parameters of the first SwinTransformer model that can predict the positive or negative category of the image patch, and obtaining the first Swin Transformer model that can predict the positive or negative category of the image patch.

[0121] Step 4: Use the trained first Swin Transformer model to remove negative image patches, and use the remaining positive image patches to generate a T-stage image patch dataset:

[0122] Step 4.1: Extract full-slice images with a class label other than "1" from the preprocessed full-slice image dataset X, thus obtaining the filtered full-slice image dataset X′={X1′,X2′,...,X′ m ,...,X′ M} and its corresponding label Y′={Y1′,Y2′,...,Y m′,...,Y M ′};where X′ m Y represents the m-th filtered full slice image. m ′ represents X′ m Corresponding category tags;

[0123] Step 4.2: Remove the m-th filtered full-slice image X′ m After removing the blank background, the image is divided into blocks to obtain Q image blocks and their category labels, denoted as . in, X′ represents the m-th filtered full-slice image. m The q-th image patch, Represents the q-th image patch Corresponding category tags, q = 1, 2, ..., Q; Q represents the number of image patches;

[0124] Step 4.3, the m-th filtered full-slice image X′ m Q image blocks The input is processed by a pre-trained Swin Transformer model that can predict the positive or negative category of image patches to remove negative image patches, thus obtaining a T-stage image patch dataset consisting of positive image patches, denoted as . in, X′ represents the m-th filtered full-slice image. m The k-th image block after removing negative image blocks Represents the k-th image patch Corresponding category tags, k = 1, 2, ..., K; K represents the number of image patches;

[0125] Step 5: Establish a second Swin Transformer model capable of predicting the T-stage category of the entire slice. The second Swin Transformer model consists of a second block layer, a fifth stage module, a sixth stage module, a seventh stage module, an eighth stage module, a second normalization layer, a second pooling layer, a feature encoding layer, an attention mechanism layer, a feature aggregation layer, and a second classification layer. The fifth stage module includes a linear embedding layer and a Swin Transformer block, while the other three stage modules each include an image patch fusion layer and a Swin Transformer block.

[0126] Step 5.1: The second block layer processes the input image blocks. After block processing, the transformed feature map is flattened along the channel dimension.

[0127] Step 5.2, the linear embedding layer in the fifth stage module for feature maps Linear feature embedding is performed to obtain the feature map of the fifth stage module.

[0128] Step 5.3: In the fifth stage module, the Swing Transformer block follows the process in step 2.3. After processing, the output feature map of the fifth stage module is obtained, denoted as...

[0129] Processing of the Swing Transformer block in the fifth stage module:

[0130] Step 5.3.1: Use equation (12) to obtain the output s of the l-th multi-head self-attention layer based on the window. l ′;

[0131] s l =W_MSA(LN(s) l-1 ))+s l-1 (12)

[0132] In equation (12), W_MSA(·) represents the processing of the window-based multi-head self-attention mechanism layer; LN(·) represents the processing of the normalization layer; s l-1 This represents the output feature map of the (l-1)th layer multilayer perceptron. When l = 1, let...

[0133] Step 5.3.2: Use equation (13) to obtain the output feature map s of the l-th layer multilayer perceptron. l ;

[0134] s l =MLP(LN(s) l ′))+s l (13)

[0135] In equation (13), MLP(·) represents the processing of the multilayer perceptron;

[0136] Step 5.3.3: Use equation (14) to obtain the output s of the l-th multi-head self-attention layer based on the moving window. l ′ +1 ;

[0137] s l ′ +1 =SW_MSA(LN(s l ))+s l (14)

[0138] In equation (14), SW_MSA(·) represents the processing of the multi-head self-attention layer based on the moving window;

[0139] Step 5.3.4: Use equation (15) to obtain the output feature map s of the l-th layer multilayer perceptron. l+1 ;

[0140] s l+1 =MLP(LN(s) l ′ +1 ))+s l ′ +1 (15)

[0141] In equation (15), MLP(·) represents the processing of the multilayer perceptron;

[0142] Step 5.3.5: After performing L-layer processing according to steps 5.3.1-5.3.4, the final output feature map of the L-th multilayer perceptron is used as the output feature map of the fifth stage module, denoted as... In this embodiment, L = 2 in the Swing Transformer block of the fifth Stage module.

[0143] Step 5.4, the image patch fusion layer in the sixth stage module for feature maps Slice fusion processing is performed to obtain the feature map of the sixth stage module.

[0144] Step 5.5: In the sixth stage module, the Swing Transformer block follows the process in step 5.3. The process is performed to obtain the output feature map of the sixth stage module. In this embodiment, L = 2 in the Swing Transformer block of the sixth Stage module.

[0145] Step 5.6, Image patch fusion layer in the seventh stage module Slice fusion is performed to obtain the feature map of the seventh stage module.

[0146] Step 5.7: The Swing Transformer block in the seventh Stage module follows the process in step 5.3. The process is performed to obtain the output feature map of the seventh stage module. In this embodiment, L = 6 in the Swing Transformer block of the seventh Stage module.

[0147] Step 5.8, Image patch fusion layer in the eighth stage module Slice fusion is performed to obtain the feature map of the eighth stage module.

[0148] Step 5.9: The Swing Transformer block in the eighth Stage module follows the process in step 5.3. The process is performed to obtain the output feature map of the eighth stage module. In this embodiment, L = 6 in the Swing Transformer block of the eighth Stage module.

[0149] Step 5.10, Second normalization layer After normalization, the normalized feature map is obtained.

[0150] Step 5.11, the second pooling layer for features Perform global pooling to obtain the pooled vector.

[0151] Step 5.12, the feature coding layer uses equation (16) to... Perform a linear transformation to obtain the eigenvectors after the linear transformation.

[0152]

[0153] In equation (16), c represents the dimension of the feature vector;

[0154] Step 5.13: The attention mechanism layer uses equation (17) to calculate the m-th full slice X′. m The k-th image patch corresponds to the gating attention weight.

[0155]

[0156] In equation (17), and All are parameters of a linear transformation; tanh(·) and sigm(·) are linear activation functions; T denotes transpose. Let X′ represent the m-th full slice. m The feature encoding vector of the j-th image block, where j = 1, 2, ..., K;

[0157] Step 5.14: The feature aggregation layer uses equation (18) to obtain the m-th full slice X′. m Features after aggregation

[0158]

[0159] Step 5.15: The second classification layer aggregates the features H. m Perform a linear transformation to obtain the m-th full slice X′. m The output result p m ∈{2,…,α};

[0160] Step 6: Train the second Swing Transformer model offline:

[0161] The cross-entropy loss function L is constructed using equation (20), and the second Swing Transformer model is trained using the gradient descent algorithm, so that the cross-entropy loss function L converges, thereby obtaining the trained second Swing Transformer model.

[0162]

[0163] In equation (20), y m For the m-th full slice X′ m The corresponding T-stage category label; In this embodiment, the Adam optimizer is used to optimize the loss value L, thereby adjusting all parameters of the second Swin Transformer model that can predict the T-stage category of the whole slice, and obtaining the second Swin Transformer model that can predict the T-stage category of the whole slice.

[0164] Step 7: Predict the T-stage category of gastric cancer tissue pathology whole slides using two trained Swin Transformer models:

[0165] Step 7.1: Obtain a whole-section image of the histopathology tissue to be predicted, remove the blank background area, and then process it into blocks to obtain multiple image block sequences (x1, x2, ..., x...). j ,…,x m ); where x j This represents the j-th image patch in the whole-section image of the histopathology to be predicted;

[0166] Step 7.2: Divide the image patch sequence (x1, x2, ..., x...) into... j ,…,x m The images are fed into the first Swin Transformer model, which is trained to predict the positive or negative categories of image patches, to predict the category of the image patches and remove negative image patches, thus obtaining the sequence of positive image patches (x1′, x′2, ..., x′). j ,…,x′ n ), where x′ j Let j represent the j-th image patch in the sequence of whole-section histopathological images after removing negative image patches; calculate the proportion of positive image patches. And compare it with the set classification threshold t, if t pos If ≥t, it means the whole tissue pathology slide image to be predicted is positive, and step 7.3 is executed; otherwise, it means the whole tissue pathology slide image to be predicted is negative, that is, the whole slide category is "1";

[0167] Step 7.3: Separate the positive image patch sequence (x1′, x′2, ..., x′) j ,…,x′ n The T stage of the whole histopathological slice image is obtained by processing the trained second Swin Transformer model that can predict the T stage category of the whole histopathological slice image.

[0168] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor in executing the above-described method for predicting the T-staging category of gastric cancer tissue pathology whole slides. The processor is configured to execute the program stored in the memory.

[0169] In this embodiment, a computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the above-described method for predicting the T-staging category of gastric cancer tissue pathological whole slides.

Claims

1. A method for predicting T-staging categories of gastric cancer pathological whole slides based on Swing Transformer and weak supervision, characterized in that, The procedure is as follows: Step 1: Image Acquisition and Preprocessing Step 1.1: Obtain a dataset of whole-slice images of gastric cancer tissue pathology with category labels and perform preprocessing to obtain the preprocessed whole-slice image dataset X = {X1, X2, ..., X...} n ,...,X N } and the corresponding label Y = {Y1,Y2,...,Y} n ,...,Y N }, where X n Let Y represent the nth full slice image. n X represents the nth full slice image. n Corresponding category label; Y m ∈{1,2,…,α}; α represents the category rank; when Y m When = 1, it means X n For negative whole-section images, when Y m ≠1, indicating X n This is a positive whole-section image; Step 1.2: Remove the nth full-slice image X. n After removing the blank background, the image is divided into blocks to obtain S image blocks, denoted as Sblock. in, X represents the nth full slice image. n The i-th image block, c represents the number of channels in the image block, and p×p represents the width and height of each image block; Represents the i-th image patch Corresponding category tags; when When, it means A negative image patch, i.e. when When, it means A positive image patch, i.e. S represents the number of image patches; Step 2: Build a first Swin Transformer model that can predict the positive or negative category of an image patch, and use it to obtain the input image patch. Classification results Step 3: Train the first Swing Transformer model offline: The cross-entropy loss function L is constructed using equation (11), and the first Swing Transformer model is trained using the gradient descent algorithm, so that the cross-entropy loss function L converges, thereby obtaining the trained first Swing Transformer model. In equation (11), represents the positive and negative pseudo-labels corresponding to the image patches, and S is the total number of image patches; Step 4: Use the trained first Swin Transformer model to remove negative image patches, and use the remaining positive image patches to generate a T-stage image patch dataset: Step 4.1: Extract full-slice images with a class label other than "1" from the preprocessed full-slice image dataset X, thus obtaining the filtered full-slice image dataset X′={X′1,X′2,...,X′ m ,...,X′ M } and its corresponding label Y′={Y′1,Y′2,...,Y′ m ,...,Y′ M }; where X′ m Y′ represents the m-th filtered full slice image. m Represents X′ m Corresponding category tags; Step 4.2: Remove the m-th filtered full-slice image X′ m After removing the blank background, the image is divided into blocks to obtain Q image blocks and their category labels, denoted as . in, X′ represents the m-th filtered full-slice image. m The q-th image patch, Represents the q-th image patch Corresponding category tags, Q represents the number of image patches; Step 4.3, the m-th filtered full-slice image X′ m Q image blocks The input is processed by a pre-trained Swin Transformer model that can predict the positive or negative category of image patches to remove negative image patches, thus obtaining a T-stage image patch dataset consisting of positive image patches, denoted as . in, X′ represents the m-th filtered full-slice image. m The k-th image block after removing negative image blocks Represents the k-th image patch Corresponding category tags, K represents the number of image patches; Step 5: Build a second Swin Transformer model that can predict the T-stage category of the whole slice, and use it to obtain the m-th whole slice X′. m The output result p m ∈{2,…,α}; Step 6: Train the second Swing Transformer model offline: The cross-entropy loss function L is constructed using equation (20), and the second Swing Transformer model is trained using the gradient descent algorithm, so that the cross-entropy loss function L converges, thereby obtaining the trained second Swing Transformer model. In equation (20), y m For the m-th full slice X′ m The corresponding T-installment category label; Step 7: Predict the T-stage category of gastric cancer tissue pathology whole slides using two trained Swin Transformer models: Step 7.1: Obtain a whole-section image of the histopathology tissue to be predicted, remove the blank background area, and then process it into blocks to obtain multiple image block sequences (x1, x2, ..., x...). j ,…,x m ); where x j This represents the j-th image patch in the whole-section image of the histopathology to be predicted; Step 7.2: Divide the image patch sequence (x1, x2, ..., x...) into... j ,…,x m The images are fed into the first Swin Transformer model, which is trained to predict the positive or negative categories of image patches, to predict the category of the image patches and remove negative image patches, thus obtaining the sequence of positive image patches (x′1, x′2, ..., x′). j ,…,x′ n ), where x′ j Let j represent the j-th image patch in the sequence of whole-section histopathological images after removing negative image patches; calculate the proportion of positive image patches. And compare it with the set classification threshold t, if t pos If ≥t, it means the whole tissue pathology slide image to be predicted is positive, and step 7.3 is executed; otherwise, it means the whole tissue pathology slide image to be predicted is negative, that is, the whole slide category is "1". Step 7.3: Separate the positive image patch sequence (x′1, x′2, ..., x′) j ,…,x′ n The T stage of the whole histopathological slice image is obtained by processing the trained second Swin Transformer model that can predict the T stage category of the whole histopathological slice image.

2. The method for predicting the T-staging category of gastric cancer pathological whole slides based on Swing Transformer and weak supervision according to claim 1, characterized in that, The first Swin Transformer model in step 2 consists of a first block layer, a first Stage module, a second Stage module, a third Stage module, a fourth Stage module, a first normalization layer, a first pooling layer, and a first classification layer. The first Stage module includes a linear embedding layer and a Swin Transformer block, while the other three Stage modules each include an image block fusion layer and a Swin Transformer block. Step 2.1: The first block layer processes the input image blocks. After block processing, the transformed feature map is flattened along the channel dimension. Step 2.2: The linear embedding layer in the first Stage module modulates the feature map. Linear feature embedding is performed to obtain the feature map of the first Stage module. C represents the size of the feature map; Step 2.3, the Swing Transformer block in the first Stage module The process is performed to obtain the output feature map, denoted as... Step 2.4: The image patch fusion layer in the second Stage module affects the feature map. Slice fusion processing is performed to obtain the feature map of the second Stage module. Step 2.5: The Swing Transformer block in the second Stage module follows the process in step 2.

3. The process is performed to obtain the output feature map of the second Stage module. Step 2.6, Image patch fusion layer in the third Stage module Slice fusion is performed to obtain the feature map of the third stage module. Step 2.7: The Swing Transformer block in the third Stage module follows the process in step 2.

3. The process is performed to obtain the output feature map of the third stage module. Step 2.8, Image patch fusion layer in the fourth Stage module Slice fusion is performed to obtain the feature map of the fourth stage module. Step 2.9: The Swing Transformer block in the fourth Stage module follows the process in step 2.

3. The process is performed to obtain the output feature map of the fourth stage module. Step 2.10, the normalization layer... After normalization, the normalized feature map is obtained. Step 2.11, the pooling layer for features Perform global pooling to obtain the pooled vector. Step 2.12, the classification layer uses formula (5) to... Perform a linear transformation to obtain the input image patch. Classification results In equation (10), Linear(·) represents a linear classification function; 3. The method for predicting the T-staging category of gastric cancer pathological whole slides based on Swing Transformer and weak supervision according to claim 2, characterized in that, Step 2.3 is performed as follows: Step 2.3.1: Use equation (1) to obtain the output z of the l-th multi-head self-attention layer based on the window. l ′; z l ′=W_MSA(LN(z l-1 ))+z l-1 (1) In equation (1), W_MSA(·) represents the processing of the window-based multi-head self-attention mechanism layer; LN(·) represents the processing of the normalization layer; z l-1 This represents the output feature map of the (l-1)th layer multilayer perceptron. When l = 1, let... Step 2.3.2: Use equation (2) to obtain the output feature map z of the l-th layer multilayer perceptron. l ; With l =MLP(LN(z l ′))+z l ′ (2) In equation (2), MLP(·) represents the processing of the multilayer perceptron; Step 2.3.3: Use equation (3) to obtain the output z of the l-th multi-head self-attention layer based on the moving window. l ′ +1 ; z l ′ +1 =SW_MSA(LN(z l ))+z l (3) In equation (3), SW_MSA(·) represents the processing of the multi-head self-attention layer based on the moving window; Step 2.3.4: Use equation (4) to obtain the output feature map z of the l-th layer multilayer perceptron. l+1 ; With l+1 =MLP(LN(z l ′ +1 ))+z l ′ +1 (4) In equation (4), MLP(·) represents the processing of the multilayer perceptron; Step 2.3.5: After performing L-layer processing according to steps 2.3.1-2.3.4, the final output feature map of the L-th multilayer perceptron is used as the output feature map of the first Stage module.

4. The method for predicting the T-staging category of gastric cancer pathological whole slides based on Swing Transformer and weak supervision according to claim 3, characterized in that, The second Swin Transformer model in step 5 consists of a second block layer, a fifth stage module, a sixth stage module, a seventh stage module, an eighth stage module, a second normalization layer, a second pooling layer, a feature encoding layer, an attention mechanism layer, a feature aggregation layer, and a second classification layer. The fifth stage module includes a linear embedding layer and a Swin Transformer block, while the other three stage modules each include an image patch fusion layer and a Swin Transformer block. Step 5.1: The second block layer processes the input image blocks. After block processing, the transformed feature map is flattened along the channel dimension. Step 5.2: The linear embedding layer in the fifth stage module pairs the feature map. Linear feature embedding is performed to obtain the feature map of the fifth stage module. Step 5.3: The Swing Transformer block in the fifth Stage module performs the process described in step 2.

3. After processing, the output feature map of the fifth stage module is obtained, denoted as... Step 5.4, the image patch fusion layer in the sixth stage module for feature maps Slice fusion processing is performed to obtain the feature map of the sixth stage module. Step 5.5: In the sixth stage module, the Swing Transformer block follows the process in step 5.

3. The process is performed to obtain the output feature map of the sixth stage module. Step 5.6, Image patch fusion layer in the seventh stage module Slice fusion is performed to obtain the feature map of the seventh stage module. Step 5.7: The Swing Transformer block in the seventh Stage module follows the process in step 5.

3. The process is performed to obtain the output feature map of the seventh stage module. Step 5.8, Image patch fusion layer in the eighth stage module Slice fusion is performed to obtain the feature map of the eighth stage module. Step 5.9: The Swing Transformer block in the eighth Stage module follows the process in step 5.

3. The process is performed to obtain the output feature map of the eighth stage module. Step 5.10, the second normalization layer... After normalization, the normalized feature map is obtained. Step 5.11, the second pooling layer on features Perform global pooling to obtain the pooled vector. Step 5.12, the feature coding layer uses equation (16) to... Perform a linear transformation to obtain the eigenvectors after the linear transformation. In equation (16), c represents the dimension of the feature vector; Step 5.13: The attention mechanism layer uses equation (17) to calculate the m-th full slice X′. m The k-th image patch Corresponding Gating Attention Weights In equation (17), and All are parameters of a linear transformation; tanh(·) and sigm(·) are linear activation functions; T denotes transpose. Let X′ represent the m-th full slice. m The feature encoding vector of the j-th image block, where j = 1, 2, ..., K; Step 5.14: The feature aggregation layer uses equation (18) to obtain the m-th full slice X′. m Features after aggregation Step 5.15: The second classification layer aggregates the features H. m Perform a linear transformation to obtain the m-th full slice X′. m The output result p m ∈{2,…,α}.

5. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing any of the methods for predicting the T-staging category of gastric cancer pathological whole slides based on Swin Transformer and weak supervision as described in claims 1-4, and the processor is configured to execute the programs stored in the memory.

6. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by the processor, performs the steps of the gastric cancer pathology whole slide T-staging category prediction method based on Swing Transformer and weak supervision as described in any of claims 1-4.

Images