Artificial intelligence-based method and device for typing thymoma histopathology
By using a multi-path, multi-scale visual Transformer model, and comprehensively utilizing the multi-scale information of whole-slide images of thymoma histopathology, the problem of insufficient classification results in existing technologies is solved, and high-precision thymoma classification is achieved.
Patent Information
- Application Number
- CN202210843559.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-18
- Publication Date
- 2026-02-24
- Estimated Expiration
- 2042-07-18
AI Technical Summary
Existing visual Transformer models struggle to effectively integrate multi-scale information from whole-slide images of thymoma tissue pathology, resulting in inaccurate classification results.
A multi-path, cross-scale visual Transformer model is adopted, including a cross-scale perception Transformer model and a full-slide image Transformer model. By magnifying and segmenting images at various scales, and combining local guided branches, feature aggregation branches, and global guided branches, the model utilizes a multi-head self-attention mechanism and a cross-correlation attention module to comprehensively utilize multi-scale information for pathological information classification and typing.
It achieves accurate typing of thymoma histopathological images, improves the accuracy of typing results, and achieves a Top-1 accuracy of 0.951 and an area under the mean receiver operating characteristic curve of 0.96, which is significantly better than existing methods.
Smart Images
Figure CN115471448B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of neural network technology, specifically relating to a method and apparatus for the classification of thymoma tissue pathology based on artificial intelligence. Background Technology
[0002] With the rapid development of deep learning technology in recent years, the analysis of whole-slide images of thymoma histopathology no longer relies entirely on time-consuming and laborious visual observation and manual annotation by doctors. It can now be achieved through efficient computer-aided diagnostic systems, helping doctors obtain detailed pathological information and accurate thymoma classification results. Existing technologies mainly include methods based on convolutional neural networks and methods based on visual Transformers, as detailed below:
[0003] Convolutional Neural Networks (CNNs): CNNs have long been one of the mainstream networks in computer vision, demonstrating excellent performance in most computer vision tasks, such as medical image processing. However, CNNs have some inherent limitations, such as difficulty in modeling long-range dependencies and local receptive fields.
[0004] Visual Transformers: Addressing the problems of convolutional neural networks (CNNs), visual transformers have emerged, including SwinTransformer, Pyramid Visual Transformer, LeViT, TNT, T2T-ViT, IPT, and Uformer. Compared to CNNs, visual transformers can better model global feature relationships and have demonstrated superior performance on various high- and low-level vision tasks. However, existing visual transformers are also difficult to effectively use for the classification of thymoma histopathological whole-slide images, mainly due to the following two problems: First, due to the existence of multi-head self-attention mechanisms, the computational cost of visual transformers is usually high, making them unsuitable for directly processing whole-slide images with millions of resolutions. Second, thymoma histopathological whole-slide images generally have multiple scales of information, such as images magnified 10x, 20x, and 40x. While existing visual transformers can effectively model global and local feature relationships, most of them do not consider the complementarity between multi-scale features of the image and fail to comprehensively utilize the multi-scale information of the image. Therefore, current histopathological analysis of thymoma cannot integrate multi-scale information, resulting in inaccurate classification results. Summary of the Invention
[0005] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and device for typing thymoma histopathology based on artificial intelligence, so as to solve the problem that the analysis of thymoma histopathology in the prior art cannot integrate multi-scale information, resulting in inaccurate typing results.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: a thymoma histopathological typing method based on artificial intelligence, comprising:
[0007] Obtain whole-slide images of the histopathological tissue of the thymoma to be classified;
[0008] The whole slide image of the thymoma tissue pathology to be classified is input into the trained multi-path cross-scale visual Transformer model to obtain the classification result;
[0009] The multi-path cross-scale visual Transformer model includes a cross-scale perception Transformer model and a full-slide image Transformer model. The cross-scale perception Transformer model is used to classify case information in full-slide images of thymoma histopathology, and the full-slide image Transformer model is used to output thymoma subtyping based on the thymoma histopathology full-slide images classified according to the case information.
[0010] Furthermore, the training method for the multi-path cross-scale visual Transformer model includes:
[0011] The whole slide images of thymoma tissue pathology were obtained by scanning the hematoxylin-eosin stained slides of the pre-acquired thymic carcinoma patients using a high-throughput digital scanner.
[0012] The full-slide image is magnified at various ratios, and the magnified full-slide image is divided into a preset number of non-overlapping patch full-slide images;
[0013] The patch whole slide images were labeled with thymoma subtypes, and the labeled patch whole slide images formed a training set;
[0014] The training set is input into the pre-constructed multi-path cross-scale visual Transformer model for training, resulting in a trained multi-path cross-scale visual Transformer model.
[0015] Furthermore, the cross-scale-aware Transformer model includes three input paths: a local guiding branch, a feature aggregation branch, and a global guiding branch. Each input path receives full-slide images at different scales.
[0016] The local guiding branch, feature aggregation branch, and global guiding branch each include a convolutional unit, a first stage, a second stage, a third stage, a connecting layer, a global average pooling layer, and a fully connected layer connected in sequence; the global average pooling layer is used to predict the pathological information category of the image patch;
[0017] The convolution unit is used to perform convolution processing on the input patched all-glass slide image;
[0018] The first, second, and third stages are used to extract multi-scale features from the convolutional processed patched glass slide image;
[0019] The connection layer is used to fuse the multi-scale features output from the feature extraction of the three branches through three stages;
[0020] The global average pooling layer is used to predict the pathological information category of the output image patch;
[0021] The fully connected layer is used to output the predicted pathological information category of the image patch.
[0022] Furthermore, the first stage includes: a patch merging layer, multiple basic Transformer blocks, a shape reshaping layer, and a patch merging layer connected in sequence; the second stage includes: multiple basic Transformer blocks, a shape reshaping layer, and a patch merging layer connected in sequence; the third stage includes: multiple basic Transformer blocks;
[0023] The patch merging layer is used to segment and unfold the input patched whole glass slide image into non-overlapping one-dimensional features, and map the one-dimensional features to features of a preset dimension. The patch merging layer is also used to downsample the features after the last basic Transformer block in each stage to generate hierarchical feature representations.
[0024] Furthermore, the first, second, and third stages of the feature aggregation branch also include sub-connection layers, each of which is used to fuse the multi-scale features output by the patch merging layer of the three branches.
[0025] Furthermore, the basic Transformer block of the local guiding branch is a local Transformer block with a window-based multi-head self-attention mechanism;
[0026] The basic Transformer block of the feature aggregation branch is a cross-correlation attention block with a spatially reduced attention mechanism;
[0027] The basic Transformer block of the global guiding branch is a global Transformer block with a multi-head attention mechanism.
[0028] Furthermore, the local Transformer block with a window-based multi-head self-attention mechanism includes: a window-based multi-head self-attention module, a moving window-based multi-head self-attention module, multiple layer normalizations, and multiple multilayer perceptrons.
[0029] The cross-correlation attention block with spatial reduction attention mechanism includes: a cross-correlation attention module, a spatial reduction attention mechanism, multiple layer normalization, and a multilayer perceptron.
[0030] The global Transformer block with multi-head attention mechanism includes a multi-head self-attention module, a multilayer perceptron, and layer normalization.
[0031] Furthermore, the cross-correlation attention module is used to connect the features from the three branches, each of size c×hw, and then resize them to 3×c×h×w. After passing through a 1×1 convolution, feature f1 is obtained.
[0032] Reshape the features from the feature aggregation branch into a feature f2 of size c×1×h×w;
[0033] Features f1 and f2 are batch multiplied and then an attention map of size 3×h×w is generated by passing the Sigmoid activation function.
[0034] The attention map is divided into three separate attention maps, each with a size of 1×h×w, and the feature f1 is divided into three separate features, each with a size of c×h×w, and the corresponding attention maps are multiplied element by element.
[0035] The results are then concatenated together, and a 3×3 convolution is used to generate features of size c×h×w. These features are then reshaped into features of size c×hw, which is the final output.
[0036] Furthermore, the all-glass slide image Transformer model includes:
[0037] The layers are connected in sequence: input layer, linear layer, first stage, second stage, third stage, global average pooling layer, fully connected layer, and output layer.
[0038] The first, second, and third phases all include a global Transformer block with a multi-head attention mechanism.
[0039] This application provides an artificial intelligence-based thymoma histopathological typing device, comprising:
[0040] The acquisition module is used to acquire whole-slide images of thymoma tissue pathology to be classified;
[0041] The classification module is used to input the whole slide image of the thymoma histopathology to be classified into a trained multi-path cross-scale visual Transformer model to obtain the classification result;
[0042] The multi-path cross-scale visual Transformer model includes a cross-scale perception Transformer model and a full-slide image Transformer model. The cross-scale perception Transformer model is used to classify case information in full-slide images of thymoma histopathology, and the full-slide image Transformer model is used to output thymoma subtyping based on the thymoma histopathology full-slide images classified according to the case information.
[0043] The beneficial effects that can be achieved by adopting the above technical solution in this invention include:
[0044] (1) The multi-path cross-scale visual Transformer of the present invention comprehensively utilizes multi-scale information and local pathological information in the whole slide image of thymoma histopathology. It effectively simulates the process by which pathologists comprehensively observe multiple multi-scale (10x, 20x and 40x) local image patches in a whole slide image of histopathology to determine the pathological information category and then diagnose the corresponding thymoma subtype. Therefore, it can accurately predict the pathological information category and thymoma subtype.
[0045] (2) The whole slide image Transformer of the present invention uses pathological information category as prior information to convert two-dimensional whole slide images into one-dimensional feature matrices and input them into the network, which solves the problem of high computational cost caused by million-level resolution whole slide images.
[0046] (3) The multi-path cross-scale visual Transformer and the whole slide image Transformer proposed in this invention achieved Top-1 accuracy of 0.939 and 0.951 in pathological information classification and thymoma typing, respectively. Attached Figure Description
[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a schematic diagram illustrating the steps of the thymoma histopathological typing method based on artificial intelligence of the present invention;
[0049] Figure 2This is a flowchart illustrating the classification method for thymoma histopathology based on artificial intelligence according to the present invention.
[0050] Figure 3 This is a schematic diagram illustrating the workflow of the cross-correlation attention module of the present invention;
[0051] Figure 4 This is a schematic diagram of the thymoma histopathology typing device based on artificial intelligence according to the present invention;
[0052] Figure 5 This is a schematic diagram of the hardware structure of the implementation environment for the thymoma histopathological typing method based on artificial intelligence of the present invention. Detailed Implementation
[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0054] Thymic epithelial tumors (i.e., thymomas) are rare primary anterior mediastinal tumors. The World Health Organization classifies thymomas into types A, AB, B1, B1+B2, B2, B2+B3, B3, and C. Due to the complex histological morphology of thymomas in whole-slide images and the frequent mixing of various features, thymomas are among the most challenging tumors for histological classification.
[0055] The following describes, with reference to the accompanying drawings, a specific artificial intelligence-based thymoma histopathological classification method and apparatus provided in an embodiment of this application.
[0056] like Figure 1 As shown in the embodiments of this application, the artificial intelligence-based histopathological classification method for thymoma tissue includes:
[0057] S101, Obtain whole-slide images of thymoma tissue pathology to be classified;
[0058] In this application, hematoxylin-eosin stained slides from thymic carcinoma patients were obtained and scanned using a high-throughput digital scanner to create whole slide images of thymoma tissue pathology.
[0059] S102, input the whole slide image of the thymoma histopathology to be classified into the trained multi-path cross-scale visual Transformer model to obtain the classification result.
[0060] The multi-path cross-scale visual Transformer model includes a cross-scale perception Transformer model and a full-slide image Transformer model. The cross-scale perception Transformer model is used to classify case information in full-slide images of thymoma histopathology, and the full-slide image Transformer model is used to output thymoma subtyping based on the thymoma histopathology full-slide images classified according to the case information.
[0061] The working principle of the AI-based thymoma histopathological classification method is as follows: This application first trains a multi-path cross-scale visual Transformer model, which includes a cross-scale perception Transformer model and a whole-slide image Transformer model. The cross-scale perception Transformer model is used to classify case information from whole-slide images of thymoma histopathology. The whole-slide image Transformer model is used to output thymoma classification based on the classified case information, ultimately obtaining the classification result. Specifically, this application first uses the cross-scale perception Transformer model to classify pathological information related to thymoma classification, and then uses this pathological information as prior knowledge to assist the whole-slide image Transformer model in thymoma classification. To fully utilize the multiple scale information in the whole-slide image, image patches magnified by 10x, 20x, and 40x are input into the cross-scale perception Transformer through three paths, respectively. The cross-scale perceptual Transformer not only employs parallel multiple paths to capture receptive field features at different scales from whole-slide images, but also introduces a cross-correlation attention module to aggregate these multi-scale features, achieving complementarity between cross-scale spatial information. Furthermore, the whole-slide image Transformer model can effectively convert two-dimensional whole-slide images into a one-dimensional feature matrix with pathological information category labels as input, and then output the specific subtype of thymoma.
[0062] The multi-path, multi-scale visual Transformer of this invention comprehensively utilizes multi-scale and local pathological information from whole-slide images of thymoma histopathology. It effectively simulates the process by which pathologists comprehensively observe multiple multi-scale (10x, 20x, and 40x) local image patches within a single whole-slide image to determine the pathological information category and subsequently diagnose the corresponding thymoma subtype. Therefore, it can accurately predict the pathological information category and thymoma subtype. The whole-slide image Transformer of this invention uses the pathological information category as prior information, converting the two-dimensional whole-slide image into a one-dimensional feature matrix input into the network, thus solving the problem of high computational cost associated with million-resolution whole-slide images.
[0063] The multi-path, multi-scale visual Transformer and whole-slide image Transformer proposed in this invention achieved Top-1 accuracy of 0.939 and 0.951 in pathological information classification and thymoma subtyping, respectively. Experiments show that the proposed method outperforms state-of-the-art convolutional neural network-based classification methods and Transformer-based classification methods.
[0064] The multi-path, cross-scale visual Transformer provided in this invention combines a cross-scale perception Transformer and a full-slide image Transformer for thymoma classification. It can effectively distinguish between AB, B1, B1+B2, B2, B2+B3, B3, and C type thymomas, with classification results largely consistent with expert-annotated results. Its Top-1 precision is 0.951, the area under the mean receiver operating characteristic (ROC) curve is 0.96, the mean sensitivity is 0.875 (95% confidence interval: 0.528-0.970), and the mean specificity is 0.982 (95% confidence interval: 0.911-0.992). Compared with existing combinations of visual Transformer, TNT, LeViT, CrossViT, ResNet-101, DenseNet-121, GuSA-Net, ROPsNet, CPWA-Net, IL-MCAM, and all-slide image Transformer, it achieves the best results in four metrics: recall, Top-1 precision, mean precision, and F-score, and the second-best results in accuracy.
[0065] This application employs a multi-path, multi-scale visual Transformer to perform specific classification of whole-slide images of thymoma histopathology. The classification results can be applied to the diagnosis of thymoma, which can help patients assess prognosis and provide effective cancer treatment strategies and targeted therapy options.
[0066] In some embodiments, the training method of the multi-path cross-scale visual Transformer model includes:
[0067] The whole slide images of thymoma tissue pathology were obtained by scanning the hematoxylin-eosin stained slides of the pre-acquired thymic carcinoma patients using a high-throughput digital scanner.
[0068] The full-slide image is magnified at various ratios, and the magnified full-slide image is divided into a preset number of non-overlapping patch full-slide images;
[0069] The patch whole slide images were labeled with thymoma subtypes, and the labeled patch whole slide images formed a training set;
[0070] The training set is input into the pre-constructed multi-path cross-scale visual Transformer model for training, resulting in a trained multi-path cross-scale visual Transformer model.
[0071] Specifically, this application collected 323 hematoxylin-eosin stained slides from 129 patients with thymoma. These slides were scanned using a high-throughput digital scanner to create whole-slide images for thymoma histopathology. Each whole-slide image had three magnification ratios: 10x, 20x, and 40x, and each whole-slide image was divided into 3000 non-overlapping patches. Experienced pathologists labeled the whole-slide images with eight thymoma subtypes and the image patches with ten categories of pathological information related to the thymoma subtypes, ultimately forming a large-scale dataset of whole-slide images for thymoma histopathology.
[0072] The dataset contains 323 full-slide images, with 243 images used as the training set and the remaining 80 images as the test set. Both sub-networks use the cross-entropy loss function and an Adam optimizer with momentum β1 = 0.9 and β2 = 0.999, and are trained for 160 epochs. For the cross-scale perception Transformer model, the batch size is 64 and the initial learning rate is 2e-3; for the full-slide image Transformer, the batch size is 8 and the initial learning rate is 1e-3, resulting in a pre-trained multi-path cross-scale visual Transformer model.
[0073] In some embodiments, such as Figure 2 As shown, the cross-scale-aware Transformer model includes three input paths: a local guiding branch, a feature aggregation branch, and a global guiding branch. Each input path receives full-slide images at different scales.
[0074] The local guiding branch, feature aggregation branch, and global guiding branch each include a convolutional unit, a first stage, a second stage, a third stage, a connecting layer, a global average pooling layer, and a fully connected layer connected in sequence; the global average pooling layer is used to predict the pathological information category of the image patch;
[0075] The convolution unit is used to perform convolution processing on the input patched all-glass slide image;
[0076] The first, second, and third stages are used to extract multi-scale features from the convolutional processed patched glass slide image;
[0077] The connection layer is used to fuse the multi-scale features output from the feature extraction of the three branches through three stages;
[0078] The global average pooling layer is used to predict the pathological information category of the output image patch;
[0079] The fully connected layer is used to output the predicted pathological information category of the image patch.
[0080] Specifically, the cross-scale-aware Transformer model provided in this application is a three-branch structure, including a local guiding branch, a feature aggregation branch, and a global guiding branch. These branches take image patches magnified by 40x, 20x, and 10x as input, respectively, and then extract and fuse multi-scale features to achieve classification and typing of multi-scale features, making the typing results more accurate.
[0081] In a preferred embodiment, the first stage includes: a patch merging layer, multiple basic Transformer blocks, a shape reshaping layer, and a patch merging layer connected in sequence; the second stage includes: multiple basic Transformer blocks, a shape reshaping layer, and a patch merging layer connected in sequence; and the third stage includes: multiple basic Transformer blocks.
[0082] The patch merging layer is used to segment and unfold the input patched whole glass slide image into non-overlapping one-dimensional features, and map the one-dimensional features to features of a preset dimension. The patch merging layer is also used to downsample the features after the last basic Transformer block in each stage to generate hierarchical feature representations.
[0083] The first, second, and third stages of the feature aggregation branch also include sub-connection layers, each of which is used to fuse the multi-scale features output by the patch merging layer of the three branches.
[0084] Preferably, the basic Transformer block of the local guiding branch is a local Transformer block with a window-based multi-head self-attention mechanism;
[0085] The basic Transformer block of the feature aggregation branch is a cross-correlation attention block with a spatially reduced attention mechanism;
[0086] The basic Transformer block of the global guiding branch is a global Transformer block with a multi-head attention mechanism.
[0087] Specifically, such as Figure 2 As shown, the local guiding branch inputs an image patch magnified 40 times. After being processed by a convolutional unit, it enters the first stage, which includes a patch merging layer, a local Transformer block, a shape reshaping layer, and another patch merging layer connected in sequence. Then it enters the second stage, which includes a local Transformer block, a shape reshaping layer, and another patch merging layer connected in sequence. Then it enters the third stage, which includes a local Transformer block. The output of the third stage is sent to a connection layer, a global average pooling layer, and a fully connected layer.
[0088] The feature aggregation branch takes a 20x magnified image patch as input. After being processed by a convolutional unit, it enters the first stage, which includes a patch merging layer, a sub-connection layer, a cross-correlation attention block, a shape reshaping layer, and a patch merging layer connected in sequence. Then it enters the second stage, which also includes a sub-connection layer, a cross-correlation attention block, a shape reshaping layer, and a patch merging layer connected in sequence. Finally, it enters the third stage, which includes a sub-connection layer and a cross-correlation attention block. The output of the third stage is fed to a connection layer, a global average pooling layer, and a fully connected layer.
[0089] The global pilot branch takes in an image patch magnified 10 times. After being processed by a convolutional unit, it enters the first stage, which includes a patch merging layer, all Transformer blocks, a shape reshaping layer, and another patch merging layer connected in sequence. Then it enters the second stage, which includes all Transformer blocks, a shape reshaping layer, and another patch merging layer connected in sequence. Then it enters the third stage, which includes local Transformer blocks. The output of the third stage is sent to a connection layer, a global average pooling layer, and a fully connected layer.
[0090] In this process, the patch merging layers at each stage of the local guiding branch, feature aggregation branch, and global guiding branch output multi-scale features, which are then fused into sub-connection layers within the feature aggregation branch to output multi-scale features. Finally, the fully connected layer outputs the classification results, which include spindle-shaped thymic epithelial cells, B1 thymic epithelial cells, B2 thymic epithelial cells, B3 thymic epithelial cells, fibrous septa, erythrocytes, lymphocytes, perivascular spaces, medullary regions, and tumors.
[0091] The local Transformer block with a window-based multi-head self-attention mechanism includes: a window-based multi-head self-attention module, a moving window-based multi-head self-attention module, multiple layer normalizations, and multiple multilayer perceptrons.
[0092] The cross-correlation attention block with spatial reduction attention mechanism includes: a cross-correlation attention module, a spatial reduction attention mechanism, multiple layer normalization, and a multilayer perceptron.
[0093] The global Transformer block with multi-head attention mechanism includes a multi-head self-attention module, a multilayer perceptron, and layer normalization.
[0094] Specifically, this application first uses a patch merging layer to segment and expand the input patch into non-overlapping one-dimensional features, and then maps these one-dimensional features to features of the desired dimension size, constructing local guiding branches, feature aggregation branches, and global guiding branches respectively to adapt to the characteristics of multi-scale image input. The local guiding branch, feature aggregation branch, and global guiding branch use local Transformer blocks with a window-based multi-head self-attention mechanism, cross-scale-aware Transformer blocks with a spatial reduction attention mechanism, and global Transformer blocks with a multi-head self-attention mechanism as basic Transformer blocks, respectively. To effectively predict the pathological information category of the input image patch, the feature aggregation branch differs from the local and global guiding branches by fusing multi-scale features from each Transformer block in the three different branches. The feature aggregation branch also includes a cross-correlation attention module, which models the relationship between multi-scale features and potential pathological information categories, thereby promoting the fusion of multi-scale features. Each branch contains three stages; after the last basic transformer block in each stage, a patch merging layer is used to downsample the features to generate hierarchical feature representations. Finally, global average pooling layers and fully connected layers are used to predict the pathological information category of the output image patch. The cross-scale-aware Transformer can capture richer global information through three different non-local self-attention mechanisms and make full use of the multi-scale information of the image.
[0095] Among them, the global Transformer block with multi-head self-attention mechanism has a configuration similar to the classic Transformer block, including a multi-head self-attention, a multi-layer perceptron, and two layer normalizations.
[0096] The local Transformer block with a window-based multi-head self-attention mechanism employs two consecutive SwingTransformer blocks, using window-based multi-head self-attention and shift-window-based multi-head self-attention respectively, instead of the traditional multi-head self-attention, thus maintaining the advantages of SwingTransformer. Compared to the traditional multi-head self-attention mechanism, the window-based multi-head self-attention mechanism focuses more on modeling feature relationships within non-overlapping local windows, effectively improving the extraction rate of local image information while significantly reducing the computational cost of the Transformer block.
[0097] The cross-scale perceptual Transformer block with spatial reduction attention mechanism consists of a cross-correlation attention module, a spatial reduction attention module, a multilayer perceptron, and two layer normalization modules. Unlike the global and local Transformer blocks, this Transformer block employs a cross-correlation attention module to aggregate multi-scale features from different branches, which can effectively enhance the representation of potential pathological information.
[0098] In some embodiments, such as Figure 3 As shown, the cross-correlation attention module is used to connect features of size c×hw from three branches, then resize them to 3×c×h×w, and obtain feature f1 after 1×1 convolution;
[0099] Reshape the features from the feature aggregation branch into a feature f2 of size c×1×h×w;
[0100] Features f1 and f2 are batch multiplied and then an attention map of size 3×h×w is generated by passing the Sigmoid activation function.
[0101] The attention map is divided into three separate attention maps, each with a size of 1×h×w, and the feature f1 is divided into three separate features, each with a size of c×h×w, and the corresponding attention maps are multiplied element by element.
[0102] The results are then concatenated together, and a 3×3 convolution is used to generate features of size c×h×w. These features are then reshaped into features of size c×hw, which is the final output.
[0103] Understandably, the cross-correlation attention module can comprehensively consider the global and local features of different receptive fields, and enhance the multi-scale fusion features through the spatial attention map generated by the cross-correlation attention mechanism, thereby further improving the classification accuracy of potential pathological information categories.
[0104] In some embodiments, such as Figure 2As shown, the all-glass slide image Transformer model includes:
[0105] The layers are connected in sequence: input layer, linear layer, first stage, second stage, third stage, global average pooling layer, fully connected layer, and output layer.
[0106] The first, second, and third phases all include a global Transformer block with a multi-head attention mechanism.
[0107] like Figure 2 As shown, the Transformer model for the all-glass slide image includes a receiver feature matrix layer, a linear layer, a first stage, a second stage, a third stage, a global average pooling layer, and a fully connected layer connected in sequence.
[0108] The first, second, and third stages all include a basic Transformer block. The structure of the basic Transformer block in the all-slide image Transformer model is the same as that of the global Transformer block.
[0109] Specifically, the typing process of the whole-slide image Transformer model is as follows: a fixed number of image patches are randomly selected for each whole-slide image. A cross-scale-aware Transformer generates multi-scale feature codes and pathological information category labels, which are then concatenated along the channel dimension. Finally, the features of all patches in the image are concatenated. Each whole-slide image is transformed into a feature matrix of size M×769 containing prior pathological information, where M represents the number of segmentation patches for each whole-slide image. Therefore, the computational cost of the Transformer is significantly reduced. The whole-slide image Transformer also consists of three stages, each containing two classic transformer blocks. Finally, a global average pooling layer and a fully connected layer are used to output the typing results of the thymoma histopathological whole-slide image.
[0110] The technical solution provided in this application can accurately predict the pathological information category and thymoma classification, helping doctors to effectively assess the prognosis and select treatment options for patients. It includes the following beneficial effects:
[0111] This application utilizes a cross-scale perceptual Transformer model to classify pathological information of thymoma. It effectively distinguishes spindle-shaped thymic epithelial cells, B1 thymic epithelial cells, B2 thymic epithelial cells, B3 thymic epithelial cells, fibrous septa, erythrocytes, lymphocytes, perivascular spaces, medullary regions, and tumors using only hematoxylin and eosin stained whole-slide images without immunohistochemical imaging. Its Top-1 precision is 0.939, and the area under the mean receiver operating characteristic (AUC) curve is 0.93. Compared with existing visual Transformers, TNT, LeViT, CrossViT, ResNet-101, DenseNet-121, GuSA-Net, ROPsNet, CPWA-Net, and IL-MCAM, this model achieves the best results in recall, Top-1 precision, mean precision, and F-score, and the second-best result in accuracy. Since the classification results of pathological information are closely related to the thymoma subtyping, the high-precision pathological information classification results output by this model help improve the accuracy of subsequent thymoma subtyping.
[0112] like Figure 4 As shown in the figure, this application provides an artificial intelligence-based thymoma histopathological typing device, comprising:
[0113] The acquisition module 201 is used to acquire whole-slide images of thymoma tissue pathology to be classified;
[0114] The classification module 201 is used to input the whole slide image of the thymoma histopathology to be classified into a trained multi-path cross-scale visual Transformer model to obtain the classification result.
[0115] The multi-path cross-scale visual Transformer model includes a cross-scale perception Transformer model and a full-slide image Transformer model. The cross-scale perception Transformer model is used to classify case information in full-slide images of thymoma histopathology, and the full-slide image Transformer model is used to output thymoma subtyping based on the thymoma histopathology full-slide images classified according to the case information.
[0116] The working principle of the artificial intelligence-based thymoma histopathology typing device provided in this application is as follows: the acquisition module 201 acquires the whole slide image of the thymoma histopathology to be typed; the typing module 201 inputs the whole slide image of the thymoma histopathology to be typed into a trained multi-path cross-scale visual Transformer model to obtain the typing result; wherein, the multi-path cross-scale visual Transformer model includes a cross-scale perception Transformer model and a whole slide image Transformer model; the cross-scale perception Transformer model is used to classify the case information of the whole slide image of the thymoma histopathology, and the whole slide image Transformer model is used to output the thymoma typing based on the thymoma histopathology whole slide image classified according to the case information.
[0117] This application provides a computer device, including a memory and a processor, and may further include a network interface. The memory stores a computer program and may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The computer device stores an operating system, and the memory is an example of a computer-readable medium. When the computer program is executed by the processor, it causes the processor to perform an artificial intelligence-based thymoma histopathological typing method. Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0118] In one embodiment, the AI-based thymoma histopathological typing method provided in this application can be implemented as a computer program, which can be implemented in, for example... Figure 5 It runs on the computer device shown.
[0119] In some embodiments, when the computer program is executed by the processor, the processor performs the following steps: acquiring a whole-slide image of a thymoma histopathology specimen to be classified; inputting the whole-slide image of the thymoma histopathology specimen to be classified into a trained multi-path cross-scale visual Transformer model to obtain a classification result; wherein the multi-path cross-scale visual Transformer model includes a cross-scale perception Transformer model and a whole-slide image Transformer model; the cross-scale perception Transformer model is used to classify case information in the whole-slide image of the thymoma histopathology specimen, and the whole-slide image Transformer model is used to output a thymoma classification based on the thymoma histopathology specimen classified according to the case information.
[0120] This application also provides a computer storage medium, examples of which include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital optical disc (DVD) or other optical storage, magnetic tape storage or other magnetic storage devices, or any other non-transfer medium, which can be used to store information that can be accessed by a computing device.
[0121] In some embodiments, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, acquires a whole-slide image of a thymoma histopathology specimen to be classified; inputs the whole-slide image of the thymoma histopathology specimen to be classified into a trained multi-path cross-scale visual Transformer model to obtain a classification result; wherein, the multi-path cross-scale visual Transformer model includes a cross-scale perception Transformer model and a whole-slide image Transformer model; the cross-scale perception Transformer model is used to classify case information in the whole-slide image of the thymoma histopathology specimen, and the whole-slide image Transformer model is used to output a thymoma classification based on the thymoma histopathology specimen classified according to the case information.
[0122] It is understood that the method embodiments provided above correspond to the device embodiments described above, and the specific details can be referred to each other, which will not be repeated here.
[0123] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0124] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0127] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A classification method for thymoma histopathology based on artificial intelligence, characterized in that, include: Obtain whole-slide images of the histopathological tissue of the thymoma to be classified; The whole slide image of the thymoma tissue pathology to be classified is input into the trained multi-path cross-scale visual Transformer model to obtain the classification result; The multi-path cross-scale visual Transformer model includes a cross-scale perception Transformer model and a full slide image Transformer model. The cross-scale perception Transformer model is used to classify case information in full slide images of thymoma histopathology, and the full slide image Transformer model is used to output thymoma subtyping based on the thymoma histopathology full slide images classified according to the case information. The cross-scale-aware Transformer model includes three input paths: a local guiding branch, a feature aggregation branch, and a global guiding branch. Each input path receives full-slide images at different scales. The local guiding branch, feature aggregation branch, and global guiding branch each include a convolutional unit, a first stage, a second stage, a third stage, a connecting layer, a global average pooling layer, and a fully connected layer connected in sequence; the global average pooling layer is used to predict the pathological information category of the image patch; The convolution unit is used to perform convolution processing on the input patched all-glass slide image; The first, second, and third stages are used to extract multi-scale features from the convolutional processed patched glass slide image; The connection layer is used to fuse the multi-scale features output from the feature extraction of the three branches through three stages; The global average pooling layer is used to predict the pathological information category of the output image patch; The fully connected layer is used to output the predicted pathological information category of the image patch; The all-glass slide image Transformer model includes: The layers are connected in sequence: input layer, linear layer, first stage, second stage, third stage, global average pooling layer, fully connected layer, and output layer. The first, second, and third phases all include a global Transformer block with a multi-head attention mechanism; The training method for the multi-path, cross-scale visual Transformer model includes: The whole slide images of thymoma tissue pathology were obtained by scanning the hematoxylin-eosin stained slides of the pre-acquired thymic carcinoma patients using a high-throughput digital scanner. The full-slide image is magnified at various ratios, and the magnified full-slide image is divided into a preset number of non-overlapping patch full-slide images; The patch whole slide images were labeled with thymoma subtypes, and the labeled patch whole slide images formed a training set; The training set is input into the pre-constructed multi-path cross-scale visual Transformer model for training, and a trained multi-path cross-scale visual Transformer model is obtained. The first stage includes: a patch merging layer, multiple basic Transformer blocks, a shape reshaping layer, and a patch merging layer connected in sequence; the second stage includes: multiple basic Transformer blocks, a shape reshaping layer, and a patch merging layer connected in sequence; the third stage includes: multiple basic Transformer blocks; The patch merging layer is used to segment and unfold the input patched whole glass slide image into non-overlapping one-dimensional features, and map the one-dimensional features to features of a preset dimension. The patch merging layer is also used to downsample the features after the last basic Transformer block in each stage to generate hierarchical feature representations.
2. The method according to claim 1, characterized in that, The first, second, and third stages of the feature aggregation branch also include sub-connection layers, each of which is used to fuse the multi-scale features output by the patch merging layer of the three branches.
3. The method according to claim 1, characterized in that, The basic Transformer block of the local guiding branch is a local Transformer block with a window-based multi-head self-attention mechanism; The basic Transformer block of the feature aggregation branch is a cross-correlation attention block with a spatially reduced attention mechanism; The basic Transformer block of the global guiding branch is a global Transformer block with a multi-head attention mechanism.
4. The method according to claim 3, characterized in that, The local Transformer block with a window-based multi-head self-attention mechanism includes: a window-based multi-head self-attention module, a moving window-based multi-head self-attention module, multiple layer normalizations, and multiple multilayer perceptrons. The cross-correlation attention block with spatial reduction attention mechanism includes: a cross-correlation attention module, a spatial reduction attention mechanism, multiple layer normalization, and a multilayer perceptron. The global Transformer block with multi-head attention mechanism includes a multi-head self-attention module, a multilayer perceptron, and layer normalization.
5. The method according to claim 4, characterized in that, The cross-correlation attention module is used to connect features of size c×hw from three branches, then resize them to 3×c×h×w, and obtain feature f1 after 1×1 convolution; Reshape the features from the feature aggregation branch into a feature f2 of size c×1×h×w; Features f1 and f2 are batch multiplied and then an attention map of size 3×h×w is generated by passing the Sigmoid activation function. The attention map is divided into three separate attention maps, each with a size of 1×h×w, and the feature f1 is divided into three separate features, each with a size of c×h×w, and the corresponding attention maps are multiplied element by element. The results are then concatenated together, and a 3×3 convolution is used to generate features of size c×h×w. These features are then reshaped into features of size c×hw, which is the final output.
6. A typing device for thymoma histopathology based on artificial intelligence, characterized in that, include: The acquisition module is used to acquire whole-slide images of thymoma tissue pathology to be classified; The classification module is used to input the whole slide image of the thymoma histopathology to be classified into a trained multi-path cross-scale visual Transformer model to obtain the classification result; The multi-path cross-scale visual Transformer model includes a cross-scale perception Transformer model and a full slide image Transformer model. The cross-scale perception Transformer model is used to classify case information in full slide images of thymoma histopathology, and the full slide image Transformer model is used to output thymoma subtyping based on the thymoma histopathology full slide images classified according to the case information. The training method for the multi-path, cross-scale visual Transformer model includes: The whole slide images of thymoma tissue pathology were obtained by scanning the hematoxylin-eosin stained slides of the pre-acquired thymic carcinoma patients using a high-throughput digital scanner. The full-slide image is magnified at various ratios, and the magnified full-slide image is divided into a preset number of non-overlapping patch full-slide images; The patch whole slide images were labeled with thymoma subtypes, and the labeled patch whole slide images formed a training set; The training set is input into the pre-constructed multi-path cross-scale visual Transformer model for training, and a trained multi-path cross-scale visual Transformer model is obtained. The cross-scale-aware Transformer model includes three input paths: a local guiding branch, a feature aggregation branch, and a global guiding branch. Each input path receives full-slide images at different scales. The local guiding branch, feature aggregation branch, and global guiding branch each include a convolutional unit, a first stage, a second stage, a third stage, a connecting layer, a global average pooling layer, and a fully connected layer connected in sequence; the global average pooling layer is used to predict the pathological information category of the image patch; The convolution unit is used to perform convolution processing on the input patched all-glass slide image; The first, second, and third stages are used to extract multi-scale features from the convolutional processed patched glass slide image; The connection layer is used to fuse the multi-scale features output from the feature extraction of the three branches through three stages; The global average pooling layer is used to predict the pathological information category of the output image patch; The fully connected layer is used to output the predicted pathological information category of the image patch; The all-glass slide image Transformer model includes: The layers are connected in sequence: input layer, linear layer, first stage, second stage, third stage, global average pooling layer, fully connected layer, and output layer. The first, second, and third phases all include a global Transformer block with a multi-head attention mechanism; The first stage includes: a patch merging layer, multiple basic Transformer blocks, a shape reshaping layer, and a patch merging layer connected in sequence; the second stage includes: multiple basic Transformer blocks, a shape reshaping layer, and a patch merging layer connected in sequence; the third stage includes: multiple basic Transformer blocks; The patch merging layer is used to segment and unfold the input patched whole glass slide image into non-overlapping one-dimensional features, and map the one-dimensional features to features of a preset dimension. The patch merging layer is also used to downsample the features after the last basic Transformer block in each stage to generate hierarchical feature representations.