Pathological image rapid analysis method and device based on scale alignment transformer

Abstract

The application discloses a pathological image rapid analysis method and device based on a magnification alignment Transformer, and comprises the following steps: identifying and segmenting a pathological image dataset to obtain pathological image blocks; screening and obtaining a Patch set; constructing a MAT classification network model comprising a self-supervised magnification alignment module and a global-local Transformer classification module; performing self-supervised magnification alignment training on the MAT classification network model by using the Patch set in the self-supervised magnification alignment module; training the MAT classification network model by using a CNN-Transformer; and obtaining a pathological image classification prediction result by using the trained MAT classification network model. The application proposes a MAT classification network model, aligns low-magnification images to high-magnification images at a feature level by using a self-supervised magnification alignment mechanism, simultaneously predicts pathological image features by using a CNN-Transformer attention mechanism, fully utilizes the information of the low-magnification images, and greatly reduces the time and space costs required for model prediction.

Description

Technical Field

[0001] This invention belongs to the technical field of image analysis, specifically relating to a method and apparatus for rapid analysis of pathological images based on magnification alignment Transformer. Background Technology

[0002] Histopathological slides can be converted into whole-slide images (WSIs) using a digital pathology scanner. With the development of image processing technology, intelligent assisted diagnosis using computer algorithms has become possible, giving rise to computational pathology. In recent years, with the rapid development of artificial intelligence technology, deep learning has achieved remarkable results in many computational pathology tasks, such as cancer diagnosis, prognosis, and risk stratification. Although deep learning models achieve excellent performance in the aforementioned prediction tasks, their speed and efficiency in image processing remain relatively low. For example, existing methods require 5 to 10 minutes to predict and analyze a complete WSI, which is difficult to meet the practical needs of clinical diagnosis.

[0003] The primary factor affecting computational efficiency is the gigabit-level image resolution of WSI. To acquire sufficient image information, existing methods typically process WSI at high magnification (e.g., 400x or 200x). Despite some image compression, the required computational resources and time remain enormous. Conversely, while using lower resolution images (e.g., 100x or 50x) can significantly reduce computation time and resources, it leads to severe image information loss, drastically reducing model accuracy. Therefore, the balance between model performance and efficiency is essentially a trade-off between model performance and resolution.

[0004] To fully utilize low-magnification images, two issues need to be addressed: 1. Do low-magnification images possess diagnostic value? 2. Can deep learning recover predictive information from low-magnification images? Clinically, doctors can make preliminary judgments at lower magnifications, indicating that low-magnification images have some diagnostic value. Technically, numerous studies have demonstrated that deep learning super-resolution algorithms can generate high-magnification images from low-magnification images, proving that deep learning algorithms can achieve information restoration from low-magnification images.

[0005] However, while using low-magnification images can save a lot of time and computational resources, low-magnification images lose a lot of detailed information, which will lead to serious degradation of model performance. On the other hand, existing methods for analyzing pathological images at high magnification (such as 400x or 200x) require a lot of time and computational resources, which is difficult to meet the requirements of practical applications. Therefore, how to fully combine the advantages of high and low magnification images for rapid analysis of pathological images is an urgent problem to be solved. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a method and apparatus for rapid analysis of pathological images based on magnification alignment Transformer. This invention proposes a MAT classification network model that utilizes a self-supervised magnification alignment mechanism to align low-magnification images to high-magnification images at the feature level. Simultaneously, it employs a CNN-Transformer attention mechanism to predict pathological image features, fully utilizing the information from low-magnification images and significantly reducing the time and space costs required for model prediction.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a rapid analysis method for pathological images based on magnification alignment Transformer, comprising the following steps:

[0009] Acquire a pathological image dataset, identify and segment tissue regions, obtain a mask for WSI tissue regions, and use the mask to perform patching operations on the WSI tissue regions; filter the WSI tissue regions after patching operations, remove patches whose internal tissue region area is too small, and obtain a patch set; the tissue region area is calculated based on the tissue area contained in the mask at the same position as the patch.

[0010] A MAT classification network model is constructed, comprising a self-supervised scalar alignment module and a global-local Transformer classification module. The self-supervised scalar alignment module includes two structurally identical feature extractors, used to extract high-scalar features and generate semantically highly similar alignment features, respectively. The global-local Transformer classification module includes a global attention module and a local attention module. The global attention module utilizes the Transformer's ability to learn global information to discover global features between instances; the local attention mechanism utilizes the CNN's ability to learn detailed information to discover local features of instances. The features discovered by the global and local attention modules are aggregated, and a prediction result is obtained through inference by a fully connected layer.

[0011] The MAT classification network model is trained by performing self-supervised magnification alignment training using a patch set, aligning low-magnification images to high-magnification images at the feature level to obtain a magnification-aligned patch set. In the global attention module, a Transformer is used to extract global information from the magnification-aligned patch set, while in the local attention module, a CNN is used to capture local information from the same patch set. Then, pathological image features are aggregated based on the global and local information, and prediction results are inferred through fully connected layers. The prediction loss is then calculated, and the trained MAT classification network model is obtained through backpropagation. The self-attention mechanism in the Transformer covers all patches, thus obtaining global attention. The fixed-size convolutional kernels and sliding window mechanism in the local attention module are more sensitive to neighboring patches.

[0012] The trained MAT classification network model is used to obtain the classification prediction results of pathological images.

[0013] As a preferred technical solution, the step of identifying and segmenting the tissue region, obtaining a mask of the WSI tissue region, and using the mask to perform a patching operation on the WSI tissue region specifically involves:

[0014] ImageNet was used to perform binary classification on the pathological image dataset to identify normal tissue areas and blank and contaminated areas, and to obtain the mask of WSI tissue areas;

[0015] The mask of the WSI tissue region is patched, and all images are cropped into patches of a set size. The patches are then input into a ResNet50 pre-trained on ImageNet for feature extraction. The size of the patch varies proportionally with the magnification of the image used.

[0016] As a preferred technical solution, the step of using a patch set to perform self-supervised scaling alignment training on the MAT classification network model specifically involves:

[0017] Using two task-dependent feature extractors Φ with identical structures High (·) and Φ Align (·), freeze Φ High The parameters (·) are used to extract high-magnification features and generate semantically highly similar alignment features, i.e., Φ. High (·) Accepts high-magnification image input and outputs high-magnification features, Φ Align (·) Accept low-resolution images with the same field of view as high-magnification images and generate semantically highly similar alignment features;

[0018] L1 loss is used to reduce the absolute distance between features at different magnifications and to achieve semantic alignment. The loss function is shown below:

[0019]

[0020] Where X i It is Φ High (·) Output features of the i-th block, x i It is Φ Align (·) Output features of the i-th block.

[0021] As a preferred technical solution, the step of utilizing Transformer to discover global information of the scale-aligned patch set in the global attention module specifically involves:

[0022] In the global attention module, the features F of the patch set are... B×(N+1)×L Input the global attention module, and use the Transformer within the global attention module to discover global information about the scale-aligned patch set. B×(N+1)×L The process involves first generating Q, K, and V vectors using a fully connected layer, then using the Q and K vectors to generate global attention using a self-attention mechanism, followed by a vector dot product with V, and finally concatenating with randomly initialized Class Tokens to obtain the final output. The complete process is defined as follows:

[0023] Q = MLP(F B×N×L )

[0024] K = MLP(F B×N×L )

[0025] V = MLP(F B×N×L )

[0026] Att B×N×L =Softmax(Transpose) 0，2，1 (Q×K))

[0027] Att B×N×L =Concat(V×Transpose) 0，2，1 (Att B×N×L ))

[0028]

[0029] Where B represents the batch size, N represents the number of features, L represents the feature length, 1 represents the class label feature, MLP represents a multilayer perceptron, and Q, K, and V all represent features F. B×N×L The intermediate variable for the transformation, Att B×N×L Representing intermediate variables in the attention mechanism. The Class Token is generated using a random initialization strategy and is used to learn all instance information; Transpose 0，2，1 (·) indicates that the 0th, 1st, and 2nd dimensions of the feature are adjusted to 0, 2, and 1.

[0030] As a preferred technical solution, the step of using a CNN to capture local information of a scale-aligned patch set in the local attention module specifically involves:

[0031] In the local attention module, dilated convolutions with dilation levels of 1, 3, and 5 from CNNs are used to construct an instance-level feature pyramid to capture instance information at three scales.

[0032] The instance information f1, f2, and f3 at the three scales are fused using an averaging method:

[0033] f1 = Conv1(f)

[0034] f2 = Conv3(f)

[0035] f3 = Conv5(f)

[0036] f out =Mean(f1, f2, f3)

[0037] Where f represents the input feature, Conv i (·) denotes a dilated convolution with a dilation of value i, Mean(·) denotes an average pooling operation, and f out For output features;

[0038] Get local information about the scale-aligned patch set.

[0039] As a preferred technical solution, the step of inferring the prediction result through a fully connected layer and then calculating the prediction loss specifically involves:

[0040] During training, the aggregated image feature loss is calculated using the cross-entropy loss function through a fully connected layer to form enhanced data, as shown in the following formula:

[0041]

[0042] in It is the output of the i-th WSI, y i These are the corresponding tags;

[0043] The enhanced data are arranged in the order before the pathological image features were aggregated, and the prediction results of the MAT classification network model are inferred.

[0044] As a preferred technical solution, training the MAT classification network model further includes the following steps:

[0045] Before training, the input scalar-aligned patch set is expanded and class labels are embedded. That is, the features of the scalar-aligned patch set are filled by random sampling. After filling, it is concatenated with randomly initialized class labels and used as the input of the first transformation layer to maintain the normality of the feature matrix.

[0046] As a preferred technical solution, the mean operation is used to aggregate pathological image features based on global and local information.

[0047] Secondly, the present invention also provides a pathological image rapid analysis system based on magnification alignment Transformer, which is applied to the aforementioned pathological image rapid analysis method based on magnification alignment Transformer, including a feature engineering module, a model building module, an alignment and attention training module, and an image testing module;

[0048] The feature engineering module is used to acquire pathological image datasets, identify and segment tissue regions, obtain masks of WSI tissue regions, and perform patching operations on WSI tissue regions using the masks; filter the masks of WSI tissue regions after patching operations, remove patches whose internal tissue region area is too small, and obtain a patch set; the tissue region area is calculated based on the tissue area contained in the mask at the same position as the patch.

[0049] The model building module is used to construct a MAT classification network model, which includes a self-supervised scalar alignment module and a global-local Transformer classification module. The self-supervised scalar alignment module includes two structurally identical feature extractors, used to extract high-scalar features and generate semantically highly similar alignment features, respectively. The global-local Transformer classification module includes a global attention module and a local attention module. The global attention module utilizes the Transformer's ability to learn global information to discover global features between instances. The local attention mechanism utilizes the CNN's ability to learn detailed information to discover local features of instances. The features discovered by the global attention module and the local attention module are aggregated, and a prediction result is obtained through inference by a fully connected layer.

[0050] The alignment and attention training module is used to train the MAT classification network model. It utilizes a patch set for self-supervised magnification alignment training, aligning low-magnification images to high-magnification images at the feature level to obtain a magnification-aligned patch set. In the global attention module, a Transformer is used to extract global information from the magnification-aligned patch set. In the local attention module, a CNN is used to capture local information from the magnification-aligned patch set. Then, pathological image features are aggregated based on the global and local information, and the prediction results are generated through a fully connected layer. The prediction loss is then calculated, and the trained MAT classification network model is obtained through backpropagation. The self-attention mechanism in the Transformer covers all patches, thus obtaining global attention. The fixed-size convolutional kernels and sliding window mechanism in the local attention module are more sensitive to neighboring patches.

[0051] The image testing module is used to obtain pathological image classification prediction results using a trained MAT classification network model.

[0052] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0053] At least one processor; and,

[0054] A memory communicatively connected to the at least one processor; wherein,

[0055] The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the rapid pathological image analysis method based on magnification alignment Transformer.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] (1) The present invention adopts a self-supervised magnification alignment mechanism to align low-magnification images to high-magnification images at the feature level, thereby repairing the lost information of low-magnification images and the information loss caused by the decrease in magnification.

[0058] (2) This invention utilizes the CNN-Transformer attention mechanism, that is, in the global attention module, the Transformer is used to discover the global information of the scalar-aligned Patch set, and in the local attention module, the CNN is used to capture the local information of the scalar-aligned Patch set. Then, based on the global and local information, the pathological image features are aggregated to predict the pathological image features, which greatly reduces the time and space costs required for model prediction. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart of a rapid pathological image analysis method based on magnification alignment Transformer according to an embodiment of the present invention;

[0061] Figure 2 This is a flowchart of pathological image tissue segmentation in an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the model structure of MAT in an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the magnification alignment module in an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of feature expansion, local attention module, and global attention module in an embodiment of the present invention;

[0065] Figure 6 This is a schematic diagram of the structure of the pathological image rapid analysis system based on magnification alignment Transformer according to an embodiment of the present invention;

[0066] Figure 7 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0067] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0068] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0069] The Transformer is a neural network model based on the attention mechanism, originally proposed by Vaswani et al. in 2017. It has achieved significant breakthroughs in Natural Language Processing (NLP) tasks and has been widely applied in machine translation, text generation, language understanding, and other fields. By introducing self-attention mechanisms and other key technologies, the Transformer model effectively addresses the limitations of traditional neural networks in processing long sequences of data, becoming an important model in the field of natural language processing.

[0070] Scale alignment refers to aligning pathological images at different magnifications in the feature space to repair information loss caused by reduced resolution.

[0071] The Magnification-Aligned Transformer (MAT) designed in this invention is an integrated, fully automatic, and time-saving WSI classification method. It is a two-stage hybrid WSI classification model based on CNN-transformer, consisting of a self-supervised Magnification-Aligned (SSMA) module and a global-local transformer (GLT), which are used for feature alignment tasks from low to high magnification and WSI classification tasks, respectively.

[0072] Inheriting the concept of multi-instance learning, MAT classification treats a WSI (Web Sample Image) as a bag, and its patches as instances. A bag is defined as positive if a positive instance exists, otherwise it is negative. The input WSI is first cropped into non-overlapping patches, and then feature extraction is performed to compress pixel information into high-order semantic information. When the WSI is at a low magnification and the goal is to achieve prediction performance close to that of high-magnification images, a magnification-aligned model is used to extract features; otherwise, a model pre-trained on ImageNet is used. Next, the high-order semantic features are input into the WSI classification model (GLT) to predict the WSI.

[0073] Please see Figure 1 This embodiment provides a rapid pathological image analysis method based on magnification alignment Transformer, which includes the following steps:

[0074] S1. Obtain the pathological image dataset, identify and segment the tissue regions, obtain the mask of the WSI tissue regions, and use the mask to perform patching operations on the WSI tissue regions; filter the WSI tissue regions after the patching operation, remove patches that contain too small tissue regions, and obtain the patch set.

[0075] Optionally, the area of ​​the tissue region is calculated based on the area of ​​the tissue contained in the mask at the same location as the patch;

[0076] Optionally, step S1 involves identifying and segmenting the tissue region to obtain a mask for the WSI tissue region, and then using the mask to perform a patching operation on the WSI tissue region, such as... Figure 2 As shown, specifically:

[0077] S11. Use ImageNet to perform binary classification on the pathological image dataset to identify normal tissue areas and blank and contaminated areas, and obtain the mask of WSI tissue area;

[0078] Furthermore, the Adam optimizer is used in step S11. The Adam optimizer adaptively adjusts the learning rate based on the gradient of the parameters and the square of the historical gradients, thereby better adapting to different parameters and datasets. Of course, other types of optimizers are also applicable to the technical solution of this invention.

[0079] S12. The mask of the WSI tissue region is cropped into patches of a set size. The patches are then input into ResNet50 pre-trained on ImageNet for feature extraction.

[0080] Preferably, the size of the patch varies proportionally with the magnification of the image used; that is, the smaller the magnification of the image, the smaller the patch; and the larger the magnification of the image, the larger the patch.

[0081] For example, the image magnification is low: when the image magnification is 100x, the patch size is 112*112 pixels, and when the image magnification is 50x, it is 56*56 pixels.

[0082] For example, the image magnification is high: when the image magnification is 200 times, the patch size is 224*224 pixels.

[0083] WSIs typically contain many non-organic areas, such as blanks, contamination from slide preparation, and human markings. Traditional thresholding methods and texture-based analysis methods are prone to misclassification when processing WSIs that differ significantly in color and morphology. Therefore, this invention designs a WSI tissue segmentation process, please refer to [link to previous document]. Figure 2 The following is a specific example of WSI organization segmentation:

[0084] First, several tissue region images and several non-tissue region images (such as blank and contaminated areas) were randomly selected from the TCGA pathological image library.

[0085] Secondly, all images are cropped into 224×224 pixel patches and then randomly sorted as input to the ResNet18 network.

[0086] Next, the ResNet18 network was pre-trained on ImageNet for binary classification of organized and unorganized data. The Adam optimizer was used during training, with a binary cross-entropy loss function, and the training data was divided into training and validation sets.

[0087] Finally, the trained model was used to segment the tissue regions of all WSIs involved in this invention.

[0088] S2. Construct a MAT classification network model, which includes a self-supervised magnification alignment module and a global-local Transformer classification module. The self-supervised magnification alignment module is trained through self-supervised learning to align low-magnification images to high-magnification images at the feature level with almost no information loss. The global-local Transformer classification module mainly includes a global attention module and a local attention module. The global attention module utilizes the Transformer's ability to learn global information to discover global information between instances. The local attention mechanism utilizes the CNN's ability to learn detailed information to discover local information at the instance time. Finally, the features of the two modules are aggregated, and the prediction result is obtained through inference by a fully connected layer. Step S2 includes the following specific contents:

[0089] S21, such as Figure 3 As shown in part a, the self-supervised scaling alignment module includes two structurally identical feature extractors Φ High (·) and Φ Align (·), which are used to extract high-magnification features and generate semantically highly similar alignment features, respectively. In this embodiment, both feature extractors use the ResNet50 model pre-trained on ImageNet. Of course, the feature extractor used is not limited to ResNet50. Any network with feature extraction capabilities can be used as a feature extractor for an image.

[0090] Furthermore, such as Figure 4 As shown, Φ High (·) is considered a standard feature extractor, and Φ is frozen during training. HighThe parameters (·) are used to accept high-magnification images I. High As input, and output feature f High As a standard for alignment. In contrast, Φ Align (·) Acceptance and I High Low-resolution images with the same field of view I Low and output the aligned feature f Align The alignment operation is performed on each image patch to ensure that the model learns complete image information, but ultimately only the feature vectors output by the feature extractor are used for the subsequent WSI prediction task.

[0091] S22. The Global-Local Transformer classification module (GLT model) is a CNN-Transformer hybrid neural network based on Multiple Instance Learning (MIL) design. In this model, the entire WSI is processed as a bag, and each patch is considered an instance within that bag. MIL alleviates the memory overload problem caused by the high resolution of WSI, allowing deep learning models to process the entire image at once. However, existing MIL models primarily focus on establishing relationships between instances and labels, neglecting the correlations between instances and between instances and the global context. To address this issue, the Global-Local Transformer classification module incorporates local and global attention modules. It leverages the sensitivity of CNNs to local information and the Transformer's ability to model global dependencies to explore the correlations between patches and between patches and the WSI. This is effectively integrated through Transformer layers, improving prediction accuracy.

[0092] Please refer to it again. Figure 3 Part a further elaborates on the specific content of the local attention module and the global attention module:

[0093] S211. In the local attention module, capturing the interactions between different regions of the tissue is crucial for accurately predicting the WSI task. To this end, the local attention module utilizes convolutional operations in CNNs to capture local information, and a local attention module has been designed, such as... Figure 5 The local attention module utilizes convolutional kernels with fixed receptive fields, which are sensitive to local information but limit the interaction of information over a larger scope. Therefore, this module uses dilated convolutions with spacings of 1, 3, and 5 to construct an instance-level feature pyramid to capture instance information at multiple scales. Finally, the information from the three scales is fused using an averaging method. In this module, class labels do not participate in the attention calculation. This can be represented as follows:

[0094] f1 = Conv1(f)

[0095] f2 = Conv3(f)

[0096] f3 = Conv5(f)

[0097] f out =Mean(f1, f2, f3)

[0098] Where f represents the input feature, Conv i (·) denotes a dilated convolution with a dilation of value i, Mean(·) denotes an average pooling operation, and f out For output features.

[0099] S222. In the global attention module, the traditional MIL model considers the correlation between instances and labels, but lacks the ability to capture global correlations, resulting in an incomplete consideration of the semantic features of WSI. This invention designs a patch-level global attention module to discover features with high predictive power within the bag, such as... Figure 5 The global attention module is shown in the diagram.

[0100] In the global attention module, the features F of the patch set are... B×(N+1)×L Input the global attention module, and use the Transformer within the global attention module to discover global information about the scale-aligned patch set. B×(N+1)×L First, a fully connected layer is used to generate Q, K, and V vectors. Then, the Q and K vectors are used to generate global attention using a self-attention mechanism. Next, a vector dot product is performed with V. Finally, the output is concatenated with randomly initialized Class Tokens. See [link to relevant documentation]. Figure 3 Part b defines the process of the global attention module algorithm:

[0101] Q = MLP(F B×N×L )

[0102] K = MLP(F B×N×L )

[0103] V = MLP(F B×N×L )

[0104] Att B×N×L =Softmax(Transpose) 0，2，1 (Q×K))

[0105] Att B×N×L =Concat(V×Transpose) 0,2,1 (Att B×N×L ))

[0106]

[0107] B represents the batch size, N represents the number of features, L represents the feature length, 1 represents the class label feature, MLP represents a multilayer perceptron, and Q, K, and V all represent features F. B×N×L The intermediate variable for the transformation, Att B×N×L Representing intermediate variables in the attention mechanism. The Class Token is generated using a random initialization strategy and is used to learn all instance information; Transpose 0,2,1 (·) indicates that the 0th, 1st, and 2nd dimensions of the feature are adjusted to 0, 2, and 1.

[0108] The program is defined as follows:

[0109]

[0110] S223, Feature expansion and class label embedding;

[0111] CNNs cannot handle irregular feature matrices, so feature augmentation is necessary to maintain their regularity. Therefore, the GLT module uses random sampling to pad the input feature set. After padding, it is concatenated with randomly initialized class labels and used as the input to the first transformation layer, such as... Figure 5 The extended features are shown in the diagram.

[0112] S3. Train the MAT classification network model. The training process includes two parts: the first part is training the self-supervised scaling alignment module, and the second part is training the global-local Transformer classification module, as detailed below:

[0113] S31. Training of the self-supervised scaling alignment module:

[0114] During training, the dataset is randomly divided into training and validation sets. The self-supervised magnification alignment module uses L1 loss to reduce the absolute distance between features of different magnification levels and achieves near-lossless semantic alignment. The loss function is shown below:

[0115]

[0116] Where X i It is Φ High (·) Output features of the i-th block, x i It is Φ Align (·) The output features of the i-th block are not augmented during training because the samples required for training are sufficient for the model.

[0117] For example, the parameters of the self-supervised scaling-aligned module are updated using the Adam optimizer with an initial learning rate of 1×10⁻⁶. -4The learning strategy is linear decay with a decay coefficient of 0.9.

[0118] S32, Training of the global-local Transformer classification module;

[0119] The global-local Transformer classification module uses the cross-entropy loss function, as shown below:

[0120]

[0121] in It is the model output of the i-th WSI, y i The labels are used to randomly initialize the model parameters in the global-local Transformer classification module. During training, instances in the input bag are randomly shuffled as a form of data augmentation, while during model inference, they are arranged according to the order of the feature extraction stages.

[0122] S4. Use the trained MAT classification network model to obtain the classification prediction results of pathological images.

[0123] Unlike traditional methods that require high-resolution input images (400x (40X) or 200x (20X)), MAT only requires low-resolution input images (100x (10X), 50x (5X), or even 25x (2.5X)). At the same time, MAT maintains the same high-performance prediction results as advanced models, with a 20 to 40x improvement in computation time, and requires only 1 / 16th the amount of data to read.

[0124] In one specific embodiment of the present invention, the following technical solution is adopted:

[0125] Step 1: After acquiring the pathological image, perform image tissue segmentation and obtain a mask for the WSI tissue region. The processing method used is described in the tissue segmentation section above.

[0126] Step 2: After obtaining the mask, perform a patching operation on the WSI. At high magnification (e.g., 200x), each patch is 224*224 pixels. The size of the patch varies proportionally with the magnification of the image used (e.g., at low magnification (100x, it's 112*112 pixels; at 50x, it's 56*56 pixels).

[0127] Step 3: Select the acquired patches. Discard any patches containing too little tissue area. The area of ​​the tissue area is calculated based on the tissue area contained in the mask at the same location as the patch.

[0128] Step 4: Train the feature alignment module. After obtaining the patch, train the alignment model using the training strategy described in the self-supervised scaling alignment module above.

[0129] Step 5: Feature Extraction. After obtaining the patch, feature extraction is performed using the pre-trained network. If extracting raw features, they are extracted from a pre-trained ResNet50 network in ImageNet. If low-magnification alignment features are to be used, they are extracted from the feature alignment network. The specific methods have been described in the self-supervised magnification alignment module above.

[0130] Step 6: Train the MAT classification network. Train the GLT model according to the GLT model training strategy described above.

[0131] Step 7: Model Testing. After collecting the test data, obtain the input features in the order of Steps 3 and 5 above, and then input them into the MAT classification model trained in Step 6 for prediction.

[0132] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0133] Based on the same idea as the pathological image rapid analysis method based on magnification alignment Transformer in the above embodiments, the present invention also provides a pathological image rapid analysis system based on magnification alignment Transformer. This system can be used to execute the above-described pathological image rapid analysis method based on magnification alignment Transformer. For ease of explanation, the structural schematic diagram of the embodiment of the pathological image rapid analysis system based on magnification alignment Transformer only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0134] Please see Figure 6 In another embodiment of this application, a method system 10 for rapid analysis of pathological images based on magnification alignment Transformer is provided. The system includes a feature engineering module 11, a model building module 12, an alignment and attention training module 13, and an image testing module 14.

[0135] The feature engineering module 11 is used to acquire pathological image datasets, identify and segment tissue regions, obtain masks of WSI tissue regions, and perform patching operations on WSI tissue regions using the masks; filter the masks of WSI tissue regions after patching operations, remove patches whose internal tissue region area is too small, and obtain a patch set; the tissue region area is calculated based on the tissue area contained in the mask at the same position as the patch.

[0136] The model building module 12 is used to construct a MAT classification network model, which includes a self-supervised scalar alignment module and a global-local Transformer classification module. The self-supervised scalar alignment module includes two structurally identical feature extractors, used to extract high-scalar features and generate semantically highly similar alignment features, respectively. The global-local Transformer classification module includes a global attention module and a local attention module. The global attention module utilizes the Transformer's ability to learn global information to discover global features between instances. The local attention mechanism utilizes the CNN's ability to learn detailed information to discover local features of instances. The features discovered by the global attention module and the local attention module are aggregated, and a prediction result is obtained through inference by a fully connected layer.

[0137] The alignment and attention training module 13 is used to train the MAT classification network model. It utilizes a patch set to perform self-supervised magnification alignment training on the MAT classification network model, aligning low-magnification images to high-magnification images at the feature level to obtain a magnification-aligned patch set. In the global attention module, a Transformer is used to discover global information from the magnification-aligned patch set. In the local attention module, a CNN is used to capture local information from the magnification-aligned patch set. Then, pathological image features are aggregated based on global and local information, and prediction results are inferred through fully connected layers. The prediction loss is then calculated, and the trained MAT classification network model is obtained through backpropagation. The self-attention mechanism in the Transformer covers all patches, thus obtaining global attention. The fixed-size convolutional kernels and sliding window mechanism in the local attention module are more sensitive to neighboring patches.

[0138] The image testing module 14 is used to test the trained MAT classification network model using a patch set and a patch set aligned with magnification, and to obtain pathological image classification prediction results.

[0139] It should be noted that the pathological image rapid analysis system based on magnification alignment Transformer of the present invention corresponds one-to-one with the pathological image rapid analysis method based on magnification alignment Transformer of the present invention. The technical features and beneficial effects described in the embodiments of the pathological image rapid analysis method based on magnification alignment Transformer described above are applicable to the embodiments of the pathological image rapid analysis method based on magnification alignment Transformer. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0140] Furthermore, in the embodiments of the pathological image rapid analysis system based on magnification alignment Transformer described above, the logical division of each program module is merely illustrative. In practical applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or software implementation convenience. That is, the internal structure of the pathological image rapid analysis system based on magnification alignment Transformer can be divided into different program modules to complete all or part of the functions described above.

[0141] Please see Figure 7 In one embodiment, an electronic device is provided for implementing a rapid pathological image analysis method based on magnification alignment Transformer. The electronic device 20 may include a first processor 21, a first memory 22, and a bus, and may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as a rapid pathological image analysis program 23 based on magnification alignment Transformer.

[0142] The first memory 22 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as the portable hard drive of the electronic device 20. In other embodiments, the first memory 22 can be an external storage device of the electronic device 20, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 20. Furthermore, the first memory 22 can include both internal storage units and external storage devices of the electronic device 20. The first memory 22 can be used not only to store application software and various types of data installed on the electronic device 20, such as the code of a pathological image rapid analysis program 23 based on magnification alignment Transformer, but also to temporarily store data that has been output or will be output.

[0143] In some embodiments, the first processor 21 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 21 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 22 and calls data stored in the first memory 22 to perform various functions of the electronic device 20 and process data.

[0144] Figure 7 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 7 The structure shown does not constitute a limitation on the electronic device 20, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0145] The pathological image rapid analysis program 23 based on magnification alignment Transformer stored in the first memory 22 of the electronic device 20 is a combination of multiple instructions, which, when run in the first processor 21, can achieve the following:

[0146] Acquire a pathological image dataset, identify and segment tissue regions, obtain a mask for WSI tissue regions, and use the mask to perform patching operations on the WSI tissue regions; filter the WSI tissue regions after patching operations, remove patches whose internal tissue region area is too small, and obtain a patch set; the tissue region area is calculated based on the tissue area contained in the mask at the same position as the patch.

[0147] A MAT classification network model is constructed, comprising a self-supervised scalar alignment module and a global-local Transformer classification module. The self-supervised scalar alignment module includes two structurally identical feature extractors, used to extract high-scalar features and generate semantically highly similar alignment features, respectively. The global-local Transformer classification module includes a global attention module and a local attention module. The global attention module utilizes the Transformer's ability to learn global information to discover global features between instances; the local attention mechanism utilizes the CNN's ability to learn detailed information to discover local features of instances. The features discovered by the global and local attention modules are aggregated, and a prediction result is obtained through inference by a fully connected layer.

[0148] The MAT classification network model is trained by performing self-supervised magnification alignment training using a patch set, aligning low-magnification images to high-magnification images at the feature level to obtain a magnification-aligned patch set. In the global attention module, a Transformer is used to extract global information from the magnification-aligned patch set, while in the local attention module, a CNN is used to capture local information from the same patch set. Then, pathological image features are aggregated based on the global and local information, and prediction results are inferred through fully connected layers. The prediction loss is then calculated, and the trained MAT classification network model is obtained through backpropagation. The self-attention mechanism in the Transformer covers all patches, thus obtaining global attention. The fixed-size convolutional kernels and sliding window mechanism in the local attention module are more sensitive to neighboring patches.

[0149] The trained MAT classification network model is used to obtain the classification prediction results of pathological images.

[0150] Furthermore, if the modules / units integrated in the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0151] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0152] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A rapid pathological image analysis method based on magnification-aligned Transformer, characterized in that, Includes the following steps: Acquire pathological image datasets, identify and segment tissue regions, obtain masks of WSI tissue regions, and use the masks to perform patch operations on WSI tissue regions; Filter the WSI tissue regions that have undergone the Patch cutting operation, and remove the Patch with too small a tissue region area to obtain a Patch set; the tissue region area is calculated based on the tissue area contained in the mask at the same position as the Patch. A MAT classification network model is constructed, comprising a self-supervised scalar alignment module and a global-local Transformer classification module. The self-supervised scalar alignment module includes two structurally identical feature extractors, used to extract high-scalar features and generate semantically highly similar alignment features, respectively. The global-local Transformer classification module includes a global attention module and a local attention module. The global attention module utilizes the Transformer's ability to learn global information to discover global features between instances; the local attention module utilizes the CNN's ability to learn detailed information to discover local features between instances. The features discovered by the global and local attention modules are aggregated, and a prediction result is obtained through inference by a fully connected layer. The MAT classification network model is trained by performing self-supervised magnification alignment training using a patch set, aligning low-magnification images to high-magnification images at the feature level to obtain a magnification-aligned patch set. In the global attention module, a Transformer is used to extract global information from the magnification-aligned patch set, while in the local attention module, a CNN is used to capture local information from the same patch set. Then, pathological image features are aggregated based on the global and local information, and prediction results are inferred through fully connected layers. The prediction loss is then calculated, and the trained MAT classification network model is obtained through backpropagation. The self-attention mechanism in the Transformer covers all patches, thus obtaining global attention. The fixed-size convolutional kernels and sliding window mechanism in the local attention module are more sensitive to neighboring patches. The trained MAT classification network model is used to obtain the classification prediction results of pathological images; The method of using a patch set to perform self-supervised scalar alignment training on the MAT classification network model specifically involves: Using two task-dependent feature extractors with identical structures and ,freeze The parameters are used to extract high-magnification features and generate semantically highly similar alignment features, i.e. It accepts high-magnification image input and outputs high-magnification features. It takes a low-resolution image with the same field of view as the high-magnification image and generates semantically highly similar alignment features. L1 loss is used to reduce the absolute distance between features at different magnifications and to achieve semantic alignment. The loss function is shown below: in yes No. The output features of each block yes No. The output characteristics of each block.

2. The rapid pathological image analysis method based on magnification alignment Transformer according to claim 1, characterized in that, The process of identifying and segmenting tissue regions to obtain a mask for WSI tissue regions, and then using the mask to perform a patching operation on the WSI tissue regions, specifically involves: ImageNet was used to perform binary classification on the pathological image dataset to identify normal tissue areas and blank and contaminated areas, and to obtain the mask of WSI tissue areas; The mask of the WSI tissue region is patched, and all images are cropped into patches of a set size. The patches are then input into a ResNet50 pre-trained on ImageNet for feature extraction. The size of the patch varies proportionally with the magnification of the image used.

3. The rapid pathological image analysis method based on magnification alignment Transformer according to claim 1, characterized in that, The method of using Transformer to discover global information of the scaling-aligned patch set in the global attention module specifically includes: Features of the patch set are incorporated into the global attention module. Input the global attention module, and use the Transformer within the global attention module to discover global information in the scale-aligned patch set. That is, first use a fully connected layer to generate Vectors, then use The vector generates global attention using a self-attention mechanism, and then... Perform vector dot product, and finally concatenate with randomly initialized Class Tokens to obtain the final output. The complete process is defined as follows: Where B represents the batch size, N represents the number of features, L represents the feature length, and 1 represents the class label feature. This represents a multilayer perceptron. K All represent features Intermediate variables in the transformation , Representing intermediate variables in the attention mechanism. It is a Class Token, generated using a random initialization strategy, used to learn all instance information; This means adjusting the 0th, 1st, and 2nd dimensions of the feature to 0, 2, and 1.

4. The rapid pathological image analysis method based on magnification alignment Transformer according to claim 1, characterized in that, The method of using CNN to capture local information of the scale-aligned patch set in the local attention module specifically includes: In the local attention module, dilated convolutions with dilation levels of 1, 3, and 5 from CNNs are used to construct an instance-level feature pyramid to capture instance information at three scales. Instance information at three scales , , Fusion was performed using the averaging method: in Indicates input features, Indicates a void as Hollow convolution, This indicates the average pooling operation. For output features; Get local information about the scale-aligned patch set.

5. The rapid pathological image analysis method based on magnification alignment Transformer according to claim 1, characterized in that, The process of inferring and predicting results through a fully connected layer, and then calculating the prediction loss, specifically involves: During training, the cross-entropy loss function is used in fully connected layers to calculate the aggregated image feature loss, resulting in enhanced data, as shown in the following formula: in It is the first The output of each WSI, These are the corresponding tags; The enhanced data are arranged in the order before the pathological image features were aggregated, and the prediction results of the MAT classification network model are inferred.

6. The rapid pathological image analysis method based on magnification alignment Transformer according to claim 1, characterized in that, Training the MAT classification network model also includes the following steps: Before training, the input scalar-aligned patch set is expanded and class labels are embedded. That is, the features of the scalar-aligned patch set are filled by random sampling. After filling, it is concatenated with randomly initialized class labels and used as the input of the first transformation layer to maintain the normality of the feature matrix.

7. The rapid pathological image analysis method based on magnification alignment Transformer according to claim 1, characterized in that, The mean operation is used to aggregate global and local information to aggregate pathological image features.

8. A rapid pathological image analysis system based on magnification alignment Transformer, characterized in that, The rapid pathological image analysis method based on magnification alignment Transformer, as described in any one of claims 1-7, includes a feature engineering module, a model building module, an alignment and attention training module, and an image testing module; The feature engineering module is used to acquire pathological image datasets, identify and segment tissue regions, obtain masks of WSI tissue regions, and use the masks to perform patch operations on WSI tissue regions. The masks of WSI tissue regions that have undergone the patching operation are filtered out, and patches with insufficient tissue region area are removed to obtain a patch set; the tissue region area is calculated based on the tissue area contained in the mask at the same position as the patch. The model building module is used to construct a MAT classification network model, which includes a self-supervised scalar alignment module and a global-local Transformer classification module. The self-supervised scalar alignment module includes two structurally identical feature extractors, used to extract high-scalar features and generate semantically highly similar alignment features, respectively. The global-local Transformer classification module includes a global attention module and a local attention module. The global attention module utilizes the Transformer's ability to learn global information to discover global features between instances. The local attention module utilizes the CNN's ability to learn detailed information to discover local features between instances. The features discovered by the global attention module and the local attention module are aggregated, and a prediction result is obtained through inference by a fully connected layer. The alignment and attention training module is used to train the MAT classification network model. It utilizes a patch set for self-supervised magnification alignment training, aligning low-magnification images to high-magnification images at the feature level to obtain a magnification-aligned patch set. In the global attention module, a Transformer is used to extract global information from the magnification-aligned patch set. In the local attention module, a CNN is used to capture local information from the magnification-aligned patch set. Then, pathological image features are aggregated based on the global and local information, and prediction results are inferred through fully connected layers. The prediction loss is then calculated, and the trained MAT classification network model is obtained through backpropagation. The self-attention mechanism in the Transformer covers all patches, thus obtaining global attention. The fixed-size convolutional kernels and sliding window mechanism in the local attention module are more sensitive to neighboring patches. The image testing module is used to obtain pathological image classification prediction results using a trained MAT classification network model.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the rapid pathological image analysis method based on magnification alignment Transformer as described in any one of claims 1-7.

Images