Rapid pathological image analysis method and apparatus based on magnification-aligned transformer

Abstract

Disclosed in the present invention are a rapid pathological image analysis method and apparatus based on a magnification-aligned transformer (MAT). The method comprises: identifying and segmenting a pathological image data set, so as to obtain pathological image patches; performing screening to acquire a patch set; constructing an MAT classification network model comprising a self-supervised magnification-aligned module and a global-local Transformer classification module; in the self-supervised magnification-aligned module, performing self-supervised magnification-aligned training on the MAT classification network model by using the patch set; training the MAT classification network model by using CNN-Transformer; and acquiring a pathological image classification prediction result by using the trained MAT classification network model. By means of the MAT classification network model provided in the present invention, a low-magnification image is aligned to a high magnification at a feature level by using a self-supervised magnification-aligned mechanism, and pathological image features are predicted by using a CNN-Transformer attention mechanism, such that information of the low-magnification image is fully utilized, and the time and space costs required by model prediction are greatly reduced.

Description

Rapid analysis method and device for pathological images based on magnification alignment Transformer Technical Field

[0001] The present invention belongs to the technical field of image analysis, and in particular relates to a method and device for rapid analysis of pathological images based on magnification alignment Transformer. Background Art

[0002] Histopathology slides can be converted into whole-slide pathology images (WSIs) using a pathology digitizing scanner. With the development of image processing technology, it has become possible to use computer algorithms to achieve intelligent auxiliary diagnosis of pathology, and computational pathology has been derived from this. In recent years, with the rapid development of artificial intelligence technology, deep learning has achieved remarkable results in many computational pathology tasks, such as cancer diagnosis, prognosis, and risk stratification. Although deep learning models can achieve excellent performance in the above-mentioned prediction tasks, their image processing speed and efficiency are still relatively low. For example, existing methods take 5 to 10 minutes to perform predictive analysis on a complete WSI, which is difficult to meet the effectiveness of clinical diagnosis.

[0003] The main factor affecting computational efficiency is the gigabyte-level image resolution of WSI. In order to obtain sufficient image information, existing methods usually process WSI at high magnification (such as 400x or 200x). Although the image has been compressed to a certain extent, the required computing resources and time are still very large. On the other hand, although using lower resolution images (such as 100x or 50x) can greatly reduce the required computing time and resources, it will lead to serious image information loss, which greatly reduces the accuracy of the model. Therefore, the balance between model performance and efficiency is actually a trade-off between model performance and resolution.

[0004] To fully utilize low-magnification images, two questions must be addressed. First, do low-magnification images have diagnostic value? Second, can deep learning be used to recover predictive information from low-magnification images? Clinically, doctors can make preliminary judgments at relatively low magnifications, demonstrating the diagnostic value of low-magnification images. Technically, extensive research has demonstrated that deep learning super-resolution algorithms can generate high-magnification images from low-magnification images, demonstrating that deep learning algorithms can restore information from low-magnification images.

[0005] However, although using low-magnification images can save a lot of time and computing resources, low-magnification images lose a lot of detail information, which will cause serious degradation of model performance; and existing methods analyze pathological images at high magnifications (such as 400 times or 200 times), which requires a lot of time and computing resources and is difficult to meet the requirements of practical applications. Therefore, how to fully combine the advantages of high and low magnification images to quickly analyze pathological images is an urgent problem to be solved.

[0006] Summary of the Invention

[0007] The main purpose of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a method and device for rapid analysis of pathological images based on magnification alignment transformers. This invention proposes a MAT classification network model that utilizes a self-supervised magnification alignment mechanism to align low-magnification images to high-magnification images at the feature level. Simultaneously, it uses a CNN-Transformer attention mechanism to predict pathological image features, fully utilizing the information from low-magnification images and significantly reducing the time and space costs required for model prediction.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a method for rapid analysis of pathological images based on magnification alignment Transformer, comprising the following steps:

[0010] Acquire a pathology image dataset, identify and segment tissue regions, obtain a mask of the WSI tissue region, and use the mask to perform a patch operation on the WSI tissue region; filter the WSI tissue regions that have undergone the patch operation, and remove patches with too small an area of ​​tissue region to obtain a patch set; the area of ​​the tissue region is calculated based on the tissue area contained in the mask at the same position as the patch;

[0011] A MAT classification network model is constructed, comprising a self-supervised rate alignment module and a global-local Transformer classification module; the self-supervised rate alignment module comprises two feature extractors with identical structures, one for extracting high-rate features and the other for generating semantically highly similar alignment features; the global-local Transformer classification module comprises a global attention module and a local attention module, wherein the global attention module utilizes the Transformer's ability to learn global information to explore global features between instances; the local attention mechanism utilizes the CNN's ability to learn detailed information to explore local features of instance time; the features explored by the global attention module and the local attention module are aggregated, and a prediction result is obtained by inference through a fully connected layer;

[0012] The MAT classification network model is trained and self-supervised magnification alignment training is performed on the MAT classification network model using the patch set, and low-magnification images are aligned to high-magnification images at the feature level to obtain a magnification-aligned patch set. The global information of the magnification-aligned patch set is explored using the Transformer in the global attention module, and the local information of the magnification-aligned patch set is captured using the CNN in the local attention module. Pathological image features are then aggregated based on the global and local information, and the prediction results are predicted through inference using the fully connected layer. The prediction loss is then calculated, and the trained MAT classification network model is obtained through the backpropagation algorithm. The self-attention mechanism in the Transformer covers all patches, thereby obtaining global attention. The fixed-size convolution kernel and sliding window mechanism of the CNN in the local attention module are more sensitive to adjacent patches.

[0013] The trained MAT classification network model is used to obtain the pathological image classification prediction results.

[0014] As a preferred technical solution, the identification and segmentation of tissue regions, obtaining a mask of the WSI tissue region, and performing a patch operation on the WSI tissue region using the mask are specifically as follows:

[0015] Use ImageNet to perform binary classification on the pathology image dataset, identify normal tissue areas, blank areas, and contaminated areas, and obtain the mask of the WSI tissue area;

[0016] The mask of the WSI tissue area was patched, and all images were cropped into patches of a set size. The patches were then input into ResNet50 pre-trained on ImageNet for feature extraction. The size of the patches varied proportionally with the magnification of the image used.

[0017] As a preferred technical solution, the self-supervised rate alignment training of the MAT classification network model using the Patch set is specifically as follows:

[0018] Using two task-dependent feature extractors with the same structure Φ High (·) and Φ Align (·), freeze Φ High (·) parameters, extracting high-magnification features and generating semantically highly similar alignment features, i.e., Φ High (·) accepts high-magnification image input and outputs high-magnification features, Φ Align (·) Accepting a low-resolution image with the same field of view as the high-magnification image, and generating semantically highly similar aligned features;

[0019] L1 loss is used for high-magnification features and alignment features to reduce the absolute distance between features with different magnifications and achieve semantic alignment. The loss function is as follows:

[0020] where X i is Φ High (·) Output features of the i-th block, x i is Φ Align (·) Output features of the i-th block.

[0021] As a preferred technical solution, the Transformer is used in the global attention module to explore the global information of the rate-aligned patch set, specifically:

[0022] In the global attention module, the feature F of the patch set is B×(N+1)×L Input global attention module, in which Transformer is used to explore the global information of the rate-aligned patch set Out B×(N+ 1 )×L , that is, first use the fully connected layer to generate Q, K, V vectors, then use Q, K vectors to generate global attention according to the self-attention mechanism, then do vector dot product with V, and finally concatenate with the randomly initialized Class Token to get the final output. The complete process definition is: Q = MLP (F B×N×L ) K=MLP(F B×N×L ) V=MLP(F B×N×L ) Att B×N×L =Softmax(Transpose 0,2,1 (Q×K)) Att B×N×L =Concat(V×Transpose 0,2,1 (Att B×N×L ))

[0023] Where B represents the batch size, N represents the number of features, L represents the feature length, 1 represents the class label feature, MLP represents the multi-layer perceptron, Q, K, and V all represent the feature F B×N×L The intermediate variable of the transformation, Att B×N×L represents the intermediate variable of the attention mechanism, Class Token, generated using a random initialization strategy, is used to learn all instance information; Transpose 0,2,1 (·) means adjusting the 0th, 1st, and 2nd dimensions of the feature to 0, 2, and 1.

[0024] As a preferred technical solution, the local information of the rate-aligned patch set is captured by using CNN in the local attention module, specifically:

[0025] In the local attention module, we use dilated convolutions with dilations of 1, 3, and 5 in CNN to construct an instance-level feature pyramid to capture instance information at three scales.

[0026] The instance information f1, f2, and f3 of the three scales are fused by the averaging method: f1=Conv1(f) f2=Conv3(f) f3=Conv5(f) f out =Mean(f1,f2,f3)

[0027] Where f represents the input feature, Conv i (·) represents the dilated convolution with hole i, Mean(·) represents the average pooling operation, and f out is the output feature;

[0028] Get the local information of the rate-aligned patch set.

[0029] As a preferred technical solution, the prediction result is inferred through the fully connected layer, and then the prediction loss is calculated, specifically:

[0030] During the training process, the cross entropy loss function is used to calculate the aggregated image feature loss through the fully connected layer to form enhanced data, as shown in the following formula:

[0031] in is the output of the i-th WSI, y i is the corresponding label;

[0032] The enhanced data are arranged in the order before the pathological image features are aggregated, and the prediction results of the MAT classification network model are inferred.

[0033] As a preferred technical solution, the training of the MAT classification network model further includes the following steps:

[0034] Before training, the input rate-aligned patch set is expanded and the class labels are embedded, that is, the features of the rate-aligned patch set are filled by random sampling; after filling, they are concatenated with the randomly initialized class labels as the input of the first conversion layer to maintain the normalization of the feature matrix.

[0035] As a preferred technical solution, the pathological image features are aggregated using mean operation of global information and local information.

[0036] In a second aspect, the present invention also provides a rapid analysis system for pathological images based on magnification alignment transformer, which is applied to the rapid analysis method for pathological images based on magnification alignment transformer, including a feature engineering module, a model building module, an alignment and attention training module, and an image testing module;

[0037] The feature engineering module is used to obtain a pathology image dataset, identify and segment tissue regions, obtain a mask of the WSI tissue region, and use the mask to perform a patch operation on the WSI tissue region; filter the mask of the WSI tissue region that has undergone the patch operation, remove patches with too small an area of ​​tissue region contained therein, and obtain a patch set; the area of ​​the tissue region is calculated based on the tissue area contained in the mask at the same position as the patch;

[0038] The model construction module is used to construct a MAT classification network model, which includes a self-supervised rate alignment module and a global-local Transformer classification module; the self-supervised rate alignment module includes two feature extractors with the same structure, which are respectively used to extract high-rate features and generate semantically highly similar alignment features; the global-local Transformer classification module includes a global attention module and a local attention module, the global attention module uses the Transformer's ability to learn global information to explore global features between instances; the local attention mechanism uses the CNN's ability to learn detailed information to explore local features of instance time; the features explored by the global attention module and the local attention module are aggregated, and a prediction result is obtained by reasoning through a fully connected layer;

[0039] The alignment and attention training module is used to train the MAT classification network model, and uses the patch set to perform self-supervised magnification alignment training on the MAT classification network model, aligning the low-magnification image to the high-magnification at the feature level to obtain the magnification-aligned patch set; in the global attention module, the Transformer is used to explore the global information of the magnification-aligned patch set, and in the local attention module, the CNN is used to capture the local information of the magnification-aligned patch set, and then the pathological image features are aggregated according to the global information and the local information, and the results are predicted through the fully connected layer, and then the prediction loss is calculated, and the trained MAT classification network model is obtained through the backpropagation algorithm; the self-attention mechanism in the Transformer covers all patches, thereby obtaining global attention, and the fixed-size convolution kernel and sliding window mechanism of the CNN in the local attention module are more sensitive to adjacent patches;

[0040] The image testing module is used to obtain pathological image classification prediction results using the trained MAT classification network model.

[0041] In a third aspect, the present invention provides an electronic device, comprising:

[0042] at least one processor; and,

[0043] a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the rapid pathological image analysis method based on magnification alignment Transformer.

[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0046] (1) The present invention adopts a self-supervised magnification alignment mechanism to align low-magnification images to high-magnification images at the feature level, thus repairing the lost information of low-magnification images and the information loss caused by magnification reduction;

[0047] (2) The present invention utilizes the CNN-Transformer attention mechanism, that is, the Transformer is used in the global attention module to explore the global information of the rate-aligned patch set, and the CNN is used in the local attention module to capture the local information of the rate-aligned patch set. Then, the pathological image features are aggregated according to the global information and the local information to predict the pathological image features, which greatly reduces the time and space costs required for model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] FIG1 is a flow chart of a method for rapid analysis of pathological images based on magnification alignment Transformer according to an embodiment of the present invention;

[0050] FIG2 is a flowchart of pathological image tissue segmentation according to an embodiment of the present invention;

[0051] FIG3 is a schematic diagram of the model structure of MAT in an embodiment of the present invention;

[0052] FIG4 is a schematic diagram of a magnification alignment module according to an embodiment of the present invention;

[0053] FIG5 is a schematic diagram of feature expansion, local attention module, and global attention module in an embodiment of the present invention;

[0054] FIG6 is a schematic structural diagram of a pathological image rapid analysis system based on magnification alignment Transformer according to an embodiment of the present invention;

[0055] FIG7 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0057] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0058] The Transformer is a neural network model based on the attention mechanism, originally proposed by Vaswani et al. in 2017. It has achieved significant breakthroughs in natural language processing (NLP) tasks and has been widely used in fields such as machine translation, text generation, and language understanding. By introducing the self-attention mechanism and other key technologies, the Transformer model effectively overcomes the limitations of traditional neural networks in processing long sequences of data, making it a key model in the field of natural language processing.

[0059] Scaling alignment refers to the alignment of pathological images of different magnifications in the feature space to repair the information loss caused by resolution degradation.

[0060] The Magnification-Aligned Transformer (MAT) designed in this paper is an integrated, fully automatic and time-saving WSI classification method. It is a two-stage CNN-transformer based hybrid WSI classification model, consisting of a self-supervised Magnification-Aligned (SSMA) module and a global-local transformer (GLT), which are used for feature alignment tasks from low to high magnifications and WSI classification tasks, respectively.

[0061] Inheriting the concept of multiple instance learning, MAT classification regards a WSI as a bag, and the patch in it as an instance in the bag. When there is a positive instance, a bag is defined as a positive bag, otherwise it is a negative bag. The input WSI is first cropped into patches without overlap, and then a feature extraction operation is performed to compress pixel information into high-order semantic information. When the WSI is low-magnification and intends to achieve prediction performance close to that of high-magnification images, a magnification alignment model is used to extract features, otherwise a model pre-trained on ImageNet is used. Next, the high-order semantic features are input into the WSI classification model (GLT) to predict the WSI.

[0062] Referring to FIG1 , this embodiment provides a rapid analysis method for pathological images based on a magnification alignment transformer, comprising the following steps:

[0063] S1. Obtain a pathology image dataset, identify and segment tissue regions, obtain a mask of the WSI tissue region, and use the mask to perform a patch operation on the WSI tissue region; filter the WSI tissue regions that have undergone the patch operation, remove patches with too small an area of ​​tissue region, and obtain a patch set;

[0064] Optionally, the area of ​​the tissue region is calculated based on the area of ​​the tissue contained in the mask at the same position as the patch;

[0065] Optionally, in step S1, the tissue region is identified and segmented, a mask of the WSI tissue region is obtained, and a patch operation is performed on the WSI tissue region using the mask, as shown in FIG2 , specifically:

[0066] S11. Use ImageNet to perform binary classification on the pathology image dataset, identify normal tissue regions, blank and contaminated regions, and obtain the mask of the WSI tissue region;

[0067] Furthermore, in step S11, an Adam optimizer is used. The Adam optimizer can adaptively adjust the learning rate according to the gradient of the parameters and the square of the historical gradient, thereby better adapting to different parameters and data sets. Of course, other types of optimizers are also applicable to the technical solution of the present invention.

[0068] S12. Patch the mask of the WSI tissue region. All images are cropped into patches of a set size and then input into a ResNet50 pre-trained on ImageNet for feature extraction.

[0069] Preferably, the size of the patch changes in geometric proportion to the magnification of the image used; that is, when the magnification of the image is smaller, the patch is smaller; when the magnification of the image is larger, the patch is larger.

[0070] For example, the image magnification is low: when the image magnification is 100 times, the size of the patch is 112*112 pixels, and when the image magnification is 50 times, the size of the patch is 56*56 pixels.

[0071] For example, the image magnification is high: when the image magnification is 200 times, the size of the patch is 224*224 pixels.

[0072] WSIs often contain many non-tissue regions, such as blank areas, contamination from slide preparation, and artifacts. Traditional thresholding and texture-based analysis methods are prone to misclassification when processing WSIs with significant differences in color and morphology. To address this, we designed a WSI tissue segmentation process. Please refer again to Figure 2. A specific example of WSI tissue segmentation is as follows:

[0073] First, several tissue region images and non-tissue region images (such as blank and contaminated areas) were randomly selected from the TCGA pathology image library;

[0074] Secondly, all images are cropped into patches of size 224×224 pixels and then randomly sorted as input to the ResNet18 network;

[0075] Next, we pre-trained a ResNet18 network on ImageNet for the binary classification of tissue and non-tissue. We used the Adam optimizer for training, with a binary cross-entropy loss function. The training data was split into a training set and a validation set.

[0076] Finally, the trained model was used to segment tissue regions in all WSIs involved in the present invention.

[0077] S2. Construct a MAT classification network model, which includes a self-supervised magnification alignment module and a global-local Transformer classification module. The self-supervised magnification alignment module is trained through self-supervised learning to align low-magnification images to high-magnification images at the feature level with almost no information loss; the global-local Transformer classification module mainly includes a global attention module and a local attention module; the global attention module uses the Transformer's ability to learn global information to explore global information between instances; the local attention mechanism uses the CNN's ability to learn detailed information to explore local information of instance time; finally, the features of the two modules are aggregated, and the prediction result is obtained by inference through a fully connected layer. The step S2 includes the following specific contents:

[0078] S21, as shown in part a of FIG3, the self-supervised rate alignment module includes two feature extractors Φ with the same structure. High (·) and Φ Align (·), which are used to extract high-magnification features and generate semantically highly similar alignment features, respectively. In this embodiment, both feature extractors use the ResNet50 model pre-trained on ImageNet. Of course, the feature extractor used is not limited to ResNet50. Any network with feature extraction capabilities can be used as an image feature extractor.

[0079] Furthermore, as shown in Figure 4, Φ High (·) is treated as a standard feature extractor and freezes Φ during training. High (·) parameter, accepts high-magnification image I High As input, and output feature f High As the alignment standard. In contrast, Φ Align (·) Acceptance and I High Low-resolution image I of the same field of view Low , and output the aligned features f Align The alignment operation is performed on each image patch to ensure that the model learns the complete image information, but ultimately only the feature vector output by the feature extractor is used for the subsequent WSI prediction task.

[0080] S22. The global-local Transformer classification module (GLT model) is a CNN-Transformer hybrid neural network designed based on Multiple Instance Learning (MIL), in which the entire WSI is processed as a package, and the Patch is regarded as an instance within the package. MIL can alleviate the memory overload problem caused by the high resolution of WSI, enabling the deep learning model to process the entire image at one time. However, the existing MIL model mainly focuses on establishing the relationship between instances and labels, ignoring the correlation between instances and between instances and the global. In order to solve this problem, the global-local Transformer classification module designs a local attention module and a global attention module, which uses CNN's sensitivity to local information and Transformer's ability to model global dependencies to explore the correlation between patches and between patches and WSI, and effectively integrates them through the Transformer layer to improve prediction accuracy.

[0081] Please refer to part a of Figure 3 again for further explanation of the specific contents of the local attention module and the global attention module:

[0082] S211. In the local attention module, capturing the interactions between different regions of the organization is crucial for accurately predicting the WSI task. To this end, the local attention module uses the convolution operation in CNN to capture local information, and designs a local attention module, such as the local attention module in Figure 5. The fixed receptive field provided by the convolution kernel is sensitive to local information, but it also limits the interaction of information in a larger range. Therefore, this module uses dilated convolutions with dilation values ​​of 1, 3, and 5 to construct an instance-level feature pyramid to capture instance information at multiple scales. Finally, the information at the three scales is fused by averaging. In this module, class labels do not participate in the calculation of attention. It can be expressed as follows: f1=Conv1(f) f2=Conv3(f) f3=Conv5(f) f out =Mean(f1,f2,f3)

[0083] Where f represents the input feature, Conv i (·) represents the dilated convolution with hole i, Mean(·) represents the average pooling operation, and f out is the output feature.

[0084] S222. In the global attention module, the traditional MIL model considers the correlation between instances and labels, but lacks the ability to capture global correlation, resulting in incomplete consideration of the semantic features of WSI. The present invention designs a patch-level global attention module to discover highly predictive features within the package, as shown in the global attention module in Figure 5.

[0085] In the global attention module, the feature F of the patch set is B×(N+1)×L Input global attention module, in which Transformer is used to explore the global information of the rate-aligned patch set Out B×(N+1)×L That is, first use the fully connected layer to generate Q, K, V vectors, then use Q, K vectors to generate global attention according to the self-attention mechanism, then do vector multiplication with V, and finally concatenate with the randomly initialized Class Token to get the final output. Refer to part b in Figure 3 for the process definition of the global attention module algorithm: Q = MLP (F B×N×L ) K=MLP(F B×N×L ) V=MLP(F B×N×L ) Att B×N×L =Softmax(Transpose 0,2,1 (Q×K)) Att B×N×L =Concat(V×Transpose 0,2,1 (Att B×N×L ))

[0086] B represents Batch size, N represents the number of features, L represents the feature length, 1 represents the class label feature, MLP represents the multi-layer perceptron, Q, K, and V all represent feature F B×N×L The intermediate variable of the transformation, Att B×N×L represents the intermediate variable of the attention mechanism, Class Token, generated using a random initialization strategy, is used to learn all instance information; Transpose 0,2,1 (·) means adjusting the 0th, 1st, and 2nd dimensions of the feature to 0, 2, and 1.

[0087] The program is defined as follows:

[0088] S223, feature expansion and class label embedding;

[0089] CNNs cannot handle irregular feature matrices, so feature augmentation is necessary to maintain their regularity. Therefore, the GLT module pads the input feature packets using random sampling. After padding, these are concatenated with randomly initialized class labels and serve as the input to the first transformation layer, as shown in the feature augmentation section in Figure 5.

[0090] S3. Train the MAT classification network model. The training process consists of two parts: the first part is the training of the self-supervised rate alignment module, and the second part is the training of the global-local Transformer classification module. The details are as follows:

[0091] S31. Training of self-supervised rate alignment module:

[0092] During training, the dataset is randomly split into a training set and a validation set. The self-supervised magnification alignment module uses L1 loss to reduce the absolute distance between features at different magnifications. The absolute distance between features at different magnifications and achieve almost lossless semantic alignment are calculated using the following loss function:

[0093] where X i is Φ High (·) Output features of the i-th block, x i is Φ Align (·) Output features of the i-th block. No data augmentation strategy is used during training because the samples required for training can meet the needs of the model.

[0094] For example, the parameters of the self-supervised rate alignment module are updated using the Adam optimizer with an initial learning rate of 1×10 -4 ; The learning strategy is linear decay with a decay coefficient of 0.9.

[0095] S32, training of the global-local Transformer classification module;

[0096] The global-local Transformer classification module uses the cross entropy loss function as follows:

[0097] in is the model output of the i-th WSI, y i is the label, and the global-local Transformer classification module randomly initializes the model parameters. During training, the instances in the input bag are randomly shuffled as a form of data augmentation, and during model inference, they are arranged in the order of the feature extraction stage.

[0098] S4. Use the trained MAT classification network model to obtain the pathological image classification prediction results.

[0099] Unlike traditional methods that require high-resolution input images (400x or 200x), MAT only requires low-resolution images (100x, 50x, or even 25x). While maintaining the same high-performance prediction results as leading models, MAT reduces computation time by 20 to 40 times, requiring only 16 times less data to be read.

[0100] In a specific embodiment of the present invention, the following technical solution is adopted:

[0101] Step 1: After acquiring the pathological image, perform tissue segmentation on the image and obtain the mask of the WSI tissue area. The processing method used is the same as that described in the tissue segmentation above.

[0102] Step 2: After obtaining the mask, perform a patch operation on the WSI. At high magnification (e.g., 200x), each patch is 224 x 224 pixels. The patch size scales proportionally with the magnification of the image being used (e.g., 112 x 112 pixels at 100x and 56 x 56 pixels at 50x).

[0103] Step 3: Select the acquired patches. If the tissue area within the patch is too small, it will be discarded. The area of ​​the tissue area is calculated based on the area of ​​the tissue contained in the mask at the same location as the patch.

[0104] Step 4: Train the feature alignment module. After obtaining the patch, the alignment model is trained using the training strategy described in the self-supervised rate alignment module above.

[0105] Step 5: Feature Extraction. After obtaining the patch, use the trained network to extract features. If extracting raw features, input them into a ResNet50 pre-trained on ImageNet. If using low-rate aligned features, input them into a feature alignment network for feature extraction. The specific method is described in the self-supervised rate alignment module above.

[0106] Step 6: Train the MAT classification network. Train the GLT model as described in the GLT model training strategy above.

[0107] Step 7: Model testing. After collecting the test data, obtain the input features according to the order of steps 3 and 5 above, and then input them into the MAT classification model trained in step 6 for prediction.

[0108] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0109] Based on the same idea as the rapid analysis method of pathological images based on magnification alignment transformer in the above-mentioned embodiment, the present invention also provides a rapid analysis system of pathological images based on magnification alignment transformer, which can be used to execute the above-mentioned rapid analysis method of pathological images based on magnification alignment transformer. For ease of explanation, the structural diagram of the embodiment of the rapid analysis system of pathological images based on magnification alignment transformer only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation of the device, and it can include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0110] Please refer to FIG6 . In another embodiment of the present application, a method system 10 for rapid analysis of pathological images based on magnification alignment transformer is provided. The system includes a feature engineering module 11 , a model building module 12 , an alignment and attention training module 13 , and an image testing module 14 .

[0111] The feature engineering module 11 is used to obtain a pathological image dataset, identify and segment tissue regions, obtain a mask of the WSI tissue region, and use the mask to perform a patch operation on the WSI tissue region; filter the mask of the WSI tissue region that has undergone the patch operation, remove patches with too small tissue area contained therein, and obtain a patch set; the tissue area is calculated based on the tissue area contained in the mask at the same position as the patch;

[0112] The model construction module 12 is used to construct a MAT classification network model, which includes a self-supervised rate alignment module and a global-local Transformer classification module; the self-supervised rate alignment module includes two feature extractors with the same structure, which are respectively used to extract high-rate features and generate semantically highly similar alignment features; the global-local Transformer classification module includes a global attention module and a local attention module, the global attention module uses the Transformer's ability to learn global information to explore global features between instances; the local attention mechanism uses the CNN's ability to learn detailed information to explore local features of instance time; the features explored by the global attention module and the local attention module are aggregated, and a prediction result is obtained by inference through a fully connected layer;

[0113] The alignment and attention training module 13 is used to train the MAT classification network model, and uses the patch set to perform self-supervised magnification alignment training on the MAT classification network model, aligning the low-magnification image to the high-magnification at the feature level to obtain the magnification-aligned patch set; in the global attention module, the Transformer is used to explore the global information of the magnification-aligned patch set, and in the local attention module, the CNN is used to capture the local information of the magnification-aligned patch set, and then the pathological image features are aggregated according to the global information and the local information, and the prediction results are predicted through the fully connected layer reasoning, and then the prediction loss is calculated, and the trained MAT classification network model is obtained through the back propagation algorithm; the self-attention mechanism in the Transformer covers all patches, thereby obtaining global attention, and the fixed-size convolution kernel and sliding window mechanism of the CNN in the local attention module are more sensitive to adjacent patches;

[0114] The image testing module 14 is used to test the trained MAT classification network model using the Patch set and the magnification-aligned Patch set to obtain the pathological image classification prediction result.

[0115] It should be noted that the pathological image rapid analysis system based on the magnification alignment Transformer of the present invention corresponds one-to-one to the pathological image rapid analysis method based on the magnification alignment Transformer of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the pathological image rapid analysis method based on the magnification alignment Transformer are applicable to the embodiment of the pathological image rapid analysis method based on the magnification alignment Transformer. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0116] In addition, in the implementation of the pathological image rapid analysis system based on the magnification alignment transformer in the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the pathological image rapid analysis system based on the magnification alignment transformer is divided into different program modules to complete all or part of the functions described above.

[0117] Please refer to Figure 7. In one embodiment, an electronic device that implements a method for rapid analysis of pathological images based on magnification alignment transformer is provided. The electronic device 20 may include a first processor 21, a first memory 22 and a bus. It may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as a rapid analysis program 23 for pathological images based on magnification alignment transformer.

[0118] Among them, the first memory 22 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as a mobile hard disk of the electronic device 20. In other embodiments, the first memory 22 can also be an external storage device of the electronic device 20, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 20. Furthermore, the first memory 22 can also include both an internal storage unit of the electronic device 20 and an external storage device. The first memory 22 can not only be used to store application software and various types of data installed on the electronic device 20, such as the code of the pathological image rapid analysis program 23 based on the magnification alignment transformer, but can also be used to temporarily store data that has been output or is to be output.

[0119] In some embodiments, the first processor 21 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The first processor 21 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 22, as well as calling data stored in the first memory 22, to perform various functions of the electronic device 20 and process data.

[0120] FIG7 only shows an electronic device with components. Those skilled in the art will understand that the structure shown in FIG7 does not constitute a limitation on the electronic device 20 , and may include fewer or more components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0121] The magnification alignment transformer-based pathology image rapid analysis program 23 stored in the first memory 22 of the electronic device 20 is a combination of multiple instructions. When running in the first processor 21, it can achieve the following:

[0122] Acquire a pathology image dataset, identify and segment tissue regions, obtain a mask of the WSI tissue region, and use the mask to perform a patch operation on the WSI tissue region; filter the WSI tissue regions that have undergone the patch operation, and remove patches with too small an area of ​​tissue region to obtain a patch set; the area of ​​the tissue region is calculated based on the tissue area contained in the mask at the same position as the patch;

[0123] A MAT classification network model is constructed, comprising a self-supervised rate alignment module and a global-local Transformer classification module; the self-supervised rate alignment module comprises two feature extractors with identical structures, one for extracting high-rate features and the other for generating semantically highly similar alignment features; the global-local Transformer classification module comprises a global attention module and a local attention module, wherein the global attention module utilizes the Transformer's ability to learn global information to explore global features between instances; the local attention mechanism utilizes the CNN's ability to learn detailed information to explore local features of instance time; the features explored by the global attention module and the local attention module are aggregated, and a prediction result is obtained by inference through a fully connected layer;

[0124] The MAT classification network model is trained and self-supervised magnification alignment training is performed on the MAT classification network model using the patch set, and low-magnification images are aligned to high-magnification images at the feature level to obtain a magnification-aligned patch set. The global information of the magnification-aligned patch set is explored using the Transformer in the global attention module, and the local information of the magnification-aligned patch set is captured using the CNN in the local attention module. Pathological image features are then aggregated based on the global and local information, and the prediction results are predicted through inference using the fully connected layer. The prediction loss is then calculated, and the trained MAT classification network model is obtained through the backpropagation algorithm. The self-attention mechanism in the Transformer covers all patches, thereby obtaining global attention. The fixed-size convolution kernel and sliding window mechanism of the CNN in the local attention module are more sensitive to adjacent patches.

[0125] The trained MAT classification network model is used to obtain the pathological image classification prediction results.

[0126] Furthermore, if the modules / units integrated in the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0127] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0128] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A fast analysis method for pathological images based on magnification alignment Transformer, characterized in that: The steps include: Obtain a pathological image dataset, identify and segment tissue regions, obtain a mask of the WSI tissue region, and use the mask to perform a patch operation on the WSI tissue region; Screening the WSI tissue regions that have undergone the patch cutting operation, removing patches with too small tissue area contained therein, and obtaining a patch set; the tissue area is calculated based on the tissue area contained in the mask at the same position as the patch; A MAT classification network model is constructed, wherein the MAT classification network model includes a self-supervised rate alignment module and a global-local Transformer classification module; the self-supervised rate alignment module includes two feature extractors with the same structure, which are respectively used to extract high rate features and generate semantically highly similar alignment features; the global-local Transformer classification module includes a global attention module and a local attention module, wherein the global attention module utilizes the learning ability of Transformer for global information to explore global features between instances; the local attention mechanism utilizes the learning ability of CNN for detail information to explore local features of instance time; the features explored by the global attention module and the local attention module are aggregated, and a prediction result is obtained by inference of a fully connected layer; The MAT classification network model is trained, and the Patch set is used to perform self-supervised magnification alignment training on the MAT classification network model, and the low-magnification image is aligned to the high-magnification at the feature level to obtain the magnification-aligned Patch set; the Transformer is used in the global attention module to explore the global information of the magnification-aligned Patch set, and the CNN is used in the local attention module to capture the local information of the magnification-aligned Patch set, and then the pathological image features are aggregated according to the global information and the local information, and the prediction results are predicted through the fully connected layer reasoning, and then the prediction loss is calculated, and the trained MAT classification network model is obtained through the back-propagation algorithm; the self-attention mechanism in the Transformer covers all patches, thereby obtaining global attention, and the fixed-size convolution kernel and sliding window mechanism of the CNN in the local attention module are more sensitive to adjacent patches; The trained MAT classification network model is used to obtain the pathological image classification prediction results.

2. The rapid analysis method of pathological images based on magnification alignment Transformer according to claim 1 is characterized in that: The identification and segmentation of the tissue region, obtaining the mask of the WSI tissue region, and using the mask to perform a patch operation on the WSI tissue region are specifically as follows: Use ImageNet to perform binary classification on the pathological image dataset, identify normal tissue areas, blank areas, and contaminated areas, and obtain the mask of the WSI tissue area; The mask of the WSI tissue area is patched, and all images are cropped into patches of a set size, and then input into ResNet50 pre-trained on Image Net for feature extraction; the size of the patch changes proportionally with the magnification of the image used.

3. The rapid analysis method of pathological images based on magnification alignment Transformer according to claim 1 is characterized in that: The use of the Patch set to perform self-supervised rate alignment training on the MAT classification network model is specifically as follows: Using two task-dependent feature extractors with the same structure Φ High (·) and Φ Align (·), freeze Φ High The parameters of (·), Extract high-rate features and generate semantically highly similar alignment features, i.e., Φ High (·) Accepts high-magnification image input and outputs high-magnification features, Φ Align (·) Accepting low-resolution images with the same field of view as high-magnification images, and generating semantically highly similar aligned features; For high-magnification features and alignment features, L1 loss is used to reduce the absolute distance of features with different magnifications to achieve semantic alignment. The loss function is as follows: Where X i is Φ High (·) Output features of the ith block, x i is Φ Align (·) Output features of the ith block.

4. The rapid analysis method of pathological images based on magnification alignment Transformer according to claim 1 is characterized in that: The Transformer is used in the global attention module to explore the global information of the rate-aligned patch set, specifically: In the global attention module, the feature F of the patch set is B×(N+1)×L Input the global attention module, and use Transformer in the global attention module to explore the global information of the rate-aligned patch set Out B×(N+1)×L That is, first use the fully connected layer to generate Q, K, V vectors, then use Q, K vectors to generate global attention according to the self-attention mechanism, then do vector dot multiplication with V, and finally concatenate with the randomly initialized Class Token to get the final output. The complete process definition is: Q=MLP(F B×N×L ) K=MLP(F B×N×L ) V=MLP(F B×N×L ) To B×N×L =Softmax(Transpose 0,2,1 (Q×K)) To B×N×L =Concat(V×Transpose 0,2,1 (To B×N×L )) Where B represents Batch size, N represents the number of features, L represents the feature length, 1 represents the class label feature, MLP represents the multi-layer perceptron, Q, K, and V all represent the feature F B×N×L The intermediate variable of the transformation, Att B×N×L represents the intermediate variable of the attention mechanism, It is a Class Token, which is generated by a random initialization strategy and is used to learn all instance information; Transpose 0,2,1 (·) means adjusting the 0th, 1st, and 2nd dimensions of the feature to 0, 2, and 1.

5. The rapid analysis method of pathological images based on magnification alignment Transformer according to claim 1 is characterized in that: The local information of the rate-aligned patch set is captured by using CNN in the local attention module, specifically: In the local attention module, a dilated convolution with dilated values ​​of 1, 3, and 5 in CNN is used to construct an instance-level feature pyramid to capture instance information at three scales. The instance information f1, f2, and f3 of the three scales are fused by the averaging method: f1=Conv1(f) f2=Conv3(f) f3=Conv5(f) f out =Mean(f1,f2,f3) Where f represents the input feature, Conv i (·) represents the dilated convolution with dilation i, Mean(·) represents the average pooling operation, and f out is the output feature; Get the local information of the aligned patch set.

6. The rapid analysis method of pathological images based on magnification alignment Transformer according to claim 1 is characterized in that: The prediction result is inferred through the fully connected layer, and then the prediction loss is calculated, specifically: During the training process, the cross entropy loss function is used to calculate the aggregated image feature loss through the fully connected layer to form enhanced data, as shown in the following formula: in is the output of the ith WSI, y i is the corresponding label; The enhanced data are arranged in the order before the pathological image features are aggregated, and the prediction results of the MAT classification network model are inferred.

7. The rapid analysis method of pathological images based on magnification alignment Transformer according to claim 1 is characterized in that: The training of the MAT classification network model further includes the following steps: Before training, the input rate-aligned patch set is expanded and the class labels are embedded, that is, the features of the rate-aligned patch set are filled by random sampling; after filling, they are concatenated with randomly initialized class labels as the input of the first conversion layer to maintain the normalization of the feature matrix.

8. The method for rapid analysis of pathological images based on magnification alignment Transformer according to claim 1, characterized in that: Pathological image features are aggregated using the mean operation for global and local information.

9. A rapid pathological image analysis system based on magnification alignment Transformer, characterized in that: A rapid analysis method for pathological images based on magnification alignment Transformer applied to any one of claims 1-7, comprising a feature engineering module, a model building module, an alignment and attention training module, and an image testing module; The feature engineering module is used to obtain a pathological image dataset, identify and segment tissue regions, obtain a mask of the WSI tissue region, and perform a patch operation on the WSI tissue region using the mask; The mask of the WSI tissue area after the patch cutting operation is screened, and the patch with too small tissue area contained therein is eliminated to obtain a patch set; the tissue area is calculated according to the tissue area contained in the mask at the same position as the patch; The model building module is used to build a MAT classification network model, which includes a self-supervised rate alignment module and a global-local Transformer classification module; the self-supervised rate alignment module includes two feature extractors with the same structure, which are used to extract high rate features and generate semantically highly similar alignment features respectively; the global-local The Transformer classification module includes a global attention module and a local attention module. The global attention module uses the Transformer's ability to learn global information to discover global features between instances; the local attention mechanism uses CNN's ability to learn detailed information to discover local features of instance time; the features discovered by the global attention module and the local attention module are aggregated, and the prediction result is obtained by inference through a fully connected layer; The alignment and attention training module is used to train the MAT classification network model, and the Patch set is used to perform self-supervised magnification alignment training on the MAT classification network model, aligning the low-magnification image to the high-magnification at the feature level to obtain the magnification-aligned Patch set; in the global attention module, the Transformer is used to explore the global information of the magnification-aligned Patch set, and in the local attention module, CNN is used to capture the local information of the magnification-aligned Patch set, and then the pathological image features are aggregated according to the global information and the local information, and the prediction results are predicted through the fully connected layer reasoning, and then the prediction loss is calculated, and the trained MAT classification network model is obtained through the back propagation algorithm; the self-attention mechanism in the Transformer covers all patches, thereby obtaining global attention, and the fixed-size convolution kernel and sliding window mechanism of the CNN in the local attention module are more sensitive to adjacent patches; The image testing module is used to obtain pathological image classification prediction results using the trained MAT classification network model.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the rapid analysis method of pathological images based on magnification alignment Transformer as described in any one of claims 1-7.

Images