Method and apparatus for predicting gene mutation based on wsi image cascaded packet level learning

Abstract

The application discloses a method and equipment for predicting gene mutation based on WSI image cascade package level learning, and the method comprises the following steps: cutting WSI images into color blocks according to maximum resolution equidistantly to obtain a color block set; using a fine-tuned ViT model to perform feature extraction on the color block set of the WSI images to generate a color block level feature matrix; regarding the color block level feature matrix of one WSI image as one package; training a first package level model based on benign and malignant package level labels and color block level features, outputting a first WSI package level feature matrix through the first package level model, and outputting an image level benign and malignant prediction through the first package level model; and training a second package level model based on gene mutation state package level labels and the first WSI package level feature matrix, and the second package level model is used for identifying the gene mutation state of the tumor region of the WSI image. The application realizes the prediction of gene mutation only through WSI images through cascade package level models.

Description

Technical Field

[0001] This invention belongs to the field of image data information mining, specifically involving a method and device for predicting gene mutations based on WSI image cascade package-level learning. Background Technology

[0002] Gene mutations refer to permanent changes in DNA sequences that occur during replication, repair, or under the influence of environmental factors, including base substitutions, insertions, deletions, duplications, inversions, or chromosomal structural variations. Essentially, they are "spelling errors" of genetic information at the molecular level, occurring in germ cells (heredity) or somatic cells (tumor development). The causes of gene mutations can generally be classified as endogenous (replication errors, spontaneous deamination) or exogenous (ultraviolet radiation, chemical carcinogens, viruses, etc.). They can lead to loss / abnormality of protein function (e.g., inactivation of the tumor suppressor gene TP53) and even confer selective advantages on cells (e.g., EGFR activating mutations driving tumors). As a source of tumor heterogeneity, they can serve as biomarkers for targeted therapy (e.g., EGFR inhibitors) and immunotherapy (e.g., TMB assessment).

[0003] In the era of precision oncology, gene mutations have become a core indicator for assessing patient survival prognosis and developing individualized treatment. Currently, the main methods for gene mutation detection include PCR, next-generation sequencing (NGS), and real-time quantitative PCR (RT-qPCR). The cost of these tests ranges from several thousand to tens of thousands of yuan, depending on the specific tests performed and the testing institution. Furthermore, the effectiveness and accuracy of the tests are highly susceptible to factors such as the laboratory environment, reagents, and operational procedures.

[0004] Current research extensively studies the morphology and genetic association of surface tumor cells. Within the same cancer type, gene mutations can significantly alter tumor cell morphology, for example: causing cells to change from a tightly packed arrangement to a loosely extended stromal morphology, exhibiting irregular spindle or star shapes, forming sheet-like and filopodia, increasing cell volume, forming membrane protrusions and vesicles, and exhibiting irregular nuclear membranes or even multinucleation, with reduced heterochromatin and enlarged nucleoli within the nucleus, thus acquiring more aggressive and adaptive characteristics. This manifests as subtle but reproducible differences in texture, karyotype, and glandular structure between cells with a certain gene mutation and those without mutation on HE sections. Based on this idea, packet-level learning can be used to quantify these differences. That is, a gigapixel WSI image is divided into tens of thousands of color patches; if a few key regions contain mutation signals, the entire section is considered "mutation-positive," greatly reducing conventional annotation time. Summary of the Invention

[0005] The purpose of this invention is to provide a method and device for predicting gene mutations based on WSI image cascaded package-level learning. This method integrates a deep learning feature extraction model for pathological images with a cascaded package-level learning method, enabling the prediction of gene mutations using only WSI images. This allows for rapid pathological diagnosis during surgery, quick screening of patients with gene mutations, and improved efficiency and success rate of clinical trials.

[0006] In a first aspect, the present invention provides a method for predicting gene mutations based on WSI image cascade package-level learning, comprising:

[0007] Obtain paraffin-embedded HE-stained sections of any disease and scan them as WSI images;

[0008] The WSI image is cut into color blocks at equal intervals according to the maximum resolution, and invalid color blocks are removed to obtain a color block set.

[0009] The ViT model (VisionTransformer), fine-tuned using the DINOv2 self-supervised learning method, is used to extract features from the color patch set of WSI images, generating a color patch-level feature matrix; the color patch-level feature matrix of a WSI image is regarded as a bag.

[0010] The good or bad condition is assigned as a package-level label to each package. A first-level package-level model is trained based on the good or bad condition package-level label and color block level features. The first-level color block attention weight is obtained through the first-level package-level model. The first-level color block attention weight is multiplied point by point with the first-level color block descriptor matrix of the WSI image to output the first-level WSI package-level feature matrix. The first-level package-level model outputs the image-level good or bad condition prediction.

[0011] Gene mutation status is assigned as a package-level label to each package. Then, a second-level package-level model is trained based on the gene mutation status package-level label and the first-level WSI package-level feature matrix. The second-level package-level model is used to identify the gene mutation status of tumor regions in WSI images.

[0012] Further preferred, the feature extraction process is as follows: after standardizing the colors of N color blocks in a WSI image, the color block size is unified, and then the ViT model, which is fine-tuned based on the DINOv2 self-supervised learning method, is input. After the tile embedding block and 24 Transformer blocks, the global and local features of each color block are extracted. After normalization, the final output is the N×1024-dimensional color block level feature matrix of the WSI image.

[0013] Further preferred, the first-level packet-level model consists of three parts: an initialization fully connected layer, a gated attention layer, and a classifier layer;

[0014] Initialize a fully connected layer to reduce the dimensionality of the original color patch level features and cache the first-level color patch descriptor matrix;

[0015] The gated attention layer generates first-level color block attention weights using the Tanh activation function and the Sigmoid activation function, respectively, and then normalizes them using Softmax.

[0016] The classifier layer multiplies the normalized first-level color patch attention weights with the color patch descriptor matrix point by point to output a first-level WSI bag-level feature matrix, which is then aggregated into a first-level global feature for image-level benign / malignant prediction.

[0017] The two-level packet-level model consists of three parts: an initialization fully connected layer, a gated attention layer, and a classifier layer.

[0018] Initialize a fully connected layer to reduce the dimensionality of the first-level WSI packet-level feature matrix and cache the second-level color block descriptor matrix;

[0019] The gated attention layer generates secondary color block attention weights using the Tanh activation function and the Sigmoid activation function, respectively, and then normalizes them using Softmax.

[0020] The classifier layer multiplies the normalized secondary color patch attention weights with the secondary color patch descriptor matrix point by point to output the secondary WSI bag-level feature matrix, which is then aggregated into secondary global features for image-level gene mutation prediction.

[0021] Further optimization involves concatenating the parameters of the first-level package-level model with those of the second-level package-level model during training, with the first-level package-level model serving as the filtering unit. If the predicted label is benign, the result is output directly; if it is malignant, the corrected feature matrix is ​​output to the second-level package-level model for gene mutation prediction.

[0022] Further optimization involves removing invalid color blocks by setting a threshold through color convolution to filter out color blocks in disorganized areas, empty cavities, and background areas.

[0023] Secondly, the present invention provides a device for predicting gene mutations based on WSI image cascade package-level learning, comprising:

[0024] The paraffin-embedded hematoxylin and eosin (HE) stained section digitization module is used to digitize paraffin-embedded HE stained sections into WSI images;

[0025] The image cropping quality control module cuts the WSI image into color blocks at equal intervals according to the maximum resolution, removes invalid color blocks, and obtains a color block set.

[0026] The color block level feature extraction module uses the ViT model, fine-tuned based on the DINOv2 self-supervised learning method, to extract features from the color block set of WSI images and generate a color block level feature matrix; the color block level feature matrix of a WSI image is regarded as a bag.

[0027] A cascaded packet-level learning module connects a first-level packet-level model and a second-level packet-level model. Each packet is assigned a packet-level label indicating whether it is benign or malignant. The first-level packet-level model is trained based on the benign / malignant packet-level labels and color block level features. First-level color block attention weights are obtained from the first-level packet-level model. These first-level color block attention weights are then multiplied point-by-point with the first-level color block descriptor matrix of the WSI image to output a first-level WSI packet-level feature matrix. Gene mutation status is assigned as a packet-level label to each packet. A second-level packet-level model is then trained based on the gene mutation status packet-level labels and the first-level WSI packet-level feature matrix. This second-level packet-level model is used to identify the gene mutation status of tumor regions in WSI images.

[0028] Thirdly, the present invention provides an electronic device, wherein the computer device includes a data processor and a professional image processor, the memory stores a visualization operating system, and the computer program is executed by the processor to describe the method for predicting gene mutations based on WSI image cascade package-level learning.

[0029] Fourthly, the present invention provides a device-readable storage medium, characterized in that the computer-readable storage medium is used to perform the method for predicting gene mutations based on WSI image cascade package-level learning.

[0030] The beneficial effects of this invention are:

[0031] A cascaded packet-level model and a ViT model, composed of a first-level packet-level model and a second-level packet-level model, can effectively transform image information from WSI into a feature matrix that can be learned at the packet-level. This provides a practical and theoretical basis for predicting gene mutation status using WSI image information. The ViT model is integrated with the packet-level learning method, and a cascaded approach is used to further improve the predictive performance of the gene mutation model. This enables the prediction of gene mutations using only WSI images, leading to rapid intraoperative pathological diagnosis, quick screening of patients with gene mutations, and improved efficiency and success rate of clinical trials.

[0032] By employing a cascaded bag-level model with a two-step "coarse-to-fine" focusing approach, the first-level bag-level model reduces the attention weights of non-tumor color patches to near zero. The second-level bag-level model then redistributes the attention weights of the second-level color patches, performing fine sorting only within tumor color patches. Compared to a simple bag-level model, this approach reduces interference from normal tissue, significantly improves the feature signal-to-noise ratio, and significantly enhances model performance. The color patch attention weight map is more focused, resulting in better clinical interpretability. Furthermore, by decomposing the training difficulty, it reduces the risk of overfitting, is compatible with any downstream mutations, and has low algorithm transfer costs. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention.

[0034] Figure 2 This is a structural diagram of a cascaded packet-level model based on a first-level packet-level model and a second-level packet-level model.

[0035] Figure 3 The ROC curve is for the first-level package model.

[0036] Figure 4 The ROC curve is for a two-level package-level model.

[0037] Figure 5 ROC curve for a standalone model.

[0038] Figure 6 The first-level color block attention weight map output by the first-level package-level model.

[0039] Figure 7 The attention weight map of the second-level color blocks output by the second-level package model. Detailed Implementation

[0040] To better understand the above-mentioned objects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to embodiments. The specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0041] like Figure 1 and Figure 2 As shown in the figure, this embodiment provides a method for predicting gene mutations based on WSI image cascade package-level learning, and the steps are as follows:

[0042] Step 1: Obtain WSI images: Obtain paraffin-embedded HE-stained sections for any disease, discarding sections without tissue, with overlapping or severely contaminated tissue, and scan the paraffin-embedded HE-stained sections as WSI images (panoramic pathological scan images). The image format should be svs, tif, tiff, kfb, png, or jpg.

[0043] Step 2, WSI Image Segmentation and Quality Control: The WSI image is segmented at equal intervals according to the maximum resolution to obtain color blocks with an actual physical distance of 128μm*128μm. Then, invalid color blocks such as unorganized areas, empty cavities, and background areas are filtered out by setting a threshold through color convolution to obtain a color block set.

[0044] Step 3: After standardizing the colors of N color patches in a WSI image and unifying the patch size, input the ViT model (Vision Transformer) fine-tuned based on the DINOv2 self-supervised learning method. After processing the patch embedding and 24 Transformer blocks, the global and local features of each color patch are extracted. After normalization, the final output is the N×1024-dimensional color patch level feature matrix of the WSI image. The color patch level feature matrix of a WSI image is regarded as a bag.

[0045] Step 4: Assign benign or malignant as a bag-level label to each bag. Train a first-level bag-level model based on the benign / malignant bag-level labels and color block level features. Obtain first-level color block attention weights through the first-level bag-level model. Multiply the first-level color block attention weights point-by-point with the first-level color block descriptor matrix of the WSI image to output the first-level WSI bag-level feature matrix. The first-level bag-level model is responsible for tumor discrimination and ROI region contribution calculation, used to identify the benign / malignant state of tissues in the WSI image. It can also output the coordinates of high-malignancy regions and the first-level WSI bag-level feature matrix. The first-level bag-level model first determines the predicted label. If it is a benign tumor, no further operation is performed, and the benign label is directly output. If it is a malignant label, the first-level bag-level model outputs the first-level WSI bag-level feature matrix to the second-level bag-level model.

[0046] Step 5: Assign the gene mutation status as a package-level label to each package, and then train a second-level package-level model based on the gene mutation status package-level label and the first-level WSI package-level feature matrix. The second-level package-level model is used to identify the gene mutation status of the tumor region in the WSI image.

[0047] Furthermore, in step one, the collected paraffin-embedded HE-stained breast tissue sections were different, with clear pathological conditions, no damage, no folds, clear images, and no staining contamination or impurities. These sections were then digitized using a slide scanner and saved as SVS, TIF, PNG, and other image formats. In this embodiment, the goal was to establish a classification model that could predict the PIK3CA gene mutation status based on WSI images of breast cancer. Therefore, TCGA breast cancer paraffin-embedded HE-stained sections and corresponding gene mutation tags were downloaded. Sections without tissue, with overlapping tissue damage, or severely contaminated staining were removed, resulting in a total of 640 cases (all malignant tumors), of which 575 cases had corresponding gene mutation tags. Paraffin-embedded HE-stained breast cancer sections from hospitals were collected and scanned as WSI images. The image format should be SVS, with a maximum resolution of 40×, totaling 311 cases, including 144 malignant tumors and 167 non-malignant tumors.

[0048] Furthermore, in step two, the WSI image is processed to remove the background, retaining the areas of the WSI image containing tissue. After being divided into several color blocks with equal physical distance according to the highest resolution, quality control is performed, and spatial location information is assigned to each color block to obtain a color block set. In this embodiment, referring to the maximum resolution of the WSI image, the WSI image is equidistantly divided into color blocks with an actual physical distance of 128μm×128μm. Quality control is then performed on the cut color blocks, including: 1. Background removal: Setting the three-channel threshold of the image, after inversion, the overall average grayscale is between 10–240, the mean variance of the three channels is greater than 280, and the median grayscale is also between 10–240. Color blocks with pure white, pure black, or near-pure white / pure black colors are removed. 2. Size control: Determining the cutting size based on the maximum resolution of the WSI image, considering "40×" resolution as using 512 pixels × 512 pixels and "20×" as using 256 pixels × 256 pixels. Color blocks with mismatched sizes are removed. 3. Sharpness control: Generally, it is considered that when the sharpness threshold is set to 150, the texture and color of the color blocks are still meaningful. Color blocks with a sharpness lower than 150 are removed, ultimately obtaining the color block set.

[0049] Further, in step three, after standardizing the colors of the color patch set, the image matrix is ​​unified to a size of 224 pixels × 224 pixels × 3 channels, with a batch size of 128 images. This image is then input into the ViT model (Vision Transformer). The ViT model used in this example is the UNI pathology pedigree model published by the mahmood laboratory. This model is trained based on the DINOv2 self-supervised learning method. Specifically, it uses random masks and class labels to allow the ViT model to learn image-level and patch-level targets through cross-entropy loss and distillation loss functions, respectively, thereby learning global and local features. After one patch embedding and 24 Transformer blocks, deep feature extraction is performed on each color patch. After all Transformer blocks, the features are normalized, and a head is used for the final output, resulting in an N×1024-dimensional color patch-level feature matrix of the WSI image. The color patch-level feature matrix of a single WSI image is considered as a bag.

[0050] Furthermore, in step four, 144 malignant tumor packets and 167 non-malignant tumor packets from hospital-sourced WSI images, as well as 65 malignant tumor packets from TCGA images, are randomly selected as the training set. The random number is set to 123, and 10 internal training, testing, and validation sets are randomly constructed in a ratio of 8:1:1, i.e., ten-fold cross-validation. The maximum number of loops is set to 200, the learning rate to 0.0001, the L2 regularization first-level color patch attention weight decay coefficient to 0.00001, the positive and negative instance sample collection limit to 8, and the ratio of packet-level loss to color patch loss in the total loss to 7:3. The first-level packet-level model consists of three parts: an initialization fully connected layer, a gated attention layer, and a classifier layer. The initialization fully connected layer first reduces the received original N×1024-dimensional color patch level features to N×512, while caching the N×1024-dimensional first-level color patch descriptor matrix. The gated attention layer generates N×256 attention points through two activation functions, "Tanh branch" and "Sigmoid branch", respectively. After dot product and full connection, the original attention score of N×1 is obtained, and then normalized to the first-level color patch attention weight α1 using Softmax. The classifier layer multiplies the first-level color patch attention weight α1 with the N×1024-dimensional first-level color patch descriptor matrix point by point, and outputs the N×1024-dimensional first-level WSI packet-level feature matrix. The same column is added to output 1×1024 first-level global features. Finally, the classifier layer gives the image-level benign / malignant prediction and outputs the predicted benign / malignant label. The performance of multiple candidate models was evaluated using 10-fold cross-validation, and the AUC-ROC values ​​(area under the operating characteristic curve) obtained from the cross-validation of each model were compared. Figure 3 As shown, Figure 3 Medium sensitivity represents the proportion of malignant (or mutated) cases correctly identified by the model; 1-specificity represents the proportion of benign (or non-mutated) cases incorrectly identified as malignant (or mutated); AUC is the area under the ROC curve, ranging from 0 to 1; CI is the 95% confidence interval; and the model with the best (usually the highest) AUC-ROC value is selected. Figure 3 Model 5 has an AUC-ROC value of 0.996, indicating that it has the best performance. Model 5 is considered the optimal first-level packet-level model. The attention weight diagram for the first-level color blocks in the first-level packet-level model is shown below. Figure 6 .

[0051] Furthermore, in step five, 575 malignant tumor packages with corresponding gene mutation tags from TCGA are used as the training set, of which 176 packages contain PIK3CA gene mutations and 399 packages do not. These 575 TCGA-sourced packages are input into the optimal first-level package-level model, and the output first-level WSI package-level feature matrix is ​​fed into the second-level package-level model for training. The random number is set to 123, and 10 internal training, testing, and validation sets are randomly constructed in a ratio of 8:1:1. The maximum number of loops is set to 200, the learning rate is 0.0001, the L2 regularization second-level color patch attention weight decay coefficient is 0.00001, the positive and negative instance sample collection is limited to 8, and the ratio of package-level loss to color patch loss in the total loss is 8:2. The second-level bag-level model is initialized based on the parameters of the optimal first-level bag-level model, enabling it to inherit the ability of the optimal first-level bag-level model to extract pathological features. The second-level bag-level model consists of three parts: an initialization fully connected layer, a gated attention layer, and a classifier layer. The initialization fully connected layer first reduces the first-level WSI bag-level feature matrix output by the optimal first-level bag-level model to N×512, while caching an N×1024-dimensional second-level color patch descriptor matrix. The gated attention layer generates N×256 attention through two activation functions, "Tanh branch" and "Sigmoid branch", respectively. After dot product and full connection, the original attention score of N×1 is obtained, and then normalized to the second-level color patch attention weight α2 using Softmax. The classifier layer multiplies the second-level color patch attention weight α2 with the N×1024-dimensional second-level color patch descriptor matrix point by point, and outputs an N×1024-dimensional second-level WSI bag-level feature matrix. The same columns are added to output 1×1024 second-level global features. Finally, the classifier layer provides image-level gene mutation prediction and outputs gene mutation labels. The performance of multiple candidate models was evaluated using 10-fold cross-validation. Model 3 showed the best performance and was selected as the optimal second-level packet-level model. The attention weight diagram of the second-level color blocks in the second-level packet-level model is shown below. Figure 7 After primary correction, the focus is more concentrated on highly malignant regions. The model's AUC-ROC value (area under the operating characteristic curve) is 0.711. Figure 4 Compared to the best model 5 (AUC-ROC value of 0.65) among independent gene mutation prediction models, the efficiency is improved by 6%. Figure 5 ).

[0052] By plotting the spatial locations of color patches and the global color patch attention weights for different gene mutation states, it can be found that the weakly supervised attention points are all located at the tumor location in the WSI image. This is consistent with the logic of laboratory detection and judgment. Therefore, it is believed that the method of the present invention has a certain degree of interpretability and can perform preliminary gene mutation state discrimination.

[0053] The present invention uses a publicly available general big data image model, which has the advantages of large model structure depth, strong feature extraction capability, strong robustness and applicability to the vast majority of pathological images.

[0054] In one implementation case, a device for predicting gene mutations based on WSI image cascade package-level learning includes a paraffin-embedded HE-stained section digitization module, an image cutting quality control module, a color patch-level feature extraction module, and a cascade package-level learning module. Among them:

[0055] The paraffin-embedded hematoxylin and eosin (HE) stained section digitization module is used to digitize paraffin-embedded HE stained sections into WSI images in svs, tif, tiff, kfb, png, and jpg formats;

[0056] The image segmentation quality control module is used to segment WSI images into color blocks with an actual physical distance of 128μm*128μm, filter out invalid color blocks such as unorganized areas, empty cavities, and background areas, assign spatial position labels to each valid color block, and obtain a color block set;

[0057] The color block level feature extraction module uses the ViT model, fine-tuned based on the DINOv2 self-supervised learning method, to extract features from the color block set of WSI images and generate a color block level feature matrix; the color block level feature matrix of a WSI image is regarded as a bag.

[0058] A cascaded packet-level learning module connects a first-level packet-level model and a second-level packet-level model. Each packet is assigned a packet-level label indicating whether it is benign or malignant. The first-level packet-level model is trained based on the benign / malignant packet-level labels and color block level features. First-level color block attention weights are obtained from the first-level packet-level model. These first-level color block attention weights are then multiplied point-by-point with the first-level color block descriptor matrix of the WSI image to output a first-level WSI packet-level feature matrix. Gene mutation status is assigned as a packet-level label to each packet. A second-level packet-level model is then trained based on the gene mutation status packet-level labels and the first-level WSI packet-level feature matrix. This second-level packet-level model is used to identify the gene mutation status of tumor regions in WSI images.

[0059] In one embodiment, a computer device includes a data processor, a specialized image processor, and a memory storing a visualization operating system, wherein a computer program is executed by the processor using any of the prediction methods described above.

[0060] In one embodiment, a computer-readable storage medium is provided for performing the prediction method described in any of the preceding embodiments.

[0061] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for predicting gene mutations based on WSI image cascade package-level learning, characterized in that, include: Obtain paraffin-embedded HE-stained sections of any disease and scan them as WSI images; The WSI image is cut into color blocks at equal intervals according to the maximum resolution, and invalid color blocks are removed to obtain a color block set. The ViT model, fine-tuned using the DINOv2 self-supervised learning method, is used to extract features from the color patch set of WSI images, generating a color patch-level feature matrix. The color block level feature matrix of a WSI image is considered as a bag; The good or bad condition is assigned as a package-level label to each package. A first-level package-level model is trained based on the good or bad condition package-level label and color block level features. The first-level color block attention weight is obtained through the first-level package-level model. The first-level color block attention weight is multiplied point by point with the first-level color block descriptor matrix of the WSI image to output the first-level WSI package-level feature matrix. The first-level package-level model outputs the image-level good or bad condition prediction. The first-level packet model consists of three parts: an initialization fully connected layer, a gated attention layer, and a classifier layer. The initialization fully connected layer is used to reduce the dimensionality of the original color patch level features and cache the first-level color patch descriptor matrix. The gated attention layer generates first-level color block attention weights through the Tanh activation function and the Sigmoid activation function, respectively, and then normalizes them using Softmax. The classifier layer multiplies the normalized first-level color patch attention weights with the color patch descriptor matrix point by point to output a first-level WSI bag-level feature matrix, which is then aggregated into a first-level global feature for image-level benign / malignant prediction. Gene mutation status is assigned as a package-level label to each package. Then, a second-level package-level model is trained based on the gene mutation status package-level label and the first-level WSI package-level feature matrix. The second-level package-level model is used to identify the gene mutation status of tumor regions in WSI images. The two-level packet-level model consists of three parts: an initialization fully connected layer, a gated attention layer, and a classifier layer. The initialization fully connected layer is used to reduce the dimensionality of the first-level WSI packet-level feature matrix and cache the second-level color block descriptor matrix. The gated attention layer generates secondary color block attention weights through the Tanh activation function and the Sigmoid activation function, respectively, and then normalizes them using Softmax. The classifier layer multiplies the normalized secondary color patch attention weights with the secondary color patch descriptor matrix point by point to output the secondary WSI bag-level feature matrix, which is then aggregated into secondary global features for image-level gene mutation prediction.

2. The method according to claim 1, characterized in that, The feature extraction process is as follows: After standardizing the colors of N color blocks in a WSI image and unifying the color block size, the model is input into the ViT model fine-tuned based on the DINOv2 self-supervised learning method. The global and local features of each color block are extracted through the tile embedding block and 24 Transformer blocks. After normalization, the final output is the N×1024-dimensional color block level feature matrix of the WSI image.

3. The method according to claim 1, characterized in that, When training the second-level packet-level model, the parameters of the first-level packet-level model are concatenated with those of the second-level packet-level model, and the first-level packet-level model performs the filtering function; if its predicted label is benign, the result is output directly. If the mutation is malignant, the corrected feature matrix will be output to the secondary package-level model for gene mutation prediction.

4. The method according to claim 1, characterized in that, Removing invalid color blocks includes filtering out color blocks in disorganized areas, empty cavities, and background areas by setting a threshold through color convolution.

5. A device for predicting gene mutations based on WSI image cascade package-level learning, characterized in that, include: The paraffin-embedded hematoxylin and eosin (HE) stained section digitization module is used to digitize paraffin-embedded HE stained sections into WSI images; The image cropping quality control module cuts the WSI image into color blocks at equal intervals according to the maximum resolution, removes invalid color blocks, and obtains a color block set. The color block level feature extraction module uses the ViT model, fine-tuned based on the DINOv2 self-supervised learning method, to extract features from the color block set of the WSI image and generate a color block level feature matrix. The color block level feature matrix of a WSI image is considered as a bag; The cascaded packet-level learning module connects the first-level packet-level model and the second-level packet-level model. It assigns benignity or malignancy as a packet-level label to each packet. The first-level packet-level model is trained based on the benignity / malignancy packet-level label and color block level features. The first-level color block attention weight is obtained through the first-level packet-level model. The first-level color block attention weight is multiplied point by point with the first-level color block descriptor matrix of the WSI image to output the first-level WSI packet-level feature matrix. Gene mutation status is assigned as a package-level label to each package. Then, a second-level package-level model is trained based on the gene mutation status package-level label and the first-level WSI package-level feature matrix. The second-level package-level model is used to identify the gene mutation status of tumor regions in WSI images. The first-level packet model consists of three parts: an initialization fully connected layer, a gated attention layer, and a classifier layer. The initialization fully connected layer is used to reduce the dimensionality of the original color patch level features and cache the first-level color patch descriptor matrix. The gated attention layer generates first-level color block attention weights through the Tanh activation function and the Sigmoid activation function, respectively, and then normalizes them using Softmax. The classifier layer multiplies the normalized first-level color patch attention weights with the color patch descriptor matrix point by point to output a first-level WSI bag-level feature matrix, which is then aggregated into a first-level global feature for image-level benign / malignant prediction. The two-level packet-level model consists of three parts: an initialization fully connected layer, a gated attention layer, and a classifier layer. The initialization fully connected layer is used to reduce the dimensionality of the first-level WSI packet-level feature matrix and cache the second-level color block descriptor matrix. The gated attention layer generates secondary color block attention weights through the Tanh activation function and the Sigmoid activation function, respectively, and then normalizes them using Softmax. The classifier layer multiplies the normalized secondary color patch attention weights with the secondary color patch descriptor matrix point by point to output the secondary WSI bag-level feature matrix, which is then aggregated into secondary global features for image-level gene mutation prediction.

6. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, the computer program being executed by the processor according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to perform the method according to any one of claims 1-4.

Images