A multi-gene mutation prediction method based on multi-task and multi-instance learning

Abstract

This invention discloses a multi-gene mutation prediction method based on a combination of multi-task and multi-instance learning. Belonging to the fields of digital image analysis, pathology, and machine learning, the specific steps are as follows: Preprocessing existing pathological image data by staining normalization; constructing a feature matrix for each pathological image, and further using a two-layer multi-instance learning method to construct a package for each pathological image; constructing a multi-task deep learning network based on transformer and MobileNet; applying the model to a test set and outputting pathological image analysis results. This invention employs a multi-task deep neural network and applies it to the task of predicting multiple gene mutations based on pathological images. Compared to traditional single-task networks, this invention can simultaneously predict the results of multiple tasks, saving computational resources while improving accuracy; furthermore, combining the transformer module with traditional convolution considers both local and global features.

Description

Technical Field

[0001] This invention belongs to the fields of digital image analysis, pathology and machine learning, and relates to a multi-gene mutation prediction method based on multi-task learning; specifically, it relates to a multi-gene mutation prediction method based on a combination of multi-task and multi-instance learning. Background Technology

[0002] Cancer is the second leading cause of death worldwide, and the number of new cancer cases globally is projected to reach 28.4 million by 2040. Therefore, accurate early diagnosis of cancer is crucial for patient treatment. Because WSI (Wide Image Sensor) technology can generate high-resolution digital images in a short time, revealing cellular morphological characteristics and the extent of cancer invasion, several WSI-based computer-aided diagnostic methods have emerged and achieved some success in recent years.

[0003] However, due to the high resolution of WSI (typically tens of thousands of pixels) and coarse annotation, automated diagnosis of WSI using human intelligence technology presents a significant challenge. Currently, many cancer diagnosis methods based on weakly supervised learning (MIL) have been proposed to address this issue. For example, Ilse et al. proposed a trainable MIL method based on an attention (gating) mechanism. By assigning a weight to each example and parameterizing it, they demonstrated the importance of different examples, improving the flexibility and interpretability of the MIL.

[0004] Because many tasks in pathological research have related components, multi-task learning is widely used in this field. Shao et al. proposed a multimodal, multi-task approach that uses genetic data and morphological features of pathological images as input, automatically discovering the task relationships between diagnosis and prognosis through their proposed multimodal, multi-task feature selection method. Recent studies have shown that cancer development is often accompanied by mutations in certain genes; therefore, determining whether a patient has gene mutations is crucial for personalized cancer treatment and medical research. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a multi-gene mutation prediction method based on multi-task and multi-example learning, which can extract and process features of pathological images using pathological image data, thereby simultaneously detecting KRAS and TP53 gene mutations in colorectal cancer.

[0006] Technical Solution: The multi-gene mutation prediction method based on multi-task and multi-instance learning combined as described in this invention has the following specific preparation steps.

[0007] Step (1): Perform preprocessing such as staining normalization on all the pre-acquired pathological image data;

[0008] Step (2): From the image data obtained in step (1), 100 image blocks of size 224*224 are randomly selected from each pathological image and input into the pre-trained ResNet101 feature extractor. Each image block obtains a 2048-dimensional feature vector, and the feature matrix of each pathological image is constructed using these feature vectors. Finally, a two-layer multi-instance learning method is used to construct a package for each pathological image.

[0009] Step (3): Construct a multi-task deep learning network based on transformer and MobileNet. This network takes the packet constructed in step (2) as input and adopts the cross-entropy loss function and Adam optimizer.

[0010] Step (4): According to the loss function in step (3), use the gradient descent method of backpropagation; with each gradient descent, the loss of the training set will become smaller and smaller. When the loss of the training set reaches the set value or the number of iterations exceeds the preset threshold, the training is completed; apply the model to the test set and output the pathological image analysis results.

[0011] Furthermore, in step (1), the process of performing staining normalization preprocessing on the pathological image data is as follows:

[0012] Considering the presence of numerous acellular areas in the pathological images and the uneven staining caused by artificial staining,

[0013] First, the background area is filtered out based on the values ​​of each pixel in the R, G, and B channels. The filtering rule is that if the pixel value in any channel is greater than 200, or if 90% of the pixels in the 200*200 neighborhood of the pixel are background areas, then the pixel is filtered out.

[0014] Then, the selected image patches are color-normalized.

[0015] Furthermore, the specific implementation process in step (2) is as follows:

[0016] Load the pre-trained ResNet101 model, remove the last fully connected layer, and use it as a feature extractor;

[0017] 100 image blocks are randomly selected from each pathological image data obtained in step (1), and 2048-dimensional features of each image block are obtained through a feature extractor. Thus, each pathological image data obtains a 100*2048 feature matrix.

[0018] A two-layer, multi-instance approach is used to construct the packages. Specifically, for each pathological image, 10 image patch features are randomly selected to form the first small package, and then 10 features are randomly selected from the remaining image patch features to form the second small package, and so on. Finally, each pathological image is a large package in the first layer, containing 10 corresponding small packages in the second layer.

[0019] Furthermore, in step (3), the multi-task deep learning network based on transformer and MobileNet consists of a transformer module, a MobileNet module, a Trans2Mobile module, a Mobile2Trans module, and a multi-task interaction module, etc.

[0020] The transformer module consists of a standard multi-head attention module, the basic form of which is shown in the following equation:

[0021]

[0022] In the formula, Q is the consultation vector, K is the key-value vector, and V is the value vector; multi-head attention enables the model to jointly attend to information from different representations of subspaces at different locations, as shown in the following formula:

[0023] MultiHead(Q,K,V)=Concat(head1,head2,...,head h W O

[0024] In the formula,

[0025] The Mobile2Trans module is a lightweight cross-multi-head attention module that transforms the MobileNet module into a transformer module, mapping local features X to global tokens, as shown in the following formula:

[0026]

[0027] In the formula, X represents the local features (initially the input is the feature matrix of the small packet, and during training, it is the output of the MobileNet module), and Z represents the global token used to learn global features, which is randomly initialized at the beginning of training; X and Z are represented in multi-head attention as follows: These represent the allocation to i. th Local features and global token of the header. W i q Indicate i th The query mapping matrix of the head, W mCombine multiple heads;

[0028] The three terms in Atten are equivalent to Q, K, and V in a standard attention module, but the mapping matrices for K and V are removed, and only the mapping matrix W corresponding to Q is retained. i q ;

[0029] Similar to the Mobile2Trans module, the Trans2Mobile module is a lightweight cross-multi-head attention module that transforms from a transformer module to a MobileNet module, implementing a transformation from global to local, as shown in the following formula:

[0030]

[0031] In the formula, and Mapping matrices for K and V respectively, W n Combine multiple heads.

[0032] Furthermore, the MobileNet module consists of a series of 1d convolutions and dynamic ReLU functions;

[0033] While retaining the standard 2D convolutions in the MobileNet architecture, the 1D convolutions are transformed into 1D convolutions; unlike the original ReLU function, the parameters of the activation function are determined by the output z of the transformer module. ′ Obtained through two MLP layers;

[0034] To achieve a multi-task network, multiple parallel deep learning networks based on Transformer and MobileNet are established according to the number of tasks. In the MobileNet modules of multiple tasks, convolutional layers with a kernel size greater than 1 are constrained for their kernel parameters. Specifically, the MMD distance (maximum mean error) between the parameters is calculated and added to the classification loss function obtained by cross-entropy and backpropagated together to achieve soft parameter sharing.

[0035] The local features X and the global token Z are then concatenated, and global average pooling and a two-layer MLP are used to predict multiple tasks to obtain the prediction results of small packets. Finally, an attention network is used to aggregate the output small packet results to obtain the prediction results of KRAS and TP53 gene mutations in pathological images.

[0036] With the continuous development of whole-slice pathological image digitization technology, more and more researchers are beginning to use computers to analyze pathological images of different types of cancer (such as lung cancer, colon cancer, breast cancer, and brain cancer). Compared with traditional manual slide reading methods, computer-aided pathological image analysis methods can significantly improve the efficiency, accuracy, and consistency of diagnosis. Cancer often occurs alongside related gene mutations, and the detection of gene mutations plays a crucial role in cancer treatment and prognosis. Medically, gene detection is often achieved through gene sequencing, which is expensive. This invention utilizes a multi-task learning method to simultaneously predict multiple genes in pathological images, providing a certain reference for clinical diagnosis and treatment. This invention first uses a two-layer multi-instance module to aggregate patches from the same WSI (Wide Sample Indicator) based on semantic and positional information between different patches. Then, a transformer module is combined with traditional convolution to consider both local and global features, ultimately outputting to a multi-task classifier to predict mutations in different genes.

[0037] Beneficial effects: Compared with the prior art, the features of this invention are: (1) It adopts a multi-task deep neural network and applies it to the task of predicting multiple gene mutations based on pathological images. Compared with the traditional single-task network, this invention can predict the results of two tasks at the same time, saving computing resources and improving accuracy; (2) It combines the transformer module with the traditional convolution, which takes into account both local features and global features. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the algorithm flow of the present invention;

[0039] Figure 2 This is a schematic diagram of the Trans-Mobile module in this invention;

[0040] Figure 3 This is a schematic diagram of the attention-based small packet aggregation module in this invention;

[0041] Figure 4 This is a schematic diagram of the ROC curve in an embodiment of the present invention. Detailed Implementation

[0042] To more clearly illustrate the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0043] like Figure 1-3 The specific preparation steps of the multi-gene mutation prediction method based on multi-task and multi-instance learning combined according to the present invention are as follows:

[0044] Step (1): Preprocess the existing data, that is, perform staining normalization preprocessing on the pre-acquired existing pathological image data;

[0045] Step (2): From each pathological image data obtained in step (1), 100 224*224 image blocks are randomly selected and sequentially fed into the pre-trained ResNet101 feature extractor to obtain feature vectors. The feature matrix is ​​constructed using the obtained feature vectors, and then a package is constructed for the pathological image using the two-layer multi-instance learning method.

[0046] Step (3): The two-layer multi-instance small packets obtained in step (2) are fed into a multi-task deep learning network based on transformer and MobileNet in units of pathological images. The network takes the packets constructed in step (2) as input and adopts the cross-entropy loss function and Adam optimizer.

[0047] Step (4): According to the loss function in step (3), use the gradient descent method of backpropagation; training is completed when the training set loss reaches the set value or the number of iterations exceeds the preset threshold; use the attention network to aggregate the output small packet results to obtain the KRAS and TP53 mutation-based prediction results of the pathological images.

[0048] Furthermore, in step (1), the process of performing staining normalization preprocessing on the pathological image data is as follows:

[0049] Considering the high resolution of pathological images, several non-overlapping 224*224 image blocks are first segmented from the upper left corner to the lower right corner of the pathological image. Considering that there are a large number of non-cellular areas (white areas) in the pathological images, the background areas are first filtered out based on the values ​​of each pixel in the R, G, and B channels. In addition, considering the color differences between different tissue pathological sections, the histogram matching method is used to achieve staining normalization.

[0050] Furthermore, the specific implementation process in step (2) is as follows:

[0051] The two-layer multi-instance learning:

[0052] A two-layer multi-instance learning model is built on the basis of traditional multi-instance learning; the pre-trained ResNet101 model is loaded and the last fully connected layer is removed and used as a feature extractor.

[0053] Given N pathological images, each containing 100 image patches (example), X n ={x n,k|k=1…100},n∈{1…N},The labels of the image patches are the same as those of the pathological images; the feature extractor obtains 2048-dimensional features for each image patch, and each pathological image data obtains a 100*2048 feature matrix;

[0054] Then, a two-layer multiple instance method is used to construct the packets, randomly and non-repeatingly distributing the 100 image patches into 10 smaller packets, namely b. n,m ={x n,i |i are selected from 0-100 without repetition, m∈{1…10}, and the labels of the small packages are the same as the labels of the pathological images; then, 10 are randomly selected from the remaining image patch features to form the second small package, and so on. Finally, each pathological image data is a large package of the first layer, containing 10 corresponding small packages in the second layer.

[0055] Furthermore, in step (3), the multi-task deep learning network based on transformer and MobileNet:

[0056] The deep learning network based on Transformer and MobileNet consists of four Trans-Mobile modules and one aggregation attention module connected in series. Each Trans-Mobile module consists of a Transformer module, a MobileNet module, a Trans2Mobile module, and a Mobile2Trans module.

[0057] First, the small packet features X representing local data and the global tokens (randomly initialized to three vectors with the same dimension as the small packet features) representing global features are fed into the Mobile2Trans module. This module acts as a lightweight cross-multi-head attention module for the transformation from the MobileNet module to the transformer module, mapping the local features X to the global token Z, as shown below:

[0058]

[0059] X and Z are represented in multi-head attention as follows: These represent the allocation to i. th Local features and global token of the header. W i q Indicate i th The query mapping matrix of the head, W m Combine multiple heads;

[0060] The three terms in Atten are equivalent to Q, K, and V in a standard attention module, but the mapping matrices for K and V are removed, and only the mapping matrix W corresponding to Q is retained.i q ;

[0061] The output of the Mobile2Trans module is fed as input into the transformer module, which consists of a standard multi-head attention module. The basic form of a standard attention module is as follows:

[0062]

[0063] Where Q is the consultation vector, K is the key-value vector, and V is the value vector; multi-head attention enables the model to jointly attend to information from different representation subspaces at different locations, as shown below:

[0064] MultiHead(Q,K,V)=Concat(head1,head2,...,head h W O

[0065] in,

[0066] The outputs Z′ and X of the transformer module are fed into the MobileNet module as input, which consists of a series of 1d convolutions and dynamic ReLU functions.

[0067] While retaining the standard 2D convolution in the MobileNet structure, the 1D convolutions are transformed into 1D convolutions.

[0068] Unlike the original ReLU function, the parameters of the activation function are obtained from the output of the transformer module through two MLP layers;

[0069] The output Z′ of the transformer module and the output of the MobileNet module are fed together as input to the Trans2Mobile module. Similar to the Mobile2Trans module, it is a lightweight cross-multi-head attention module that performs the transformation from the transformer module to the MobileNet module, realizing the transformation from global to local. It can be represented as:

[0070]

[0071] in, This indicates that the output Z′ of the transformer module is at the i-th... th Weight of the size and Mapping matrices for K and V respectively, W n Combine multiple heads.

[0072] The output Z′ of the transformer module and the output X′ of the Trans2Mobile module are fed as inputs into the next Trans-Mobile module.

[0073] Furthermore, the soft connection module of the multi-task model:

[0074] This invention uses a soft connection method to realize parameter interaction of multi-task modules. Specifically, the method is to calculate the maximum mean error of the parameters of the convolutional layers with a kernel size greater than 1 in the first two MobileNet modules of the two tasks, and minimize it as part of the total loss.

[0075] Furthermore, the attention aggregation module for small packages:

[0076] The outputs Z′ and X′ of the last Trans-Mobile module are concatenated in the row direction, and then global average pooling and two fully connected neural networks are used to predict the label of the small packet. The cross-entropy loss of the predicted result of the small packet and the label is calculated as the loss of the small packet.

[0077] The feature matrix of each small packet is used as the embedding of each small packet by global average pooling. Based on the prediction results of the small packets, the small packet with the highest prediction result is selected as the key packet. The inner product of the embedding of each small packet and the embedding of the key packet is taken, and all inner product results are normalized by the softmax() function and used as the weight of each small packet. Finally, the weighted sum of the embeddings of all small packets in a pathological image is used as the embedding of the entire pathological image. Because it is a multi-task network, each small packet will obtain the prediction results of two genes, KRAS and TP53, respectively. Therefore, a pathological image will also obtain two different embeddings, thereby realizing the attention aggregation of small packets.

[0078] Finally, the two obtained embeddings are fed into different fully connected layers to obtain the prediction results of KRAS and TP53 genes in the final pathological image. The prediction results and the labels of the pathological image are subjected to cross-entropy loss to obtain the loss of the pathological image. This loss is added to the previous loss from the small packet and the maximum mean error as the total loss for backpropagation.

[0079] Example

[0080] This invention validates its method on a colorectal cancer dataset from The Cancer Genome Atlas (TCGA). For each pathological image, 100 non-overlapping image patches are randomly sampled at a 20x multiplier, with 10 patches forming a small package. To demonstrate the effectiveness of the method, this invention employs a 5-fold cross-validation method for training, using 1 fold of data as the test set each time, and all data from the remaining 4 folds are used for model training.

[0081] For the prediction of KRAS and TP53 genes in colorectal cancer, this invention uses accuracy, F1 score, AUC (area under the ROC curve), and recall as metrics; the results are shown in the table below:

[0082] Table 1: Predicted results of KRAS gene mutations

[0083] Accuracy F1 score AUC Recall KRAS Single Task 0.742(±0.045) 0.723(±0.053) 0.728(±0.036) 0.703(±0.014) Multitasking methods 0.852(±0.058) 0.841(±0.013) 0.906(±0.022) 0.768(±0.036)

[0084] Table 2: Predicted results of TP53 gene mutations

[0085] Accuracy F1 score AUC Recall TP53 Single Task 0.652(±0.066) 0.601(±0.045) 0.652(±0.063) 0.561(±0.024) Multitasking methods 0.814(±0.043) 0.819(±0.023) 0.827(±0.052) 0.823(±0.031)

[0086] The above table demonstrates that the multi-task method of this invention achieves an improvement of 0.1 to 0.25 in accuracy and other evaluation metrics compared to single-task direct prediction of KRAS and TP53, especially in the prediction of the TP53 gene.

[0087] This can be attributed to multi-task learning, which leverages classification information from other tasks to enhance model robustness and reduce the risk of overfitting; notably, this invention boasts the highest recall rate. High recall is particularly important in the medical field because false negatives can lead to serious consequences, including patient death; specifically... Figure 4 As shown.

[0088] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A multi-gene mutation prediction method based on multi-task and multi-instance learning, characterized in that, The specific operating steps are as follows: Step (1): Perform staining normalization preprocessing on all the pre-acquired pathological image data; Step (2): Randomly select 100 image blocks from each obtained pathological image data and input them into the pre-trained ResNet101 feature extractor to obtain feature vectors. Construct a feature matrix using the obtained feature vectors and then use a two-layer multi-instance learning method to construct a package for the pathological image. The implementation process is as follows: First, load the pre-trained ResNet101 model and remove the last fully connected layer, using it as a feature extractor; 100 image blocks are randomly selected from each pathological image data obtained in step (1), and 2048-dimensional features of each image block are obtained through a feature extractor. Then, each pathological image data has a 100*2048 feature matrix. Then, a two-layer multi-instance method is used to construct the package. That is, for each pathological image data, 10 features of image blocks are randomly selected to form the first small package, and then 10 features are randomly selected from the remaining image block features to form the second small package, and so on. Finally, each pathological image data is a large package in the first layer, containing 10 corresponding small packages in the second layer. Step (3): Construct a multi-task deep learning network based on transformer and MobileNet. This network takes the packet constructed in step (2) as input and adopts the cross-entropy loss function and Adam optimizer. Step (4): According to the loss function in step (3), use the gradient descent method of backpropagation; training is completed when the training set loss reaches the set value or the number of iterations exceeds the preset threshold; apply the model to the test set and output the gene prediction results.

2. The multi-gene mutation prediction method based on multi-task and multi-instance learning combined according to claim 1, characterized in that: The implementation process of step (1) is as follows: First, the background area is filtered out according to the values ​​of each pixel in the three channels R, G, and B. The filtering rule is that if the pixel value in any channel is greater than 200 or 90% of the pixels in the 200*200 neighborhood of the pixel are background areas, then the pixel is filtered out. Then, the selected image patches are color-normalized.

3. The multi-gene mutation prediction method based on multi-task and multi-instance learning combined according to claim 1, characterized in that: In step (3), the multi-task deep learning network based on transformer and MobileNet consists of a transformer module, a MobileNet module, a Trans2Mobile module, a Mobile2Trans module, and a multi-task interaction module; The transformer module consists of a standard multi-head attention module, the basic form of which is shown in the following equation: In the formula, It is a consultation vector. It is a key-value vector. It is a value vector; Multi-head attention enables the model to jointly attend to information from different representational subspaces at different locations, as shown in the following equation: In the formula, ; The Mobile2Trans module is a lightweight cross-multi-head attention module that transforms the MobileNet module into a transformer module, mapping local features X to global tokens, as shown in the following equation: In the formula, represents local features, and Z is the global token used to learn global features, which is randomly initialized at the start of training; In multi-head attention, X and Z are represented as , , where represent the local features and global token assigned to the head, respectively; represents the query mapping matrix of the head, which combines multiple heads; The three terms in the expression are equal to Q, K, and V in the standard attention module, respectively. The mapping matrices of K and V are removed, while the mapping matrix corresponding to Q is retained. Similar to the Mobile2Trans module, the Trans2Mobile module is a lightweight cross-multi-head attention module that transforms from a transformer module to a MobileNet module, realizing the transformation from global to local, as shown in the following formula: In the formula, Z is a local feature, and Z is a global token used to learn global features. It is randomly initialized at the beginning of training. X and Z are represented in multi-head attention as follows: , , They represent the allocation to Local features and global token of the header; express The query mapping matrix of the head, Combine multiple heads; The three terms in the expression are equal to Q, K, and V in the standard attention module, respectively. The mapping matrices for K and V are removed, while the mapping matrix corresponding to Q is retained. ; Similar to the Mobile2Trans module, the Trans2Mobile module is a lightweight cross-multi-head attention module that transforms from a transformer module to a MobileNet module, realizing the transformation from global to local, as shown in the following formula: In the formula, and The mapping matrices for K and V are respectively. Combine multiple heads.

4. The multi-gene mutation prediction method based on multi-task and multi-instance learning combined according to claim 3, characterized in that: The MobileNet module consists of a series of 1d convolutions and dynamic ReLU functions; While retaining the standard 2D convolution in the MobileNet structure, the 2D convolutions are transformed into 1D convolutions; unlike the original ReLU function with fixed parameters, the parameters of the activation function are obtained from the output of the transformer module through two MLP layers. The MMD distance between convolutional kernel parameters with a size greater than 1 in the MobileNet module for all tasks is calculated and added to the classification loss function obtained by cross-entropy and backpropagated to achieve soft parameter sharing. The local feature X and the global token Z are concatenated to obtain the small packet embedding, and the MLP is used to obtain the small packet score. Based on the small packet score, the attention network is used to aggregate the small packet embedding to obtain the embedding of the entire pathological image. Finally, the prediction results of the two gene mutations, KRAS and TP53, are obtained.

Images