Multi-example multi-task learning-based pan-cancer biomarker prediction method and system

Abstract

The invention belongs to the technical field of medical pathological images, and particularly relates to a multi-instance multi-task learning pan cancer biomarker prediction method and system, and the method comprises the steps: obtaining a full-view digital pathological image and pan cancer biomarker expression data of a solid tumor patient; performing preprocessing and feature extraction based on the full-view digital pathological image to obtain a pathological feature data set; based on the pathological feature data set, inputting the pathological feature data set into a twin multi-example full convolutional network model for training to obtain pathological low-dimensional features, and calculating an attention score corresponding to each pathological feature in combination with a linear attention mechanism; and in combination with each pathologic low-dimensional feature and the corresponding attention score, calculating to obtain a pathologic depth feature, and inputting the pathologic depth feature into a multi-task classification prediction model for classification prediction to obtain a prediction result of each biomarker in the pan cancer.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical pathological images, and in particular relates to a method and system for predicting pan-cancer biomarkers based on multi-example and multi-task learning. Background Art

[0002] Molecular profiling plays a vital role in cancer research and treatment. It helps scientists better understand the mechanisms of tumorigenesis and provides a basis for precision therapy. Precision therapy relies on accurate diagnosis of genetic changes, which are usually identified by molecular biological tests such as Sanger sequencing and fluorescence in situ hybridization. However, these tests have some problems, such as high turnaround time, expertise and high cost, which to some extent limit their widespread application in oncology workflows.

[0003] In recent years, the progress of computational pathology (CPath) has provided a new perspective for cancer research. Studies have found that histological sections stained with hematoxylin and eosin (H&E) not only contain known biomarker information, but may also hide potential biomarkers. Through digital pathology sections (WSI) technology, scientists can infer the lineage of molecules directly from the sections. The advantage of this method is that it does not require complex molecular biology testing, but uses technologies such as deep learning to extract useful information from tissue sections.

[0004] However, although computational pathology (CPath) has shown great potential in research, it still has certain limitations. The manual feature engineering currently used will increase the workload and is easily affected by subjective experience bias, which will reduce the accuracy and reliability of the results; the lightweight shufflenet model used faces challenges in learning complex patterns or advanced functions, which limits its performance in tasks that require high global correlation; the traditional global attention method in computational pathology (CPath) applications requires calculating the similarity between a pixel and all other pixels, which will result in a lot of time and computing resources consumption, thereby increasing research costs and limiting the scale and speed of research; in addition, the expression of different molecular markers in tissues is often not isolated, and many will be highly or lowly expressed at the same time, while the current multi-task learning (MTL) is relatively insufficient in the study of computational pathology (CPath), which limits the ability to analyze tissue sections from multiple angles at the same time. Summary of the invention

[0005] The present invention provides a method for predicting pan-cancer biomarkers based on multi-instance multi-task learning, which preprocesses and extracts the full-field digital pathology images of solid tumor patients to obtain pathological features, and performs classification prediction after weighted calculation of the pathological features based on a twin multi-instance fully convolutional network model and a linear attention mechanism, so that multiple pan-cancer biomarkers can be predicted simultaneously, efficiently and effectively based on the full-field digital pathology images of solid tumor patients, thereby assisting doctors in clinically screening for markers of cancer patients early, replicating clinical treatment plan selection, and optimizing the diagnosis and treatment process.

[0006] A multi-instance multi-task learning method for predicting pan-cancer biomarkers, including: Obtain full-field digital pathology images and pan-cancer biomarker expression data for patients with solid tumors; Based on the full-field digital pathology images, preprocessing and feature extraction are performed to obtain a pathology feature dataset; Based on the pathological feature dataset, it is input into the twin multi-instance fully convolutional network model for training to obtain the low-dimensional features of the pathology, and combined with the linear attention mechanism, the attention score corresponding to each pathological feature is calculated; By combining each pathological low-dimensional feature and the corresponding attention score, the pathological deep feature is calculated and input into the multi-task classification prediction model for classification prediction to obtain the prediction results of each biomarker in pan-cancer.

[0007] Pathological features are obtained by preprocessing and extracting the full-field digital pathology images of patients with solid tumors. Classification and prediction are performed after weighted calculation of the pathological features based on the twin multi-instance fully convolutional network model and combined with the linear attention mechanism. This makes it possible to simultaneously, efficiently and effectively predict multiple pan-cancer biomarkers based on the full-field digital pathology images of patients with solid tumors, thereby assisting doctors in clinically screening for markers in cancer patients at an early stage, replicating clinical treatment options, and optimizing the diagnosis and treatment process.

[0008] Furthermore, based on the full-field digital pathology image, preprocessing and feature extraction are performed to obtain a pathology feature dataset, including: Based on the full-field digital pathology images, the Otsu threshold method is used for segmentation to obtain the tissue area of ​​each full-field digital pathology image; Downsampling is performed based on the tissue area of ​​each full-field digital pathology image; Based on the tissue area of ​​the full-field digital pathology image after downsampling, combined with the preset minimum threshold of the median of the RGB channels, the patch lacking tissue or with the smallest tissue is excluded to obtain the tissue area of ​​the preprocessed full-field digital pathology image; Based on the tissue area of ​​the preprocessed full-field digital pathology image, a fine-tuned pre-trained ResNet50 model was used for feature extraction to obtain the pathological features of each full-field digital pathology image, and to construct a pathological feature dataset of the full-field digital pathology image.

[0009] Furthermore, the tissue area of ​​the full-view digital pathology image is a square block, and the tissue areas of multiple full-view digital pathology images do not overlap; the edge length of each square block is , the size is Pixel.

[0010] Furthermore, based on the tissue area of ​​the preprocessed full-view digital pathology image, a fine-tuned pre-trained ResNet50 model is used to extract features to obtain the pathological features of each full-view digital pathology image, and a pathological feature dataset of the full-view digital pathology image is constructed, including: Select the ResNET50 model as the base model; The ResNET50 model is pre-trained using clustering-guided contrastive learning to obtain a pre-trained ResNET50 model; Based on the pre-trained ResNET50 model, a fully connected layer is added at the end to fine-tune the model to obtain a fine-tuned pre-trained ResNet50 model; Based on the multi-instance MIL framework, a fine-tuned pre-trained ResNet50 model was used to extract features of the tissue area of ​​each full-field digital pathology image, obtain the pathological features of each full-field digital pathology image, and construct a pathological feature dataset of full-field digital pathology images.

[0011] By introducing a fine-tuned pre-trained ResNET model for feature extraction based on the multi-instance MIL framework, the time-consuming and tedious manual labeling is eliminated, and an informative embedding representation of pathological features can be obtained.

[0012] Furthermore, based on the pathological feature dataset, it is input into the twin multi-instance fully convolutional network model for training to obtain the pathological low-dimensional features, and combined with the linear attention mechanism, the attention score corresponding to each pathological feature is calculated, including: Based on the fully convolutional neural network, a twin-structured two-stream subnet is constructed; the two-stream subnet includes two weight-sharing subnets; each of the subnets consists of one convolutional layer, two rectifier unit activation layers, one adaptive average pooling layer, and two stacked fully connected layers; Based on the pathological feature dataset, it is input into the two-stream sub-network for training to obtain the pathological low-dimensional features; The expression of the pathological low-dimensional feature is: ; In the formula, Indicates The low-dimensional feature embedding representation corresponding to the pathological embedded features is , indicates the total number of pathological features; represents the activation function; Represents the weight matrix of the first fully connected layer; Represents the bias vector of the first fully connected layer; Represents the weight matrix of the second fully connected layer; Represents the bias vector of the second fully connected layer; Indicates The unmapped embedding vector, pathological features; Based on the low-dimensional features of pathology, they are input into the focused linear attention module for training to obtain the attention score corresponding to each pathological feature; The expression of the attention score corresponding to each pathological feature is: ; In the formula, represents the first pathological features and The attention score of each task; Indicates pathological features and The query vector of each task; Indicates The key vector of tokens; Indicates The value vector of tokens; , Indicates the number of tokens in the input sequence, that is, the total number of elements that the model needs to process; represents the depth convolution function; Represents a mapping function or feature transformation function.

[0013] By designing a two-stream subnet constructed by a multi-instance fully convolutional neural network and combining it with a focused linear attention network model, multiple layers of fully convolutional layers are combined with nonlinear activation functions. This can effectively reduce the number of model parameters and enable the model to better focus on important features. By introducing nonlinear transformations, the model can capture more complex feature relationships, thereby improving the model's expressiveness and performance.

[0014] Furthermore, by combining each pathological low-dimensional feature and the corresponding attention score, the pathological deep feature is calculated and input into the multi-task classification prediction model for classification prediction to obtain the prediction results of each biomarker in pan-cancer, including: Based on the given slides, the deep features of the given task are calculated by combining the pathological low-dimensional features and their corresponding attention scores; The expression of the deep feature of the given task is: ; In the formula, Indicates deep feature representation of all blocks in a given slide for each task; Based on the deep features of a given task, input it into a multi-task classification prediction model for classification prediction, calculate the slice-level probability prediction score of each task, and obtain the prediction result of each biomarker in pan-cancer; the multi-task classification prediction model includes multiple classifiers; The expression of the slice-level probability prediction score of each task is: ; In the formula, Indicates Slice-level probability prediction score for each task; Normalization function; Indicates The weight matrix of each task in the classification layer; Indicates The bias vector of each task in the classification layer.

[0015] By designing a multi-task classification prediction model containing multiple classifiers, it is possible to output predictions for different markers simultaneously.

[0016] Furthermore, it also includes calculating a loss function based on the multi-task classification prediction model; the calculation of the loss function includes: Based on each slide, the loss function of a single task is calculated using the standard cross entropy loss function; Based on the loss function of a single task, the total loss function of all tasks is calculated by weighted sum; The expression of the total loss function is: ; In the formula, represents the total loss of all tasks; Indicates The weight of the task, , Indicates the total number of tasks, ; Indicates The loss function for each task.

[0017] A system for predicting a pan-cancer biomarker using multi-instance multi-task learning, comprising: A data acquisition module, which is used to obtain full-field digital pathology images and pan-cancer biomarker expression data of patients with solid tumors; A data processing module, which is used to perform preprocessing and feature extraction based on the full-field digital pathology image to obtain a pathology feature data set; The computing module is used to input the pathological feature dataset into the twin multi-instance fully convolutional network model for training to obtain the pathological low-dimensional features, and combine the linear attention mechanism to calculate the attention score corresponding to each pathological feature; The prediction module is used to combine each pathological low-dimensional feature and the corresponding attention score to calculate the pathological deep feature, and input the pathological deep feature into the multi-task classification prediction model for classification prediction to obtain the prediction result of each biomarker in pan-cancer.

[0018] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0019] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0020] The beneficial effects of the present invention are: The present invention obtains pathological features by preprocessing and extracting the full-field digital pathology images of patients with solid tumors, and performs classification prediction after weighted calculation of the pathological features based on the twin multi-instance fully convolutional network model and combined with the linear attention mechanism. This enables simultaneous, efficient and effective prediction of multiple pan-cancer biomarkers based on the full-field digital pathology images of patients with solid tumors, thereby assisting doctors in clinically conducting early screening of cancer patients' markers, replicating clinical treatment plan selection, and optimizing the diagnosis and treatment process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the present invention; Figure 2 This is the ROC curve of the test set; Figure 3 It is a structural schematic diagram of the system of the present invention; Figure 4 It is a schematic diagram of the structure of the computer device in the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0024] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples, and a person of ordinary skill in the art can understand the specific meanings of the above terms in this application under specific circumstances.

[0025] Example 1 Figure 1 The method shown is a multi-instance multi-task learning method for predicting pan-cancer biomarkers. The pathological features are obtained by preprocessing and extracting the full-view digital pathology images of solid tumor patients. The pathological features are weighted and calculated based on the twin multi-instance fully convolutional network model and combined with the linear attention mechanism for classification prediction. This enables simultaneous, efficient and effective prediction of multiple pan-cancer biomarkers based on the full-view digital pathology images of solid tumor patients, thereby assisting doctors in early screening of cancer patients' markers, replicating clinical treatment options, and optimizing the diagnosis and treatment process. Specifically, it includes: S1: Acquire full-field digital pathology images (WSI) and pan-cancer biomarker expression data of patients with solid tumors; In this embodiment, the full-field digital pathology images WSI and pan-cancer biomarker expression data of solid tumor patients required for the experiment were downloaded from The Cancer Genome Atlas (TCGA) and The Clinical ProteomicTumor Analysis Consortium (CPTAC) databases.

[0026] S2: Based on the full-field digital pathology image, preprocessing and feature extraction are performed to obtain a pathology feature dataset; S11: Based on the full-field digital pathology image WSI, the Otsu threshold method is used for segmentation to obtain the tissue region patch of each full-field digital pathology image WSI; In this embodiment, the tissue area of ​​the full-view digital pathology image is a square block, and the tissue areas of multiple full-view digital pathology images do not overlap; the edge length of each square block is , the size is Pixel.

[0027] S12: performing downsampling processing based on the tissue region patch of each full-view digital pathology image WSI; In this embodiment, 64× resolution is used for downsampling and reading into the memory; in actual applications, other single resolutions or multiple resolutions can be selected for sampling to obtain hierarchical pathological information, thereby obtaining rich pathological features.

[0028] S13: based on the tissue region patch of the full-view digital pathology image WSI after downsampling, combined with a preset minimum threshold of the median of the RGB channels, the patch lacking tissue or with the smallest tissue is excluded to obtain the tissue region patch of the preprocessed full-view digital pathology image WSI; In this embodiment, patches with lack of tissue or minimal tissue are excluded by setting a minimum threshold of 50 for the median of each 8-bit RGB channel.

[0029] S14: Based on the preprocessed tissue region patches of the full-view digital pathology image WSI, a fine-tuned pre-trained ResNet50 model is used to perform feature extraction to obtain the pathological features of each full-view digital pathology image WSI, and to construct a pathological feature dataset of the full-view digital pathology image WSI; S141: Select the ResNET50 model as the base model; S142: Pre-training the ResNET50 model using clustering-guided contrastive learning (CCL) to obtain a pre-trained ResNET50 model; S143: Based on the pre-trained ResNET50 model, a fully connected layer is added at the end to fine-tune the model to obtain a fine-tuned pre-trained ResNet50 model; S144: Based on the multi-instance MIL framework, a fine-tuned pre-trained ResNet50 model is used to extract features from the tissue region patch of each full-view digital pathology image WSI, obtain the pathological features of each full-view digital pathology image, and construct a pathological feature dataset of the full-view digital pathology image; In this embodiment, each full-view digital pathology image WSI is regarded as a package, in which each tissue region patch is regarded as an example, and after feature extraction, the image containing The WSI of the example, and then for the The patch-level embedding representation of the patch, that is, the pathological feature is Among them, the dimension of the pathological feature output is 2048, that is, .

[0030] S3: Based on the pathological feature dataset, it is input into the twin multi-instance fully convolutional network model for training to obtain the pathological low-dimensional features, and combined with the linear attention mechanism, the attention score corresponding to each pathological feature is calculated; S31: Based on the fully convolutional neural network FCN, a twin-architecture two-stream subnet is constructed; the two-stream subnet includes two weight-sharing subnets; each subnet consists of 1 convolutional layer, 2 rectifier unit activation layers, 1 adaptive average pooling layer, and 2 stacked fully connected layers; Each subnet is capable of learning a fully convolutional neural network (FCN) for information representation. The kernel size of the convolutional layer is 1×1, which is used to change the number of channels in the feature map; two rectifier unit activation layers are set to increase the nonlinear ability of the model; one adaptive average pooling layer is set to dynamically adjust the size of the pooling window; and two stacked fully connected layers are set to integrate the information in the feature map for tuning.

[0031] S32: Based on the pathological feature dataset, input it into the two-stream subnet for training to obtain the pathological low-dimensional features ; Among them, the expression of pathological low-dimensional features is: ; In the formula, Indicates The low-dimensional feature embedding representation corresponding to the pathological embedded features is , indicates the total number of pathological features; represents the activation function; represents the weight matrix of the first fully connected layer, and ; represents the bias vector of the first fully connected layer, and ; represents the weight matrix of the second fully connected layer, and ; represents the bias vector of the second fully connected layer, and ; Indicates The unmapped embedding vector, pathological features; S33: Based on pathological low-dimensional features , input it into the focused linear attention module for training, and obtain the attention score corresponding to each pathological feature ; In this embodiment, the pathological low-dimensional feature set , as the input of the focused linear attention module. All tasks are learned jointly and the model parameters in the second fully connected layer are shared. The focused linear attention model is trained for each patch and task Assign attention scores.

[0032] Among them, the expression of the attention score corresponding to each pathological feature is: ; In the formula, represents the first pathological features and The attention score of each task; Indicates pathological features and The query vector of each task; Indicates The key vector of tokens; Indicates The value vector of tokens; , Indicates the number of tokens in the input sequence, that is, the total number of elements that the model needs to process; represents the depth convolution function; Represents a mapping function or feature transformation function.

[0033] In this embodiment, according to the associative property of matrix multiplication, the calculation order is changed from Adjust to , thereby reducing the computational complexity of tokens from Restrict to , reducing the number of model parameters, so that the model can better focus on important features; and by introducing nonlinear transformation , which can help the model capture more complex feature relationships, thereby improving the model's expressiveness and performance.

[0034] S4: Combining each pathological low-dimensional feature And the corresponding attention score , calculate the pathological deep features, and input the pathological deep features into the multi-task classification prediction model for classification prediction to obtain the prediction results of each biomarker in pan-cancer; S41: Based on a given slide, combined with pathological low-dimensional features and its corresponding attention score , calculate the given task The depth features of Among them, given task The expression of the deep feature is: ; In the formula, Indicates deep feature representation of all blocks in a given slide for each task; S42: Based on a given task The deep features , input it into the multi-task classification prediction model for classification prediction, and calculate the slice-level probability prediction score of each task , get the prediction results of each biomarker in Pan-Cancer; In this embodiment, the multi-task classification prediction model includes multiple classifiers. By designing a multi-task classification prediction model including multiple classifiers, it is possible to facilitate simultaneous output predictions of different markers.

[0035] Among them, the slice-level probability prediction score of each task is The expression is: ; In the formula, Indicates Slice-level probability prediction score for each task; Normalization function; Indicates The weight matrix of each task in the classification layer; Indicates The bias vector of each task in the classification layer.

[0036] In this embodiment, the total loss function is calculated based on the multi-task classification prediction model. The calculation process of the total loss function includes: S51: Based on each slide, the loss function of a single task is calculated using the standard cross entropy loss function ; In this embodiment, in each task The classification layer cls is composed of the weight matrix of the classification layer , and the bias vector of the classification layer Defined, Representation Task The number of classes.

[0037] S52: Based on the loss function of a single task, the total loss function of all tasks is calculated by weighted sum ; Among them, the expression of the total loss function is: ; In the formula, represents the total loss of all tasks; Indicates The weight of the task, , Indicates the total number of tasks, ; Indicates The loss function for each task.

[0038] In this embodiment, a five-fold cross validation method and an AUC method are also used to evaluate the performance of the multi-task classification prediction model.

[0039] Example 2 In this embodiment, a method for predicting pan-cancer biomarkers using multi-example multi-task learning is provided, comprising: T1: Acquire full-field digital pathology images (WSI) and pan-cancer biomarker expression data of patients with solid tumors; T11: Based on the TCGA database, a solid tumor dataset was obtained and divided into a training set and a validation set; In this embodiment, the TCGA database contains 12 solid tumor datasets, namely, a total of 4527 WSIs of 4257 patients and corresponding pan-cancer biomarker expression data; the pan-cancer type image data are shown in Table 1.

[0040] Table 1 Number of images and patients included in the multi-task study of pan-cancer types

[0041] T12: Based on the CPTAC database, obtain the colorectal cancer dataset and use it as the test set; In this embodiment, the CPTAC database is an external test set, in which the colorectal cancer data set includes 221 WSIs of 105 patients and corresponding pan-cancer biomarker expression data.

[0042] T2: Based on the full-field digital pathology image WSI, preprocessing and feature extraction are performed to obtain a pathology feature dataset; T11: Based on the full-field digital pathology image WSI, the Otsu threshold method is used for segmentation to obtain the tissue area patch of each full-field digital pathology image WSI; T12: Downsampling processing is performed based on the tissue area patch of each full-field digital pathology image WSI; T13: Based on the tissue region patch of the full-field digital pathology image WSI after downsampling, combined with the preset minimum threshold of 50 of the 8-bit RGB channel median, the patch with lack of tissue or minimal tissue is excluded to obtain the tissue region patch of the preprocessed full-field digital pathology image WSI; T14: Based on the preprocessed tissue region patches of the full-view digital pathology image WSI, a fine-tuned pre-trained ResNet50 model is used to extract features, obtain the pathological features of each full-view digital pathology image WSI, and construct a pathological feature dataset of the full-view digital pathology image WSI; T141: Select the ResNET50 model as the base model; T142: Use clustering-guided contrastive learning (CCL) to pre-train the ResNET50 model to obtain a pre-trained ResNET50 model; T143: Based on the pre-trained ResNET50 model, a fully connected layer is added at the end to fine-tune the model to obtain a fine-tuned pre-trained ResNet50 model; T144: Based on the multi-instance MIL framework, a fine-tuned pre-trained ResNet50 model is used to extract features from the tissue region patches of each full-view digital pathology image WSI, obtain the pathological features of each full-view digital pathology image, and construct a pathological feature dataset of full-view digital pathology images; T3: Based on the pathological feature dataset, it is input into the twin multi-instance fully convolutional network model for training to obtain the pathological low-dimensional features, and combined with the linear attention mechanism, the attention score corresponding to each pathological feature is calculated; T31: Based on the fully convolutional neural network FCN, a two-stream subnet with a twin architecture is constructed; the two-stream subnet includes two weight-sharing subnets; each subnet consists of 1 convolutional layer, 2 rectifier unit activation layers, 1 adaptive average pooling layer, and 2 stacked fully connected layers; T32: Based on the pathological feature dataset, input it into the two-stream subnet for training to obtain the pathological low-dimensional features ; T33: Based on pathological low-dimensional features , input it into the focused linear attention module for training, and obtain the attention score corresponding to each pathological feature ; T4: Combine each pathological low-dimensional feature And the corresponding attention score , calculate the pathological deep features, and input the pathological deep features into the multi-task classification prediction model for classification prediction to obtain the prediction results of each biomarker in pan-cancer; T41: Based on a given slide, combined with pathological low-dimensional features and its corresponding attention score , calculate the given task The depth features of T42: Based on a given task The deep features , input it into the multi-task classification prediction model for classification prediction, and calculate the slice-level probability prediction score of each task and the total loss function to obtain the prediction results of each biomarker in Pan-Cancer; T421: Based on each slide, the loss function of a single task is calculated using the standard cross entropy loss function , and based on the loss function of a single task, the total loss function of all tasks is calculated by weighted sum ; T422: Based on a given task The deep features , input it into the multi-task classification prediction model for classification prediction, and calculate the slice-level probability prediction score of each task , and obtain the prediction results of each biomarker in Pan-Cancer.

[0043] The training process of the multi-task classification prediction model includes: W11: Use Adam optimizer, set learning rate to 1e-2, and weight decay to 4e-5; W12: Add a Dropout layer after each hidden layer and set P=0.25 to avoid over-convergence; W13: Set the number of iterations; the range of iterations is 50~200 epochs; W14: Based on the preset number of iterations, training stop judgment is performed; W141: When the total loss function of the validation set does not decrease after 20 consecutive epochs, the training is stopped and the best multi-task classification prediction model is generated; W142: When the total loss function of the validation set decreases within 20 epochs, continue training.

[0044] Before using the multi-task classification prediction model for application, the model is trained with a test set, and the AUC method is used to evaluate the model performance, including: W21: Based on the test set, samples are randomly selected with replacement, and a total of 1000 bootstrap samples are generated; W22: Calculate the AUC score based on each bootstrap sample; In this embodiment, the closer the AUC score is to 1, the better the model performance is; the closer the AUC score is to 0.5, the worse the model performance is.

[0045] W23: Based on the calculated AUC score, the AUC score distribution is formed, and the percentile method is used to extract the 2.5% as the lower limit of the confidence interval and the 97.5% quantile as the upper limit of the confidence interval to form a confidence interval for evaluating the performance stability of the model on the validation set. When the confidence interval is narrow, the evaluation result is considered to be more accurate, otherwise it is considered that the uncertainty of the evaluation result is greater.

[0046] Figure 2 Shown is the ROC curve of the test set, and the unstable detection result of colorectal cancer was obtained: AUC: 0.769 (95CI, 0.686~0.841).

[0047] Table 2 shows the average AUC performance and standard deviation for all cancer types.

[0048] Table 2 Average AUC performance and standard deviation of all implemented cancer types for multi-task learning

[0049] Based on Table 2, among the 12 solid tumor datasets, KICH has the best overall effect, with a confidence interval of [0.850-0.050, 0.850+0.050]; PAAD has the worst overall effect, with a confidence interval of [0.557-0.056, 0.557+0.056]. The effectiveness and robustness of this method can be confirmed by training and validating on internal solid tumor datasets and testing on external colorectal cancer datasets.

[0050] Example 3 Based on the same technical concept, Figure 3 As shown, this embodiment also provides a multi-instance multi-task learning pan-cancer biomarker prediction system, including a data acquisition module, a data processing module, a calculation module and a prediction module.

[0051] Specifically, a data acquisition module, which is used to acquire full-field digital pathology images and pan-cancer biomarker expression data of patients with solid tumors; Specifically, a data processing module is used to perform preprocessing and feature extraction based on the full-field digital pathology image to obtain a pathology feature data set; Specifically, the calculation module is used to input the pathological feature dataset into the twin multi-instance fully convolutional network model for training to obtain the pathological low-dimensional features, and combine the linear attention mechanism to calculate the attention score corresponding to each pathological feature; Specifically, the prediction module is used to combine each pathological low-dimensional feature and the corresponding attention score to calculate the pathological deep feature, and input the pathological deep feature into the multi-task classification prediction model for classification prediction to obtain the prediction results of each biomarker in pan-cancer.

[0052] Example 3 Based on the same technical concept, the embodiment of the present application also provides a computer device, including a memory 1 and a processor 2, such as Figure 4 As shown, the memory 1 stores a computer program, and the processor 2 implements any of the above methods when executing the computer program.

[0053] Among them, the memory 1 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 1 can be an internal storage unit of the prediction system of pan-cancer biomarkers of multi-instance multi-task learning, such as a hard disk. In other embodiments, the memory 1 can also be an external storage device of the prediction system of pan-cancer biomarkers of multi-instance multi-task learning, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 1 can also include both the internal storage unit and the external storage device of the prediction system of pan-cancer biomarkers of multi-instance multi-task learning. The memory 1 can not only be used to store application software and various types of data installed in the prediction system of pan-cancer biomarkers of multi-instance multi-task learning, such as the code of the prediction system program of pan-cancer biomarkers of multi-instance multi-task learning, but also can be used to temporarily store data that has been output or is to be output.

[0054] In some embodiments, the processor 2 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 1, such as a prediction system program for pan-cancer biomarkers that performs multi-instance multi-task learning.

[0055] The disclosed embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the steps of the method described in the above method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0056] The computer program product of the application page content refresh method provided in the disclosed embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0057] The disclosed embodiment of the present invention further provides a computer program, which implements any one of the methods of the aforementioned embodiments when executed by a processor. The computer program product can be implemented in hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0058] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0059] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0060] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0061] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0062] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0063] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0064] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0065] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0066] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for predicting pan-cancer biomarkers based on multi-instance multi-task learning, characterized in that: include: Obtain full-field digital pathology images and pan-cancer biomarker expression data for patients with solid tumors; Based on the full-field digital pathology images, preprocessing and feature extraction are performed to obtain a pathology feature dataset; Based on the pathological feature dataset, it is input into the twin multi-instance fully convolutional network model for training to obtain the low-dimensional features of the pathology, and combined with the linear attention mechanism, the attention score corresponding to each pathological feature is calculated; By combining each pathological low-dimensional feature and the corresponding attention score, the pathological deep feature is calculated and input into the multi-task classification prediction model for classification prediction to obtain the prediction results of each biomarker in pan-cancer.

2. The method for predicting pan-cancer biomarkers by multi-instance multi-task learning according to claim 1, characterized in that: Based on the full-field digital pathology images, preprocessing and feature extraction are performed to obtain the pathology feature data set, including: Based on the full-field digital pathology images, the Otsu threshold method is used for segmentation to obtain the tissue area of ​​each full-field digital pathology image; Downsampling is performed based on the tissue area of ​​each full-field digital pathology image; Based on the tissue area of ​​the full-field digital pathology image after downsampling, combined with the preset minimum threshold of the median of the RGB channels, the patch lacking tissue or with the smallest tissue is excluded to obtain the tissue area of ​​the preprocessed full-field digital pathology image; Based on the tissue area of ​​the preprocessed full-field digital pathology image, a fine-tuned pre-trained ResNet50 model was used for feature extraction to obtain the pathological features of each full-field digital pathology image, and to construct a pathological feature dataset of the full-field digital pathology image.

3. The method for predicting pan-cancer biomarkers by multi-instance multi-task learning according to claim 2, characterized in that: The tissue area of ​​the full-view digital pathology image is a square block, and the tissue areas of multiple full-view digital pathology images do not overlap; the edge length of each square block is , the size is Pixel.

4. The method for predicting pan-cancer biomarkers by multi-instance multi-task learning according to claim 2, characterized in that: Based on the tissue area of ​​the preprocessed full-view digital pathology image, a fine-tuned pre-trained ResNet50 model is used to extract features to obtain the pathological features of each full-view digital pathology image, and a pathological feature dataset of the full-view digital pathology image is constructed, including: Select the ResNET50 model as the base model; The ResNET50 model is pre-trained using clustering-guided contrastive learning to obtain a pre-trained ResNET50 model; Based on the pre-trained ResNET50 model, a fully connected layer is added at the end to fine-tune the model to obtain a fine-tuned pre-trained ResNet50 model; Based on the multi-instance MIL framework, a fine-tuned pre-trained ResNet50 model was used to extract features of the tissue area of ​​each full-field digital pathology image, obtain the pathological features of each full-field digital pathology image, and construct a pathological feature dataset of full-field digital pathology images.

5. The method for predicting pan-cancer biomarkers by multi-instance multi-task learning according to claim 1, characterized in that: Based on the pathological feature dataset, it is input into the twin multi-instance fully convolutional network model for training to obtain the pathological low-dimensional features. Combined with the linear attention mechanism, the attention score corresponding to each pathological feature is calculated, including: Based on the fully convolutional neural network, a twin-structured two-stream subnet is constructed; the two-stream subnet includes two weight-sharing subnets; each of the subnets consists of one convolutional layer, two rectifier unit activation layers, one adaptive average pooling layer, and two stacked fully connected layers; Based on the pathological feature dataset, it is input into the two-stream sub-network for training to obtain the pathological low-dimensional features; The expression of the pathological low-dimensional feature is: ； In the formula, Indicates The low-dimensional feature embedding representation corresponding to the pathological embedded features is , indicates the total number of pathological features; represents the activation function; Represents the weight matrix of the first fully connected layer; Represents the bias vector of the first fully connected layer; Represents the weight matrix of the second fully connected layer; Represents the bias vector of the second fully connected layer; Indicates The unmapped embedding vector, pathological features; Based on the low-dimensional features of pathology, they are input into the focused linear attention module for training to obtain the attention score corresponding to each pathological feature; The expression of the attention score corresponding to each pathological feature is: ； In the formula, represents the first pathological features and The attention score of each task; Indicates pathological features and The query vector of each task; Indicates The key vector of tokens; Indicates The value vector of tokens; , Indicates the number of tokens in the input sequence, that is, the total number of elements that the model needs to process; represents the depth convolution function; Represents a mapping function or feature transformation function.

6. The method for predicting pan-cancer biomarkers by multi-instance multi-task learning according to claim 1, characterized in that: Combine each pathological low-dimensional feature and the corresponding attention score to calculate the pathological deep feature, and input the pathological deep feature into the multi-task classification prediction model for classification prediction to obtain the prediction results of each biomarker in pan-cancer, including: Based on the given slides, the deep features of the given task are calculated by combining the pathological low-dimensional features and their corresponding attention scores; The expression of the deep feature of the given task is: ； In the formula, Indicates deep feature representation of all blocks in a given slide for each task; Based on the deep features of a given task, input it into a multi-task classification prediction model for classification prediction, calculate the slice-level probability prediction score of each task, and obtain the prediction result of each biomarker in pan-cancer; the multi-task classification prediction model includes multiple classifiers; The expression of the slice-level probability prediction score of each task is: ； In the formula, Indicates Slice-level probability prediction score for each task; Normalization function; Indicates The weight matrix of each task in the classification layer; Indicates The bias vector of each task in the classification layer.

7. The method for predicting pan-cancer biomarkers by multi-instance multi-task learning according to claim 6, characterized in that: The method further includes calculating a loss function based on the multi-task classification prediction model; the calculation of the loss function includes: Based on each slide, the loss function of a single task is calculated using the standard cross entropy loss function; Based on the loss function of a single task, the total loss function of all tasks is calculated by weighted sum; The expression of the total loss function is: ； In the formula, represents the total loss of all tasks; Indicates The weight of the task, , Indicates the total number of tasks, ; Indicates The loss function for each task.

8. A system for the method for predicting pan-cancer biomarkers using multi-instance multi-task learning according to claim 1, characterized in that: include: A data acquisition module, which is used to obtain full-field digital pathology images and pan-cancer biomarker expression data of patients with solid tumors; A data processing module, which is used to perform preprocessing and feature extraction based on the full-field digital pathology image to obtain a pathology feature data set; The computing module is used to input the pathological feature dataset into the twin multi-instance fully convolutional network model for training to obtain the pathological low-dimensional features, and combine the linear attention mechanism to calculate the attention score corresponding to each pathological feature; The prediction module is used to combine each pathological low-dimensional feature and the corresponding attention score to calculate the pathological deep feature, and input the pathological deep feature into the multi-task classification prediction model for classification prediction to obtain the prediction result of each biomarker in pan-cancer.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images