Biomarker Prediction Method, Device, Equipment and Storage Medium

Abstract

The present invention discloses a method, device, equipment and storage medium for predicting biomarkers, including: collecting pathological images of patients and their corresponding annotation information, and performing preprocessing to establish a biomarker dataset and a tumor segmentation dataset; training a self-supervised learning model using the biomarker dataset; training a tumor segmentation model using the image patch features of the tumor segmentation dataset; using the trained tumor segmentation model to predict the tumor probability of each image patch and the corresponding image slice in the biomarker dataset; based on the prediction results, screening and optimizing the image patches in the biomarker dataset to generate a new biomarker dataset; training a biomarker prediction model using the image patch features of the new biomarker dataset; and applying the trained model to the pathological images to be analyzed. The present invention can effectively predict the status of biomarkers, with high prediction accuracy, and can effectively reduce the burden on clinical pathologists.

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image processing, and particularly relates to a method, device, equipment and storage medium for predicting biomarkers. Background Art

[0002] Timely and accurate diagnosis of malignant tumors is crucial for the treatment options and prognosis assessment of patients. Biomarkers play a crucial role in the diagnosis and treatment of cancer. They can not only improve the accuracy of diagnosis, but also play an important role in risk assessment, treatment decision-making, efficacy monitoring, etc. Currently, the mainstream methods for detecting biomarkers are usually polymerase chain reaction (PCR), sequencing or immunohistochemical analysis. However, for many patients in low-income and middle-income countries, the detection of gene biomarkers has problems of high costs and complex infrastructure. At the same time, due to the complexity of the detection methods of gene biomarkers, the detection cycle is long, resulting in the postponement of treatment plans.

[0003] For example, the prostate cancer biomarker AR-V7 (androgen receptor splice variant 7) is an important focus in the current field of prostate cancer treatment. AR-V7 is a splice variant of the androgen receptor (AR), and it plays a key role in the treatment resistance and disease progression of prostate cancer. qRT-PCR is the most commonly used method for detecting AR-V7 mRNA at present, but the technical route is complex, requiring professional laboratory conditions and technical personnel, which limits its wide application in clinics.

[0004] The application of artificial intelligence (AI) in predicting biomarkers is developing rapidly, mainly including data mining, machine learning, deep learning, etc. Deep learning technology can process and analyze a large amount of bioinformatics and clinical data, mine the association between biomarkers and diseases, and discover new biomarkers and potential treatment targets. However, the existing artificial intelligence prediction methods have the following problems.

[0005] First, it is difficult to determine tumor and normal tissues, and the technical difficulty is high. Classification errors are prone to occur at the boundary between tumor and normal tissues, reducing the accuracy of diagnosis. Conventional classification methods fail to accurately reflect the actual distribution of tumor cells and cannot provide the specific proportion of tumor cells. The lack of key information affects treatment decisions.

[0006] Second, high annotation costs. Precise pixel-level segmentation requires a large amount of manual annotation by experts, which is costly and time-consuming.

[0007] Third, poor adaptability. The high resolution and staining inconsistency of pathological images make precise pixel-level segmentation extremely challenging. Individual differences in cell morphology and staining quality make it difficult for algorithms to be generally applicable to different samples. Summary of the Invention

[0008] To solve the above technical problems, the present invention proposes a biomarker prediction method, apparatus, device and storage medium.

[0009] To achieve the above object, the technical solution of the present invention is as follows:

[0010] In a first aspect, the present invention discloses a biomarker prediction method, including:

[0011] Step S1: Collect the pathological images of patients and their corresponding annotation information;

[0012] Step S2: Divide each collected pathological image into several image patches and perform preprocessing;

[0013] Step S3: Based on the preprocessed image patches, establish a biomarker dataset and a tumor segmentation dataset;

[0014] The biomarker dataset includes: image patches labeled with biomarker diagnosis results;

[0015] The tumor segmentation dataset includes: image patches labeled with tumors and normal tissues;

[0016] Step S4: Use the biomarker dataset to train a self-supervised learning model, which is used to divide each image patch into several image slices and extract the image slice features of each image slice;

[0017] Step S5: Use the trained self-supervised learning model to divide each image patch in the tumor segmentation dataset into several image slices and extract the image slice features of each image slice;

[0018] Step S6: Use the image slice features of the tumor segmentation dataset obtained in Step S5 to train a tumor segmentation model, which is used to predict the tumor probability of each image patch and its corresponding image slices;

[0019] Step S7: Use the trained tumor segmentation model to predict the tumor probability of each image patch and its corresponding image slices in the biomarker dataset;

[0020] Step S8: Based on the prediction results of Step S7, screen and optimize the image patches in the biomarker dataset to generate a new biomarker dataset;

[0021] Step S9: Use the trained self-supervised learning model to divide each image patch in the new biomarker dataset into several image slices and extract the image slice features of each image slice;

[0022] Step S10: Train a biomarker prediction model using the image patch features of the new biomarker dataset obtained in Step S9. The biomarker prediction model is used to predict the biomarker status of a patient;

[0023] Step S11: Apply the trained model to the pathological image to be analyzed to predict the biomarker status.

[0024] Based on the above technical solution, the following improvements can be made:

[0025] As a preferred solution, Step S2 includes:

[0026] Step S2.1: Divide each collected pathological image into several image patches according to a fixed size;

[0027] Step S2.2: Adjust and unify the resolution of all image patches;

[0028] Step S2.3: Screen and exclude invalid image patches.

[0029] As a preferred solution, Step S11 includes:

[0030] Step S11.1: Collect the pathological image to be analyzed of the patient;

[0031] Step S11.2: Divide each collected pathological image into several image patches and perform preprocessing;

[0032] Step S11.3: Based on the preprocessed image patches, establish a dataset to be analyzed;

[0033] Step S11.4: Use the trained self-supervised learning model to divide each image patch in the dataset to be analyzed into several image slices and extract the image patch features of each image slice;

[0034] Step S11.5: Based on the image patch features of the dataset to be analyzed, use the trained tumor segmentation model to predict the tumor probability of each image patch and the corresponding image slice;

[0035] Step S11.6: Based on the prediction results of Step S11.5, screen and optimize the image patches in the dataset to be analyzed to generate a new dataset to be analyzed;

[0036] Step S11.7: Use the trained self-supervised learning model to divide each image patch in the new dataset to be analyzed into several image slices and extract the image patch features of each image slice;

[0037] Step S11.8: Based on the image patch features of the new dataset to be analyzed, use the trained biomarker prediction model to predict the biomarker status of the patient.

[0038] As a preferred solution, steps S8 and S11.6 respectively screen and optimize the image patches in the corresponding data sets through the following steps, specifically including:

[0039] Step A: According to the predicted tumor probability of each image patch, determine whether each image patch is a tumor image patch or a normal tissue image patch;

[0040] According to the predicted tumor probability of each image slice, determine whether each image slice is a tumor image slice or a normal tissue image slice;

[0041] Step B: Screen out all the image patches determined to be normal tissue in the corresponding data set;

[0042] Step C: Evaluate the tumor content t of each remaining image patch in the corresponding data set,

[0043] t = m / n;

[0044] Where: m is the number of image slices determined to be tumor image slices in this image patch;

[0045] n is the total number of image slices in this image patch;

[0046] Step D: According to the tumor content of each image patch, screen out the image patches with a tumor content exceeding the content threshold from the remaining image patches in the corresponding data set;

[0047] Step E: For the screened image patches, use the masking technique to remove all the image slices determined to be normal tissue to obtain new image patches, forming a new corresponding data set.

[0048] In a second aspect, the present invention also discloses a biomarker prediction device, including:

[0049] A collection module for collecting the pathological images of patients and their corresponding annotation information;

[0050] A preprocessing module for dividing each collected pathological image into several image patches and performing preprocessing;

[0051] A data set establishment module for establishing a biomarker data set and a tumor segmentation data set based on the preprocessed image patches;

[0052] The biomarker data set includes: image patches labeled with biomarker diagnosis results;

[0053] The tumor segmentation data set includes: image patches labeled with tumors and normal tissues;

[0054] A self-supervised learning model training module for training a self-supervised learning model using the biomarker data set, and the self-supervised learning model is used to divide each image patch into several image slices and extract the image slice features of each image slice;

[0055] The first feature extraction module is used to divide each image patch in the tumor segmentation data set into several image slices by using a trained self-supervised learning model, and extract the image slice features of each image slice;

[0056] The tumor segmentation model training module is used to train a tumor segmentation model by using the image slice features of the tumor segmentation data set obtained by the first feature extraction module. The tumor segmentation model is used to predict the tumor probability of each image patch and the corresponding image slice;

[0057] The segmentation prediction module is used to predict the tumor probability of each image patch and the corresponding image slice in the biomarker data set by using the trained tumor segmentation model;

[0058] The screening and optimization module is used to screen and optimize the image patches in the biomarker data set based on the prediction results of the segmentation prediction module, and generate a new biomarker data set;

[0059] The second feature extraction module is used to divide each image patch in the new biomarker data set into several image slices by using a trained self-supervised learning model, and extract the image slice features of each image slice;

[0060] The biomarker prediction model training module is used to train a biomarker prediction model by using the image slice features of the new biomarker data set obtained by the second feature extraction module. The biomarker prediction model is used to predict the biomarker status of a patient;

[0061] The application module is used to apply the trained model to the pathological image to be analyzed and predict the biomarker status.

[0062] As a preferred solution, the preprocessing module includes:

[0063] The image patch division unit is used to divide each collected pathological image into several image patches according to a fixed size;

[0064] The resolution adjustment unit is used to adjust and unify the resolution of all image patches;

[0065] The invalid image patch screening unit is used to screen and exclude invalid image patches.

[0066] As a preferred solution, the application module includes:

[0067] The application collection unit is used to collect the pathological image to be analyzed of a patient;

[0068] The application preprocessing unit is used to divide each collected pathological image into several image patches and perform preprocessing;

[0069] An application dataset building unit, configured to build a dataset to be analyzed based on the preprocessed image patches;

[0070] An application first feature extraction unit, configured to use a trained self-supervised learning model to divide each image patch in the dataset to be analyzed into several image slices, and extract the image slice features of each image slice;

[0071] An application segmentation prediction unit, configured to predict the tumor probability of each image patch and its corresponding image slices based on the image slice features of the dataset to be analyzed by using a trained tumor segmentation model;

[0072] An application screening and optimization unit, configured to screen and optimize the image patches in the dataset to be analyzed based on the prediction results of the application segmentation prediction unit, and generate a new dataset to be analyzed;

[0073] An application second feature extraction unit, configured to use a trained self-supervised learning model to divide each image patch in the new dataset to be analyzed into several image slices, and extract the image slice features of each image slice;

[0074] An application prediction unit, configured to predict the biomarker status of a patient based on the image slice features of the new dataset to be analyzed by using a trained biomarker prediction model.

[0075] As a preferred solution, the screening and optimization module and the application screening and optimization unit respectively include:

[0076] A judgment unit, configured to judge whether each image patch is a tumor image patch or a normal tissue image patch according to the predicted tumor probability of each image patch;

[0077] Judge whether each image slice is a tumor image slice or a normal tissue image slice according to the predicted tumor probability of each image slice;

[0078] An elimination unit, configured to eliminate all the image patches judged as normal tissues in the corresponding dataset;

[0079] A tumor content evaluation unit, configured to evaluate the tumor content t of each remaining image patch in the corresponding dataset,

[0080] t = m / n;

[0081] Where: m is the number of image slices judged as tumor image slices in this image patch;

[0082] n is the number of all image slices in this image patch;

[0083] A screening unit, configured to screen out the image patches with tumor content exceeding the content threshold from the remaining image patches in the corresponding dataset according to the tumor content of each image patch;

[0084] A forming unit, which is configured to, for the selected image patches, use a masking technique to remove all image slices determined to be normal tissues, obtain new image patches, and form a new corresponding data set.

[0085] In a third aspect, the present invention also discloses a computing device, including:

[0086] One or more processors;

[0087] A memory;

[0088] And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for any of the above biomarker prediction methods.

[0089] In a fourth aspect, the present invention also discloses a storage medium, characterized in that the storage medium stores one or more computer-readable programs, and the one or more programs include instructions adapted to be loaded and executed by the memory to perform any of the above biomarker prediction methods.

[0090] The present invention discloses a biomarker prediction method, device, device and storage medium, which have the following beneficial effects:

[0091] First, the present invention uses a self-supervised learning model to extract image slice features, reducing the dependence on manually labeled data.

[0092] Second, the present invention trains a tumor segmentation model based on the image slice features using a weakly supervised learning method, which can accurately segment tumor tissues and normal tissues.

[0093] Third, the present invention screens and optimizes the image patches according to the prediction results of the tumor segmentation model, making the prediction of biomarkers more accurate.

[0094] In summary, the present invention can effectively predict the status of biomarkers, with high prediction accuracy, can effectively reduce the burden on clinical pathologists, improve the prediction accuracy of tumor-related biomarkers, and has significant clinical significance. Description of the Drawings

[0095] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0096] Figure 1 It is a flowchart of the biomarker prediction method provided by the embodiment of the present invention.

[0097] Figure 2 This is a schematic flowchart of pathological image division and preprocessing provided by an embodiment of the present invention.

[0098] Figure 3 This is a schematic flowchart of image block feature extraction provided by an embodiment of the present invention.

[0099] Figure 4 This is a schematic flowchart of a tumor segmentation model provided by an embodiment of the present invention.

[0100] Figure 5(a) is the original image block provided by an embodiment of the present invention;

[0101] Figure 5(b) is the image block of the predicted tumor region provided by an embodiment of the present invention.

[0102] Figure 6 This is the AUROC curve provided by an embodiment of the present invention.

[0103] Figure 7 This is a schematic flowchart of the application stage provided by an embodiment of the present invention.

[0104] Figure 8 This is a block diagram of a biomarker prediction device provided by an embodiment of the present invention.

[0105] Figure 9 This is a block diagram of a computing device provided by an embodiment of the present invention.

[0106] Wherein: 201 - collection module, 202 - preprocessing module, 203 - dataset establishment module, 204 - self-supervised learning model training module, 205 - first feature extraction module, 206 - tumor segmentation model training module, 207 - segmentation prediction module, 208 - screening and optimization module, 209 - second feature extraction module, 210 - biomarker prediction model training module, 211 - application module, 301 - processor, 302 - memory. Detailed implementation manners

[0107] The preferred implementation manners of the present invention will be described in detail below with reference to the accompanying drawings.

[0108] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0109] The use of ordinal numbers such as "first", "second", "third", etc. to describe ordinary objects only indicates different instances of similar objects and does not intend to imply that the objects so described must have a given order in terms of time, space, ranking, or in any other way.

[0110] In addition, the expression "comprising" an element is an "open-ended" expression, which only means the existence of corresponding components or steps and should not be construed as excluding additional components or steps.

[0111] To achieve the object of the present invention, in some embodiments of the biomarker prediction method, the prostate cancer biomarker AR-V7 is taken as an example. As Figure 1 shown, the biomarker prediction method includes:

[0112] Step S101: Collect the pathological images of the patient and their corresponding annotation information;

[0113] Step S102: Divide each of the collected pathological images into several image patches and perform preprocessing;

[0114] Step S103: Based on the preprocessed image patches, establish a biomarker data set and a tumor segmentation data set;

[0115] The biomarker data set includes: image patches labeled with biomarker diagnosis results;

[0116] The tumor segmentation data set includes: image patches labeled with tumors and normal tissues;

[0117] Step S104: Use the biomarker data set to train a self-supervised learning model, and the self-supervised learning model is used to divide each image patch into several image slices and extract the image slice features of each image slice;

[0118] Step S105: Use the trained self-supervised learning model to divide each image patch in the tumor segmentation data set into several image slices and extract the image slice features of each image slice;

[0119] Step S106: Use the image slice features of the tumor segmentation data set obtained in Step S105 to train a tumor segmentation model, and the tumor segmentation model is used to predict the tumor probability of each image patch and the corresponding image slice;

[0120] Step S107: Use the trained tumor segmentation model to predict the tumor probability of each image patch and the corresponding image slice in the biomarker data set;

[0121] Step S108: Based on the prediction results of Step S107, screen and optimize the image patches in the biomarker data set to generate a new biomarker data set;

[0122] Step S109: Use the trained self-supervised learning model to divide each image patch in the new biomarker dataset into several image slices, and extract the image slice features of each image slice;

[0123] Step S110: Use the image slice features of the new biomarker dataset obtained in Step S109 to train a biomarker prediction model, which is used to predict the biomarker status of patients;

[0124] Step S111: Apply the trained model to the pathological images to be analyzed to predict the biomarker status.

[0125] The following elaborates on each of the above steps in detail.

[0126] Step S101 collects H&E stained pathological images and biomarker diagnosis results of prostate cancer patients.

[0127] Specifically, in this embodiment, the detection status of AR-V7 in the pathological images of 400 cases of prostate cancer in the TCGA dataset is collected, and the status of each sample is determined: AR-V7 positive or AR-V7 negative, where: 100 cases are positive and 300 cases are negative.

[0128] Step S102 divides each collected pathological image into several image patches and performs preprocessing, specifically including:

[0129] Step S102.1: Divide each collected pathological image into several image patches according to a fixed size;

[0130] Step S102.2: Adjust and unify the resolution of all image patches;

[0131] Step S102.3: Screen and exclude invalid image patches.

[0132] As Figure 2 shown, specifically, in this embodiment, the pathological image is divided into several image patches (tlie) according to a fixed physical size of 256um × 256um. The resolution of the image patches is adjusted to 512x512 pixels, that is, the image patches are normalized to 0.5um / pixel. Image patches with a tissue content ratio lower than a preset threshold (e.g., 30%) are identified as invalid image patches, and image patches higher than the preset threshold are identified as valid image patches, indicating rich information. Screen and exclude invalid image patches to reduce the amount of computation.

[0133] Step S103 establishes a biomarker dataset and a tumor segmentation dataset.

[0134] Specifically, in this embodiment,

[0135] The biomarker dataset is used for training the model and the final evaluation of biomarker prediction. The biomarker dataset is divided into a training set and a test set according to a ratio of 7:3. The training set contains 280 cases, including 70 positive cases and 210 negative cases; the test set contains 120 cases, including 30 positive cases and 90 negative cases.

[0136] The tumor segmentation dataset is used for training the tumor segmentation model. 500 tumor image patches and 500 normal tissue image patches are selected from the image patches of the training set of the biomarker dataset.

[0137] The tumor segmentation dataset is divided according to a ratio of 7:3. The training set contains 700 image patches, including 350 tumor image patches and 350 normal tissue image patches, and the test set contains 300 image patches, including 150 tumor images and 150 normal tissue images.

[0138] Step S104 uses the biomarker dataset to train the self-supervised learning model. The Dinov2 model is used as the backbone network of the self-supervised learning model. The self-supervised learning model can efficiently extract multi-level pathological features from pathological images.

[0139] Specifically, in this embodiment, as Figure 3 shown, the feature extraction of the self-supervised learning model includes:

[0140] First, extract the image patch , , where h represents the height of the image patch and w represents the width of the image patch, and the scaled image patch is obtained through the scaling function.

[0141]

[0142] Among them, represents the scaling function. In the embodiment, the size of the scaled image patch is 224×224.

[0143] Then, input the scaled image patch into the pre-trained model to extract features. The pre-trained model divides the image patch into non-overlapping image patches (patches) of 16×16×3. Each image patch is flattened into a vector and undergoes mapping to generate an initial feature representation:

[0144]

[0145] Among them, represents the th non-overlapping image patch of 16×16;

[0146] is the linear mapping matrix,

[0147] is the bias term, , where d is the feature dimension;

[0148] represents the dot product operation of vectors.

[0149] The pre-trained model considers the relative positions between each image patch and performs positional encoding on the feature vector of each image patch as follows:

[0150]

[0151] where, is the positional encoding, ;

[0152] N is the number of image patches.

[0153] Finally, the feature vector after positional encoding is input into a multi-layer Transformer through a multi-layer attention mechanism, and the feature vector of each image patch is obtained in the penultimate layer .

[0154]

[0155] where MAS represents the multi-head self-attention mechanism, represents the number of layers of the Transformer.

[0156] Self-supervised learning can extract valuable features from unlabeled data. In the few-shot learning environment with scarce samples, self-supervised learning generates supervision signals by constructing proxy tasks, effectively reducing the dependence on manually labeled data. In addition, the feature vector of the image patch not only contains the position and detail information of the image, but also covers the detailed information inside the cell, which helps to train the tumor segmentation model subsequently.

[0157] Step S105 uses the trained self-supervised learning model to divide each image block in the tumor segmentation data set into several image patches and extract the image patch features of each image patch.

[0158] As Figure 4 shown, specifically, in this embodiment, each image block obtains the feature vectors of 196 image patches , which are used as a feature bag.

[0159] Step S106 trains a tumor segmentation model using the weakly supervised learning method, and segments tumors and normal tissues through the feature vectors of image patches.

[0160] Train according to the training set divided from the tumor segmentation dataset, evaluate on the test set, and save the optimal tumor segmentation model.

[0161] The weakly supervised learning method is through the feature vectors in the feature bag to affect the prediction result. The feature bag first passes through a multi-layer perceptron (MLP) for encoding, then through an attention mechanism for weighting, and finally through a decoder for classification.

[0162] Output the predicted tumor probability of each image patch at the middle layer of the tumor segmentation model and output the predicted tumor probability of each image block at the last layer. .

[0163] The pixels of pathological images are usually in the hundreds of thousands and millions. It is difficult to process pathological images using a pixel-level segmentation model. Based on the image patch features, a weakly supervised learning method is used to train a tumor segmentation model. This method does not require manual delineation of labels, only the category of the image block needs to be defined. This not only solves the problem of high annotation cost and time consumption, but also through the decoding of the tumor segmentation model, obtains a pixel-level segmentation result, accurately distinguishing normal tissues and tumor tissues.

[0164] Furthermore, Figure 5(a) is the original image patch, Figure 5(b) is the image patch showing the predicted tumor area, the red part represents the predicted tumor area, and the blue part represents the predicted normal tissue area. The present invention can achieve pixel-level segmentation.

[0165] Step S107 uses the trained tumor segmentation model to predict the tumor probability of each image block and the corresponding image patch in the biomarker dataset.

[0166] Step S108 screens and optimizes the image patches in the biomarker dataset based on the prediction results of Step S107 to generate a new biomarker dataset.

[0167] Step S108 includes:

[0168] Step S108.1: According to the predicted tumor probability of each image block , judge whether each image block is a tumor image block or a normal tissue image block;

[0169] According to the predicted tumor probability of each image patch , judge whether each image patch is a tumor image patch or a normal tissue image patch;

[0170] For example: and are respectively judged against a probability threshold (e.g., 0.5). When the probability threshold is exceeded, it is judged as a tumor image patch or a tumor image slice; otherwise, it is a normal tissue image patch or a normal tissue image slice;

[0171] Step S108.2: Screen out all the image patches judged as normal tissues in the biomarker dataset;

[0172] Step S108.3: Evaluate the tumor content t of each remaining image patch in the biomarker dataset,

[0173] t = m / n;

[0174] where: m is the number of image slices judged as tumor image slices in this image patch;

[0175] n is the number of all image slices in this image patch;

[0176] Step S108.4: According to the tumor content t of each image patch, screen out the image patches with a tumor content exceeding the content threshold from the remaining image patches in the biomarker dataset;

[0177] For example: Judge the tumor content t against the content threshold (e.g., 0.85). When the content threshold is exceeded, screen out the corresponding image patches;

[0178] Step S108.5: For the screened-out image patches, use the masking technique to remove all the image slices judged as normal tissues to obtain new image patches, forming a new biomarker dataset.

[0179] For the image patches screened out in step S108.4, apply the masking technique to remove all the image slices predicted as normal tissues. This step is achieved by setting a masking value for each image slice which depends on the prediction result of this image slice:

[0180] The masking value corresponding to the tumor image slice , = 1;

[0181] For the masking value of the normal tissue image slice , = 0.

[0182] Then, use these masking values to modify the representation of the image patch to ensure that it only contains information of tumor tissues:

[0183]

[0184] where: T represents the pixel value corresponding to the image patch, and M represents the mask The formed matrix, represents a new image patch that only contains the pixel values of tumor tissues.

[0185] Using a screened and optimized dataset can more accurately predict biomarkers.

[0186] In step S109, the trained self-supervised learning model divides each image patch in the new biomarker dataset into several image slices and extracts the image slice features of each image slice.

[0187] In step S110, the image slice features of the new biomarker dataset obtained in step S109 are used to train a biomarker prediction model, and the biomarker prediction model is used to predict the biomarker status of patients with prostate cancer.

[0188] Specifically, in this embodiment, the biomarker prediction model consists of a feature encoder (Encoder), an attention mechanism (Attention), and an output layer (Head).

[0189] The feature encoder converts the input feature vector into a low-dimensional vector.

[0190] Let the input feature be , and , where B is the batch size, N is the number of instances in each feature packet, and F represents the feature dimension. After passing through the encoder, the output of the feature is:

[0191]

[0192] Among them: d represents the encoded feature dimension. The specific structure of the feature encoder is as follows:

[0193]

[0194] Among them: represents the weight matrix, ;

[0195] represents the bias term, .

[0196] The attention mechanism is used to calculate the importance weights of each instance in order to weight the instance features in subsequent steps. It calculates the importance score by mapping the features to a smaller dimension.

[0197] Let the input feature be , and the calculation of the attention score can be expressed as:

[0198]

[0199] The attention mechanism involves three parts: linear transformation, activation function, and output layer.

[0200] Among them, the linear transformation maps the input features to a low dimension. Let the input feature be , and the output feature after linear transformation is:

[0201]

[0202] In the above formula , represents the dimension of the hidden layer features. The linear transformation is input into the activation function:

[0203]

[0204] The output of the activation function is passed into another linear transformation layer to obtain the final attention score of the attention mechanism. In the following formula .

[0205]

[0206] For each instance in the feature packet, the attention mechanism performs masking according to the number of instances (length).

[0207] The masked attention score is normalized by the Softmax function to ensure that the sum of the weights is 1:

[0208]

[0209] The weights obtained through the attention mechanism are used to perform weighted summation on the encoded features to generate the global weighted feature representation of each feature packet:

[0210]

[0211] In the above formula, is the global weighted feature representation of each feature packet, ; is the i-th encoded feature, is the attention score corresponding to the i-th encoded feature.

[0212] The output layer is used to classify the weighted global features to obtain the classification result of each feature packet. The output layer contains a linear layer and is trained using cross-entropy loss. The output calculation formula for classification is:

[0213]

[0214] In the above formula is the weight matrix of the output layer, is the bias term, is the number of output categories.

[0215] During the training process, cross-entropy loss is used to measure the difference between the model's predictions and the true labels:

[0216]

[0217] In the above formula, is the true label, is the predicted probability of the model.

[0218] The prediction results of the biomarker prediction model are verified, and the model is optimized according to the verification results to improve the accuracy and reliability of the prediction.

[0219] As Figure 6 shown, the AUROC (Area Under the Receiver Operating Characteristic Curve) of the biomarker AR-v7 is the area under the ROC curve, which is used to evaluate the performance of a binary classification model and reaches 0.772.

[0220] As Figure 7 shown, step S111 is the clinical application step, which specifically includes:

[0221] Step S111.1: Collect the pathological images to be analyzed of the patient;

[0222] Step S111.2: Divide each collected pathological image into several image patches and perform preprocessing;

[0223] Step S111.3: Based on the preprocessed image patches, establish a dataset to be analyzed;

[0224] Step S111.4: Use the trained self-supervised learning model to divide each image patch in the dataset to be analyzed into several image slices and extract the image slice features of each image slice;

[0225] Step S111.5: Based on the image slice features of the dataset to be analyzed, use the trained tumor segmentation model to predict the tumor probability of each image patch and the corresponding image slices;

[0226] Step S111.6: Based on the prediction results of step S111.5, screen and optimize the image patches in the dataset to be analyzed to generate a new dataset to be analyzed;

[0227] Step S111.7: Use the trained self-supervised learning model to divide each image patch in the new dataset to be analyzed into several image slices and extract the image slice features of each image slice;

[0228] Step S111.8: Based on the image patch features of the new dataset to be analyzed, use the trained biomarker prediction model to predict the biomarker status of the patient, which is positive or negative.

[0229] Based on the prediction results, doctors can make reasonable clinical decisions.

[0230] The above step S111.6 can screen and optimize the image patches in the dataset to be analyzed through the following steps, specifically including:

[0231] Step S111.6.1: According to the predicted tumor probability of each image patch, determine whether each image patch is a tumor image patch or a normal tissue image patch;

[0232] According to the predicted tumor probability of each image slice, determine whether each image slice is a tumor image slice or a normal tissue image slice;

[0233] Step S111.6.2: Screen out all the image patches judged as normal tissue in the dataset to be analyzed;

[0234] Step S111.6.3: Evaluate the tumor content t of each remaining image patch in the dataset to be analyzed,

[0235] t = m / n;

[0236] Where: m is the number of image slices judged as tumor image slices in this image patch;

[0237] n is the total number of image slices in this image patch;

[0238] Step S111.6.4: According to the tumor content of each image patch, screen out the image patches with tumor content exceeding the content threshold from the remaining image patches in the dataset to be analyzed;

[0239] Step S111.6.5: For the screened image patches, use the masking technique to remove all the image slices judged as normal tissue to obtain new image patches and form a new dataset to be analyzed.

[0240] Specifically, it is similar to the above step S108 and will not be elaborated here.

[0241] In some other embodiments, as Figure 8 shown, the present invention also discloses a biomarker prediction device, including:

[0242] A collection module 201, configured to collect the pathological images of the patient and their corresponding annotation information;

[0243] A preprocessing module 202, configured to divide each collected pathological image into several image patches and perform preprocessing;

[0244] The dataset building module 203 is used to build a biomarker dataset and a tumor segmentation dataset based on the preprocessed image patches;

[0245] The biomarker dataset includes: image patches labeled with biomarker diagnosis results;

[0246] The tumor segmentation dataset includes: image patches labeled with tumors and normal tissues;

[0247] The self-supervised learning model training module 204 is used to train a self-supervised learning model using the biomarker dataset. The self-supervised learning model is used to divide each image patch into several image slices and extract the image slice features of each image slice;

[0248] The first feature extraction module 205 is used to divide each image patch in the tumor segmentation dataset into several image slices using the trained self-supervised learning model and extract the image slice features of each image slice;

[0249] The tumor segmentation model training module 206 is used to train a tumor segmentation model using the image slice features of the tumor segmentation dataset obtained by the first feature extraction module. The tumor segmentation model is used to predict the tumor probability of each image patch and the corresponding image slices;

[0250] The segmentation prediction module 207 is used to predict the tumor probability of each image patch and the corresponding image slices in the biomarker dataset using the trained tumor segmentation model;

[0251] The screening and optimization module 208 is used to screen and optimize the image patches in the biomarker dataset based on the prediction results of the segmentation prediction module to generate a new biomarker dataset;

[0252] The second feature extraction module 209 is used to divide each image patch in the new biomarker dataset into several image slices using the trained self-supervised learning model and extract the image slice features of each image slice;

[0253] The biomarker prediction model training module 210 is used to train a biomarker prediction model using the image slice features of the new biomarker dataset obtained by the second feature extraction module. The biomarker prediction model is used to predict the biomarker status of a patient;

[0254] The application module 211 is used to apply the trained model to the pathological image to be analyzed to predict the biomarker status.

[0255] Furthermore, the preprocessing module includes:

[0256] The image patch division unit is used to divide each collected pathological image into several image patches according to a fixed size;

[0257] A resolution adjustment unit for adjusting and unifying the resolution of all image patches;

[0258] An invalid image patch screening unit for screening and excluding invalid image patches.

[0259] Furthermore, the application module includes:

[0260] An application collection unit for collecting the pathological images to be analyzed of patients;

[0261] An application preprocessing unit for dividing each collected pathological image into several image patches and performing preprocessing;

[0262] An application dataset establishment unit for establishing a dataset to be analyzed based on the preprocessed image patches;

[0263] An application first feature extraction unit for using a trained self-supervised learning model to divide each image patch in the dataset to be analyzed into several image slices and extracting the image slice features of each image slice;

[0264] An application segmentation prediction unit for predicting the tumor probability of each image patch and the corresponding image slices based on the image slice features of the dataset to be analyzed by using a trained tumor segmentation model;

[0265] An application screening and optimization unit for screening and optimizing the image patches in the dataset to be analyzed based on the prediction results of the application segmentation prediction unit to generate a new dataset to be analyzed;

[0266] An application second feature extraction unit for using a trained self-supervised learning model to divide each image patch in the new dataset to be analyzed into several image slices and extracting the image slice features of each image slice;

[0267] An application prediction unit for predicting the biomarker status of the patient based on the image slice features of the new dataset to be analyzed by using a trained biomarker prediction model.

[0268] Furthermore, the screening and optimization module and the application screening and optimization unit respectively include:

[0269] A judgment unit for judging whether each image patch is a tumor image patch or a normal tissue image patch according to the predicted tumor probability of each image patch;

[0270] Judging whether each image slice is a tumor image slice or a normal tissue image slice according to the predicted tumor probability of each image slice;

[0271] A screening unit for screening and removing all image patches judged as normal tissue in the corresponding dataset;

[0272] A tumor content evaluation unit for evaluating the tumor content t of each remaining image patch in the corresponding dataset

[0273] t = m / n;

[0274] Where: m is the number of image slices determined to be tumor image slices in the image block;

[0275] n is the number of all image slices in the image block;

[0276] A screening unit for screening out image blocks with a tumor content exceeding a content threshold from the remaining image blocks of the corresponding data set according to the tumor content of each image block;

[0277] A forming unit for, for the screened image blocks, using a masking technique to remove all image slices determined to be normal tissues to obtain new image blocks and form a new corresponding data set.

[0278] Furthermore, it should be noted that: when the biomarker prediction device provided in the above embodiment performs biomarker prediction, only the above division of each functional module is used as an example for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the biomarker prediction device is divided into different functional modules to complete all or part of the functions described above.

[0279] In addition, the biomarker prediction device provided in the above embodiment and the embodiment of the biomarker prediction method belong to the same concept. For the specific implementation process, please refer to the method embodiment and will not be elaborated here.

[0280] In addition, in some other embodiments, as Figure 9 shown, the present invention also discloses a computing device, including:

[0281] One or more processors 301;

[0282] A memory 302;

[0283] And one or more programs, where one or more programs are stored in the memory 302 and are configured to be executed by one or more processors 301, and one or more programs include the instructions of the biomarker prediction method disclosed in the above embodiment.

[0284] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 301 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0285] The memory 302 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 302 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 301 to implement the biomarker prediction method provided in the method embodiments of the present invention.

[0286] In addition, the computing device may optionally further include: a peripheral device interface and at least one peripheral device. The processor 301, the memory 302, and the peripheral device interface may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface through a bus, signal lines, or a circuit board. Schematically, the peripheral devices include, but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply, etc.

[0287] Of course, the computing device may also include fewer or more components, and this embodiment does not limit this.

[0288] In addition, in some other embodiments, the present invention also discloses a storage medium, and the storage medium stores one or more computer-readable programs. The one or more programs include instructions, and the instructions are adapted to be loaded and executed by the memory to perform the biomarker prediction method disclosed in the above embodiments.

[0289] The present invention discloses a method, device, equipment and storage medium for predicting biomarkers, which have the following beneficial effects:

[0290] First, the present invention uses a self-supervised learning model to extract image patch features, reducing the dependence on manually labeled data.

[0291] Second, the present invention trains a tumor segmentation model using a weakly supervised learning method based on the image patch features, which can accurately segment tumor tissues and normal tissues.

[0292] Third, the present invention screens and optimizes the image patches according to the prediction results of the tumor segmentation model, making the prediction of biomarkers more accurate.

[0293] In summary, the present invention can effectively predict the status of biomarkers, with high prediction accuracy, which can effectively reduce the burden on clinical pathologists and improve the prediction accuracy of tumor-related biomarkers, having significant clinical significance.

[0294] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification only illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A method for predicting biomarkers, characterized in that, Including: Step S1: Collect the pathological images of the patient and their corresponding annotation information; Step S2: Divide each collected pathological image into several image patches and perform preprocessing; Step S3: Based on the preprocessed image patches, establish a biomarker dataset and a tumor segmentation dataset; The biomarker dataset includes: image patches labeled with biomarker diagnosis results; The tumor segmentation dataset includes: image patches labeled with tumors and normal tissues; Step S4: Use the biomarker dataset to train a self-supervised learning model, which is used to divide each image patch into several image slices and extract the image slice features of each image slice; Step S5: Use the trained self-supervised learning model to divide each image patch in the tumor segmentation dataset into several image slices and extract the image slice features of each image slice; Step S6: Use the image slice features of the tumor segmentation dataset obtained in Step S5 to train a tumor segmentation model, which is used to predict the tumor probability of each image patch and its corresponding image slice; Step S7: Use the trained tumor segmentation model to predict the tumor probability of each image patch and its corresponding image slice in the biomarker dataset; Step S8: Based on the prediction results of Step S7, screen and optimize the image patches in the biomarker dataset to generate a new biomarker dataset; Step S9: Use the trained self-supervised learning model to divide each image patch in the new biomarker dataset into several image slices and extract the image slice features of each image slice; Step S10: Use the image slice features of the new biomarker dataset obtained in Step S9 to train a biomarker prediction model, which is used to predict the biomarker status of the patient; Step S11: Apply the trained self-supervised learning model, tumor segmentation model, and biomarker prediction model to the pathological images to be analyzed to predict the biomarker status.

2. The biomarker prediction method according to claim 1, wherein The said Step S2 includes: Step S2.1: Divide each collected pathological image into several image patches according to a fixed size; Step S2.2: Adjust and unify the resolution of all image patches; Step S2.3: Screen and exclude invalid image patches.

3. The biomarker prediction method according to claim 1 or 2, characterized in that The said Step S11 includes: Step S11.1: Collect the pathological images to be analyzed of the patient; Step S11.2: Divide each collected pathological image into several image patches and perform preprocessing; Step S11.3: Based on the preprocessed image patches, establish a dataset to be analyzed; Step S11.4: Use the trained self-supervised learning model to divide each image patch in the dataset to be analyzed into several image slices and extract the image slice features of each image slice; Step S11.5: Based on the image slice features of the dataset to be analyzed, use the trained tumor segmentation model to predict the tumor probability of each image patch and its corresponding image slice; Step S11.6: Based on the prediction results of Step S11.5, screen and optimize the image patches in the dataset to be analyzed to generate a new dataset to be analyzed; Step S11.7: Use the trained self-supervised learning model to divide each image patch in the new dataset to be analyzed into several image slices, and extract the image slice features of each image slice; Step S11.8: Based on the image slice features of the new dataset to be analyzed, use the trained biomarker prediction model to predict the biomarker status of the patient.

4. The biomarker prediction method according to claim 3, wherein The steps S8 and S11.6 respectively screen and optimize the image patches in the corresponding datasets through the following steps, specifically including: Step A: According to the predicted tumor probability of each image patch, determine whether each image patch is a tumor image patch or a normal tissue image patch; According to the predicted tumor probability of each image slice, determine whether each image slice is a tumor image slice or a normal tissue image slice; Step B: Screen out all the image patches determined to be normal tissue in the corresponding dataset; Step C: Evaluate the tumor content t of each remaining image patch in the corresponding dataset, t = m / n; where: m is the number of image slices determined to be tumor image slices in this image patch; n is the total number of image slices in this image patch; Step D: According to the tumor content of each image patch, screen out the image patches with a tumor content exceeding the content threshold from the remaining image patches in the corresponding dataset; Step E: For the screened image patches, use the masking technique to remove all the image slices determined to be normal tissue, obtain new image patches, and form a new corresponding dataset.

5. Biomarker prediction device, characterized in that, Including: A collection module for collecting the pathological images of patients and their corresponding annotation information; A preprocessing module for dividing each collected pathological image into several image patches and performing preprocessing; A dataset establishment module for establishing a biomarker dataset and a tumor segmentation dataset based on the preprocessed image patches; The biomarker dataset includes: image patches labeled with biomarker diagnosis results; The tumor segmentation dataset includes: image patches labeled with tumors and normal tissues; A self-supervised learning model training module for training a self-supervised learning model using the biomarker dataset. The self-supervised learning model is used to divide each image patch into several image slices and extract the image slice features of each image slice; A first feature extraction module for using the trained self-supervised learning model to divide each image patch in the tumor segmentation dataset into several image slices and extract the image slice features of each image slice; A tumor segmentation model training module for training a tumor segmentation model using the image slice features of the tumor segmentation dataset obtained by the first feature extraction module. The tumor segmentation model is used to predict the tumor probability of each image patch and the corresponding image slice; A segmentation prediction module for using the trained tumor segmentation model to predict the tumor probability of each image patch and the corresponding image slice in the biomarker dataset; A screening and optimization module for screening and optimizing the image patches in the biomarker dataset based on the prediction results of the segmentation prediction module to generate a new biomarker dataset; A second feature extraction module for using the trained self-supervised learning model to divide each image patch in the new biomarker dataset into several image slices and extract the image slice features of each image slice; A biomarker prediction model training module, which is used to train a biomarker prediction model by using the image patch features of the new biomarker dataset obtained by the second feature extraction module. The biomarker prediction model is used to predict the biomarker status of a patient; An application module, which is used to apply the trained self-supervised learning model, tumor segmentation model, and biomarker prediction model to the pathological images to be analyzed to predict the biomarker status.

6. The biomarker prediction device according to claim 5, wherein The preprocessing module includes: An image patch division unit, which is used to divide each collected pathological image into several image patches according to a fixed size; A resolution adjustment unit, which is used to adjust and unify the resolutions of all image patches; An invalid image patch screening unit, which is used to screen and exclude invalid image patches.

7. The biomarker prediction device according to claim 5 or 6, characterized in that The application module includes: An application collection unit, which is used to collect the pathological images to be analyzed of a patient; An application preprocessing unit, which is used to divide each collected pathological image into several image patches and perform preprocessing; An application dataset establishment unit, which is used to establish a dataset to be analyzed based on the preprocessed image patches; An application first feature extraction unit, which is used to divide each image patch in the dataset to be analyzed into several image slices by using the trained self-supervised learning model and extract the image patch features of each image slice; An application segmentation prediction unit, which is used to predict the tumor probability of each image patch and the corresponding image slices based on the image patch features of the dataset to be analyzed by using the trained tumor segmentation model; An application screening and optimization unit, which is used to screen and optimize the image patches in the dataset to be analyzed based on the prediction results of the application segmentation prediction unit to generate a new dataset to be analyzed; An application second feature extraction unit, which is used to divide each image patch in the new dataset to be analyzed into several image slices by using the trained self-supervised learning model and extract the image patch features of each image slice; An application prediction unit, which is used to predict the biomarker status of a patient based on the image patch features of the new dataset to be analyzed by using the trained biomarker prediction model.

8. The biomarker prediction device according to claim 7, wherein, The screening and optimization module and the application screening and optimization unit respectively include: A judgment unit, which is used to judge whether each image patch is a tumor image patch or a normal tissue image patch according to the predicted tumor probability of each image patch; Judge whether each image slice is a tumor image slice or a normal tissue image slice according to the predicted tumor probability of each image slice; A screening unit, which is used to screen and remove all the image patches judged to be normal tissues in the corresponding dataset; A tumor content evaluation unit, which is used to evaluate the tumor content t of each remaining image patch in the corresponding dataset, t = m / n; where: m is the number of image slices judged to be tumor image slices in this image patch; n is the total number of image slices in this image patch; A screening unit, which is used to screen out the image patches with a tumor content exceeding the content threshold from the remaining image patches in the corresponding dataset according to the tumor content of each image patch; A formation unit, which is used to remove all the image slices judged to be normal tissues from the screened image patches by using the mask technology to obtain new image patches and form a new corresponding dataset.

9. A computing device, characterized in that, Includes: One or more processors; A memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and one or more of the programs include instructions for the biomarker prediction method according to any one of claims 1-4 above.

10. Storage medium, characterized in that, The storage medium stores one or more computer-readable programs, and one or more of the programs include instructions adapted to be loaded and executed by the memory for the biomarker prediction method according to any one of claims 1-4 above.

Images