Tumor tissue tracing method and equipment based on deep learning

Through deep learning methods, squamous cell carcinoma tissue is processed in multiple stages, combined with multi-classification and category prediction models, gender information is used to improve the accuracy of tumor origin determination, solving the problems of complex operation, long cycle and low accuracy in traditional methods, and realizing the Efficient and accurate identification of the origin of squamous cell carcinoma tissue.

CN118690289BActive Publication Date: 2024-11-12CANHELP GENOMICS (SUZHOU) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411196097.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-12
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

Traditional methods are difficult to effectively identify the origin of squamous cell carcinoma tissue, and there are problems such as complex operation, long detection cycle, high cost and limited accuracy. The early method based on deep learning model is simple, making it difficult to distinguish highly similar pathological characteristics.

Method used

Using deep learning-based tumor tissue traceability method, through multi-stage processing of pathological images, including image block division, preprocessing, multi-classification model and category prediction model, the patient's gender information is used to improve the judgment accuracy, and the comprehensive score is calculated to determine Tumor origin category.

Benefits of technology

It simplifies the operation process, shortens the detection time, improves the detection accuracy and efficiency, reduces costs, and is suitable for clinical applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118690289B_ABST
    Figure CN118690289B_ABST
Patent Text Reader

Abstract

This invention discloses a method and device for tracing the origin of tumor tissue based on deep learning, comprising: classifying tumor tissue into k tumor origin categories according to its origin; collecting a large number of pathological images, their tumor tissue origination tags, and patient gender information; dividing each collected pathological image into several image patches and preprocessing them; training a multi-classification model using the preprocessed image patches, which is used to predict the probability of each tumor origin category to which the image patch belongs; training k category prediction models using the preprocessed image patches based on prior knowledge of the patient's gender, which are used to predict the distance between each image patch and the standard image patch corresponding to its tumor origin category; and using the trained models to predict the tumor tissue origin of new pathological images. This invention employs a multi-stage model to determine the tumor origin category, achieving high prediction accuracy, low cost, and short cycle time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image processing, and specifically relates to a tumor tissue tracing method and device based on deep learning. Background Art

[0002] The main task of tumor pathology diagnosis is to classify tumors, among which clarifying the origin of tumor tissue is crucial for tumor staging, treatment and prognosis. Tumor types that were once neglected, such as unknown primary lesions and multiple primary tumors, have gradually become the focus of clinical attention, among which squamous cell carcinoma accounts for a considerable proportion.

[0003] Squamous cell carcinoma, or SCC for short, is one of the most common cancers in the human body. In recent years, the incidence of squamous cell carcinoma has increased dramatically due to the effects of carcinogenic factors such as ultraviolet radiation, smoking, drinking, or infection with pathogens (human papillomavirus, HPV; Epstein-Barr virus, EBV). Squamous cell carcinoma often occurs in the skin, head and neck, esophagus, lungs, and cervix. It can originate from squamous epithelium (such as skin and esophageal epithelium) or non-squamous epithelium (such as airway epithelium). At present, tissue morphology and immunophenotype have certain tissue and organ specificity, and are the basis for classification widely used in pathology.

[0004] However, squamous cell carcinomas of different primary sites are highly similar in tissue morphology and immunophenotype, and traditional methods cannot effectively identify the origin. For a long time, due to the lack of effective organ-specific identification methods, the identification of the tissue origin of squamous cell carcinoma has always been a blind spot in tumor pathological diagnosis and has become a bottleneck in the precision treatment of tumors.

[0005] Recently, researchers have proposed a variety of new methods to determine the origin of tumor tissues, providing a new direction for the study of squamous cell carcinoma. First, the study of determining the tissue origin of squamous cell carcinoma by analyzing the molecular characteristics of squamous cell carcinoma has been reported. Researchers have found that tumors of different tissue origins have specific molecular characteristics associated with the tissue origin. By analyzing the gene variation spectrum, DNA methylation, gene expression spectrum or proteomics of tumor tissues, it can be used to determine the tissue origin of tumors including squamous cell carcinoma.

[0006] However, the above method has the following problems:

[0007] 1) It requires complex experimental operations and has almost stringent requirements on equipment, personnel and laboratory environment, which greatly increases the difficulty and cost of implementation.

[0008] 2) The detection cycle is long, often taking weeks or even months, which is not conducive to timely treatment of patients.

[0009] 3) Expensive. Gene variation profiles, DNA methylation, gene expression profiles, or proteomics are not routinely available due to the high cost and complex infrastructure required for testing.

[0010] 4) Accuracy is affected by many factors, such as the quality of the sample, the standardization of the experimental operation, etc., which may lead to inaccurate results.

[0011] With the development of artificial intelligence, deep learning-based digital pathology image analysis can also be used to determine the tissue origin of squamous cell carcinoma. By designing a targeted network architecture and autonomous learning of H&E pathology images, the trained neural network model can be directly predicted through pathology images, which makes up for the shortcomings of traditional methods to a certain extent. Ming et al. developed an artificial intelligence model (TOAD) based on digital pathology images to identify 18 different tumor types, with an overall compliance rate of 83% and 61% in the validation set and CUP cohort, respectively. At the same time, the study also evaluated the performance of the TOAD model in the differential diagnosis of four types of squamous cell carcinoma (head and neck, lung, cervix, and esophagus), with an overall compliance rate of approximately 80%.

[0012] However, this series of methods also has the following problems:

[0013] 1) The model structure is too simple. For the classification of squamous cell carcinoma, only an end-to-end multi-classification model is used for prediction, which is unable to distinguish pathologies that are highly similar in tissue morphology and immune phenotype.

[0014] 2) Easily affected by adjacent tissues: Without distinguishing between cancer cells and normal tissues, the model is easily affected by normal tissues.

[0015] 3) The prediction accuracy is not enough to achieve clinical application. These limitations of the current algorithm also make it difficult for its classification results to meet the requirements of clinical application. Summary of the invention

[0016] In order to solve the above technical problems, the present invention proposes a tumor tissue tracing method and device based on deep learning.

[0017] In order to achieve the above object, the technical solution of the present invention is as follows:

[0018] In one aspect, the present invention discloses a method for tracing tumor tissue origin based on pathological images, comprising:

[0019] Step S1: Classify tumor tissues according to their origins and divide them into k tumor origin categories;

[0020] Step S2: Collect a large number of pathological images, their tumor tissue traceability labels, and patient gender information;

[0021] Step S3: Divide each collected pathological image into several image blocks and preprocess them;

[0022] Step S4: using the preprocessed image blocks to train a multi-classification model, the multi-classification model is used to predict the probability of each tumor origin category to which the image block belongs based on the image block;

[0023] Step S5: Based on the prior knowledge of the patient's gender, k category prediction models are trained using the preprocessed image blocks. One category prediction model corresponds to one tumor origin category.

[0024] The category prediction model is used to predict the distance between an image block and a standard image block of the corresponding tumor origin category;

[0025] Step S6: Use the trained multi-classification model and category prediction model to predict the new pathological image and determine its tumor tissue origin, that is, the tumor origin category to which it belongs, including:

[0026] Step S6.1: Collect pathological images of the patient to be tested and the patient's gender information;

[0027] Step S6.2: Divide the pathological image of the patient to be detected into a number of image blocks and perform preprocessing;

[0028] Step S6.3: Input the preprocessed image block into the multi-classification model to predict the probability of each tumor origin category to which it belongs, and obtain the probability Pt, t=1,2,…k;

[0029] Step S6.4: input the preprocessed image blocks into k category prediction models respectively, predict the distance between them and the standard image blocks of the corresponding tumor origin category, and obtain dt, t=1,2,…k;

[0030] Step S6.5: Calculate the comprehensive score corresponding to each tumor origin category;

[0031] St = Pt*wt / dt;

[0032] Where: Pt is the probability of belonging to the tth tumor origin category;

[0033] wt is the corresponding weight between the patient's gender and the tth tumor origin category;

[0034] dt is the distance from the standard image patch of the t-th tumor origin category;

[0035] St is the comprehensive score of the tth tumor origin category;

[0036] Step S6.6: Rank the comprehensive scores, and the tumor origin category corresponding to the highest comprehensive score is determined as the tumor tissue origin of the patient.

[0037] Based on the above technical solution, the following improvements can be made:

[0038] As a preferred solution, step S3 includes:

[0039] Step S3.1: Divide each pathological image into a number of image blocks according to a fixed size, and each image block has the same resolution;

[0040] Step S3.2: Filter out qualified image blocks from all image blocks according to the screening conditions;

[0041] Step S3.3: Classify the qualified image blocks into tumor image blocks and normal tissue image blocks according to their contents.

[0042] As a preferred solution, step S4 includes:

[0043] Step S4.1: extracting features from the tumor image block and the normal tissue image block of each patient to obtain tumor features and normal tissue features;

[0044] Step S4.2: combining the tumor features and normal tissue features of each patient to form a feature package;

[0045] Step S4.3: Train a multi-classification model using several feature packages.

[0046] As a preferred solution, the loss function used by the multi-classification model is as follows:

[0047] ;

[0048] Among them: Loss is the loss function value;

[0049] n is the sample size;

[0050] k is the number of tumor origin categories;

[0051] is the probability that the i-th sample in the training set is of category j;

[0052] The probability of predicting the i-th sample in the training set as category j.

[0053] As a preferred solution, step S5 includes:

[0054] Step S5.1: selecting data from the tumor image blocks of the patient corresponding to one of the tumor origin categories as in-distribution data;

[0055] According to the relationship between tumor origin category and patient gender, data is selected from tumor image blocks of patients corresponding to other tumor origin categories, and data is selected from normal tissue image blocks of all patients, and mixed as out-of-distribution data;

[0056] Step S5.2: extracting features from in-distribution data and out-of-distribution data;

[0057] Step S5.3: using the extracted features, training a category prediction model corresponding to the tumor origin category;

[0058] Step S5.4: Repeat steps S5.1 to S5.3 until the category prediction models corresponding to all tumor origin categories are trained.

[0059] In addition, on the other hand, the present invention also discloses a deep learning tumor tissue tracing device, comprising:

[0060] The origin classification module is used to classify tumor tissues according to their origins and divide them into k tumor origin categories;

[0061] The collection module is used to collect a large number of pathological images, their tumor tissue traceability labels and the patient's gender information; the preprocessing module is used to divide each collected pathological image into several image blocks and preprocess them;

[0062] The first training module is used to train a multi-classification model using the preprocessed image blocks, and the multi-classification model is used to predict the probability of each tumor origin category to which the image block belongs based on the image block;

[0063] The second training module is used to train k category prediction models based on the prior knowledge of the patient's gender using the preprocessed image patches. One category prediction model corresponds to one tumor origin category.

[0064] The category prediction model is used to predict the distance between an image block and a standard image block of the corresponding tumor origin category;

[0065] The application module is used to use the trained multi-classification model and category prediction model to predict new pathological images and determine their tumor tissue origin, that is, the tumor origin category to which they belong, including:

[0066] A collecting unit, used to collect pathological images of patients to be tested and gender information of the patients;

[0067] A preprocessing unit, used for dividing the pathological image of the patient to be detected into a plurality of image blocks and performing preprocessing;

[0068] The first prediction unit is used to input the preprocessed image block into the multi-classification model to predict the probability of each tumor origin category to which it belongs, and obtain the probability Pt, t=1,2,…k;

[0069] The second prediction unit is used to input the preprocessed image blocks into k category prediction models respectively, predict the distance between the image blocks and the standard image blocks of the corresponding tumor origin category, and obtain dt, t=1, 2, ...k;

[0070] A calculation unit, used to calculate the comprehensive score corresponding to each tumor origin category;

[0071] St = Pt*wt / dt;

[0072] Where: Pt is the probability of belonging to the tth tumor origin category;

[0073] wt is the corresponding weight between the patient's gender and the tth tumor origin category;

[0074] dt is the distance from the standard image patch of the t-th tumor origin category;

[0075] St is the comprehensive score of the tth tumor origin category;

[0076] The determination unit is used to rank the comprehensive scores, and the tumor origin category corresponding to the highest comprehensive score is determined as the tumor tissue origin of the patient.

[0077] As a preferred solution, the preprocessing module includes:

[0078] A division unit, used for dividing each pathological image into a plurality of image blocks according to a fixed size, and each image block has the same resolution;

[0079] A screening unit, used for screening qualified image blocks from all image blocks according to screening conditions;

[0080] The classification unit is used to classify the qualified image blocks into tumor image blocks and normal tissue image blocks according to their contents.

[0081] As a preferred solution, the first training module includes:

[0082] A first feature extraction unit is used to extract features from the tumor image block and the normal tissue image block of each patient to obtain tumor features and normal tissue features;

[0083] A combination unit, used for combining the tumor features and normal tissue features of each patient to form a feature package;

[0084] The first training unit is used to train a multi-classification model using a plurality of feature packages.

[0085] As a preferred solution, the loss function used by the multi-classification model is as follows:

[0086] ;

[0087] Among them: Loss is the loss function value;

[0088] n is the sample size;

[0089] k is the number of tumor origin categories;

[0090] is the probability that the i-th sample in the training set is of category j;

[0091] The probability of predicting the i-th sample in the training set as category j.

[0092] As a preferred solution, the second training module includes:

[0093] A training data collection unit, used for selecting data from a tumor image block of a patient corresponding to one of the tumor origin categories as in-distribution data;

[0094] According to the relationship between tumor origin category and patient gender, data is selected from tumor image blocks of patients corresponding to other tumor origin categories, and data is selected from normal tissue image blocks of all patients, and mixed as out-of-distribution data;

[0095] A second feature extraction unit is used to extract features from the in-distribution data and the out-of-distribution data;

[0096] The second training unit is used to train a category prediction model corresponding to the tumor origin category by using the features extracted from the features;

[0097] The repetitive execution unit is used to repeatedly execute the methods in the training data collection unit, the second feature extraction unit, and the second training unit until the category prediction models corresponding to all tumor origin categories are trained.

[0098] The present invention discloses a tumor tissue tracing method and device based on deep learning, which can determine the origin of tumor tissue based on pathological images and determine the tumor origin category to which it belongs.

[0099] The present invention has the following beneficial effects:

[0100] First, simplify the operation. The present invention uses a deep learning method to predict the tissue origins of unknown primary and multiple primary tumors through pathological images, which greatly simplifies the experimental operation and reduces the high requirements for equipment, personnel and laboratory environment, making this technology easier to implement.

[0101] Second, rapid detection. Compared with the existing technology which requires a detection cycle of several weeks or even months, the present invention can obtain the detection results in a short time, greatly improving the detection efficiency and facilitating the timely treatment of patients.

[0102] Third, improve accuracy. (1) The present invention adopts a deep learning method, which can train and learn a large amount of pathological image data, thereby improving the accuracy of prediction. (2) Since the present invention does not require complex experimental operations, it reduces the influence of human factors and further improves the accuracy of the results. (3) The present invention combines the differences between male and female patients. Some tumors may only occur in the corresponding gender. By using this clinical prior knowledge, the data set can be made more accurate and the model can be prevented from learning more tasks, thereby making the model's prediction accuracy higher and further improving the accuracy of the overall model.

[0103] Fourth, reducing costs. Since the present invention simplifies experimental operations and reduces the high requirements for equipment, personnel and laboratory environment, compared with the prior art, the present invention can significantly reduce the cost of implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0105] Figure 1 A flow chart of a tumor tissue tracing method provided in an embodiment of the present invention.

[0106] Figure 2 This is a flowchart of step S6 provided in an embodiment of the present invention.

[0107] Figure 3 This is an example diagram of the preprocessing of a pathological image provided by an embodiment of the present invention.

[0108] Figure 4 A schematic diagram of training a multi-classification model provided in an embodiment of the present invention.

[0109] Figure 5 A schematic diagram of training a category prediction model provided in an embodiment of the present invention.

[0110] Figure 6 A schematic diagram of a confusion matrix provided by an embodiment of the present invention.

[0111] Figure 7 A schematic diagram of AUROC provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0112] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0113] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0114] Using ordinal numbers “first,” “second,” “third,” etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0115] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0116] In order to achieve the purpose of the present invention, in some embodiments of a method and device for tracing the origin of tumor tissue based on deep learning, as Figure 1 As shown, the tumor tissue tracing method for determining squamous cell carcinoma tissue includes:

[0117] Step S1: Classify tumor tissues according to their origins and divide them into k tumor origin categories;

[0118] Step S2: Collect a large number of pathological images, their tumor tissue traceability labels, and patient gender information;

[0119] Step S3: Divide each collected pathological image into several image blocks and preprocess them;

[0120] Step S4: using the preprocessed image blocks to train a multi-classification model, the multi-classification model is used to predict the probability of each tumor origin category to which the image block belongs based on the image block;

[0121] Step S5: Based on the prior knowledge of the patient's gender, k category prediction models are trained using the preprocessed image blocks. One category prediction model corresponds to one tumor origin category.

[0122] The category prediction model is used to predict the distance between an image block and a standard image block of the corresponding tumor origin category;

[0123] Step S6: Use the trained multi-classification model and category prediction model to predict the new pathological image and determine its tumor tissue origin, that is, the tumor origin category to which it belongs, such as Figure 2 As shown, including:

[0124] Step S6.1: Collect pathological images of the patient to be tested and the patient's gender information;

[0125] Step S6.2: Divide the pathological image of the patient to be detected into a number of image blocks and perform preprocessing;

[0126] Step S6.3: Input the preprocessed image block into the multi-classification model to predict the probability of each tumor origin category to which it belongs, and obtain the probability Pt, t=1,2,…k;

[0127] Step S6.4: input the preprocessed image blocks into k category prediction models respectively, predict the distance between them and the standard image blocks of the corresponding tumor origin category, and obtain dt, t=1,2,…k;

[0128] Step S6.5: Calculate the comprehensive score corresponding to each tumor origin category;

[0129] St = Pt*wt / dt;

[0130] Where: Pt is the probability of belonging to the tth tumor origin category;

[0131] wt is the corresponding weight between the patient's gender and the tth tumor origin category;

[0132] dt is the distance from the standard image patch of the t-th tumor origin category;

[0133] St is the comprehensive score of the tth tumor origin category;

[0134] Step S6.6: Rank the comprehensive scores, and the tumor origin category corresponding to the highest comprehensive score is determined as the tumor tissue origin of the patient.

[0135] Each step of the present invention is described in detail below.

[0136] Step S1 is a classification step, in which tumor tissues are classified according to their origins into six tumor origin categories, specifically: head and neck squamous cell carcinoma, esophageal squamous cell carcinoma, cervical squamous cell carcinoma, nasopharyngeal carcinoma, lung squamous cell carcinoma and urothelial carcinoma.

[0137] Step S2 is used to collect data, including a large amount of pathological image data. The pathological images should cover various types of tumor tissues, including pathological slice images of unknown primary and multiple primary squamous cell carcinoma tissues, and their tissue traceability labels. At the same time, due to the physiological differences between men and women, in order to further improve the performance of the model, the patient's gender information is collected simultaneously when collecting data.

[0138] Specifically, step S2 collects H&E stained pathological section images (whole-field pathological images WSI) of patients with squamous cell carcinoma, as well as clinical diagnosis results of their tissue tracing.

[0139] In a specific embodiment, a total of 600 patients with head and neck squamous cell carcinoma, esophageal squamous cell carcinoma, cervical squamous cell carcinoma, nasopharyngeal carcinoma, lung squamous cell carcinoma and urothelial carcinoma were collected, and the gender information of each patient was collected.

[0140] For each type of case, the collected data set was divided into a training set and a test set in a ratio of 70%:30%. 420 patients were included in the training set and 180 patients were included in the test set.

[0141] For the 420 patients in the training set, data preprocessing was performed, such as Figure 3 As shown, step S3 includes:

[0142] Step S3.1: Divide each pathological image into a number of image tiles according to a fixed size, and each image tile has the same resolution;

[0143] For example, the physical size of each image tile is 256um in length and width, and the classification rate of the image tile is 512x512, that is, the tiles are normalized to 0.5um / pixel;

[0144] Step S3.2: Filter out qualified image tiles from all image tiles according to the screening conditions;

[0145] Step S3.3: Classify the qualified image tiles into tumor tiles and normal tissue tiles according to their contents.

[0146] In step S3.2, qualified image tiles are screened out from all image tiles according to the screening conditions, and quality control is performed on each image tile to delete image tiles with low tissue ratio, handwriting, blur, and dirt.

[0147] The specific screening methods are as follows:

[0148] First, exclude image tiles whose tissue ratio does not exceed a threshold (e.g., 0.5);

[0149] Secondly, the Laplace operator is used to eliminate blurred image tiles;

[0150] Finally, exclude tiles containing handwriting or dirt;

[0151] The remaining image blocks are qualified image blocks tiles.

[0152] Step S3.3 divides the qualified image tiles into tumor tiles and normal tissue tiles according to their contents.

[0153] Step S3.3 can be performed by pathologists to circle out the tumor tissue in advance, or it can be classified through a pre-trained model, which will not be elaborated here.

[0154] Step S4 and step S5 use the preprocessed data for training. The training is divided into two stages. The first stage is the coarse classification stage, which trains the multi-classification model. The second stage is the training of the category prediction model for a single tumor tissue, which is used to calculate the distance between the tumor image block and the training set.

[0155] like Figure 4 As shown, step S4 includes:

[0156] Step S4.1: extract features from tumor tiles and normal tissue tiles of each patient to obtain tumor features and normal tissue features;

[0157] Step S4.2: combining the tumor features and normal tissue features of each patient to form a feature bag F_bag, and saving it in h5 file format;

[0158] Step S4.3: Each feature package is a sample and is input into the multi-classification model for training.

[0159] Because there are 6 tumor origin categories, the multi-classification model is a six-classification model. Further, the above multi-classification model can be, but is not limited to, a MIL model. Multi-Instance Learning (MIL) is a common technique for weakly supervised learning in deep learning, so it will not be described here. In this example, the MIL classifier used consists of an attention Backbone and an MLP (multi-layer perceptron) to classify samples into 6 categories.

[0160] The loss function used by the multi-classification model is as follows:

[0161] ;

[0162] Among them: Loss is the loss function value;

[0163] n is the sample size;

[0164] k is the number of tumor origin categories, k=6, which are head and neck squamous cell carcinoma, esophageal squamous cell carcinoma, cervical squamous cell carcinoma, nasopharyngeal carcinoma, lung squamous cell carcinoma and urothelial carcinoma;

[0165] is the probability that the i-th sample in the training set is of category j;

[0166] The probability of predicting the i-th sample in the training set as category j.

[0167] like Figure 5 As shown, step S5 includes:

[0168] Step S5.1: Select data from the tumor tiles of the patient corresponding to one of the tumor origin categories as in-distribution data;

[0169] According to the relationship between tumor origin category and patient gender, data is selected from tumor tiles corresponding to patients of other tumor origin categories, and data is selected from normal tissue tiles of all patients. After mixing, the data is used as out-of-distribution data.

[0170] Further, it is worth noting that for the tumor origin category that may only occur in a certain gender, data are selected only from the tumor image patches and normal tissue image patches corresponding to that gender as out-of-distribution data;

[0171] For example, head and neck squamous cell carcinoma has little correlation with the patient's gender, and both men and women are at risk of cancer.

[0172] Collect data from tumor tiles of head and neck squamous cell carcinoma patients as in-distribution data;

[0173] Collect data from tumor tiles of another 5 tumor origin categories and normal tissue tiles of 6 tumor origin categories, mix them together and use them as out-of-distribution data;

[0174] Furthermore, for example, cervical squamous cell carcinoma is a type of cancer that is unique to female patients. When preparing out-of-distribution data for cervical squamous cell carcinoma, in order to reduce the difficulty of model training, only the data of female patients can be collected. Then:

[0175] Collect data from tumor tiles of patients with cervical squamous cell carcinoma as in-distribution data;

[0176] Collect data from tumor tiles of women with another 5 tumor origin categories and normal tissue tiles of women with 6 tumor origin categories, and mix them together as out-of-distribution data;

[0177] Step S5.2: extracting features from in-distribution data and out-of-distribution data;

[0178] Step S5.3: using the extracted features, training a category prediction model corresponding to the tumor origin category;

[0179] Step S5.4: Repeat steps S5.1 to S5.3 until the category prediction models corresponding to all tumor origin categories are trained.

[0180] The above-mentioned category prediction model may be, but is not limited to, an OoD model.

[0181] When selecting data in step S5.1, in order to ensure the accuracy of the model, the number of data within the distribution should not be less than the number of data outside the distribution.

[0182] Step S6 is to use the trained model to predict the new squamous cell carcinoma pathology image and determine its tumor tissue origin, that is, the tumor origin category to which it belongs.

[0183] Step S6 includes:

[0184] Step S6.1: Collect pathological images of the patient to be tested and the patient's gender information;

[0185] Step S6.2: Divide the pathological image of the patient to be detected into a number of image tiles and perform preprocessing. The preprocessing operation is similar to the above, and tumor tiles and normal tissue tiles are found;

[0186] Extract features from tumor tiles and normal tissue tiles;

[0187] The features extracted from tumor tiles and normal tissue tiles are combined to form a feature bag F_bag;

[0188] After the features of tumor tiles are extracted, features are obtained;

[0189] Step S6.3: Input the feature bag F_bag into the multi-classification model to predict the probability of each tumor origin category to which it belongs, and obtain the probabilities P1, P2, P3, P4, P5, and P6;

[0190] Step S6.4: Input the features Features into the six category prediction models respectively, predict the distance between them and the standard image blocks of the corresponding tumor origin categories, and obtain d1, d2, d3, d4, d5, and d6;

[0191] Among them: the standard image block is the image block in the training set;

[0192] Step S6.5: Calculate the comprehensive score corresponding to each tumor origin category;

[0193] St = Pt*wt / dt;

[0194] Where: Pt is the probability of belonging to the tth tumor origin category;

[0195] wt is the corresponding weight between the patient's gender and the tth tumor origin category;

[0196] dt is the distance from the standard image patch of the t-th tumor origin category;

[0197] St is the comprehensive score of the tth tumor origin category;

[0198] Step S6.6: Rank the comprehensive scores, and the tumor origin category corresponding to the highest comprehensive score is determined as the tumor tissue origin of the patient.

[0199] Among the above six types of cancer, cervical squamous cell carcinoma is a cancer unique to female patients. The weight factor w = 0 for cervical squamous cell carcinoma in male patients, and the weight factor w = 1 for cervical squamous cell carcinoma in female patients.

[0200] Head and neck squamous cell carcinoma, esophageal squamous cell carcinoma, nasopharyngeal carcinoma, lung squamous cell carcinoma and urothelial carcinoma are types of cancer that may occur in both male and female patients, so the weight factor w=1.

[0201] In the test set of 180 squamous cell carcinoma FFPE specimens divided in this dataset, the prediction results were compared with the gold standard pathological diagnosis, such as Figure 6 , 7 As shown: Different types of AUROC (Area Under the Receiver Operating Characteristic Curve) are the areas under the ROC curve, which are used to evaluate the performance of the binary classification model, which are: 0.9207 (head and neck squamous cell carcinoma), 0.9207 (esophageal squamous cell carcinoma), 0.9467 (cervical squamous cell carcinoma), 0.9513 (nasopharyngeal carcinoma), 0.9056 (lung squamous cell carcinoma) and 0.9982 (urothelial carcinoma).

[0202] Furthermore, in some embodiments, in order to simplify the computational complexity, the computational time is shortened.

[0203] Step S6 includes:

[0204] Step S6.1: Collect pathological images of the patient to be tested and the patient's gender information;

[0205] Step S6.2: Divide the pathological image of the patient to be detected into a number of image blocks and perform preprocessing;

[0206] Step S6.3: input the preprocessed image block into the multi-classification model to predict the probability of each tumor origin category to which it belongs;

[0207] Rank the predicted probabilities of each tumor origin category, and select the category prediction models corresponding to the tumor origin categories with the top probabilities (e.g., 3, specifically: P1, P2, P3);

[0208] Step S6.4: input the preprocessed image blocks into the three selected category prediction models respectively, predict the distance between them and the standard image blocks of the corresponding tumor origin category, and obtain d1, d2, and d3;

[0209] Step S6.5: Calculate the comprehensive scores corresponding to the three selected tumor origin categories;

[0210] S1 = P1*w1 / d1;

[0211] S2= P2*w2 / d2;

[0212] S3 = P3*w3 / d3;

[0213] Step S6.6: Rank the comprehensive scores, and the tumor origin category corresponding to the highest comprehensive score is determined as the tumor tissue origin of the patient.

[0214] The present invention classifies tumor tissues according to their origins and divides them into several tumor origin categories. For those skilled in the art, they would not think of classifying according to origins. The present invention adopts a multi-stage model. In the first stage, a multi-classification model is trained for classification. In the second stage, a category prediction model for different tumor origin categories is used for judgment to improve the prediction accuracy.

[0215] The present invention determines the tumor origin category through a multi-stage model, with high prediction accuracy, low cost and short cycle, and is of great significance in application fields such as medical image processing, tumor diagnosis and bioinformatics. The details are as follows:

[0216] First, the present invention uses pathological images and a multi-stage deep learning method to predict the tissue origin of unknown primary and multiple primary squamous cell carcinomas, avoiding the complex experimental operations and high costs in traditional methods, and greatly improving the efficiency and accuracy of diagnosis. Secondly, the present invention effectively solves the problems of existing methods based on analyzing the gene variation spectrum, DNA methylation, gene expression spectrum or proteomics of tumor tissues, such as high experimental difficulty, long cycle and high cost.

[0217] Then, the present invention uses a multi-stage deep learning method to process different tasks at different stages, thereby avoiding the interference of normal tissue on the model and improving the accuracy of model prediction.

[0218] Finally, the present invention has relatively low requirements on equipment, personnel and laboratory environment, which is conducive to its promotion and application in more medical institutions.

[0219] Due to the high incidence of tumors and the threat to human health, the present invention is widely needed in the fields of clinical diagnosis and treatment, scientific research, and public health.

[0220] In summary, the present invention can change the current situation that only wet experiments can be used to trace the tissue origin of unknown primary and multiple primary squamous cell carcinomas. It uses a multi-stage model and adopts different training strategies at different stages, which greatly improves the prediction accuracy of the model. This technical invention can greatly reduce the workload of pathologists and improve their work efficiency, and has significant advantages and broad application prospects.

[0221] The embodiment of the present invention further discloses a tumor tissue tracing device based on pathological images, comprising:

[0222] The origin classification module is used to classify tumor tissues according to their origins and divide them into k tumor origin categories;

[0223] The collection module is used to collect a large number of pathological images, their tumor tissue traceability labels, and the patient's gender information;

[0224] A preprocessing module, used for dividing each collected pathological image into a number of image blocks and preprocessing them;

[0225] The first training module is used to train a multi-classification model using the preprocessed image blocks, and the multi-classification model is used to predict the probability of each tumor origin category to which the image block belongs based on the image block;

[0226] The second training module is used to train k category prediction models based on the prior knowledge of the patient's gender using the preprocessed image patches. One category prediction model corresponds to one tumor origin category.

[0227] The category prediction model is used to predict the distance between an image block and a standard image block of the corresponding tumor origin category;

[0228] The application module is used to use the trained multi-classification model and category prediction model to predict new pathological images and determine their tumor tissue origin, that is, the tumor origin category to which they belong, including:

[0229] A collecting unit, used to collect pathological images of patients to be tested and gender information of the patients;

[0230] A preprocessing unit, used for dividing the pathological image of the patient to be detected into a plurality of image blocks and performing preprocessing;

[0231] The first prediction unit is used to input the preprocessed image block into the multi-classification model to predict the probability of each tumor origin category to which it belongs, and obtain the probability Pt, t=1,2,…k;

[0232] The second prediction unit is used to input the preprocessed image blocks into k category prediction models respectively, predict the distance between the image blocks and the standard image blocks of the corresponding tumor origin category, and obtain dt, t=1, 2, ...k;

[0233] A calculation unit, used to calculate the comprehensive score corresponding to each tumor origin category;

[0234] St = Pt*wt / dt;

[0235] Where: Pt is the probability of belonging to the tth tumor origin category;

[0236] wt is the corresponding weight between the patient's gender and the tth tumor origin category;

[0237] dt is the distance from the standard image patch of the t-th tumor origin category;

[0238] St is the comprehensive score of the tth tumor origin category;

[0239] The determination unit is used to rank the comprehensive scores, and the tumor origin category corresponding to the highest comprehensive score is determined as the tumor tissue origin of the patient.

[0240] Furthermore, the preprocessing module includes:

[0241] A division unit, used for dividing each pathological image into a plurality of image blocks according to a fixed size, and each image block has the same resolution;

[0242] A screening unit, used for screening qualified image blocks from all image blocks according to screening conditions;

[0243] The classification unit is used to classify the qualified image blocks into tumor image blocks and normal tissue image blocks according to their contents.

[0244] As a preferred solution, the first training module includes:

[0245] A first feature extraction unit is used to extract features from the tumor image block and the normal tissue image block of each patient to obtain tumor features and normal tissue features;

[0246] A combination unit, used for combining the tumor features and normal tissue features of each patient to form a feature package;

[0247] The first training unit is used to train a multi-classification model using a plurality of feature packages.

[0248] As a preferred solution, the loss function used by the multi-classification model is as follows:

[0249] ;

[0250] Among them: Loss is the loss function value;

[0251] n is the sample size;

[0252] k is the number of tumor origin categories;

[0253] is the probability that the i-th sample in the training set is of category j;

[0254] The probability of predicting the i-th sample in the training set as category j.

[0255] As a preferred solution, the second training module includes:

[0256] A training data collection unit, used for selecting data from a tumor image block of a patient corresponding to one of the tumor origin categories as in-distribution data;

[0257] According to the relationship between tumor origin category and patient gender, data is selected from tumor image blocks of patients corresponding to other tumor origin categories, and data is selected from normal tissue image blocks of all patients, and mixed as out-of-distribution data;

[0258] A second feature extraction unit is used to extract features from the in-distribution data and the out-of-distribution data;

[0259] The second training unit is used to train a category prediction model corresponding to the tumor origin category by using the features extracted from the features;

[0260] The repetitive execution unit is used to repeatedly execute the methods in the training data collection unit, the second feature extraction unit, and the second training unit until the category prediction models corresponding to all tumor origin categories are trained.

[0261] The present invention discloses a tumor tissue tracing method and device based on deep learning, which can determine the origin of tumor tissue based on pathological images and determine the tumor origin category to which it belongs.

[0262] The present invention has the following beneficial effects:

[0263] First, simplify the operation. The present invention uses a deep learning method to predict the tissue origins of unknown primary and multiple primary tumors through pathological images, which greatly simplifies the experimental operation and reduces the high requirements for equipment, personnel and laboratory environment, making this technology easier to implement.

[0264] Second, rapid detection. Compared with the existing technology which requires a detection cycle of several weeks or even months, the present invention can obtain the detection results in a short time, greatly improving the detection efficiency and facilitating the timely treatment of patients.

[0265] Third, improve accuracy. (1) The present invention adopts a deep learning method, which can train and learn a large amount of pathological image data, thereby improving the accuracy of prediction. (2) Since the present invention does not require complex experimental operations, it reduces the influence of human factors and further improves the accuracy of the results. (3) The present invention combines the differences between male and female patients. Some tumors may only occur in the corresponding gender. By using this clinical prior knowledge, the data set can be made more accurate and the model can be prevented from learning more tasks, thereby making the model's prediction accuracy higher and further improving the accuracy of the overall model.

[0266] Fourth, reducing costs. Since the present invention simplifies experimental operations and reduces the high requirements for equipment, personnel and laboratory environment, compared with the prior art, the present invention can significantly reduce the cost of implementation.

[0267] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or portions of the method and apparatus of the present invention, can take the form of program codes (i.e., instructions) embedded in a tangible medium, such as a floppy disk, a CD-ROM, a hard disk drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0268] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A tumor tissue tracing method based on deep learning, characterized in that: include: Step S1: Classify tumor tissues according to their origins and divide them into k tumor origin categories; Step S2: Collect a large number of pathological images, their tumor tissue traceability labels, and patient gender information; Step S3: Divide each collected pathological image into several image blocks and preprocess them; The step S3 comprises: Step S3.1: Divide each pathological image into a number of image blocks according to a fixed size, and each image block has the same resolution; Step S3.2: Filter out qualified image blocks from all image blocks according to the screening conditions; Step S3.3: Classify the qualified image blocks into tumor image blocks and normal tissue image blocks according to their contents; Step S4: using the preprocessed image blocks to train a multi-classification model, wherein the multi-classification model is used to predict the probability of each tumor origin category to which the image block belongs based on the image block; The step S4 comprises: Step S4.1: extracting features from the tumor image block and the normal tissue image block of each patient to obtain tumor features and normal tissue features; Step S4.2: combining the tumor features and normal tissue features of each patient to form a feature package; Step S4.3: training a multi-classification model using several feature packages; Step S5: Based on the prior knowledge of the patient's gender, k category prediction models are trained using the preprocessed image blocks. One category prediction model corresponds to one tumor origin category. The category prediction model is used to predict the distance between the image block and the standard image block of the corresponding tumor origin category based on the image block; The step S5 comprises: Step S5.1: selecting data from the tumor image blocks of the patient corresponding to one of the tumor origin categories as in-distribution data; According to the relationship between tumor origin category and patient gender, data is selected from tumor image blocks of patients corresponding to other tumor origin categories, and data is selected from normal tissue image blocks of all patients, and mixed as out-of-distribution data; Step S5.2: extracting features from in-distribution data and out-of-distribution data; Step S5.3: using the extracted features, training a category prediction model corresponding to the tumor origin category; Step S5.4: repeat steps S5.1 to S5.3 until the category prediction models corresponding to all tumor origin categories are trained; Step S6: Use the trained multi-classification model and category prediction model to predict the new pathological image and determine its tumor tissue origin, that is, the tumor origin category to which it belongs; The step S6 comprises: Step S6.1: Collect pathological images of the patient to be tested and the patient's gender information; Step S6.2: Divide the pathological image of the patient to be detected into a number of image blocks and perform preprocessing; Step S6.3: Input the preprocessed image block into the multi-classification model to predict the probability of each tumor origin category to which it belongs, and obtain the probability Pt, t = 1, 2, ... k; Step S6.4: input the preprocessed image blocks into k category prediction models respectively, predict the distance between them and the standard image blocks of the corresponding tumor origin category, and obtain dt, t = 1, 2, ... k; Step S6.5: Calculate the comprehensive score corresponding to each tumor origin category; St = Pt*wt / dt; Where: Pt is the probability of belonging to the tth tumor origin category; wt is the corresponding weight between the patient's gender and the tth tumor origin category; dt is the distance from the standard image patch of the t-th tumor origin category; St is the comprehensive score of the tth tumor origin category; Step S6.6: Rank the comprehensive scores, and the tumor origin category corresponding to the highest comprehensive score is determined as the tumor tissue origin of the patient.

2. The tumor tissue tracing method according to claim 1, characterized in that: The loss function used by the multi-classification model is as follows: Among them: Loss is the loss function value; n is the sample size; k is the number of tumor origin categories; Y ij is the probability that the i-th sample in the training set is of category j; y ij The probability of predicting the i-th sample in the training set as category j.

3. Tumor tissue tracing equipment based on deep learning, characterized by: include: The origin classification module is used to classify tumor tissues according to their origins and divide them into k tumor origin categories; The collection module is used to collect a large number of pathological images, their tumor tissue traceability labels, and the patient's gender information; A preprocessing module, used for dividing each collected pathological image into a number of image blocks and preprocessing them; The preprocessing module comprises: A division unit, used for dividing each pathological image into a plurality of image blocks according to a fixed size, and each image block has the same resolution; A screening unit, used for screening qualified image blocks from all image blocks according to screening conditions; A classification unit, used for classifying qualified image blocks into tumor image blocks and normal tissue image blocks according to the contents thereof; A first training module is used to train a multi-classification model using the preprocessed image blocks, wherein the multi-classification model is used to predict the probability of each tumor origin category to which the image block belongs based on the image block; The first training module includes: A first feature extraction unit is used to extract features from the tumor image block and the normal tissue image block of each patient to obtain tumor features and normal tissue features; A combination unit, used for combining the tumor features and normal tissue features of each patient to form a feature package; A first training unit is used to train a multi-classification model using a plurality of feature packages; The second training module is used to train k category prediction models based on the prior knowledge of the patient's gender using the preprocessed image patches. One category prediction model corresponds to one tumor origin category. The category prediction model is used to predict the distance between the image block and the standard image block of the corresponding tumor origin category based on the image block; The second training module includes: A training data collection unit, used for selecting data from a tumor image block of a patient corresponding to one of the tumor origin categories as in-distribution data; According to the relationship between tumor origin category and patient gender, data is selected from tumor image blocks of patients corresponding to other tumor origin categories, and data is selected from normal tissue image blocks of all patients, and mixed as out-of-distribution data; A second feature extraction unit is used to extract features from the in-distribution data and the out-of-distribution data; The second training unit is used to train a category prediction model corresponding to the tumor origin category by using the features extracted from the features; A repeated execution unit, used to repeatedly execute the methods in the training data collection unit, the second feature extraction unit, and the second training unit until the category prediction models corresponding to all tumor origin categories are trained; The application module is used to use the trained multi-classification model and category prediction model to predict new pathological images and determine their tumor tissue origin, that is, the tumor origin category to which they belong, including: A collecting unit, used to collect pathological images of patients to be tested and gender information of the patients; A preprocessing unit, used for dividing the pathological image of the patient to be detected into a plurality of image blocks and performing preprocessing; The first prediction unit is used to input the preprocessed image block into the multi-classification model to predict the probability of each tumor origin category to which the image block belongs, and obtain the probability Pt, t = 1, 2, ...k; The second prediction unit is used to input the preprocessed image blocks into k category prediction models respectively, predict the distance between the image blocks and the standard image blocks of the corresponding tumor origin category, and obtain dt, t = 1, 2, ... k; A calculation unit, used to calculate the comprehensive score corresponding to each tumor origin category; St = Pt*wt / dt; Where: Pt is the probability of belonging to the tth tumor origin category; wt is the corresponding weight between the patient's gender and the tth tumor origin category; dt is the distance from the standard image patch of the t-th tumor origin category; St is the comprehensive score of the tth tumor origin category; The determination unit is used to rank the comprehensive scores, and the tumor origin category corresponding to the highest comprehensive score is determined as the tumor tissue origin of the patient.

4. The tumor tissue tracing device according to claim 3, characterized in that: The loss function used by the multi-classification model is as follows: Among them: Loss is the loss function value; n is the sample size; k is the number of tumor origin categories; Y ij is the probability that the i-th sample in the training set is of category j; y ij The probability of predicting the i-th sample in the training set as category j.