Tumor digital pathological section image management and diagnosis system based on deep learning

Abstract

The invention provides a tumor digital pathological section image management and diagnosis system based on deep learning, and relates to the technical field of image diagnosis. The tumor digital pathological section image management and diagnosis system based on deep learning comprises a data management module which comprises a data acquisition module, a data storage module and a data preprocessing module, wherein the data acquisition module is used for acquiring clinical data, a pathological report and a digital pathological section image of a patient from a hospital information system and a pathological information system; the data storage module is used for efficiently storing and managing massive case slice images by adopting a distributed storage technology; the data preprocessing module is used for cutting, normalizing and de-noising the case slice image, and performing data enhancement on the image at the same time; and through a visualization technology and a diagnosis interpretation module, a diagnosis basis is provided, the interpretability of a model prediction result is enhanced, and a pathologist is assisted to make more accurate diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of imaging diagnosis, and in particular to a tumor digital pathological slice image management and diagnosis system based on deep learning. Background Art

[0002] Tumors are new growths formed by the proliferation of local tissue cells. The acidity of tumors is slightly stronger than that of healthy body tissues. Tumors can form an "acid wall" to protect themselves from attacks by the immune system. According to the cell characteristics of the new growth and the degree of harm to the body, tumors are divided into two categories: benign tumors and malignant tumors. Malignant tumors can be divided into carcinomas and sarcomas. Carcinomas refer to malignant tumors originating from epithelial tissues. Sarcomas refer to malignant tumors that occur in mesenchymal tissues, including fibrous connective tissues, fat, muscles, blood vessels, bones and cartilage tissues. For example, malignant tumors formed by the mucosal epithelium of the large intestine are called mucosal epithelial carcinoma of the large intestine, or simply colon cancer. Those formed by the skin epithelium are called skin epithelial carcinoma, or simply skin cancer, and so on. Therefore, if a doctor says that someone has cancer, it means that the patient has a malignant tumor; if someone says that someone has gastric cancer, it means that the patient has a cancer formed by the gastric mucosal epithelium. If the patient is said to have gastric sarcoma, it means that this malignant tumor is not formed by mucosal epithelial cells, but may be caused by the malignant transformation of smooth muscle cells, or belong to malignant lymphoma of the stomach, etc. But it can also be said in general terms that he had cancer.

[0003] With the rapid development of artificial intelligence technology, deep learning has shown great application potential in the field of medical image analysis, especially in the early screening, diagnosis and treatment evaluation of tumors. Traditional tumor diagnosis mainly relies on the experience of pathologists, who observe pathological slice images under a microscope to identify and judge the type and malignancy of the tumor. This method has the following problems: it is highly subjective, and different pathologists may have different diagnostic results for the same pathological slice, which makes it difficult to ensure the consistency of the diagnostic results. The pathological diagnosis process relies on manual operation, which is inefficient and difficult to cope with the growing demand for pathological diagnosis. The diagnostic results of pathologists are highly dependent on their professional knowledge and experience. For inexperienced pathologists, the risk of misdiagnosis and missed diagnosis is high. Therefore, technicians in this field provide a tumor digital pathological slice image management and diagnosis system based on deep learning to solve the problems raised in the above background technology. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides a tumor digital pathology section image management and diagnosis system based on deep learning, which solves the problems of strong subjectivity, low efficiency and high experience dependence of traditional methods.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: A tumor digital pathology slice image management and diagnosis system based on deep learning, comprising:

[0008] The data management module includes a data acquisition module, a data storage module and a data preprocessing module. The data acquisition module is used to collect patients' clinical data, pathology reports and digital pathology slice images from the hospital information system and the pathology information system. The data storage module uses distributed storage technology to efficiently store and manage massive case slice images. The data preprocessing module performs cropping, normalization and denoising on the case slice images, and performs data enhancement on the images, including rotation, flipping and scaling operations, to expand the amount of training data and improve the generalization ability of the model.

[0009] The model training module includes a feature extraction module and a classifier training module. The pre-trained deep convolutional neural network is used as a feature extractor to extract high-dimensional features from the case slice images. The extracted features are input into the fully connected layer and the Softmax layer to construct a classifier. The classifier is trained using the cross entropy loss function and the optimization algorithm. The transfer learning technology is used to fine-tune the parameters of the pre-trained model to speed up the training and improve the model performance.

[0010] The model reasoning module includes an image segmentation module and a diagnosis prediction module, which segments the tumor area of ​​the pathological slice image, extracts the region of interest, inputs the ROI into the trained classifier, predicts the tumor type and malignancy, and outputs the diagnosis result, including information on the tumor type, malignancy, and diagnostic confidence;

[0011] The visualization and interpretation module includes a result visualization module and a diagnosis interpretation module. It visualizes the pathology slice images, segmentation results, and diagnosis results to facilitate pathologists to view and analyze. It provides an interactive interface to allow pathologists to modify and confirm the diagnosis results. It uses visualization technology to interpret the model prediction results and generates a heat map to indicate the image area that the model focuses on, providing a basis for diagnosis and assisting pathologists in making more accurate diagnoses.

[0012] The system management module, including the user management module, log management module and model management module, implements user registration, login and permission management functions, records system operation logs and user operation logs, facilitates system maintenance and troubleshooting, implements model version control, deployment and update, and supports online model training and inference services.

[0013] Preferably, the data management module comprises the following steps:

[0014] S1. Data collection, obtaining data from hospital information systems and pathology information systems: obtaining clinical information of patients from HIS, including age, gender, medical history and imaging examination results; obtaining pathology reports from PIS, including pathological diagnosis results, tumor type and malignancy, and obtaining digital pathology slice images from PIS, usually high-resolution WSI files; data cleaning: removing duplicate data, missing data and data that does not meet the requirements, and checking the consistency of data to ensure that data from different sources correspond to the same patient;

[0015] S2. Data storage, select a storage solution: Considering the huge amount of pathology image data, a distributed storage system is used, and a database is used to store the patient's clinical information and pathology report; Data storage: The cleaned WSI images are stored in a distributed file system, and a file index is established to facilitate subsequent query and management. The clinical information and pathology report are stored in the database and associated with the WSI image. Assuming that the average size of each WSI image is S WSI GB, storage N WSI The storage space required for each image is calculated as follows:

[0016] S total =S WSI ×N WSI

[0017] Assume that the average size of each record of clinical information and medical records stored in the database is S record KB, then store N clean The storage space required for each record is:

[0018] S DB =S record ×N clean / 1024

[0019] S3. Data preprocessing: perform preprocessing operations on images through cropping, normalization and denoising, enhance data through rotation, flipping, scaling and color jittering, and use data enhancement technology to expand the training data set and improve the generalization ability of the model; annotate the preprocessed images according to the pathology report, correspond the annotated data with the image data to form training samples.

[0020] Preferably, the model training module comprises the following steps:

[0021] S1. Model selection: select a pre-trained deep convolutional neural network as a feature extractor, use the feature representation learned by the pre-trained model on a large-scale dataset, and fine-tune it in combination with tumor pathology slice image data to improve model training efficiency and performance;

[0022] S2. Model construction: input the preprocessed pathological slice image into the pre-trained model, extract the high-dimensional feature vector, add the fully connected layer and the Softmax layer on the basis of the pre-trained model, and build the classifier;

[0023] S3. Model training, including data preparation, hyperparameter setting, model training process and model evaluation, including:

[0024] Data preparation: group the annotated pathological slide images and corresponding labels to form training sets, validation sets, and test sets. The commonly used data division ratio is 70% training set, 15% validation set, and 15% test set.

[0025] Hyperparameter settings, including learning rate, batch size, optimization algorithm, and regularization;

[0026] The model training process includes forward propagation, back propagation, parameter update and iterative training. The input image data is input into the model, and the model output is obtained after passing through the convolution layer, pooling layer and fully connected layer. The loss value between the model output and the true label is calculated by using the cross entropy loss function, and the calculation formula is:

[0027]

[0028] Among them, N is the number of samples, N classes is the number of classification categories, y ij is the true label, p ij The probability predicted by the model;

[0029] According to the loss value, the back propagation algorithm is used to calculate the gradient of the loss function to the model parameters. According to the optimization algorithm, the model parameters are updated using the calculated gradient. The forward propagation, back propagation and parameter update process are repeated until the model converges or the predetermined number of training rounds is reached.

[0030] Model evaluation,During the training process, the validation set is used to evaluate the performance of the model on unseen data to avoid overfitting of the model. After the model training is completed, the test set is used to perform a final evaluation of the model.

[0031] Preferably, the model reasoning module comprises the following steps:

[0032] S1. Image preprocessing. Since the resolution of WSI images is extremely high, direct processing will result in huge consumption of computing resources. Therefore, it is necessary to crop the image, extract the region of interest, normalize the image pixel values ​​to eliminate the brightness difference between different images, and use filtering algorithms to remove image noise and improve image quality.

[0033] S2. Image segmentation: segment the tumor area from the pathological slice image and extract the ROI;

[0034] S3. Feature extraction: input the ROI into the pre-trained deep convolutional neural network to extract high-dimensional feature vectors;

[0035] S4. Classification prediction: the extracted feature vector is input into the classifier to predict the tumor type and malignancy. The feature vector is input into the fully connected layer to obtain the score of each category. After the Softmax layer, the score is converted into a probability distribution to obtain the prediction probability of each category.

[0036] S5. Output the results. The classification prediction results are output and visualized. The information on tumor type, malignancy and diagnostic confidence is output. The pathological slice images, segmentation results and diagnostic results are visualized to facilitate the pathologist to view and analyze.

[0037] (III) Beneficial effects

[0038] The present invention provides a tumor digital pathology slice image management and diagnosis system based on deep learning. It has the following beneficial effects:

[0039] 1. In the present invention, distributed storage and data preprocessing technology are used to achieve efficient storage and management of massive pathological slice images.

[0040] 2. In the present invention, deep learning technology is used to realize automatic diagnosis of tumor pathological section images, assisting pathologists to improve diagnostic accuracy and efficiency.

[0041] 3. In the present invention, visualization technology and diagnostic interpretation modules are used to provide diagnostic basis, enhance the interpretability of model prediction results, and assist pathologists in making more accurate diagnoses. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the flow of the overall system of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Embodiment 1:

[0045] like Figure 1 As shown, an embodiment of the present invention provides a tumor digital pathology slice image management and diagnosis system based on deep learning, including:

[0046] The data management module includes a data acquisition module, a data storage module and a data preprocessing module. The data acquisition module is used to collect patients' clinical data, pathology reports and digital pathology slice images from the hospital information system and the pathology information system. The data storage module uses distributed storage technology to efficiently store and manage massive case slice images. The data preprocessing module performs cropping, normalization and denoising on the case slice images, and performs data enhancement on the images, including rotation, flipping and scaling operations, to expand the amount of training data and improve the generalization ability of the model.

[0047] The model training module includes a feature extraction module and a classifier training module. The pre-trained deep convolutional neural network is used as a feature extractor to extract high-dimensional features from the case slice images. The extracted features are input into the fully connected layer and the Softmax layer to construct a classifier. The classifier is trained using the cross entropy loss function and the optimization algorithm. The transfer learning technology is used to fine-tune the parameters of the pre-trained model to speed up the training and improve the model performance.

[0048] The model reasoning module includes an image segmentation module and a diagnosis prediction module, which segments the tumor area of ​​the pathological slice image, extracts the region of interest, inputs the ROI into the trained classifier, predicts the tumor type and malignancy, and outputs the diagnosis result, including information on the tumor type, malignancy, and diagnostic confidence;

[0049] The visualization and interpretation module includes a result visualization module and a diagnosis interpretation module. It visualizes the pathology slice images, segmentation results, and diagnosis results to facilitate pathologists to view and analyze. It provides an interactive interface to allow pathologists to modify and confirm the diagnosis results. It uses visualization technology to interpret the model prediction results and generates a heat map to indicate the image area that the model focuses on, providing a basis for diagnosis and assisting pathologists in making more accurate diagnoses.

[0050] The system management module, including the user management module, log management module and model management module, implements user registration, login and permission management functions, records system operation logs and user operation logs, facilitates system maintenance and troubleshooting, implements model version control, deployment and update, and supports online model training and inference services.

[0051] The data management module includes the following steps:

[0052] S1. Data collection, obtaining data from hospital information systems and pathology information systems: obtaining clinical information of patients from HIS, including age, gender, medical history and imaging examination results; obtaining pathology reports from PIS, including pathological diagnosis results, tumor type and malignancy, and obtaining digital pathology slice images from PIS, usually high-resolution WSI files; data cleaning: removing duplicate data, missing data and data that does not meet the requirements, and checking the consistency of data to ensure that data from different sources correspond to the same patient;

[0053] S2. Data storage, select a storage solution: Considering the huge amount of pathology image data, a distributed storage system is used, and a database is used to store the patient's clinical information and pathology report; Data storage: The cleaned WSI images are stored in a distributed file system, and a file index is established to facilitate subsequent query and management. The clinical information and pathology report are stored in the database and associated with the WSI image. Assuming that the average size of each WSI image is S WSI GB, storage N WSI The storage space required for each image is calculated as follows:

[0054] S total =S WSI ×N WSI

[0055] Assume that the average size of each record of clinical information and medical records stored in the database is S record KB, then store N clean The storage space required for each record is:

[0056] S DB =S record ×N clean / 1024

[0057] S3. Data preprocessing: perform preprocessing operations on images through cropping, normalization and denoising, enhance data through rotation, flipping, scaling and color jittering, and use data enhancement technology to expand the training data set and improve the generalization ability of the model; annotate the preprocessed images according to the pathology report, correspond the annotated data with the image data to form training samples.

[0058] The model training module includes the following steps:

[0059] S1. Model selection: select a pre-trained deep convolutional neural network as a feature extractor, use the feature representation learned by the pre-trained model on a large-scale dataset, and fine-tune it in combination with tumor pathology slice image data to improve model training efficiency and performance;

[0060] S2. Model construction: input the preprocessed pathological slice image into the pre-trained model, extract the high-dimensional feature vector, add the fully connected layer and the Softmax layer on the basis of the pre-trained model, and build the classifier;

[0061] S3. Model training, including data preparation, hyperparameter setting, model training process and model evaluation, including:

[0062] Data preparation: group the annotated pathological slide images and corresponding labels to form training sets, validation sets, and test sets. The commonly used data division ratio is 70% training set, 15% validation set, and 15% test set.

[0063] Hyperparameter settings, including learning rate, batch size, optimization algorithm, and regularization;

[0064] The model training process includes forward propagation, back propagation, parameter update and iterative training. The input image data is input into the model, and the model output is obtained after passing through the convolution layer, pooling layer and fully connected layer. The loss value between the model output and the true label is calculated by using the cross entropy loss function, and the calculation formula is:

[0065]

[0066] Among them, N is the number of samples, N classes is the number of classification categories, y ij is the true label, p ij The probability predicted by the model;

[0067] According to the loss value, the back propagation algorithm is used to calculate the gradient of the loss function to the model parameters. According to the optimization algorithm, the model parameters are updated using the calculated gradient. The forward propagation, back propagation and parameter update process are repeated until the model converges or the predetermined number of training rounds is reached.

[0068] Model evaluation,During the training process, the validation set is used to evaluate the performance of the model on unseen data to avoid overfitting of the model. After the model training is completed, the test set is used to perform a final evaluation of the model.

[0069] The model inference module includes the following steps:

[0070] S1. Image preprocessing. Since the resolution of WSI images is extremely high, direct processing will result in huge consumption of computing resources. Therefore, it is necessary to crop the image, extract the region of interest, normalize the image pixel values ​​to eliminate the brightness difference between different images, and use filtering algorithms to remove image noise and improve image quality.

[0071] S2. Image segmentation: segment the tumor area from the pathological slice image and extract the ROI;

[0072] S3. Feature extraction: input the ROI into the pre-trained deep convolutional neural network to extract high-dimensional feature vectors;

[0073] S4. Classification prediction: input the extracted feature vector into the classifier to predict the tumor type and malignancy. Input the feature vector into the fully connected layer to obtain the score of each category. After the Softmax layer, the score is converted into a probability distribution to obtain the prediction probability of each category.

[0074] S5. Output the results. The classification prediction results are output and visualized. The information on tumor type, malignancy and diagnostic confidence is output. The pathological slice images, segmentation results and diagnostic results are visualized to facilitate the pathologist to view and analyze.

[0075] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A tumor digital pathology slice image management and diagnosis system based on deep learning, characterized by: include: The data management module includes a data acquisition module, a data storage module and a data preprocessing module, wherein the data acquisition module is used to collect the patient's clinical data, pathology report and digital pathology slice images from the hospital information system and the pathology information system; The data storage module uses distributed storage technology to efficiently store and manage massive case slice images; The data preprocessing module crops, normalizes and denoises the case slice images, and performs data enhancement on the images, including rotation, flipping and scaling operations, to expand the amount of training data and improve the generalization ability of the model; The model training module includes a feature extraction module and a classifier training module. The pre-trained deep convolutional neural network is used as a feature extractor to extract high-dimensional features from the case slice images. The extracted features are input into the fully connected layer and the Softmax layer to construct a classifier. The classifier is trained using the cross entropy loss function and the optimization algorithm. The transfer learning technology is used to fine-tune the parameters of the pre-trained model to speed up the training and improve the model performance. The model reasoning module includes an image segmentation module and a diagnosis prediction module, which segments the tumor area of ​​the pathological slice image, extracts the region of interest, inputs the ROI into the trained classifier, predicts the tumor type and malignancy, and outputs the diagnosis result, including information on the tumor type, malignancy, and diagnostic confidence; The visualization and interpretation module includes a result visualization module and a diagnosis interpretation module. It visualizes the pathology slice images, segmentation results, and diagnosis results to facilitate pathologists to view and analyze. It provides an interactive interface to allow pathologists to modify and confirm the diagnosis results. It uses visualization technology to interpret the model prediction results and generates a heat map to indicate the image area that the model focuses on, providing a basis for diagnosis and assisting pathologists in making more accurate diagnoses. The system management module, including the user management module, log management module and model management module, implements user registration, login and permission management functions, records system operation logs and user operation logs, facilitates system maintenance and troubleshooting, implements model version control, deployment and update, and supports online model training and inference services.

2. According to claim 1, a tumor digital pathology slice image management and diagnosis system based on deep learning is characterized by: The data management module includes the following steps: S1. Data collection, obtaining data from hospital information systems and pathology information systems: obtaining clinical information of patients from HIS, including age, gender, medical history and imaging examination results; obtaining pathology reports from PIS, including pathological diagnosis results, tumor type and malignancy, and obtaining digital pathology slice images from PIS, usually high-resolution WSI files; data cleaning: removing duplicate data, missing data and data that does not meet the requirements, and checking the consistency of data to ensure that data from different sources correspond to the same patient; S2. Data storage, select a storage solution: Considering the huge amount of pathology image data, a distributed storage system is used, and a database is used to store the patient's clinical information and pathology report; Data storage: The cleaned WSI images are stored in a distributed file system, and a file index is established to facilitate subsequent query and management. The clinical information and pathology report are stored in the database and associated with the WSI image. Assuming that the average size of each WSI image is S WSI GB, storage N WSI The storage space required for each image is calculated as follows: S total =S WSI ×N WSI Assume that the average size of each record of clinical information and medical records stored in the database is S record KB, then store N clean The storage space required for each record is: S DB =S record ×N clean / 1024 S3. Data preprocessing: perform preprocessing operations on images through cropping, normalization and denoising, enhance data through rotation, flipping, scaling and color jittering, and use data enhancement technology to expand the training data set and improve the generalization ability of the model; annotate the preprocessed images according to the pathology report, correspond the annotated data with the image data to form training samples.

3. According to the deep learning-based tumor digital pathology slice image management and diagnosis system of claim 1, it is characterized by: The model training module includes the following steps: S1. Model selection: select a pre-trained deep convolutional neural network as a feature extractor, use the feature representation learned by the pre-trained model on a large-scale dataset, and fine-tune it in combination with tumor pathology slice image data to improve model training efficiency and performance; S2. Model construction: input the preprocessed pathological slice image into the pre-trained model, extract the high-dimensional feature vector, add the fully connected layer and the Softmax layer on the basis of the pre-trained model, and build the classifier; S3. Model training, including data preparation, hyperparameter setting, model training process and model evaluation, including: Data preparation: group the annotated pathological slide images and corresponding labels to form training sets, validation sets, and test sets. The commonly used data division ratio is 70% training set, 15% validation set, and 15% test set. Hyperparameter settings, including learning rate, batch size, optimization algorithm, and regularization; The model training process includes forward propagation, back propagation, parameter update and iterative training. The input image data is input into the model, and the model output is obtained after passing through the convolution layer, pooling layer and fully connected layer. The loss value between the model output and the true label is calculated by using the cross entropy loss function, and the calculation formula is: Among them, N is the number of samples, N classes is the number of classification categories, y ij is the true label, p ij The probability predicted by the model; According to the loss value, the back propagation algorithm is used to calculate the gradient of the loss function to the model parameters. According to the optimization algorithm, the model parameters are updated using the calculated gradient. The forward propagation, back propagation and parameter update process are repeated until the model converges or the predetermined number of training rounds is reached. Model evaluation,During the training process, the validation set is used to evaluate the performance of the model on unseen data to avoid overfitting of the model. After the model training is completed, the test set is used to perform a final evaluation of the model.

4. According to the deep learning-based tumor digital pathology slice image management and diagnosis system of claim 1, it is characterized by: The model reasoning module includes the following steps: S1. Image preprocessing. Since the resolution of WSI images is extremely high, direct processing will result in huge consumption of computing resources. Therefore, it is necessary to crop the image, extract the region of interest, normalize the image pixel values ​​to eliminate the brightness difference between different images, and use filtering algorithms to remove image noise and improve image quality. S2. Image segmentation: segment the tumor area from the pathological slice image and extract the ROI; S3. Feature extraction: input the ROI into the pre-trained deep convolutional neural network to extract high-dimensional feature vectors; S4. Classification prediction: input the extracted feature vector into the classifier to predict the tumor type and malignancy. Input the feature vector into the fully connected layer to obtain the score of each category. After the Softmax layer, the score is converted into a probability distribution to obtain the prediction probability of each category. S5. Output the results. The classification prediction results are output and visualized. The information on tumor type, malignancy and diagnostic confidence is output. The pathological slice images, segmentation results and diagnostic results are visualized to facilitate the pathologist to view and analyze.

Images