Multi-modal pathological data conjoint analysis method based on deep learning

Abstract

The invention belongs to the technical field of medical image processing, and particularly relates to a multi-modal pathological data conjoint analysis method based on deep learning. According to the method, a ResNet50 network is utilized, a pathological section image, a fluorescence image and a lasing image are respectively collected for deep learning, pathological tissues are divided into five tissue types, and super-multi-classification is realized; and through a weight voting system, carrying out weight comprehensive voting on the three-mode 5 classification results to obtain voting results with different colors representing different organization types. According to the multi-modal data conjoint analysis method for the pathological tissue, high-dimensional information orthogonal to original two-dimensional pathological section imaging is introduced, so that a relatively rough data set can be adopted to realize optimization judgment, and the judgment robustness is improved. And pathologists can be efficiently assisted to diagnose and analyze pathological sections in real time.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a multimodal pathology data joint analysis method. Background Art

[0002] Cancer is a major public health problem and a leading cause of death worldwide. The main cause of cancer is that normal cells in the human body undergo lesions, forming cancerous cells that differentiate and reproduce indefinitely and uncontrollably. According to the World Cancer Report, the number of new cancer cases in the world will reach 20 million in 2022, and the number of deaths will reach 9.7 million. The World Health Organization's International Agency for Research on Cancer (IARC) summarized that the "three major types of cancer in 2022" are lung cancer, breast cancer, and colorectal cancer. To date, morphological evaluation by hematoxylin and eosin staining (H&E) remains the gold standard for cancer diagnosis. However, due to the lack of quantitative capabilities, current microscopic histopathological analysis is subjective in many cases. Therefore, even experienced pathologists may have different diagnostic opinions and conclusions. Therefore, there is an urgent need for an effective, accurate, time-saving and labor-saving method to assist pathologists in diagnosing cancer.

[0003] In recent years, deep learning technology has demonstrated its powerful capabilities in a wide range of complex data analysis. Therefore, using deep learning to distinguish cancerous and non-cancerous tissues in pathological sections has become a mainstream auxiliary means. However, in supervised learning, high-accuracy classification prediction models often require high-quality data sets. The production of such high-quality data sets requires strict control by pathologists, which is not easy in actual operation. When the quality of data sets is uneven, supervised learning is prone to overfitting, and robustness will also decrease. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a multimodal pathology data joint analysis method based on deep learning to solve the problems existing in the above-mentioned prior art.

[0005] The multimodal pathology data joint analysis method based on deep learning proposed in the present invention comprises the following specific steps:

[0006] (1) constructing a pathology data set; the pathology data includes pathology slice images, fluorescence images, and laser emission images (referred to as trimodal data); dividing the pathology data according to different tissue types and tumor differentiation degrees to obtain a case data set (also referred to as a multimodal data set, specifically a trimodal data set); and manually annotating the data set; dividing the data set into a training data set and a validation data set in a ratio of 8:2; and then expanding the data by data enhancement to balance the number of tissue classifications in the pathology data set;

[0007] (2) Construct a multimodal classification prediction model; specifically, three identical ResNet50 networks are used to construct multiple classification prediction models; the ResNet50 network is a deep learning network used to learn multimodal data in parallel according to different modalities; the ResNet50 network includes a fully connected layer and a convolutional layer (including a residual module); the three ResNet50 networks are referred to as Model 1, Model 2, and Model 3, where Model 1 is used for pathological slice images, Model 2 is used for fluorescence images, and Model 3 is used for laser images;

[0008] (3) A weighted voting module is used to vote on the prediction results of multiple classification prediction models to obtain a large number of classification results of pathological tissues of cancer patients.

[0009] Further:

[0010] In step (1), the pathological section image is collected from the whole slide image of the cancer patient, and the fluorescence image and the laser image are collected from the sensor image of the cancer patient's pathological tissue impregnated with YOPRO fluorescent dye in the Fabry-Perot cavity. The sensing information of the fluorescence image and the laser image is obtained by a laser emission microscopy (LEM) multi-channel detection system.

[0011] Based on the multiple images collected (also called multi-channel images), pathologists annotate the various tissue categories of the pathological images and divide them into different tissue type areas. Each area is further divided into 400μm×400μm non-overlapping image blocks, and they are included in the folders of their respective tissue types according to the tissue category to construct a classification prediction data set;

[0012] The data set is divided into a training data set and a validation data set in a ratio of 8:2;

[0013] The multi-classification categories of the pathological tissue include five categories, specifically: blank tissue, tumor tissue, normal tissue, necrotic tissue and lymphoid tissue;

[0014] The tumor differentiation degree categories include: poorly differentiated cancer, moderately differentiated cancer and well-differentiated cancer.

[0015] In the present invention, the lasing image introduces physiological and pathological high-dimensional information that is orthogonal to the original two-dimensional imaging, and the stacked laser transverse modes do not need to be split by means of a grating mirror, and the model can directly learn and analyze the data;

[0016] In the laser image, the relationship between the pump power and the laser point density is related to the rich information in the tissue cells. In the process of increasing the pump power alone, the original laser points (such as mode, intensity, etc.) undergo nonlinear changes.

[0017] In the present invention, the data enhancement methods include: rotation angle, color enhancement, brightness enhancement and contrast enhancement. The enhancement methods are used to expand the training data sets of different categories to achieve a balance in the amount of data for each category;

[0018] The prediction results obtained by the classification prediction model are voted on by the weighted voting module in a multi-modal comprehensive manner, according to the principle of majority rule. If the prediction results are different, the pathological section shall prevail.

[0019] The multimodal comprehensive voting performs weighted comprehensive voting on the five tissue classification results of the three modal data;

[0020] In the present invention, the accuracy rate is specifically used to evaluate the result of the voting module, and the accuracy rate after voting of each category is calculated separately, so as to refine the improvement effect of the weighted voting module on the accuracy rate and robustness of each category.

[0021] In the present invention, the classification prediction model is trained and the training effect is evaluated, specifically including:

[0022] The prediction model is trained with the data set, the model parameters are adjusted, and the excellent model is selected according to the accuracy of the training set. The model is evaluated based on the accuracy and loss function of the validation set, and the ROC curve (Receiver Operating Characteristic Curve) and AUC (Area Under the Curve) value are used to evaluate the model performance.

[0023] The present invention also includes a multimodal pathology data joint analysis system based on deep learning, which specifically includes: a pathology data set construction module, a multimodal classification prediction model module, and a weighted voting module; the three modules are used to execute the operation contents of the three steps in the multimodal pathology data joint analysis method to realize the joint analysis of multimodal pathology data.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] Based on the original two-dimensional information of a single pathological slice, the present invention utilizes the advantages of multi-channel detection of the LEM system to obtain fluorescence images and laser images, and introduces high-dimensional information orthogonal to the original two-dimensional pathological slice imaging.

[0026] The present invention utilizes the ResNet50 network to learn multimodal information separately, achieving accurate recognition of multiple classifications of tissue types and degree of tumor differentiation not only in pathological section images but also in fluorescence images and laser images.

[0027] The present invention uses a weighted voting module to vote on the results obtained by the learning model, realizes information optimization judgment when the data set is rough, and improves the robustness of the corresponding multi-classification judgment, which is expected to efficiently and accurately assist pathologists in the immediate diagnosis and analysis of pathological sections. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of a method for joint analysis of multimodal pathology data based on deep learning in an embodiment of the present invention.

[0029] Figure 2 It is a diagram of the ResNet50 model architecture adopted in the present invention.

[0030] Figure 3 It is a schematic diagram of the stacking of lasing modes in the present invention.

[0031] Figure 4 It is a schematic diagram of the variation of the lasing mode with the pump power in the present invention.

[0032] Figure 5 Schematic diagram of prediction results and comprehensive voting results of three modes in an embodiment of the present invention.

[0033] Figure 6 It is the accuracy rate during the training process of the prediction model corresponding to the three modes in the embodiment of the present invention.

[0034] Figure 7 : It is the ROC curve of the prediction model corresponding to the three modes in the embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings.

[0036] The method for joint analysis of multimodal pathological data based on deep learning provided by the present invention is as follows: Figure 1 As shown, including:

[0037] Data preparation

[0038] First, the pathological slice image, fluorescence image and laser image data of the cancer patient are obtained; the pathological slice image is collected from the whole slide image of the cancer patient, and the fluorescence image and laser image are collected from the sensor image of the cancer patient's pathological tissue impregnated with YOPRO fluorescent dye in the Fabry-Perot cavity by taking advantage of the multi-channel detection of the LEM system. (For this invention, only squamous cell lung cancer is included as a sample)

[0039] Data processing

[0040] Pathologists annotated the pathological images for each tissue category and divided them into different tissue type regions. Each region was further divided into 400 μm × 400 μm non-overlapping image blocks, which were classified into folders of different tissue types, to construct a classification prediction dataset, and the dataset was divided into a training set and a validation set in a ratio of 8:2.

[0041] The multi-classification categories of pathological tissue include: blank tissue, tumor tissue, normal tissue, necrotic tissue and lymphoid tissue;

[0042] The tumor differentiation degree categories include: poorly differentiated cancer, moderately differentiated cancer and well-differentiated cancer.

[0043] Among them, the laser image introduces high-dimensional information that is orthogonal to the original two-dimensional pathological imaging. Figure 3 What is shown is the sensing information presented by the local pathological tissue after passing through the LEM system and lasing in the resonant cavity, that is, the stacking of multiple laser transverse modes. This stacked laser transverse mode does not need to be split by the grating mirror, and the model can directly learn and analyze the data;

[0044] In the laser image, the relationship between pump power and laser point density is related to the rich information in tissue cells, such as Figure 4 The results show that different pump powers (400 μJ / mm 2 、800μJ / mm 2 and 1600μJ / mm 2 ). It can be seen that in the process of increasing the pump power alone, the original lasing point (such as mode, intensity, etc.) undergoes nonlinear changes.

[0045] In the present invention, the data enhancement methods include: rotation angle, color enhancement, brightness enhancement and contrast enhancement. The enhancement methods are used to expand the training data sets of different categories to achieve a balance in the amount of data for each category.

[0046] Model Building

[0047] In order to achieve the best classification task, the present invention uses the ResNet50 network to classify and learn the data of each modality. Figure 2As shown in the figure, the network model uses the Bottleneck Block structure as its basic residual module, which can be divided into five stages, including: using a 7×7 convolution kernel, the number of convolution kernels is 64, and the step size is 2, then passing through the BatchNormalization (BN) layer and the Relu activation function, and then passing through the maximum pooling layer, the pooling layer size is 3×3, the step size is 2; through batch normalization, the activation of each neuron becomes a Gaussian distribution. Then after each convolution, the BN layer and Relu layer are activated.

[0048] The test set images are predicted by the ResNet50 model, and the prediction results are input into the voting module for multi-modal comprehensive voting. When the classification results of two or more modalities in the pathological section image, fluorescence image and laser image are the same, voting is carried out according to the principle of minority obeys majority. When the classification results in the pathological section image, fluorescence image and laser image are different, the voting result is based on the pathological section.

[0049] Model Evaluation

[0050] The present invention uses ROC curve and AUC value to evaluate the model performance and verifies the ResNet50 model. Figure 7 As shown, it can be seen that the 5-classification models of the three modalities all have excellent classification capabilities. In HE mode, the AUC value of blank tissue 0 was 0.98, the AUC value of tumor tissue 1 was 0.95, the AUC value of normal tissue 2 was 0.93, the AUC value of necrotic tissue 3 was 0.87, the AUC value of lymphatic tissue 4 was 0.95, and the average AUC value was 0.94; in fluorescence mode, the AUC value of blank tissue 0 was 0.98, the AUC value of tumor tissue 1 was 0.95, the AUC value of normal tissue 2 was 0.91, the AUC value of necrotic tissue 3 was 0.88, the AUC value of lymphatic tissue 4 was 0.96, and the average AUC value was 0.94; in lasing mode, the AUC value of blank tissue 0 was 0.97, the AUC value of tumor tissue 1 was 0.92, the AUC value of normal tissue 2 was 0.87, the AUC value of necrotic tissue 3 was 0.79, the AUC value of lymphatic tissue 4 was 0.9, and the average AUC value was 0.89.

[0051] The multi-classification accuracy and individual classification accuracy are used to evaluate the multimodal model after the integration of the voting module and the ResNet50 model.

[0052] Model Description

[0053] In order to better explain the model of the present invention, a 400μm×400μm small image of each classification will be output, and small images of different colors will be output according to different tissue types. And according to the output probability, the depth of different colors is represented. The darker the color, the higher the probability. Among them, black represents blank tissue 0, red represents tumor tissue 1, yellow represents normal tissue 2, blue represents necrotic tissue 3, and purple represents lymphatic tissue 4. Therefore, by using color small images, not only can each tissue type be classified, but also the areas of different tissue types can be successfully segmented, and the number of small images of each type can be quantitatively counted. Figure 5 As shown in the figure, the three-modal data are predicted by the prediction model, and the color diagram predicted by each modality and after weighted voting is obtained. The lower right corner is the pathologist's diagnosis mark result. Figure 5 In the test, we took a complete tissue slice and found that the classification accuracy of the pathological slice image was 87.99%, the classification accuracy of the fluorescence image was 70.29%, and the classification accuracy of the lasing image was 71.43%. After the voting system, the accuracy rate can be as high as 91.07%.

Claims

1. A method for joint analysis of multimodal pathological data based on deep learning, characterized in that: The specific steps are: (1) constructing a pathology data set; the pathology data includes pathology slice images, fluorescence images, and laser emission images, and the pathology data is divided according to different tissue types and tumor differentiation degrees to obtain a case data set, also called a multimodal data set; And perform manual annotation; divide the data set into a training data set and a validation data set; then expand the data by data enhancement to balance the number of each tissue classification in the pathology data set; (2) Construct a multimodal classification prediction model; specifically, three identical ResNet50 networks are used to construct multiple classification prediction models; the ResNet50 network is a deep learning network used to learn multimodal data in parallel according to different modalities; the ResNet50 network includes a fully connected layer and a convolutional layer; the three ResNet50 networks are referred to as Model 1, Model 2, and Model 3, where Model 1 is used for pathological slice images, Model 2 is used for fluorescence images, and Model 3 is used for laser images; (3) A weighted voting module is used to vote on the prediction results of multiple classification prediction models to obtain a large number of classification results of pathological tissues of cancer patients.

2. The method according to claim 1, characterized in that In step (1), the pathological section image is collected from the whole slide image of the cancer patient, and the fluorescence image and the laser image are collected from the sensor image of the cancer patient's pathological tissue impregnated with YOPRO fluorescent dye in the Fabry-Perot cavity; the sensing information of the fluorescence image and the laser image is obtained by a laser emission microscopy (LEM) multi-channel detection system; Based on the collected multiple images, also called multi-channel images, pathologists annotate the various tissue categories of the pathological images and divide them into different tissue type regions; each region is further divided into 400 μm×400 μm non-overlapping image blocks, and the blocks are respectively included in the folders of the respective tissue types according to the tissue categories, so as to construct a classification prediction data set; The division ratio of the training data set to the validation data set is 8:2; The multi-classification categories of the pathological tissue include: blank tissue, tumor tissue, normal tissue, necrotic tissue and lymphoid tissue; The tumor differentiation degree categories include: poorly differentiated cancer, moderately differentiated cancer, and well-differentiated cancer; The data enhancement methods include: rotation angle, color enhancement, brightness enhancement and contrast enhancement; the data enhancement method is used to expand the training data sets of different categories to achieve a balance in the amount of data for each category.

3. The method according to claim 2, characterized in that The prediction results obtained by the classification prediction model are subjected to multimodal comprehensive voting through the weighted voting module, according to the principle of minority obeys majority; if the prediction results are different, the pathological section shall prevail.

4. The method according to claim 3, characterized in that Specifically, the accuracy rate is used to evaluate the results of the voting module, and the accuracy rate of each category after voting is calculated separately, so as to refine the effect of the weighted voting module on improving the accuracy and robustness of each category.

5. The method according to claim 1, characterized in that The process of training and evaluating the classification prediction model includes: training the prediction model with the data set, adjusting the model parameters, and screening excellent models according to the accuracy of the training set; and evaluating the model based on the accuracy and loss function of the validation set, and using the ROC curve and AUC value to evaluate the model performance.

6. The method according to claim 5, characterized in that The ResNet50 network adopts the Bottleneck Block structure as its basic residual module, including: 7×7 convolution kernels, the number of convolution kernels is 64, and the step size is 2; followed by a BN layer and a Relu activation function; then a maximum pooling layer, the size of the pooling layer is 3×3, and the step size is 2; then batch normalization, the activation of each neuron becomes a Gaussian distribution; and then the BN layer and Relu layer are activated after each convolution.

7. A multimodal pathology data joint analysis system based on the method according to any one of claims 1 to 6, specifically comprising: Pathology dataset construction module, multimodal classification prediction model module, weighted voting module; The three modules are used to execute the operation contents of the three steps in the multimodal pathology data joint analysis method to realize the joint analysis of the multimodal pathology data.

Images