Deep learning-based lung adenocarcinoma growth mode pathological image tumor region segmentation method

Abstract

The invention relates to a lung adenocarcinoma growth mode pathological image tumor region segmentation method based on deep learning. The method comprises the following steps: S1, queue: training a model; s2, data preprocessing: segmenting each digital pathological image into image blocks; s3, a digital pathological image segmentation system based on integrated deep learning; and S4, implementing detail and model evaluation. The method is used for automatically segmenting the growth modes of lung adenocarcinoma, and main subtypes can be determined and an IASLC grading system can be applied by outputting the proportion of the six growth modes. The quantitative output mode of the deep learning model is expected to solve the subjectivity of semi-quantitative evaluation of the six growth modes by a pathologist, and can improve the consistency among observers, so that the proportion of the six growth modes of the patient can be judged more accurately, and prognosis layering can be performed more effectively. In addition, the deep learning model is expected to reduce heavy workload of pathologists, and the pathologists can focus on more complicated cases through AI auxiliary evaluation, so that diagnosis and treatment of lung cancer patients are optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and artificial intelligence, in particular to a lung adenocarcinoma growth pattern pathological image tumor region segmentation method based on deep learning. BACKGROUND

[0002] Lung adenocarcinoma is histologically heterogeneous, presenting a combination of multiple growth patterns and proportions. In 2015, the World Health Organization (WHO) described the main subtypes of lung adenocarcinoma and divided its prognosis into three groups, corresponding to different patient outcomes (low grade: mainly with adhesion-like growth; intermediate grade: mainly with acinar or papillary growth; high grade: mainly with solid or micropapillary growth). However, even a small proportion of high-grade growth patterns can lead to worsening patient outcomes. In addition, studies have shown that complex glandular type, like solid or micropapillary type, also indicates poor prognosis. Taking these factors into consideration, the International Association for the Study of Lung Cancer (IASLC) proposed a new grading system that includes both the main growth patterns and sets the critical value of high-grade growth patterns at 20%. This grading system requires pathologists to read HE-stained sections and assess the proportion of different growth patterns (adhesion type, papillary type, acinar type, micropapillary type, solid type, complex glandular type) in increments of 5%, and classify the tumor as: grade 1, mainly with adhesion type, no high-grade growth pattern or high-grade growth pattern (micropapillary type, solid type, complex glandular type) proportion less than 20%; grade 2, mainly with acinar type or papillary type, no high-grade growth pattern or high-grade growth pattern proportion less than 20%; grade 3, any tumor containing 20% and above high-grade growth pattern. It has been proven that this grading system has strong prognostic value, especially for stage I lung adenocarcinoma. The new pathological grading system for lung adenocarcinoma proposed by the International Association for the Study of Lung Cancer (IASLC) requires the naked eye to assess the proportion of different pathological subtypes, and the process is complex and has large inter-observer variability. It is difficult to identify complex glandular type from traditional acinar type, which depends on the professional experience of pathologists. In addition, the semi-quantitative assessment of all growth patterns in increments of 5%, and the setting of the critical value of high-grade patterns at 20%, has poor inter-observer consistency. Therefore, there is an urgent need for a diagnostic tool to improve the reproducibility in clinical practice.

[0003] Chinese patent document CN 110288613A discloses a super-pixel histopathological image segmentation method, which belongs to the field of image processing and artificial intelligence. The method comprises the following steps: S1: randomly selecting a fixed window size of pathological section image block on a super-pixel histopathological image which has completed histopathological annotation, forming a pathological image block training dataset; S2: preprocessing the pathological section image block; S3: establishing a multi-scale spatial full convolution network and its class activation mapping model, jointly connecting the full connection layer, and training the network parameters of the model using the pathological image block training set to realize accurate classification and identification of lesions based on image blocks; S4: using the multi-scale spatial full convolution network structure, inputting the super-pixel histopathological image to be analyzed, and outputting the image section segmentation result with lesion location information. The present application can efficiently and accurately realize the pixel-level accurate region segmentation of super-pixel histopathological images.

[0004] Chinese patent document CN 114565761A discloses a method for segmenting tumor regions in renal clear cell carcinoma pathological images based on deep learning, which includes data acquisition and preprocessing, classification network SENet building and training, and tumor region prediction. The present application studies renal clear cell carcinoma based on pathological images, which has higher credibility compared with CT and MRI image judgment. The present application solves the problem that previous research on renal clear cell carcinoma was only limited to judging whether it exists or not, and can intuitively give the position and size of the tumor region, facilitating medical personnel to better study the pathogenesis and treatment trend of renal clear cell carcinoma. The present application starts from the whole pathological image, automatically segments the tumor region of the whole pathological image, breaks through the previous research on pathological image blocks, and can give a relatively complete and intelligent diagnosis result.

[0005] However, there is no report on the precise diagnosis of lung adenocarcinoma tumor grading by using artificial intelligence to quantify the growth pattern. SUMMARY

[0006] The purpose of the present application is to overcome the deficiencies in the prior art, and to provide a method for segmenting tumor regions in lung adenocarcinoma growth pattern pathological images based on deep learning.

[0007] In a first aspect, the present application provides a method for segmenting tumor regions in lung adenocarcinoma growth pattern pathological images based on deep learning, which comprises the following steps:

[0008] S1: Queue: training model, independently labeling six growth patterns;

[0009] S2: data preprocessing: segmenting each digital pathology image into image blocks;

[0010] S3: digital pathology image segmentation system based on integrated deep learning;

[0011] The digital pathology image segmentation system is a six-binary deep learning model based on an integrated deep neural network, which identifies and segments six growth patterns in digital pathology images at the pixel level. The model uses SE-residual blocks to extract deep features of input image blocks and updates weights based on limited data. Finally, the output of the six identification results is merged and processed by a post-processing module to obtain a unified segmentation result.

[0012] S4: Implementation details and model evaluation: The proposed integrated deep neural network uses the Adam optimizer during training, with a learning rate of 0.0001 to update the weights.

[0013] As a preferred example, the digital pathology image segmentation system includes the following modules

[0014] Single growth pattern segmentation module: a convolutional neural network using six binary classification models for instance segmentation of digital pathology images.

[0015] Residual block: a residual block formula is used based on the convolutional neural network as follows:

[0016] y = F(X) + X y = F(X) + G(X)

[0017] Where X represents input information, F(·) and G(·) represent nonlinear functions, and y represents output features.

[0018] SE-residual block: integrates a squeeze-excitation module, which models the importance of feature channels by reweighting the channels.

[0019] Post-processing module: the steps are as follows: first, define a class set including background class and six growth patterns; second, based on the read pixel value, directly correct the background information, and correct the pixel value of 255 to background class; in addition, for the pixels with confusion, assign them to the class corresponding to the binary classification model with the highest probability value at that pixel position

[0020] As a preferred example, S1: queue: the six growth patterns are adherent, papillary, ductal, micro-papillary, solid, and complex glandular.

[0021] S2: Data preprocessing: each digital pathology image is segmented into 256x256 pixel image blocks for subsequent segmentation processing.

[0022] In a second aspect, the present application provides a digital pathology image segmentation system based on integrated deep learning, which is a six-binary deep learning model based on an integrated deep neural network, and is used to identify and segment six growth patterns in a digital pathology image at the pixel level.

[0023] As a preferred example, the digital pathology image segmentation system is divided into the following modules:

[0024] Single growth pattern segmentation module: a convolutional neural network is used, and six binary classification models are used for instance segmentation of digital pathology images;

[0025] Residual block: a residual block formula is used on the basis of a convolutional neural network as follows:

[0026] y = F(X) + X y = F(X) + G(X)

[0027] Where X represents input information, F(·) and G(·) represent nonlinear functions, and y represents output features;

[0028] SE-residual block: an SE module is fused into the residual block, which models the importance of feature channels by reweighting the channels;

[0029] Post-processing module: the steps are as follows: first, define a class set including a background class and six growth patterns, second, directly correct the background information based on the read pixel value, correct the pixel value of 255 to the background class, and third, for the pixels with confusion, assign them to the class corresponding to the binary classification model with the highest probability value at the pixel position

[0030] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, when the computer program is executed in a computer, the computer is caused to execute a deep learning based segmentation method for a tumor region of a lung adenocarcinoma growth pattern pathology image.

[0031] The present application has the advantages that the present application is used for automatically segmenting the growth patterns of lung adenocarcinoma. By outputting the proportions of six growth patterns (mural type, acinar type, papillary type, solid type, micro-papillary type and complex gland), the main subtype can be determined and the IASLC classification system can be applied. The quantitative output mode of the deep learning model is expected to solve the subjectivity of the semi-quantitative evaluation of the six growth patterns by pathologists, and at the same time improve the inter-observer consistency, so as to more accurately judge the proportion of the six growth patterns of the patient, thereby more effectively performing prognosis stratification. In addition, the deep learning model is also expected to reduce the heavy workload of pathologists, and through AI-assisted evaluation, the pathologists can focus on more complex cases, thereby optimizing the diagnosis and treatment of lung cancer patients. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Digital pathology image segmentation system based on integrated deep learning. DETAILED DESCRIPTION

[0033] The present application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that after reading the content described in the present application, those skilled in the art can make various modifications or modifications to the present application, and these equivalent forms also fall within the scope defined by the claims attached to the present application.

[0034] Example 1

[0035] Materials and methods:

[0036] (1) Cohort: For training the model, we selected 67 digital pathology images (WSIs) from 67 patients, and two expert pathologists with more than 20 years of experience in thoracic tumors independently labeled six growth patterns (mural type, papillary type, acinar type, micro-papillary type, solid type, and complex gland type). Subsequently, we evaluated the diagnostic performance of the model in two retrospective cohorts. Validation cohort 1 included 639 patients with a total of 1033 digital pathology images; validation cohort 2 included 375 patients from a multicenter cohort of three medical centers in other regions of China, with a total of 909 digital pathology images. In addition, from October 15, 2024 to November 15, 2024, a multicenter clinical trial (NCT05925764) was conducted in these four centers to prospectively verify the effectiveness of the model. Patients with stage I-III invasive lung adenocarcinoma who underwent surgical resection were included. Exclusion criteria include: stage IV patients, pre-invasive lung adenocarcinoma patients, mucinous adenocarcinoma and its variants, patients with a history of neoadjuvant therapy, patients with no available tumor tissue sections or poor quality sections, patients with no available clinical and pathological data, and patients with no available follow-up information.

[0037] (2) Data preprocessing: Since each digital pathology image contains billions of pixels, and most of the content is irrelevant background interference information, deep learning (DL) models cannot be directly used to effectively extract deep features, nor can they run on most actual workstations. In addition, a large amount of background information will seriously interfere with the segmentation performance of the model, and thus affect the final histological grading results. Therefore, in the preprocessing stage, the digital pathology image needs to be down-sampled to reduce the number of pixels per image. At the same time, each digital pathology image is divided into 256x256 pixel image blocks for subsequent segmentation processing.

[0038] (3) Digital pathology image segmentation system based on integrated deep learning Figure 1 ): Based on digital pathology image diagnosis of lung adenocarcinoma histological grading is a multi-label image segmentation problem, which aims to effectively classify the histological features of six growth patterns (adherent type, papillary type, glandular type, micro-papillary type, solid type, and complex glandular type). Since the number of digital pathology images for each patient is different, and the proportion of growth patterns contained in each digital pathology image varies, deep learning-based models need to have robust recognition ability for different growth patterns. Therefore, the present application proposes an integrated deep neural network (EDNN) to process digital pathology images to achieve accurate mapping of lung adenocarcinoma growth patterns. The integrated deep neural network we developed uses six binary deep learning models to identify and segment the six growth patterns in digital pathology images at the pixel level. The model uses SE-residual blocks to extract deep features of input image blocks, and updates weights based on limited data. Finally, the output of the six recognition results is processed by the post-processing module to obtain a unified segmentation result. The specific details of the model are as follows: 1) Single growth pattern segmentation module: This application introduces a convolutional neural network (CNN) and designs an SE-residual block to avoid the problem of gradient disappearance during model training. In addition, since the growth patterns on each digital pathology image differ, we designed six binary classification models for instance segmentation of digital pathology images. 2) Residual block: Generally, increasing the depth of the network can significantly improve the performance of the model, and the performance of deep networks is generally better than that of shallow networks. However, experiments have shown that as the network depth increases, the model performance will first improve and then decline, and then the gradient disappearance phenomenon will occur. Therefore, this study designs a residual block based on convolutional neural networks (CNNs) to avoid the problem of gradient disappearance during training. The details of the residual block are as follows:

[0039] y = F(X) + X y = F(X) + G(X)

[0040] where X represents the input information, F(·) and G(·) represent nonlinear functions, and y represents the output feature. 3) SE-Residual Block: A Squeeze-and-Excitation (SE) module is fused to enhance the performance of the residual block. The SE module reweights the channels to enable the network to adaptively focus on the importance of different channels. By modeling the importance of feature channels, the SE-Residual Block improves the sensitivity of the model to key features, thereby enhancing its representation ability and helping to mine more valuable deep features.

[0041] 4) Post-processing module: Each digital pathology image usually contains multiple tissue growth patterns, and each binary classification model has a specific classification boundary. For digital pathology images with multiple growth patterns, there may be a single pixel classification confusion problem in different binary segmentation results. Therefore, for the output of the binary segmentation result, this study designs a set of post-processing rules to synthesize a complete multi-instance segmentation image. The specific steps are as follows:

[0042] First, define a class set including the background class and six growth patterns. Second, based on the read pixel value, directly correct the background information, and modify the pixel value of 255 to the background class. In addition, for the pixels with confusion, assign them to the class corresponding to the binary classification model with the highest probability value at the pixel position.(4) Implementation details and model evaluation: The proposed integrated deep neural network (EDNN) uses the Adam optimizer during training, with a learning rate of 0.0001 to update the weights. In addition, this study selects a batch size of 16 to balance the training speed and workstation memory usage. We use the Dice coefficient to evaluate the segmentation performance of the proposed model at the patch level. To achieve this evaluation, a pathologist annotated another 6 digital pathology images, each corresponding to a major subtype (adherent, papillary, ductal, micro-papillary, solid, and complex glandular).

[0043] 2. Results:

[0044] (1) Diagnostic performance of lung adenocarcinoma grading: In the grading system based on major subtypes of lung adenocarcinoma, the agreement rate of artificial intelligence (AI) and pathologists was 75.6%, 75.2%, and 75.0% in validation cohort 1, validation cohort 2, and prospective cohort, respectively. In addition, the Kappa statistics of the three cohorts were 0.665, 0.622, and 0.649, respectively, indicating that the diagnostic results of AI and pathologists had a high degree of consistency. In the International Association for the Study of Lung Cancer (IASLC) grading system, the agreement rate of AI and pathologists was 78.7%, 80.5%, and 81.6% in validation cohort 1, validation cohort 2, and prospective cohort, respectively. The Kappa statistics of the three cohorts were 0.651, 0.648, and 0.658, respectively, also reflecting that the diagnostic results of AI and pathologists had a high degree of consistency.

[0045] (2) Application of IASLC pathological grading predicted by the model in survival prediction: The International Association for the Study of Lung Cancer (IASLC) grading system mainly affects the prognosis of stage I tumors, so we first focused on stage I tumor patients. Overall, in validation cohort 1 and validation cohort 2, both the IASLC grading diagnosed by artificial intelligence and pathologists successfully stratified the disease-free survival (DFS) of stage I patients. In addition, multivariate Cox analysis showed that after adjusting for age, pathological tumor size, and adjuvant chemotherapy, the IASLC grading diagnosed by AI (validation cohort 1: grade 1 vs. grade 3: hazard ratio [HR] = 0.08, 95% confidence interval [CI] = 0.02-0.33, p = 0.001; grade 2 vs. grade 3: HR = 0.30, 95% CI = 0.18-0.53, p < 0.0001; validation cohort 2: grade 1 vs. grade 3: HR = 0.15, 95% CI = 0.02-1.08, p = 0.059; grade 2 vs. grade 3: HR = 0.34, 95% CI = 0.19-0.60, p < 0.0001) and the IASLC grading diagnosed by pathologists (validation cohort 1: grade 1 vs. grade 3: HR = 0.03, 95% CI = 0.01-0.19, p < 0.001; grade 2 vs. grade 3: HR = 0.33, 95% CI = 0.19-0.57, p < 0.001; validation cohort 2: grade 1 vs. grade 3: HR = 0.13, 95% CI = 0.02-0.97, p = 0.046; grade 2 vs. grade 3: HR = 0.32, 95% CI = 0.18-0.60, p < 0.001) were determined as independent predictors of survival.

[0046] Furthermore, the AI-defined IASLC classification was found to be comparable to pathologists in its ability to predict disease-free survival in univariate analysis (Validation Cohort 1: C-index of AI-diagnosed IASLC classification = 0.709, 95% CI = 0.664-0.754; pathologist 1-diagnosed IASLC classification = 0.732, 95% CI = 0.691-0.773; Validation Cohort 2: C-index of AI-diagnosed IASLC classification = 0.651, 95% CI = 0.592-0.710; pathologist 2-diagnosed IASLC classification = 0.653, 95% CI = 0.596-0.70) and in multivariate Cox models (Validation Cohort 1: C-index of baseline characteristics + AI-diagnosed IASLC classification = 0.746, 95% CI = 0.701-0.791; baseline characteristics + pathologist 1-diagnosed IASLC classification = 0.760, 95% CI = 0.715-0.805; Validation Cohort 2: C-index of baseline characteristics + AI-diagnosed IASLC classification = 0.710, 95% CI = 0.645-0.775; baseline characteristics + pathologist 2-diagnosed IASLC classification = 0.720, 95% CI = 0.661-0.779). The AI-diagnosed IASLC classification also achieved comparable predictive performance in terms of 3- and 5-year area under the curve (AUC) for predicting disease-free survival. Similar results were obtained when analyzing the overall population of stage I-III patients, collectively confirming the prognostic value of AI-diagnosed IASLC classification.

[0047] 0.720, 95% CI = 0.661-0.779). The AI-diagnosed IASLC classification also achieved comparable predictive performance in terms of 3- and 5-year area under the curve (AUC) for predicting disease-free survival. Similar results were obtained when analyzing the overall population of stage I-III patients, collectively confirming the prognostic value of AI-diagnosed IASLC classification.

[0048] (3) Differentiation of complex glandular patterns from traditional acinar patterns: The identification of complex glandular components from traditional acinar patterns is crucial but also extremely challenging in diagnosing IASLC classification. It was found that, among the cases misdiagnosed by the model, tumors dominated by complex glands accounted for the highest proportion of misdiagnosis as tumors dominated by acinar (7.8%, 10.2%, and 9.8% in Validation Cohort 1, Validation Cohort 2, and Prospective Cohort, respectively). In addition, tumors dominated by acinar also accounted for the highest proportion of misdiagnosis as tumors dominated by complex glands (12.6%, 12.9%, and 11.4% in the three cohorts, respectively).

[0049] We evaluated the ability of the proposed model to distinguish between the two patterns. At the patch level, the model performed best in segmenting the two patterns (Dice coefficient of 0.843 for acinar pattern; Dice coefficient of 0.75 for complex gland pattern). At the tumor level, the AI-predicted proportion of acinar component (Spearman rank correlation coefficients of 0.828, 0.788, and 0.789 for validation cohort 1, validation cohort 2, and prospective cohort, respectively, all p < 0.0001) and complex gland component (Spearman rank correlation coefficients of 0.783, 0.717, and 0.804 for the three cohorts, respectively, all p < 0.0001) had the highest correlation with the proportion assessed by pathologists. In terms of the main subtype, the diagnostic agreement rate for acinar-dominant pattern (consistency rates of 83.3%, 78.0%, and 79.4% for the three cohorts, respectively) and complex gland-dominant pattern (consistency rates of 75.7%, 76.5%, and 80.0% for the three cohorts, respectively) was also higher than that for other main patterns.

[0050] In addition, in acinar-dominant tumors, the difference between the proportion of acinar pattern minus the proportion of complex gland pattern was significantly different between tumors that were correctly diagnosed as acinar-dominant and those that were misdiagnosed as complex gland-dominant. Similarly, in complex gland-dominant tumors, the difference between the proportion of complex gland pattern minus the proportion of acinar pattern was also significantly different between tumors that were correctly diagnosed as complex gland-dominant and those that were misdiagnosed as acinar-dominant. These results suggest that the model is promising in assisting the diagnosis of the IASLC classification system, specifically in distinguishing between complex gland pattern and acinar pattern.

[0051] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the method of the present application, several improvements and supplements can also be made, which should be considered as the protection scope of the present application.

Claims

1. A deep learning-based lung adenocarcinoma growth pattern pathological image tumor region segmentation method, characterized by: The method comprises the following steps: S1: queue: train the model, and independently label six growth patterns; S2: data preprocessing: divide each digital pathology image into image blocks; S3: digital pathology image segmentation system based on integrated deep learning; The digital pathology image segmentation system is obtained by using six binary deep learning models of the integrated deep neural network to identify and segment the six growth patterns in the digital pathology image at the pixel level, the model extracts deep features of the input image block by taking the SE-residual block as the backbone network, and updates the weight based on limited data, finally, the output of the six identification results is combined by the post-processing module, and a unified segmentation result is obtained. S4: implementation details and model evaluation: the proposed integrated deep neural network uses the Adam optimizer in the training process, and the learning rate is set to 0.0001 to update the weight.

2. The method of claim 1, wherein: The digital pathology image segmentation system comprises the following modules Single growth pattern segmentation module: a convolutional neural network is used, and six binary classification models are used for instance segmentation of digital pathology images; Residual block: the residual block formula is as follows based on the convolutional neural network: y=F(X)+X y=F(X)+G(X) Wherein, X represents input information, F(·) and G(·) represent nonlinear functions, and y represents output features. SE-residual block: the squeeze-excitation module is fused, and the SE module models the importance of feature channels by reweighting the channels; Post-processing module: the steps are as follows: first, define a class set including background class and six growth patterns; second, directly correct the background information based on the read pixel value, and correct the pixel value of 255 to the background class; in addition, for the pixels with confusion, the pixels are assigned to the class corresponding to the binary classification model with the highest probability value at the pixel position.

3. The method of claim 1, wherein: S1: queue: the six growth patterns are adherent type, papillary type, ductal type, micro-papillary type, solid type, and complex glandular type; S2: data preprocessing: each digital pathology image is divided into 256x256 pixel image blocks for subsequent segmentation processing.

4. An integrated deep learning-based digital pathology image segmentation system, characterized by: The digital pathology image segmentation system is obtained by using six binary deep learning models of the integrated deep neural network to identify and segment the six growth patterns in the digital pathology image at the pixel level, the model extracts deep features of the input image block by taking the SE-residual block as the backbone network, and updates the weight based on limited data; finally, the output of the six identification results is combined by the post-processing module, and a unified segmentation result is obtained.

5. The digital pathology image segmentation system of claim 4, wherein: The digital pathology image segmentation system comprises the following modules: Single growth pattern segmentation module: a convolutional neural network is used, and six binary classification models are used for instance segmentation of digital pathology images; Residual block: the residual block formula is as follows based on the convolutional neural network: y=F(X)+X y=F(X)+G(X) Wherein, X represents input information, F(·) and G(·) represent nonlinear functions, and y represents output features; SE-residual block: fused with squeeze-and-excitation module, the SE module reweights the channels by modeling the importance of the feature channels; Post-processing module: the steps are as follows: firstly, a class set is defined, including a background class and six growth modes; secondly, based on the read pixel value, the background information is directly corrected, the pixel with a value of 255 is corrected as the background class; in addition, for the pixels with confusion, the pixels are assigned to the class corresponding to the binary classification model with the highest probability value at the pixel position. 6.A computer readable storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-3.

Images