Method for establishing multi-drug-resistant pulmonary tuberculosis auxiliary diagnosis model based on deep learning and large model

Abstract

The invention relates to a method for establishing a multi-drug-resistant pulmonary tuberculosis auxiliary diagnosis model based on deep learning and a large model. The method comprises the following steps: S1, automatically marking a CT image of an MDR-PTB patient; s2, image omics feature extraction based on deep learning and a large model; s3, feature screening and radiomics label establishment; and S4, establishing a diagnosis model for early prognosis recognition and judgment of MDR-PTB. According to the method, the disease auxiliary diagnosis model is constructed based on AI + medical image + pre-trained large model, and by means of the technical advantages of AI in sensory cognition and deep learning, the efficiency and accuracy of identifying patients with poor MDR-PTB prognosis are improved, so that clinical diagnosis is assisted, and the probability of missed diagnosis and misdiagnosis is reduced. Meanwhile, through accurate AI image segmentation, the clinical treatment mode and path of MDR-PTB are changed, an optimal treatment scheme is provided according to the specific condition of a patient, adverse reactions of treatment are reduced, and reference is provided for auxiliary decision making of doctors.

Description

Technical Field

[0001] This application relates to the fields of deep neural networks and medical image processing technology, specifically to a method for establishing an auxiliary diagnostic model for multidrug-resistant tuberculosis based on deep learning and a large model. Background Technology

[0002] Drug-resistant tuberculosis (DR-PTB) refers to pulmonary tuberculosis caused by Mycobacterium tuberculosis that has been shown to be resistant to specific anti-tuberculosis drugs in vitro. It includes single-drug resistant tuberculosis, multidrug-resistant tuberculosis, multidrug-resistant tuberculosis (MDR-TB), rifampin-resistant tuberculosis (RR-TB), quasi-extensively drug-resistant tuberculosis, and extensively drug-resistant tuberculosis. DR-PTB is a major medical challenge in the global tuberculosis control field. According to the 2023 WHO Global Tuberculosis Report, there were approximately 410,000 (95% CI: 370,000) tuberculosis cases globally in 2022. ～ 450,000 patients were diagnosed with MDR / RR-TB[1]. The global success rate of MDR / RR-TB treatment is only 63%, while the success rate of MDR / RR-TB treatment in my country is only 51%. There is still a long way to go to achieve the goal and task of "ending tuberculosis" by 2030[1].

[0003] The prevention and treatment of drug-resistant tuberculosis (MDR-PTB) still faces many challenges: ① Complex anti-tuberculosis treatment regimens, long treatment courses, high costs, and poor treatment adherence; ② Low clinical cure rates, uneven regional development of diagnosis and treatment, and numerous adverse drug reactions. The urgent need for MDR-PTB patients in the early stages of treatment is to predict treatment outcomes in order to identify patients with poor prognoses and change treatment methods as early as possible to improve the cure rate.

[0004] The existing strategy to address this situation is to use radiomics to assist in diagnosis [2]. The radiomics process includes manually annotating segmented images and extracting omics features, but manual annotation is time-consuming and laborious. Therefore, automatic segmentation technology can help radiologists and clinicians shorten the workflow time. With the deepening of research, large models have been extended to the field of medical image segmentation. However, research shows that directly using large segmentation models for medical image segmentation has very poor results. However, the difficulty in obtaining medical data and the high cost of annotation urgently require a basic model to open up the situation, not only in the image segmentation level, but also in the data annotation level. How to transfer SAM's excellent zero-shot capability to the field of medical images is a challenge. At the same time, single omics features may have poor performance in analysis. In order to optimize the advantages of radiomics and CNN, we tried multimodal feature extraction and analysis.

[0005] Chinese patent document CN112308888 A discloses a method for grouping multimodal medical image sequences based on deep learning-based vital sign structures. The method includes: acquiring medical image information; extracting information from the acquired medical image information; establishing a multimodal deep learning AI sequence matching system; performing sequence matching processing; transmitting the processed medical image sequence groups to a display unit; and displaying the multimodal medical image sequence groups on the display unit. This invention utilizes a deep learning neural network, based on precise anatomical information from human CT and MR scans, to accurately identify and segment human bones into relatively fixed local regions. Precise human body positioning is achieved using specific bone locations, and precise positioning is achieved based on CT or MR images in a dual-modal molecular imaging system. The images are then converted to the corresponding level of the multimodal image for automatic and precise registration and display, reducing diagnostic errors caused by differences in technical capabilities and improving physician efficiency.

[0006] Currently, there are no reports on methods for establishing an auxiliary diagnostic model for multidrug-resistant tuberculosis based on deep learning and large models. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and diagnostic model for establishing an auxiliary diagnostic model for multidrug-resistant pulmonary tuberculosis based on deep learning and large models.

[0008] In a first aspect, the present invention provides a method for establishing an auxiliary diagnostic model for multidrug-resistant pulmonary tuberculosis based on deep learning and large models, comprising:

[0009] S1: Automatic annotation of CT images of MDR-PTB patients;

[0010] S2: Image omics feature extraction based on deep learning and large models;

[0011] S3: Feature selection and establishment of radiomics labels;

[0012] S4: Establish a diagnostic model for early prognostic assessment of MDR-PTB;

[0013] S1: Manually label a small amount of data on ITK-Snap, evaluate the labeled dataset, and optimize the automatic segmentation algorithm. The automatic segmentation algorithm optimization involves training the manually labeled data using the algorithm and then automatically segmenting the unlabeled data. The algorithm consists of two parts: the first part is to construct a SAM module for extracting image priors; the second part is to use nnUNet to customize the learning of the dataset to achieve medical image segmentation.

[0014] As a preferred example, the SAM image prior extraction module: by fine-tuning the SAM and generating segmentation masks for all possible locations of the image under grid hint settings, drawing the external boundary prior map of each segmentation mask based on the mask, and directly superimposing the prior map onto the original image to generate an enhanced image.

[0015] As another preferred example, nnUNet was used to customize the learning of the dataset to achieve medical image segmentation; the image input was enhanced by a network consisting of two parallel encoders.

[0016] As another preferred example, the residual UNet encoder adds residual blocks to the convolutional layers of UNet, and the fine-tuned SAM3D encoder is a pre-trained ViT fine-tuned by LoRa. The network runs in the nnUNet framework, and the process iterates continuously to obtain the region of interest.

[0017] As another preferred example, in step S2: the CT slice with the largest lung cavity is input into a deep convolutional structure, and a layered transformation is performed based on the trained convolutional parameters. Then, a label map is generated after the final convolutional layer. After removing the global max pooling layer, the label map is matched with the segmented ROI to obtain the label map of the lung cavity region. Subsequently, the statistical distribution of the lung cavity label map is quantified into DL features. Image information is extracted for each label separately, and omics features are extracted for each segmentation label. Each category extracts 107-dimensional features. Then, the information extracted by the segmentation network SAM3D in step S1 is converted into features, and then the data information is merged.

[0018] As another preferred example, each of the labels is cavity morphology, cavity wall calcification, nodule, granuloma, outer wall spiky protrusion, and treatment outcome.

[0019] As another preferred example, S3: Features with ICCs greater than 0.8 and CVs less than 15% are selected and incorporated into the construction of radiomics labels.

[0020] As another preferred example, S4: The first part uses deep learning, pre-trained large models, and radiomics to extract all features of radiomics information, and uses algorithms to build and screen models to identify feature information that is useful for early prognosis; the second part uses algorithms to build and screen models based on the multidimensional information of MDR-PTB patients at the time of enrollment, and makes early judgments on the prognosis of MDR-PTB patients, ultimately generating four models, namely the radiomics model at the time of enrollment, the dynamic model of influencing omics, the fusion model of medical big data indicators and radiomics, and the dynamic change model of the fusion model of medical big data indicators and radiomics.

[0021] As another preferred example, S4 consists of: first, constructing and screening models using support vector machine, random forest, and logistic regression algorithms; second, constructing and screening models using algorithms based on CT imaging information, disease information, TCM symptoms, epidemiological information, and clinical manifestations extracted by CNN; finally, comparing the four models to determine the optimal model, and then, within the optimal model, calculating the risk score for each patient based on the radiomics labels included in the nomota and the different weights of each independent clinical risk factor, thereby achieving risk stratification of MDR-PTB non-cure.

[0022] Secondly, the present invention provides an auxiliary diagnostic model for multidrug-resistant pulmonary tuberculosis based on deep learning and a large model, which is constructed using the above-mentioned method.

[0023] The advantages of this invention are as follows: This invention makes significant innovations in segmentation and classification technologies related to intelligent diagnosis of MDR-PTB. Firstly, regarding segmentation algorithms, addressing the issue of limited training on structures such as cavities and bronchi, a segmentation algorithm based on a large-scale SAM segmentation model and a general large-scale segmentation model is designed, achieving high-precision segmentation with only a small number of training samples. Secondly, regarding classification algorithms related to treatment efficacy prediction and recommendation, a multi-level, multi-dimensional classification framework based on radiomics features and high-dimensional model features is designed. The high-dimensional model features are intended to be obtained by fine-tuning existing large-scale medical model features, thereby improving feature representation capabilities. Currently, there is a gap in the field of early prognostic models for MDR-PTB patients. This invention constructs a disease-aided diagnostic model based on "AI + medical imaging + pre-trained large-scale model," leveraging the technological advantages of AI in sensory cognition and deep learning to improve the efficiency and accuracy of identifying patients with poor prognoses in MDR-PTB, thereby assisting clinical diagnosis and reducing the probability of missed or misdiagnosed cases. At the same time, precise AI image segmentation can also change the clinical treatment methods and pathways of MDR-PTB, providing the optimal treatment plan based on the patient's specific situation, reducing adverse reactions to treatment, and providing reference for doctors' decision-making. Attached Figure Description

[0024] Figure 1 Technology roadmap.

[0025] Figure 2 Hollow segmentation based on deep learning.

[0026] Figure 3 Feature extraction based on deep learning. Detailed Implementation

[0027] The present invention will be further described below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the description of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0028] Example 1

[0029] A method for establishing an auxiliary diagnostic model for multidrug-resistant pulmonary tuberculosis based on deep learning and large models includes the following steps, see below. Figure 1 :

[0030] Step 1: Automatic Annotation of CT Images of MDR-PTB Patients; Addressing the difficulty of annotating cavitary data in pulmonary tuberculosis, a small amount of data was first manually annotated using the ITK-Snap software. Then, doctors evaluated the annotated dataset and selected key images for manual annotation. To alleviate the burden of manual annotation for doctors and accelerate the annotation process, we designed and optimized an automatic segmentation algorithm. The manually annotated labeled data was trained using the algorithm, and then the unlabeled data was automatically segmented. The algorithm consists of two parts, see... Figure 2 .

[0031] Part 1: To address the challenges of varying lesion sizes and difficult localization, and to improve overall segmentation performance, we leverage the powerful zero-shot generalization capability of the SAM segmentation model to construct a general SAM-based image prior extraction module. Module details: By fine-tuning the model and generating segmentation masks for all possible locations in the image with a 64×64 grid cue setting, we generate segmentation masks at all reasonable locations in the image. The generated segmentation masks are then stored in a list. We further generate boundary prior maps based on the masks provided by SAM. We draw the outer boundaries of each segmentation mask in the mask list and combine all boundaries to form the boundary prior map. After generating the previous map, our next step is to use the generated boundary prior map to enhance the input image X. We choose a simple method for this enhancement: directly overlaying the prior map onto the original image to generate the enhanced image.

[0032] Part 2: Addressing the issues of limited sample size and features in segmented medical images, we fully leverage the robust feature extraction capabilities of large-scale models while utilizing nnUNet's auto-configuration capabilities for customized learning on the dataset, achieving more accurate and robust medical image segmentation. Module Details: Unlike current methods based on large-scale segmentation models, which segment 3D data by converting 3D images into 2D slices for separate analysis, the SAM3D encoder is a Vision Transformer model pre-trained on a large dataset of 3D medical image segmentation, processing the entire 3D volumetric image in a unified manner. The model trained on this massive dataset possesses powerful zero-shot capabilities. However, its segmentation ability is supervised, making it unable to self-identify segmentation targets and underlying semantics. Therefore, we use the SAM3D encoder to integrate its feature extraction advantages while leaving the dataset-specific task (identifying the region of interest to be segmented) to nnUNet. Image Input Enhancement: A network consisting of two parallel encoders. The residual UNet encoder adds residual blocks to the convolutional layers of UNet, and the fine-tuned SAM3D encoder is a pre-trained Vision Transformer (ViT) fine-tuned by LoRa. The network runs within the nnUNet framework. The process iterates continuously to obtain the region of interest (ROI).

[0033] Step 2: Imageomics Feature Extraction Based on Deep Learning and Large Models, see... Figure 3 To optimize the advantages of radiomics and CNNs while addressing their limitations in global and local features, we introduced and evaluated a hybrid architecture. The CT slice with the largest lung cavity was input into a deep convolutional structure, undergoing hierarchical transformation based on trained convolutional parameters. Six label maps were then generated after the final convolutional layer. After removing the global max-pooling layer, the label maps were directly matched with the segmented ROIs to obtain the label maps for the lung cavity regions. Subsequently, the statistical distribution of the lung cavity label maps was quantified into deep learning features. Image information was extracted separately for each label (cavity morphology, cavity wall calcification, nodule, granuloma, lateral wall spiky protrusions, treatment outcome). Simultaneously, the pyradiomics library was used to extract radiomics features for each segmentation label, yielding 107-dimensional features for each category. The information extracted by the SAM3D segmentation network in step one was then converted into features, and finally, the data was merged to provide a data foundation for building an image recognition data model. This novel architecture demonstrates the remarkable classification performance of radiomics and CNNs, and exhibits outstanding results through the synergistic effect of radiomics and deep convolutional features.

[0034] Step 3: Feature Selection and Radiomics Labeling; After extracting three types of features, the features are stitched together. Because the dimensionality of the stitched image features is too high, many features may be redundant or irrelevant. Therefore, feature dimensionality reduction is needed to remove redundant or irrelevant features, retaining only the most important features, thereby achieving data compression and simplification, and improving the efficiency and accuracy of data processing. This study uses the training set for feature selection. First, based on the re-segmentation set and the simulated slice thickness set, the repeatability and robustness of feature extraction are evaluated using ICCs and coefficients of variation (CV). Only features with ICCs greater than 0.8 and CVs less than 15% are included in the construction of radiomics labels. Feature selection, dimensionality reduction, and redundancy elimination are crucial. Feature selection algorithms include the minimum absolute shrinkage selection operator algorithm, principal component analysis, and cluster analysis. This study uses the minimum absolute shrinkage selection operator algorithm for dimensionality reduction to screen valuable radiomics features for patients with poor prognosis of MDR-TB, thereby improving the prediction accuracy and performance of the classification model. Finally, LASSO (Least Absolute Shrinkage and Selection Operator) feature selection is used to further reduce the dimensionality of high-dimensional features. LASSO is a regularization method for linear regression models. It adds an L1 regularization term to the loss function, causing the model coefficients of some unimportant feature variables to tend to zero, thereby achieving the effect of feature selection.

[0035] The LASSO feature selection process can be achieved by minimizing a loss function of the following form:

[0036]

[0037] Where Y is the target variable, X is the feature matrix, n is the number of features, θ is the model coefficients, and λ is the regularization parameter. When λ is large enough, LASSO will cause the coefficients of some features to become zero, thereby eliminating these features and achieving the purpose of feature selection.

[0038] Step Four: Establishing a diagnostic model for early prognostic assessment of MDR-PTB. The first part predicts treatment outcomes using CT images at MDR-PTB enrollment. This involves using deep learning, a pre-trained large model, and all features extracted from radiomics to construct and screen models using algorithms such as Support Vector Machine (SVM), Random Forest, and Logistic Regression to identify key features for early prognosis. The second part utilizes multidimensional information from MDR-PTB patients at enrollment, including CT images extracted by CNN, disease information, TCM symptoms, epidemiological information, and clinical manifestations. Models are constructed and screened using algorithms such as SVM, Bayesian, KNN, Random Forest, ANN, CART, or Xgboost to make early prognostic judgments for MDR-PTB patients. Ultimately, our research will generate four models: a radiomics model at enrollment, a dynamic model of influencing omics, a fusion model of medical big data indicators and radiomics, and a dynamic change model of the fusion model of medical big data indicators and radiomics. Then, the accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (ROC) of these four models were compared. The predictive performance of the models was further validated using indicators such as calibration curves and clinical decision curves (DCA) to determine the optimal model. Finally, in the optimal model, based on the radiomics labels incorporated into the nomograph and the different weights of each independent clinical risk factor, a risk score was calculated for each patient, achieving risk stratification of MDR-PTB non-cure status.

[0039] 1. This invention constructs a model for automatic medical image segmentation, intelligent feature extraction, and disease-aided diagnosis. It integrates pulmonary anatomy, medical big data, deep learning, a large model framework, and multi-dimensional information such as TCM syndromes, epidemiological information, and disease information in the early stages of MDR-PTB patient treatment. Utilizing computational aids such as digital imaging and feature point recognition, it performs high-throughput computational extraction of quantitative features from specific regions of interest in medical images, converting medical digital images into deep data for quantitative analysis. Ultimately, this data can be used for disease-aided diagnosis and classification. The model can extract numerous subtle quantitative features from medical images, establishing an early prediction model for MDR-TB clinical outcomes, assisting doctors in the auxiliary diagnosis, efficacy evaluation, and clinical prognosis prediction of MDR-PTB patients. Machine learning classifiers such as Support Vector Machine, Random Forest, and Logistic Regression are used to classify the radiomics feature dataset. The classification results are shown in Table 1. As can be seen from the table, the closer the AUC is to 1.0, the higher the realism. The best experimental results were obtained using the logistic regression classifier, with an AUC of 0.732.

[0040] Table 1. Classification performance of different algorithms on radiomics datasets

[0041]

[0042] 2. This invention provides an AI-assisted diagnostic system based on "AI + medical imaging + medical big data": (1) This system combines CT image segmentation and visualization reconstruction technology to identify factors affecting the prognosis of MDR-PTB patients early, and to intervene early in patients with risk factors, increasing drug dosage or changing from taking anti-tuberculosis drugs to CT-guided cavitary puncture injection, significantly improving the effective drug concentration of local lesions, thereby improving the success rate of MDR-PTB treatment. (2) This system can assist clinicians in accurately locating and segmenting the patient's position, and to conduct intuitive and rapid diagnostic evaluation. It fundamentally changes the traditional image reading mode that is highly dependent on labor, alleviates the pressure of medical image diagnosis to a certain extent, and helps radiologists and clinicians shorten the diagnostic workflow time and improve work efficiency.

[0043] 3. In this embodiment of the invention, a standardized, large-sample MDR-PTB data center is established through a reasonable data sharing and communication mechanism, providing high-quality data suitable for AI training and enabling its application in the medical field to have more possibilities. Relying on the project team's national collaborative platform for tuberculosis, the application of the AI-assisted diagnostic system is promoted, enhancing the domestic diagnostic and treatment capabilities for MDR-PTB.

[0044] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and additions without departing from the method of the present invention, and these improvements and additions should also be considered within the scope of protection of the present invention.

[0045] References:

[0046] [1].World Health Organization.2023Global tuberculosis report.

[0047] https: / / www.who.int / teams / global-tuberculosis-programme / tb-reports / global-tuberculosis-report-2023

[0048] [2]Li,Y.,Wang,B.,Wen,L.et al.Machine learning and radiomics for theprediction of multidrug resistance in cavitary pulmonary tuberculosis:amulticentre study.Eur Radiol 33,391–400(2023).https: / / doi.org / 10.1007 / s00330-022-08997-9。

Claims

1. A method for establishing an auxiliary diagnostic model for multidrug-resistant pulmonary tuberculosis based on deep learning and large models, characterized in that, include: S1: Automatic annotation of CT images of MDR-PTB patients; S2: Image omics feature extraction based on deep learning and large models; S3: Feature selection and establishment of radiomics labels; S4: Establish a diagnostic model for early prognostic assessment of MDR-PTB; S1: Manually label a small amount of data on ITK-Snap, evaluate the labeled dataset, and optimize the automatic segmentation algorithm. The automatic segmentation algorithm optimization involves training the manually labeled data using the algorithm and then automatically segmenting the unlabeled data. The algorithm consists of two parts: the first part is to construct a SAM module for extracting image priors; the second part is to use nnUNet to customize the learning of the dataset to achieve medical image segmentation.

2. The method for establishing according to claim 1, characterized in that, The SAM image prior extraction module: by fine-tuning the SAM and generating segmentation masks for all possible positions of the image under grid hint settings, the module draws the external boundary prior map of each segmentation mask according to the mask, and directly superimposes the prior map onto the original image to generate an enhanced image.

3. The method for establishing according to claim 1 or 2, characterized in that, Medical image segmentation was achieved by custom learning of the dataset using nnUNet; the image input was enhanced by a network consisting of two parallel encoders.

4. The method for establishing according to claim 3, characterized in that, The residual UNet encoder adds residual blocks to the convolutional layers of UNet. The fine-tuned SAM3D encoder is a pre-trained ViT that has been fine-tuned by LoRa. The network runs within the nnUNet framework, and the process iterates continuously to obtain the region of interest.

5. The method for establishing according to claim 1, characterized in that, S2: The CT slice with the largest lung cavity is input into a deep convolutional structure, and a layered transformation is performed based on the trained convolutional parameters. Then, a label map is generated after the final convolutional layer. After removing the global max pooling layer, the label map is matched with the segmented ROI to obtain the label map of the lung cavity region. Subsequently, the statistical distribution of the lung cavity label map is quantified into DL features. Image information is extracted for each label separately, and omics features are extracted for each segmentation label. Each category extracts 107-dimensional features. Then, the information extracted by the SAM3D segmentation network in step S1 is converted into features, and then the data information is merged.

6. The method for establishing according to claim 5, characterized in that, Each of the aforementioned labels is: cavity morphology, cavity wall calcification, nodule, granuloma, spiky protrusions on the outer wall, and treatment outcome.

7. The method for establishing according to claim 1, characterized in that, S3: Features with ICCs greater than 0.8 and CVs less than 15% were selected and incorporated into the construction of radiomics labels.

8. The method for establishing according to claim 1, characterized in that, S4: The first part uses deep learning, pre-trained large models, and radiomics to extract all features of radiomics information. Through algorithms, models are constructed and screened to identify feature information that is useful for early prognosis. The second part uses multidimensional information of MDR-PTB patients at the time of enrollment to construct and screen models to make early prognostic judgments for MDR-PTB patients. Ultimately, four models are generated: radiomics model at the time of enrollment, dynamic model of influencing omics, fusion model of medical big data indicators and radiomics, and dynamic change model of fusion model of medical big data indicators and radiomics.

9. The method for establishing according to claim 8, characterized in that, The S4 process consists of two parts: First, models are constructed and screened using support vector machine, random forest, and logistic regression algorithms. Second, models are constructed and screened using algorithms based on CT imaging information, disease information, TCM symptoms, epidemiological information, and clinical manifestations extracted by CNN. Finally, the four models are compared to determine the optimal model. Then, within the optimal model, the risk score for each patient is calculated based on the radiomics labels included in the nomota and the different weights of each independent clinical risk factor, thus achieving risk stratification of MDR-PTB non-cure.

10. The auxiliary diagnostic model for multidrug-resistant pulmonary tuberculosis based on deep learning and a large model, constructed according to any one of claims 1-9.

Images