Chest computed tomography image-text large model construction method, device and equipment
By extracting pseudo-labels from clinical reports and combining multi-task learning with a visual encoder based on the Mamba architecture, the problem of low accuracy in feature extraction and localization in existing large chest CT image models is solved, achieving a high-performance large chest CT image model with multi-disease detection and pixel-level localization.
Patent Information
- Application Number
- CN202510763096.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-09
- Publication Date
- 2025-11-11
AI Technical Summary
Existing large-scale chest CT image models suffer from severe information loss during feature extraction, and the detection results lack accurate spatial localization information, making it difficult to achieve pixel-level lesion localization.
By extracting disease labels and organ segmentation masks from clinical reports to generate pseudo-labels, and pre-training with a visual encoder based on multi-task learning and the Mamba architecture, a large-scale chest CT image model is generated. Fine-tuning is then performed using a medical image question-answering task to improve the model's localization accuracy.
It enables accurate detection and pixel-level precise localization of various diseases in chest CT images, enhancing the diagnostic interpretability and reliability of the model.
Smart Images

Figure CN120932853A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image processing technology, and in particular to a method, apparatus and equipment for constructing a large model of chest computed tomography images. Background Technology
[0002] CT (Computed Tomography) is widely used in clinical practice to analyze the causes of symptoms related to chest diseases, especially symptoms such as cough, shortness of breath, chest pain, and fever. Chest CT plays a crucial diagnostic role in these cases. However, a single three-dimensional chest CT image contains millions of voxels, and factors such as individual anatomical differences and variations in imaging parameters mean that radiologists need to examine CT images layer by layer, a process that is both tedious and time-consuming.
[0003] To support clinical diagnostic decisions and reduce the workload of radiologists, numerous deep learning models have been developed in recent years to assist CT diagnosis. However, due to the scarcity of clinical data, most existing deep learning models are trained on small datasets and are only designed for single diseases. This raises concerns about the generalization ability and robustness of these models in clinical practice. Therefore, this paper proposes a pre-training framework for chest CT images trained on a large-scale dataset, enabling it to detect multiple diseases and generalize, thereby achieving reliable detection and accurate lesion localization of various diseases in chest CT images. The encoder obtained through this pre-training framework can be directly used to construct large-scale chest CT image models, significantly improving the overall model performance.
[0004] Currently, pre-training of large-scale, weakly supervised chest CT image-text models based on clinical reports mainly employs the CLIP (Contrastive Language-Image Pre-training) framework. These models achieve cross-modal alignment by maximizing the similarity between matched CT images and radiological reports while minimizing the similarity between mismatched samples. However, this type of pre-training method has several key drawbacks: (1) information loss during feature extraction; and (2) the detection results lack precise spatial localization information, making it difficult to achieve pixel-level lesion localization. Summary of the Invention
[0005] This application provides a method, apparatus, and device for constructing a large model of chest CT scan images, in order to solve the problems of easy loss of features and low positioning accuracy in large models of chest CT images in related technologies.
[0006] The first aspect of this application provides a method for constructing a large model of chest computed tomography (CT) images, comprising the following steps: acquiring a three-dimensional chest CT image dataset and clinical report text; extracting disease labels of chest organs from the clinical report text using a pre-trained large language model; extracting organ segmentation masks related to chest organ diseases from the three-dimensional chest CT image dataset using a pre-trained organ model; generating pseudo-labels in tabular form based on the disease labels and organ segmentation masks; using the organ segmentation masks as coarse segmentation labels; generating a training dataset based on the coarse segmentation labels, pseudo-labels, and the three-dimensional chest CT image dataset; and using the training dataset to perform visual... The visual encoder is pre-trained, with the input being a 3D chest CT image. The decoder outputs a 3D chest CT image as a disease prediction map. Based on the pre-trained large language model, visual encoder, and multimodal projection module, a large chest CT image-text model is generated. The large chest CT image-text model is then fine-tuned using a medical image question-answering task to obtain the final large chest CT image-text model. Specifically, the 3D chest CT image related to the question-answering task is input into the visual encoder, and the multimodal projection module projects the disease prediction map output by the visual encoder into the visual labels required by the large language model. The large language model outputs the question-answering results of the question-answering task based on the visual labels and clinical report text.
[0007] Optionally, the visual encoder generates a disease anomaly map and its corresponding organ segmentation mask based on the three-dimensional chest CT image, and multiplies the disease anomaly map with the corresponding organ segmentation mask elements to obtain a disease prediction map.
[0008] Optionally, the pre-trained architecture of the visual encoder includes a feature extractor, an organ segmentation decoder, and a disease anomaly map decoder. The feature extractor extracts spatial and contextual features from the 3D chest CT image, the organ segmentation decoder generates an organ segmentation mask based on the spatial and contextual features, and the disease anomaly map decoder generates a disease anomaly map based on the spatial and contextual features.
[0009] Optionally, the feature extractor includes activation functions, instance-normalized 3D convolutional layers, gated spatial convolutions, and Mamba layers. The organ segmentation decoder and the disease anomaly graph decoder have the same structure, employing an encoder-decoder structure, and connecting high-level semantic features with low-level features through skip connections.
[0010] Optionally, the calculation formula for the Mamba layer in the feature extractor is:
[0011] x i+1 =Unflatten(Mamba(Flatten(LayerNorm(x i ))))+x i ;
[0012] Where, x i For the input features of the i-th layer, LayerNorm(x) i ) is for the input feature x i Layer normalization is performed. Flatten(·) is the operation that flattens the feature tensor into a sequence form. Mamba(·) is the Mamba computation block of the Mamba state-space model. Unflatten(·) is the operation that reshapes the feature sequence back into its original dimensional structure. i+1 The output features of layer i+1;
[0013] Input feature x i After linear transformation, it is divided into a first component and a second component. The formulas for obtaining the first and second components are as follows:
[0014] y,z=split(linear(x i ));
[0015] Where, linear(x) i ) represents the linear transformation operation, y is the first component, and z is the second component.
[0016] Optionally, the visual encoder dynamically adjusts the weights of the loss function during training, wherein the dynamic adjustment formula for the weights of the loss function is:
[0017]
[0018] Where λ is the weight, initial weight For the initial weights, decay rate The attenuation rate, total epochs The total number of training rounds is 1, and the current training round is epoch.
[0019] Optionally, the multimodal projection module includes an attention pooling layer and a two-layer perceptron, wherein the attention pooling layer uses learned latent queries and multi-head cross-attention to compress the features output by the visual encoder into a fixed-length sequence of image labels.
[0020] Optionally, the large language model includes a Transformer network, an attention head, an embedding dimension, and a feedforward network dimension. During the tuning phase of the large language model, the large language model is tuned using the tags and disease labels added in the word segmenter.
[0021] A second aspect of this application provides a large-scale model construction apparatus for chest computed tomography (CT) images, comprising: an acquisition module for acquiring a three-dimensional chest CT image dataset and clinical report text; an extraction module for extracting disease labels of chest organs from the clinical report text using a pre-trained large language model, extracting organ segmentation masks related to chest organ diseases from the three-dimensional chest CT image dataset using a pre-trained organ model, and generating pseudo-labels in tabular form based on the disease labels and organ segmentation masks; and a training module for using the organ segmentation masks as coarse segmentation labels, generating a training dataset based on the coarse segmentation labels, pseudo-labels, and the three-dimensional chest CT image dataset, and using the training... The dataset is used to pre-train a visual encoder, which takes a 3D chest CT image as input and outputs a 3D chest CT image as a disease prediction map after being decoded. A generation module is used to generate a large-scale chest CT image-text model based on the pre-trained large language model, visual encoder, and multimodal projection module. This model is then fine-tuned using a medical image question-answering task to obtain the final large-scale chest CT image-text model. Specifically, the 3D chest CT image relevant to the question-answering task is input into the visual encoder, and the multimodal projection module projects the disease prediction map output by the visual encoder into visual labels required by the large language model. The large language model outputs the question-answering results based on the visual labels and clinical report text.
[0022] Optionally, the visual encoder generates a disease anomaly map and its corresponding organ segmentation mask based on the three-dimensional chest CT image, and multiplies the disease anomaly map with the corresponding organ segmentation mask elements to obtain a disease prediction map.
[0023] Optionally, the pre-trained architecture of the visual encoder includes a feature extractor, an organ segmentation decoder, and a disease anomaly map decoder. The feature extractor extracts spatial and contextual features from the 3D chest CT image, the organ segmentation decoder generates an organ segmentation mask based on the spatial and contextual features, and the disease anomaly map decoder generates a disease anomaly map based on the spatial and contextual features.
[0024] Optionally, the feature extractor includes activation functions, instance-normalized 3D convolutional layers, gated spatial convolutions, and Mamba layers. The organ segmentation decoder and the disease anomaly graph decoder have the same structure, employing an encoder-decoder structure, and connecting high-level semantic features with low-level features through skip connections.
[0025] Optionally, the calculation formula for the Mamba layer in the feature extractor is:
[0026] x i+1 =Unflatten(Mamba(Flatten(LayerNorm(x i ))))+x i;
[0027] Where, x i For the input features of the i-th layer, LayerNorm(x) i ) is for the input feature x i Layer normalization is performed. Flatten(·) is the operation that flattens the feature tensor into a sequence form. Mamba(·) is the Mamba computation block of the Mamba state-space model. Unfaltten(·) is the operation that reshapes the feature sequence back into its original dimensional structure. i+1 The output features of layer i+1;
[0028] Input feature x i After linear transformation, it is divided into a first component and a second component. The formulas for obtaining the first and second components are as follows:
[0029] y,z=split(linear(x i ));
[0030] Where, linear(x) i ) represents the linear transformation operation, y is the first component, and z is the second component.
[0031] Optionally, the visual encoder dynamically adjusts the weights of the loss function during training, wherein the dynamic adjustment formula for the weights of the loss function is:
[0032]
[0033] Where λ is the weight, initial weight For the initial weights, decay rate The attenuation rate, total epochs The total number of training rounds is 1, and the current training round is epoch.
[0034] Optionally, the multimodal projection module includes an attention pooling layer and a two-layer perceptron, wherein the attention pooling layer uses learned latent queries and multi-head cross-attention to compress the features output by the visual encoder into a fixed-length sequence of image labels.
[0035] Optionally, the large language model includes a Transformer network, an attention head, an embedding dimension, and a feedforward network dimension. During the tuning phase of the large language model, the large language model is tuned using the tags and disease labels added in the word segmenter.
[0036] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for constructing a large model of chest computed tomography images as described in the above embodiments.
[0037] Therefore, this application has the following beneficial effects:
[0038] This application embodiment can extract disease labels of thoracic organs from clinical report text using a pre-trained large language model, extract organ segmentation masks related to thoracic organ diseases from a 3D chest CT image dataset using a pre-trained organ model, generate pseudo-labels in tabular form based on disease labels and organ segmentation masks, without relying on pixel-level annotation data of real lesions, use organ segmentation masks as coarse segmentation labels, generate a training dataset based on coarse segmentation labels, pseudo-labels and 3D chest CT image dataset, train the visual encoder using the training dataset, generate a large chest CT image-text model based on the trained large language model, visual encoder and multimodal projection module, and fine-tune the large chest CT image-text model using a medical image question-answering task to improve the accuracy of the model in locating thoracic diseases.
[0039] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0040] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0041] Figure 1 This is a flowchart of a method for constructing a large model of chest computed tomography images according to an embodiment of this application;
[0042] Figure 2 This is a diagram illustrating the specific implementation process of the Y-Mamba visual encoder provided according to an embodiment of this application;
[0043] Figure 3 This is a schematic diagram of the structure of a large-scale chest CT image model provided according to an embodiment of this application;
[0044] Figure 4 This is a schematic diagram illustrating the Chest-OMDL pre-training and testing process according to an embodiment of this application;
[0045] Figure 5 A heatmap comparing the performance metrics of Chest-OMDL with other methods according to embodiments of this application;
[0046] Figure 6 A heatmap comparing the performance metrics of Chest-OMDL with other methods according to one embodiment of this application;
[0047] Figure 7 This is a schematic diagram of the anomaly location structure provided according to an embodiment of this application;
[0048] Figure 8 This is a schematic diagram of the output of a large chest CT image model according to an embodiment of this application;
[0049] Figure 9 This is a graph showing the evaluation metrics of the large chest CT image model provided according to the embodiments of this application compared with other models;
[0050] Figure 10 This is an example diagram of a large model construction device for chest computed tomography images provided according to an embodiment of this application;
[0051] Figure 11 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation
[0052] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0053] Before describing the solution of this application, let me first introduce the relevant technologies of this application.
[0054] Constructing a large-scale chest CT image-text model to achieve deep fusion and intelligent understanding of CT images and medical text is of great value in improving medical diagnostic efficiency and assisting clinical decision-making. Therefore, it is urgent to build a chest CT pre-training framework trained on a large-scale dataset, enabling it to detect multiple diseases and generalize, thereby achieving reliable detection and accurate lesion localization of various diseases in chest CT images. The encoder obtained through this pre-training framework can be directly used to construct the large-scale chest CT image-text model, significantly improving the overall performance of the model.
[0055] However, the development of large-scale pre-training faces significant challenges in data acquisition. Annotating medical images is not only time-consuming and labor-intensive, but also requires the involvement of radiology experts to ensure the professionalism of the annotations. Furthermore, technical challenges such as noise and individual differences in annotation must be addressed. To address this data bottleneck, this paper proposes a promising new approach by extracting the massive amounts of accumulated label information from clinical radiology reports and constructing a weakly supervised learning training framework based on these reports.
[0056] Currently, pre-training of large-scale, weakly supervised chest CT image models based on clinical reports primarily employs a contrastive learning framework based on CLIP (Contrastive Language-Image Pre-training). Representative works include CT-CLIP and fVLM, which achieve cross-modal alignment by maximizing the similarity between matched CT images and radiological reports while minimizing the similarity between mismatched samples. However, this type of pre-training method suffers from several key drawbacks.
[0057] First, there is a serious problem of information loss during feature extraction. CLIP-based pre-trained models typically compress the entire CT volume into a single global feature vector, and the radiology report is also encoded into a uniform text representation. This global encoding mechanism leads to a significant loss of detailed lesion information. Specifically, key lesions in chest CT (such as pulmonary nodules, ground-glass opacities, and emphysema) often occupy only a very small area of the overall image. During global feature extraction, these crucial local features are easily diluted or completely lost. Simultaneously, the model cannot establish an accurate correspondence between the textual descriptions in the clinical report and specific regions in the CT scan, resulting in a severe loss of spatial semantic information.
[0058] Secondly, the detection results of these models lack precise spatial localization information, making it difficult to achieve pixel-level lesion localization. In clinical applications, knowing only the existence of a lesion without knowing its specific location, or only having organ-level visualized diagnostic information, has limited value. Furthermore, the model's inability to provide visualized decision-making evidence significantly reduces the interpretability and credibility of this pre-training method, making it difficult for doctors to verify and trust the AI system's judgments.
[0059] Therefore, there is an urgent need to explore better pre-training methods to further improve local feature parsing and spatial localization capabilities. Through innovative technical solutions, a high-performance large-scale model of chest CT images and text can be constructed to achieve deep integration and intelligent understanding of CT images and medical text, thereby promoting the development of medical imaging AI systems towards higher interpretability and clinical applicability.
[0060] To this end, this application provides a method for constructing a large-scale chest CT image model. In this method, instead of relying on pixel-level labeled data of real lesions, weakly supervised labels with local correlations are extracted from clinical reports using advanced natural language processing technology. By combining multi-task learning, multi-instance learning, and efficient sequence modeling based on the Mamba architecture, accurate detection and pixel-level precise localization of various diseases are achieved. Finally, a pre-trained Mamba encoder is extracted to construct a high-performance large-scale chest CT image model.
[0061] Specifically, Figure 1This is a flowchart illustrating a method for constructing a large model of a chest computed tomography image provided in an embodiment of this application.
[0062] like Figure 1 As shown, the method for constructing a large model of a chest computed tomography (CT) scan image includes the following steps:
[0063] In step S101, a three-dimensional chest computed tomography (CT) image dataset and clinical report text are acquired.
[0064] In step S102, disease labels for thoracic organs are extracted from clinical report text using a pre-trained large language model, and organ segmentation masks related to thoracic organ diseases are extracted from a 3D chest CT image dataset using a pre-trained organ model. Pseudo-labels in tabular form are generated based on the disease labels and organ segmentation masks.
[0065] Among them, the organ model can be the SAT (Segment Anything by Text) model, which is a knowledge augmentation method that uses natural language prompts to segment three-dimensional medical data; the disease labels can include various labels such as cardiomegaly, pericardial effusion, and hiatal hernia; the organ segmentation mask can include segmentation masks for multiple organs such as lungs, trachea and bronchi, pleura, mediastinum, heart and esophagus.
[0066] It is understood that the embodiments of this application can use a pre-trained large language model to extract disease labels of chest organs from clinical report text, and use a pre-trained organ model to extract chest-related organ segmentation masks from a three-dimensional chest CT dataset, and generate pseudo-labels in tabular form based on the disease labels and organ segmentation masks.
[0067] It should be noted that when the dataset is large enough, the trained deep learning model is robust to noisy labels. Therefore, the labels obtained by directly using the output of existing automated methods as training labels are also called pseudo-labels.
[0068] Specifically, the embodiments of this application are used for extracting pseudo-labels from disease and organ masks, including:
[0069] To extract disease labels from CT reports, a pre-trained language model (such as the RadBERT-RoBERTa-4m model used in this application) can be fine-tuned to identify disease labels from free text reports. This application's embodiment utilizes Rad-BERT to extract 16 disease labels from CT-RATE, including: cardiomegaly, pericardial effusion, hiatal hernia, lymphadenopathy, emphysema, atelectasis, pulmonary nodules, ground-glass opacities in the lungs, sequelae of pulmonary fibrosis, pleural effusion, mosaic attenuation pattern, peribronchial thickening, consolidation, bronchiectasis, interlobular septal thickening, and coronary artery wall calcification. Based on medical knowledge, these disease labels are associated with different organs, thus providing coarse localization information. A tabular pseudo-label was created for each subject to show the correspondence between the extracted 16 disease labels and 6 specific organs, as shown in Table 1. Table 1 is a table of disease-organ correspondences.
[0070] Table 1
[0071]
[0072]
[0073] As shown in Table 1, this application retains segmentation masks for six disease-related organs in nine usable regions, including the lungs, trachea and bronchi, pleura, mediastinum, heart, and esophagus.
[0074] In step S103, the organ segmentation mask is used as the coarse segmentation label. A training dataset is generated based on the coarse segmentation label, pseudo label and three-dimensional chest CT image dataset. The visual encoder is pre-trained using the training dataset. The input of the visual encoder is the three-dimensional chest CT image, and the output of the three-dimensional chest CT image after decoding is the disease monitoring image.
[0075] The visual encoder can be a Y-Mamba visual encoder.
[0076] It is understood that in the embodiments of this application, the organ segmentation mask can be used as a coarse segmentation label, and a training dataset can be generated based on the coarse segmentation label, pseudo label and three-dimensional chest CT image dataset. The visual encoder is trained using the training dataset, wherein the input of the visual encoder is a three-dimensional chest CT image and the output of the three-dimensional chest CT image is a disease prediction map.
[0077] In this embodiment, the visual encoder generates a disease anomaly map and its corresponding organ segmentation mask based on a three-dimensional chest CT image. The disease anomaly map and its corresponding organ segmentation mask elements are multiplied together to obtain a disease prediction map.
[0078] Specifically, the Y-Mamba implementation in this application can better capture spatial correlations and complex disease-organ interaction features in chest CT scans, thereby improving the effectiveness of anomaly detection and pixel-level anatomical localization. One branch of the model is responsible for organ segmentation, outputting binary masks for multiple organs, while the other branch outputs a pixel-level anomaly probability map for each disease, thus forming task complementarity. Furthermore, by multiplying the disease anomaly map with the corresponding organ segmentation mask, an organ-specific disease prediction map with clear anatomical significance is obtained, ensuring both the accuracy of localization and enhancing diagnostic interpretability.
[0079] In this embodiment, the pre-trained architecture of the visual encoder includes a feature extractor, an organ segmentation decoder, and a disease anomaly map decoder. The feature extractor extracts spatial features and contextual features from a three-dimensional chest CT image. The organ segmentation decoder generates an organ segmentation mask based on the spatial features and contextual features. The disease anomaly map decoder generates a disease anomaly map based on the spatial features and contextual features.
[0080] Specifically, the visual encoder of this application is a deep learning model trained based on multi-task learning, designed for organ segmentation and disease anomaly detection in medical images. Y-Mamba consists of three main components: a feature extractor, an organ segmentation decoder, and a disease anomaly map generator. The feature extractor encodes multi-scale representations of the input medical image, while the two decoders utilize these representations to perform their respective tasks.
[0081] In this embodiment, the feature extractor includes an activation function, an instance-normalized 3D convolutional layer, a gated spatial convolution, and a Mamba layer. The organ segmentation decoder and the disease anomaly graph decoder have the same structure, employing an encoder-decoder structure, and connecting high-level semantic features with low-level features through skip connections.
[0082] In this embodiment, the calculation formula for the Mamba layer in the feature extractor is:
[0083] x i+1 =Unflatten(Mamba(Flatten(LayerNorm(x i ))))+x i ;
[0084] in,
[0085] x i For the input features of the i-th layer, LayerNorm(x) i ) is for the input feature x iLayer normalization is performed. Flatten(·) is the operation that flattens the feature tensor into a sequence form. Mamba(·) is the Mamba computation block of the Mamba state-space model. Unflatten(·) is the operation that reshapes the feature sequence back into its original dimensional structure. i+1 The output features of layer i+1;
[0086] Input feature x i After linear transformation, it is divided into a first component and a second component. The formulas for obtaining the first and second components are as follows:
[0087] y,z=split(linear(x i ));
[0088] Where, linear(x) i ) represents the linear transformation operation, y is the first component, and z is the second component.
[0089] In this embodiment, the visual encoder dynamically adjusts the weights of the loss function during training, wherein the dynamic adjustment formula for the weights of the loss function is:
[0090]
[0091] Where λ is the weight, initial weight For the initial weights, decay rate The attenuation rate, total epochs The total number of training rounds is 1, and the current training round is epoch.
[0092] Specifically, the implementation process of the Y-Mamba visual encoder in this application embodiment is as follows: Figure 2 As shown below:
[0093] Feature Extractor: The feature extraction module employs a multi-resolution encoding strategy, utilizing 3D convolutional layers (Conv3D) with Leaky ReLU activation and instance normalization (InstanceNorm3D) to extract spatial and contextual features. To enhance representation capabilities, this application combines GSC (Gated Spatial Convolution) and deeper layers of Mamba. The downsampling process is implemented through stride convolution, progressively reducing spatial resolution while increasing feature channels to generate hierarchical feature representations.
[0094] Gated Spatial Convolution (GSC): The GSC module enhances spatial feature learning by combining multiple receptive fields. It consists of parallel 3×3 convolutions, which are then normalized and activated. The output is gated through an additional 1×1 convolution that learns spatial importance weights. This mechanism effectively suppresses irrelevant features while enhancing key spatial patterns.
[0095] Mamba Layers: Each Mamba layer begins with Layer Normalization. This step normalizes the input sequence by calculating the root mean square value of the input activations and is crucial for preventing gradient explosion in deep networks. After normalization, the Mamba module processes the input sequence, and the resulting output is combined with the residuals of the input, as shown in the following formula:
[0096] x i+1 =Unflatten(Mamba(Flatten(LeyerNorm(x i ))))+x i ;
[0097] Initially, the input feature x i After a linear transformation, it is divided into two components: y and z. These two components are obtained through the following operation:
[0098] y,z=split(linear(x i ));
[0099] Next, segment y is processed and activated through 1D convolution, and further processed using a selective scan model (SSM), as shown in the following formula:
[0100] y i =SSM(SiLu(1DConv(y)));
[0101] At the same time, the activated z segment serves as the gating vector, and is used in conjunction with y. i Perform element-wise multiplication. After processing by the Mamba module, y i After passing through another linear layer, the module's output x is generated. i+1 .
[0102] Organ segmentation decoder: The organ segmentation decoder follows an encoder-decoder structure, where high-level semantic features are progressively upsampled and connected to corresponding low-level features via skip connections. This part uses transposed convolution (TransConv) for upsampling, while an MLP (Multi-Layer Perceptron) further refines the feature representation. The final segmentation mask contains binary segments of the six organs, accurately segmenting the location of each organ.
[0103] Disease Anomaly Map Decoder: Parallel to the segmentation decoder, it uses the same encoded features to predict anomalous regions, generating anomaly maps corresponding to each disease. Each pixel value ranges from 0 to 1, representing the probability that the location belongs to an anomalous region of a specific disease. This decoder has a similar structure to the segmentation decoder but adds an additional feature fusion mechanism to integrate deeper semantic cues. Finally, it integrates the multi-scale outputs through element-wise addition steps to generate anomaly heatmaps.
[0104] To achieve organ-specific multi-disease detection, this embodiment of the application uses Y-Mamba to perform element-wise multiplication between the anomaly map of each disease and its corresponding organ segmentation mask. For example, the anomaly map of pulmonary nodules is multiplied by the lung organ mask, and this operation yields an organ-specific disease prediction map. The specific correspondence between each disease and organ is shown in Table 1. During the evaluation process, a multi-instance learning strategy is adopted to select the predicted values of the top k pixels from the above prediction map and take their average value as the final predicted probability of the disease in the current organ, thereby achieving the disease classification task.
[0105] Furthermore, the following embodiment illustrates the adjustment of the weights of the loss function of the visual encoder in this application.
[0106] This embodiment used 22,620 CT images from the publicly available chest CT dataset CT-RATE for system training, while retaining 1,508 images for early stopping and threshold selection. Among these 1,508 images, this embodiment selected the model with the highest AUROC (Area Under the Receiver Operating Characteristic Curve) and determined the optimal threshold using the Jaccard index. The entire training framework employed the AdamW optimizer and utilized four NVIDIA A800 GPUs for computation, requiring approximately 2 days and 20 hours to complete one training cycle. The training was set to 150 epochs with a batch size of 8. During training, the weights of the loss function were dynamically adjusted according to the following formula:
[0107]
[0108] This formula ensures that the weights of the segmentation task gradually decay during training.
[0109] In step S104, a large-scale chest CT image model is generated based on the pre-trained large language model, visual encoder, and multimodal projection module. The large-scale chest CT image model is then fine-tuned using a medical image question-and-answer task to obtain the final large-scale chest CT image model. Specifically, the three-dimensional chest CT images related to the question-and-answer task are input into the visual encoder. The multimodal projection module projects the disease prediction map output by the visual encoder into the visual labels required by the large language model. The large language model outputs the question-and-answer results of the question-and-answer task based on the visual labels and clinical report text.
[0110] Among them, the large-scale model of chest CT images can be called Chest-CHAT.
[0111] It is understood that the embodiments of this application can generate a large chest CT image model based on a pre-trained large language model, visual encoder, and multimodal projection module, and then fine-tune the large chest CT image model using a medical image question-answering task to obtain the final large chest CT image model. The specific question-answering task implementation process is as follows:
[0112] The 3D chest CT images related to the question-answering task are input into the visual encoder. The multimodal projection module projects the disease prediction map output by the visual encoder into the visual labels required by the large language model. The large language model outputs the question-answering results of the question-answering task based on the visual labels and clinical report text.
[0113] In this embodiment, the multimodal projection module includes an attention pooling layer and a two-layer perceptron. The attention pooling layer uses learned latent queries and multi-head cross-attention to compress the features output by the visual encoder into a fixed-length image label sequence.
[0114] It is understood that the implementation of the multimodal projection module in this application includes an attention pooling layer and a multilayer perceptron with a two-layer structure. This attention pooling mechanism compresses the final layer features of the encoder backbone into a fixed-length sequence of image labels by using 256 learned latent queries and multi-head cross-attention. This design not only optimizes the training and inference processes but also ensures that the sequence length fits within the context window of the LLM, allowing the model to focus on key features.
[0115] In this embodiment, the large language model includes a Transformer network, an attention head, an embedding dimension, and a feedforward network dimension. During the adjustment phase of the large language model, the large language model is adjusted using the tags and disease labels added by the word segmenter.
[0116] It is understood that the large language model in this application embodiment includes a Transformer network, an attention head, an embedding dimension, and a feedforward network dimension. In the large language model adjustment stage, the large language model is adjusted using the tags and disease labels added by the word segmenter.
[0117] Specifically, the structure of the large chest CT image model Chest-CHAT in this application embodiment is as follows: Figure 3 As shown, Chest-CHAT is developed based on an adapted version of the LLaVa (Large Language and Vision Assistant) framework, aiming to combine a pre-trained Y-Mamba visual encoder, a multimodal projection module, and a large language model (LLM) to enhance its multimodal processing capabilities. First, the Y-Mamba encoder is responsible for compressing 3D chest CT volume data into low-dimensional features. This encoder, pre-trained on the CT-RATE dataset for multi-disease monitoring and localization, effectively captures and extracts key medical features. These features are then processed by the multimodal projection module, transforming them into an embedding space suitable for the large language model to achieve effective alignment with textual tags. Finally, the large language model utilizes these visual and linguistic tags to generate relevant contextual responses.
[0118] The pre-trained Y-Mamba encoder efficiently transforms 3D CT image data into low-dimensional features, providing a solid foundation for subsequent processing. Next, the implementation of the multimodal projection module includes an attention pooling layer and a two-layer multilayer perceptron. This attention pooling mechanism compresses the final layer features of the encoder backbone into a fixed-length sequence of image labels using 256 learned latent queries and multi-head cross-attention. This design not only optimizes the training and inference processes but also ensures that the sequence length fits within the context window of the LLM, allowing the model to focus on key features.
[0119] When configuring the LLM, Chest-CHAT adopted the open-source Vicuna 13B model, which is a fine-tuned version of Llama 2. Its structure includes a 40-layer Transformer network, 40 attention heads, 5,120 embedding dimensions, and 18,432 feedforward network dimensions, with a maximum context length of 4,096 tokens. To enable the model to flexibly adapt to different output formats, this application added four special tokens to the tokenizer:<long answer> ,<short answer> ,<report genera-tion> and<multiple choice> These tags guide the model to perform specific types of tasks during training and inference.
[0120] During the training phase, Chest-CHAT is fine-tuned using medical instructions. In one specific embodiment of the invention, it utilizes long-answer, short-answer, multiple-choice, and report generation question-answer pairs from the CT-RATE-curated open-source VQA dataset for training. During this process, the multimodal projection module and the LLM (Low-Rank Adaptation) weights with 128 rank and 256 alpha are fine-tuned. In addition to the plain text question and answer, four special tags are added to each question-answer pair to enable the model to effectively learn and distinguish different output patterns. These tags are used during inference to select different response modes for the model.
[0121] In summary, the method for constructing a large chest CT image model according to embodiments of this application includes a weakly supervised learning pre-training framework based on clinical reports (Chest-OMDL) and a constructed large chest CT image model (Chest-CHAT).
[0122] The complete technical solution comprises three main stages: pseudo-label extraction, multi-disease detection and localization pre-training, and large-scale image-text model fine-tuning. In the pseudo-label extraction stage, natural language processing techniques are used to extract organ-specific disease labels from clinical reports. Then, a pre-trained organ model is used to extract disease-related organ masks from the CT dataset. In the multi-disease detection and localization pre-training stage, the extracted disease labels and organ masks are used to train the proposed Y-Mamba model. This model simultaneously predicts anomaly maps for each disease and the corresponding segmentation masks for each anatomical organ. This results in multi-disease detection capabilities and pixel-level localization capabilities. In the large-scale image-text model fine-tuning stage, the encoder of the pre-trained Y-Mamba model is extracted and combined with a multimodal projection module and a large language model to form a multimodal image-text model. The multimodal image-text model is fine-tuned on the VQA (Visual Question Answering) dataset obtained from clinical reports, ultimately resulting in a multimodal model capable of handling diverse medical image question answering tasks.
[0123] In this application, the Chest-OMDL framework and Y-Mamba model architecture are proposed for the pre-training stage of disease detection and localization. The Chest-OMDL framework constructs a complete process around the goal of "organ-specific multi-disease detection and localization," including weakly supervised pseudo-label generation, multi-branch prediction, and multi-instance aggregation classification. Furthermore, since this framework can handle multimodal data and be trained on large-scale datasets, the trained backbone network can serve as a base model applicable to diverse downstream tasks.
[0124] Chest-OMDL is designed based on multi-task learning. It achieves collaborative optimization of representation layer sharing and task-customized output by jointly training a multi-branch network structure (Y-Mamba) for segmentation and anomaly detection. Benefiting from the advantages of the Mamba structure in long-sequence modeling and efficient information flow, Y-Mamba can better capture spatial correlations and complex disease-organ interactions in chest CT scans, thereby improving the effectiveness of anomaly detection and pixel-level anatomical localization. One branch of the model is responsible for organ segmentation, outputting binary masks for multiple organs, while the other branch outputs pixel-level anomaly probability maps for each disease, thus forming task complementarity. Furthermore, by multiplying the disease anomaly map with the corresponding organ segmentation mask, an organ-specific disease prediction map with clear anatomical significance is obtained, ensuring both the accuracy of localization and enhancing diagnostic interpretability.
[0125] In the prediction output and label mapping stages, Chest-OMDL introduces a multi-instance learning mechanism. Each case is treated as a bag, while pixels or regions are treated as instances. By aggregating the results of the top k high-confidence pixels in the prediction map, the model achieves bag-level (case-level) probability predictions, which are then compared with weak labels obtained from reports. This strategy significantly improves the discriminative ability under weak label conditions and ultimately achieves closed-loop processing for multi-disease classification and localization tasks.
[0126] Chest-OMDL Training and Testing Process: Training Phase as follows Figure 4 As shown, the framework first extracts pseudo-labels for 16 diseases from chest CT reports using a pre-trained RadBERT model. Simultaneously, it utilizes the SAT model to generate coarse segmentation masks for six organs from the 3D chest CT volume as weak supervision signals. Then, it employs a Y-Mamba model for multi-task learning. This model comprises three core components: a feature extractor, an organ segmentation decoder, and a disease anomaly map generator. This is achieved by combining the segmentation loss L... seg,i and anomaly detection loss (L abn,i The Y-Mamba model is trained end-to-end using a multi-task loss function. During the testing phase, the model directly infers the input 3D chest CT volume and outputs organ-level segmentation masks and disease anomaly maps. It achieves organ-specific multi-disease detection and lesion localization through element-level multiplication operations, and finally generates interpretable lesion heatmaps and classification results.
[0127] Y-Mamba is a deep learning model trained using multi-task learning, designed for organ segmentation and disease anomaly detection in medical images. Y-Mamba consists of three main components: a feature extractor, an organ segmentation decoder, and a disease anomaly map generator. The feature extractor encodes multi-scale representations of the input medical image, while the two decoders utilize these representations to perform their respective tasks.
[0128] The following is a specific embodiment illustrating the detailed process of the method for constructing a large model of chest computed tomography images according to this application:
[0129] 1. In the pseudo-label extraction stage, natural language processing technology is used to extract organ-specific disease labels from clinical reports, and organ models are used to extract disease-related organ masks from the CT dataset.
[0130] The pseudo-label extraction stage includes:
[0131] (1) Disease labels were extracted from clinical reports using a pre-trained language model, including 16 types of chest diseases such as cardiomegaly, pericardial effusion, and hiatal hernia.
[0132] (2) The trained SAT model is used to segment the three-dimensional medical data and extract the segmentation mask of six organs, namely the lung, trachea and bronchi, pleura, mediastinum, heart and esophagus.
[0133] (3) Establish the correspondence between disease labels and related organs to form pseudo-labels in tabular form.
[0134] 2. In the pre-training stage of multi-disease detection and localization, the extracted disease labels and organ masks are used to train the Y-Mamba model, while predicting the anomaly map of each disease and the segmentation mask of the corresponding organ.
[0135] The Y-Mamba model includes the following components:
[0136] (1) Feature extractor: adopts a multi-resolution coding strategy, combining gated spatial convolution and Mamba layer to extract spatial and contextual features;
[0137] (2) Organ segmentation decoder: The encoder-decoder structure is adopted. High-level semantic features are connected with low-level features through skip connections to generate binary segmentation masks for 6 organs.
[0138] (3) Disease anomaly map decoder: using the same coding features to generate an anomaly probability map for each disease, where each pixel value represents the probability that the location belongs to the anomaly region of a specific disease.
[0139] The Mamba layer in the feature extractor is calculated using the following formula:
[0140] x i+1 =Ubflatten(Mamba(Flatten(LayerNorm(x i ))))+x i ;
[0141] Where the input feature x iAfter a linear transformation, it is divided into two components: y and z. These two components are obtained through the following operation:
[0142] y,z=split(linear(x i ));
[0143] The Gated Spatial Convolution (GSC) module consists of parallel 3×3 convolutions, which are then normalized and activated. The output is gated by 1×1 convolutions to learn spatial importance weights.
[0144] Organ-specific multi-disease detection is achieved by element-wise multiplication of the abnormal map of each disease with its corresponding organ segmentation mask, and a multi-instance learning strategy is adopted to select the average of the predicted values of the first k pixels in the prediction map as the final prediction probability.
[0145] The weights of the loss function during the pre-training phase are dynamically adjusted according to the following formula:
[0146]
[0147] This formula ensures that the weights of the segmentation task gradually decay during training.
[0148] 3. In the fine-tuning stage of the large image-text model, the pre-trained Y-Mamba model encoder is extracted and combined with the multimodal projection module and the large language model to form a multimodal image-text model, which is then fine-tuned on the VQA dataset obtained from the clinical reports.
[0149] Among them, the large image and text model Chest-CHAT includes:
[0150] (1) A pre-trained Y-Mamba visual encoder compresses three-dimensional chest CT volume data into low-dimensional features;
[0151] (2) Multimodal projection module, including attention pooling layer and two-layer perceptron, transforms visual features into an embedding space that meets the requirements of large language models;
[0152] (3) Large language model, which uses visual and linguistic tags to generate relevant contextual responses.
[0153] The multimodal projection module compresses the final layer features of the encoder backbone into a fixed-length sequence of image labels by using 256 learned latent queries and multi-head cross-attention.
[0154] The large language model uses Vicuna 13B and adds four special tags to the tokenizer:<long answer> ,<short answer> ,<report generation> and<multiple choice> It is used to guide the model to complete a specific type of task.
[0155] This application also provides a large-scale chest CT image model system based on clinical reports, including:
[0156] Y-Mamba pre-trained module for compressing 3D chest CT volume data into low-dimensional features;
[0157] The image-text multimodal conversion module is used to convert visual features into representations suitable for large language models;
[0158] The medical dialogue generation module is used to generate medical responses based on CT images and text queries.
[0159] The system can handle various types of medical question-and-answer tasks, including long answers, short answers, multiple-choice questions, and report generation, without requiring large-scale manually labeled data. It can be pre-trained and fine-tuned using information from clinical reports.
[0160] Furthermore, it should be noted that the embodiments of this application also employ a comprehensive scientific evaluation system to verify system performance. In terms of multi-disease monitoring and localization, the system uses the Area Under the Receiver Operating Characteristic Curve (AUROC), F1 score, and accuracy to evaluate overall performance. When evaluating the quality and fluency of the Chest-CHAT-generated text, multiple metrics are employed, including BLEU-1 (Bilingual Evaluation Understudy), ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence), METEOR (Metric for Evaluation of Translation with Explicit Ordering), and CIDEr (Consensus-based Image Description Evaluation). Furthermore, this application introduces a Llama score based on a large language model. The Llama score uses the Llama 3.1 70B model to compare the results generated by Chest-CHAT with reference answers extracted from the original clinical report, providing a score for content similarity. This score is used to measure the clinical accuracy of the generated answer compared to the true value.
[0161] Based on the above evaluation tests, this application has significant technological breakthroughs and clinical application value compared to existing technologies, with the following specific beneficial effects:
[0162] 1. Significantly Improved Diagnostic Accuracy: To evaluate the performance and effectiveness of the model, this application reported AUROC, F1 score, and detection accuracy on the internal CT-RATE validation set and the external RAD-ChestCT validation set. The overall performance is shown in Table 2 below, which presents the overall performance of different models. The ROC curves of Chest-OMDL on both datasets demonstrate its superior performance.
[0163] Although Chest-OMDL performs worse on external datasets than on internal datasets, its average AUROC is still above 0.7, and its performance is comparable to that of the comparison methods in internal validation, demonstrating good generalization ability. Compared to fVLM, which achieves state-of-the-art performance on this task, in internal validation, the average AUROC is improved by 3.73%, the average F1 score by 10.25%, and the average accuracy by 5.01%; in external validation, the AUROC is improved by 11.80%, the F1 score by 9.05%, and the accuracy by 6.46%. Specifically, Figure 5 and Figure 6 The average AUROC, F1 score, and detection accuracy for 16 diseases (internal validation) and 13 diseases (external validation) were summarized, achieving state-of-the-art performance for the current task. Figure 5 This study presents a detailed comparison of Chest-OMDL with four other methods across 16 diseases on an internal dataset: In AUROC, Chest-OMDL performed best in all categories except pulmonary nodules; in F1, it performed best in all diseases; and in Accuracy, Chest-OMDL also performed best in all categories except atelectasis. Figure 6 The study presents a detailed comparison of Chest-OMDL with four other methods on 13 diseases on external datasets: On AUROC, Chest-OMDL performed best in all diseases except for sequelae of pulmonary fibrosis; on F1, Chest-OMDL also performed best in all diseases except for emphysema, consolidation, peribronchial wall thickening, and bronchiectasis; and on Accuracy, Chest-OMDL also performed best in all diseases except for emphysema, sequelae of pulmonary fibrosis, peribronchial wall thickening, and bronchiectasis.
[0164] Table 2
[0165]
[0166] 2. Demonstrated zero-sample lesion localization capability: Since the CT-RATE and RAD-ChestCT datasets do not contain lesion segmentation labels, this test evaluated the model's localization capabilities on 10 manually pixel-level labeled COVID-19 CT cases using an external COVID-19 CT dataset. The results are as follows: Figure 7 As shown, Figure 7This paper presents anomaly localization results on the Covid-19 CT dataset, showcasing the anomaly maps and binarized segmentation results for the top 5 cases with the highest DSC values. In CT scans, COVID-19 typically presents as lung-specific abnormalities such as ground-glass opacities (GGO), post-pulmonary fibrosis sequelae (PFS), and consolidation (CON). To segment these lesions, anomaly maps of lung-related diseases were overlaid using Chest-OMDL, and a binarization threshold of 0.1 was applied. The method described in this application achieved an average DSC of 0.450, while supervised methods using pixel-level labels reported a DSC of 0.673 ± 0.223. This demonstrates that, despite receiving only weak organ-level supervision, Chest-OMDL achieves segmentation accuracy comparable to pixel-level supervised methods, reaching 67% performance.
[0167] 3. Excellent performance of the large-scale chest CT image model: The trained Chest-CHAT model possesses multiple question-answering capabilities, flexibly handling various forms of medical questions, such as... Figure 8 As shown, Figure 8 Example outputs for Chest-CHAT include: (a) long-answer questions, (b) short-answer questions, (c) multiple-choice questions, and (d) report generation questions.
[0168] For example, in long-answer mode, the system can interpret chest CT images in detail, providing comprehensive answers such as "mild emphysema was observed in both lungs, with several non-specific small nodules present." In short-answer mode, the system can quickly respond to simple questions, such as directly identifying the disease type in the specific area of the left upper lobe as "linear atelectasis," providing concise and clear key information. Furthermore, Chest-CHAT supports interactive multiple-choice questions. For example, users can ask, "What results are normal in the right lung CT image?" and are provided with multiple options, such as "normal lung tissue" and "benign lymph nodes." This capability helps clinicians further clarify their diagnostic approach when assessing the likelihood of a lesion. Finally, through the report generation function, the model can generate comprehensive radiological reports, including "Finding" and "Impression" sections, providing both detailed descriptions and the final diagnostic result. This function greatly improves the efficiency of medical image analysis and provides strong technical support for healthcare. Chest-CHAT demonstrated superior performance across several key metrics, including BLEU-1, METEOR, ROUGE-L, CIDEr, and Llama scores, significantly outperforming other comparative models.
[0169] Specific indicators are as follows Figure 9 As shown, Figure 9The presentation shows evaluation metrics for Chest-CHAT compared to other models, including BLEU-1, METEOR, ROUGE-L, CIDEr, and Llama scores. On the BLEU-1 metric, the model achieves a significant leap forward in long-answer generation tasks, with fluency more than 2.5 times that of the comparison models; even in the most challenging report generation tasks, it maintains a significant advantage. The METEOR metric fully reveals the model's superior ability to handle complex contexts: in short-answer tasks requiring precise responses, the application scores almost twice that of the second-best methods; and in multiple-choice scenarios requiring deep reasoning, the model demonstrates near-human-level semantic understanding. The ROUGE-L metric confirms the effectiveness of the optimized design in the medical scenario—the short-answer optimization strategy tailored for medical question-answering achieved an excellent score of 0.5991, and in multiple-choice tasks, the model's content coverage completeness is highly consistent with the reference text (reaching 0.8679). On the CIDEr metric, the model demonstrated its exceptional potential for creative expression: it outperformed the best baseline by 5 times in long-answer tasks requiring creative expression, and achieved a high score of 7.6019 in multiple-choice questions. The Llama Score, the most convincing metric for evaluating the semantics of generated text, further validated the model's strength: it scored over 8 points in both long and short-answer tasks, and even in the more challenging report generation task, it robustly achieved a score of 5.97.
[0170] The method for constructing a large-scale chest CT image-text model according to the embodiments of this application can extract disease labels of chest organs from clinical report text using a pre-trained large language model, extract organ segmentation masks related to chest organ diseases from a 3D chest CT image dataset using a pre-trained organ model, generate pseudo-labels in tabular form based on disease labels and organ segmentation masks, without relying on pixel-level annotation data of real lesions, use organ segmentation masks as coarse segmentation labels, generate a training dataset based on coarse segmentation labels, pseudo-labels and 3D chest CT image dataset, train a visual encoder using the training dataset, generate a large-scale chest CT image-text model based on the trained large language model, visual encoder and multimodal projection module, and fine-tune the large-scale chest CT image-text model using a medical image question-answering task to improve the accuracy of the model in locating chest diseases.
[0171] Next, referring to the accompanying drawings, a large model construction apparatus for chest computed tomography images according to an embodiment of this application is described.
[0172] Figure 10 This is a block diagram of a chest computed tomography (CT) image large model construction device according to an embodiment of this application.
[0173] like Figure 10As shown, the large model construction device 10 for chest computed tomography images includes: an acquisition module 100, an extraction module 200, a pre-training module 300, and a fine-tuning module 400.
[0174] The acquisition module 100 acquires a dataset of three-dimensional chest computed tomography (CT) images and clinical report text. The extraction module 200 uses a pre-trained large language model to extract disease labels for chest organs from the clinical report text, and uses a pre-trained organ model to extract disease-related organ segmentation masks from the three-dimensional chest CT image dataset. It also generates tabular pseudo-labels based on the disease labels and organ segmentation masks. The pre-training module 300 uses the organ segmentation masks as coarse segmentation labels, generates a training dataset based on the coarse segmentation labels, pseudo-labels, and the three-dimensional chest CT image dataset, and uses the training dataset to pre-train the visual encoder. The input to the visual encoder is a 3D chest CT image, which is then decoded to output a 3D chest CT image as a disease prediction map. The fine-tuning module 400 is used to generate a large chest CT image-text model based on the pre-trained large language model, visual encoder, and multimodal projection module, and to fine-tune the large chest CT image-text model using a medical image question-answering task to obtain the final large chest CT image-text model. Specifically, the 3D chest CT image related to the question-answering task is input into the visual encoder, and the multimodal projection module projects the disease prediction map output by the visual encoder into the visual labels required by the large language model. The large language model outputs the question-answering results of the question-answering task based on the visual labels and clinical report text.
[0175] In this embodiment, the visual encoder generates a disease anomaly map and its corresponding organ segmentation mask based on a three-dimensional chest CT image. The disease anomaly map and its corresponding organ segmentation mask elements are multiplied together to obtain a disease prediction map.
[0176] In this embodiment, the pre-trained architecture of the visual encoder includes a feature extractor, an organ segmentation decoder, and a disease anomaly map decoder. The feature extractor extracts spatial features and contextual features from a three-dimensional chest CT image. The organ segmentation decoder generates an organ segmentation mask based on the spatial features and contextual features. The disease anomaly map decoder generates a disease anomaly map based on the spatial features and contextual features.
[0177] In this embodiment, the feature extractor includes an activation function, an instance-normalized 3D convolutional layer, a gated spatial convolution, and a Mamba layer. The organ segmentation decoder and the disease anomaly graph decoder have the same structure, employing an encoder-decoder structure, and connecting high-level semantic features with low-level features through skip connections.
[0178] In this embodiment, the calculation formula for the Mamba layer in the feature extractor is:
[0179] x i+1 =Unflatten(Mamba(Flatten(LayerNorm(x i ))))+x i ;
[0180] Where, x i For the input features of the i-th layer, LayerNorm(x) i ) is for the input feature x i Layer normalization is performed. Flatten(·) is the operation that flattens the feature tensor into a sequence form. Mamba(·) is the Mamba computation block of the Mamba state-space model. Unflatten(·) is the operation that reshapes the feature sequence back into its original dimensional structure. i+1 The output features of layer i+1;
[0181] Input feature x i After linear transformation, it is divided into a first component and a second component. The formulas for obtaining the first and second components are as follows:
[0182] y,z=split(linear(x i ));
[0183] Where, linear(x) i ) represents the linear transformation operation, y is the first component, and z is the second component.
[0184] In this embodiment, the visual encoder dynamically adjusts the weights of the loss function during training, wherein the dynamic adjustment formula for the weights of the loss function is:
[0185]
[0186] Where λ is the weight, initial weight For the initial weights, decay rate The attenuation rate, total epochs The total number of training rounds is 1, and the current training round is epoch.
[0187] In this embodiment, the multimodal projection module includes an attention pooling layer and a two-layer perceptron. The attention pooling layer uses learned latent queries and multi-head cross-attention to compress the features output by the visual encoder into a fixed-length image label sequence.
[0188] In this embodiment, the large language model includes a Transformer network, an attention head, an embedding dimension, and a feedforward network dimension. During the adjustment phase of the large language model, the large language model is adjusted using the tags and disease labels added by the word segmenter.
[0189] It should be noted that the foregoing explanation of the embodiment of the method for constructing large models of chest computed tomography images also applies to the chest computed tomography image construction device of this embodiment, and will not be repeated here.
[0190] The chest computed tomography (CT) image-text large model construction device proposed in this application can extract disease labels of chest organs from clinical report text using a pre-trained large language model, extract organ segmentation masks related to chest organ diseases from a 3D chest CT image dataset using a pre-trained organ model, generate pseudo-labels in tabular form based on disease labels and organ segmentation masks, without relying on pixel-level annotation data of real lesions, use organ segmentation masks as coarse segmentation labels, generate a training dataset based on coarse segmentation labels, pseudo-labels and 3D chest CT image dataset, train a visual encoder using the training dataset, generate a chest CT image-text large model based on the trained large language model, visual encoder and multimodal projection module, and fine-tune the chest CT image-text large model using a medical image question-answering task to improve the accuracy of the model in locating chest diseases.
[0191] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:
[0192] The memory 1101, the processor 1102, and the computer program stored on the memory 1101 and executable on the processor 1102.
[0193] When the processor 1102 executes the program, it implements the method for constructing a large model of chest computed tomography images provided in the above embodiments.
[0194] Furthermore, electronic devices also include:
[0195] Communication interface 1103 is used for communication between memory 1101 and processor 1102.
[0196] The memory 1101 is used to store computer programs that can run on the processor 1102.
[0197] The memory 1101 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.
[0198] If the memory 1101, processor 1102, and communication interface 1103 are implemented independently, then the communication interface 1103, memory 1101, and processor 1102 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0199] Optionally, in a specific implementation, if the memory 1101, processor 1102, and communication interface 1103 are integrated on a single chip, then the memory 1101, processor 1102, and communication interface 1103 can communicate with each other through an internal interface.
[0200] The processor 1102 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.
[0201] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0202] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0203] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0204] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.
[0205] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0206] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A method for constructing a large model of chest computed tomography images, characterized in that, Includes the following steps: Acquire a dataset of three-dimensional chest computed tomography (CT) images and clinical report text; Disease labels for thoracic organs are extracted from the clinical report text using a pre-trained large language model. Organ segmentation masks related to thoracic organ diseases are extracted from a 3D chest CT image dataset using a pre-trained organ model. Pseudo-labels in tabular form are generated based on the disease labels and the organ segmentation masks. The organ segmentation mask is used as a coarse segmentation label. A training dataset is generated based on the coarse segmentation label, the pseudo label, and the three-dimensional chest CT image dataset. The visual encoder is pre-trained using the training dataset. The input of the visual encoder is a three-dimensional chest CT image, and the output of the three-dimensional chest CT image after decoding is a disease prediction map. A large-scale chest CT image model is generated based on a pre-trained large language model, a visual encoder, and a multimodal projection module. The large-scale chest CT image model is then fine-tuned using a medical image question-answering task to obtain the final large-scale chest CT image model. Specifically, the three-dimensional chest CT images related to the question-answering task are input into the visual encoder. The multimodal projection module projects the disease prediction map output by the visual encoder into the visual labels required by the large language model. The large language model outputs the question-answering results of the question-answering task based on the visual labels and clinical report text.
2. The method for constructing a large model of chest computed tomography images according to claim 1, characterized in that, The visual encoder generates a disease anomaly map and its corresponding organ segmentation mask based on the three-dimensional chest CT image. The disease anomaly map and its corresponding organ segmentation mask elements are multiplied together to obtain a disease prediction map.
3. The method for constructing a large model of chest computed tomography images according to claim 2, characterized in that, The pre-trained architecture of the visual encoder includes a feature extractor, an organ segmentation decoder, and a disease anomaly map decoder. The feature extractor extracts spatial and contextual features from the three-dimensional chest CT image. The organ segmentation decoder generates an organ segmentation mask based on the spatial and contextual features. The disease anomaly map decoder generates a disease anomaly map based on the spatial and contextual features.
4. The method for constructing a large model of chest computed tomography images according to claim 3, characterized in that, The feature extractor includes an activation function, an instance-normalized 3D convolutional layer, a gated spatial convolution, and a Mamba layer. The organ segmentation decoder and the disease anomaly graph decoder have the same structure, employing an encoder-decoder structure and connecting high-level semantic features with low-level features through skip connections.
5. The method for constructing a large model of chest computed tomography images according to claim 4, characterized in that, The calculation formula for the Mamba layer in the feature extractor is as follows: x i+1 =Unflatten(Mamba(Flatten(LayerNorm(x i ))))+x i ; Where, x i For the input features of the i-th layer, LayerNorm(x) i ) is for the input feature x i Layer normalization is performed. Flatten(·) is the operation that flattens the feature tensor into a sequence form. Mamba(·) is the Mamba computation block of the Mamba state-space model. Unflatten(·) is the operation that reshapes the feature sequence back into its original dimensional structure. i+1 The output features of layer i+1; Input feature x i After linear transformation, it is divided into a first component and a second component. The formulas for obtaining the first component and the second component are as follows: y,z=split(linear(x i )); Where, linear(x) i ) represents the linear transformation operation, y is the first component, and z is the second component.
6. The method for constructing a large model of chest computed tomography images according to claim 1, characterized in that, The visual encoder dynamically adjusts the weights of the loss function during training, wherein the dynamic adjustment formula for the weights of the loss function is: Where λ is the weight, initial weight For the initial weights, decay rate The attenuation rate, total epochs The total number of training rounds is 1, and the current training round is epoch.
7. The method for constructing a large model of chest computed tomography images according to claim 1, characterized in that, The multimodal projection module includes an attention pooling layer and a two-layer perceptron. The attention pooling layer uses learned latent queries and multi-head cross-attention to compress the features output by the visual encoder into a fixed-length image label sequence.
8. The method for constructing a large model of chest computed tomography images according to claim 1, characterized in that, The large language model includes a Transformer network, an attention head, an embedding dimension, and a feedforward network dimension. During the adjustment phase of the large language model, the large language model is adjusted using the tags added in the word segmenter and the disease labels.
9. A device for constructing a large model of chest computed tomography images, characterized in that, include: The acquisition module is used to acquire a dataset of three-dimensional chest computed tomography (CT) images and clinical report text. The extraction module is used to extract disease labels of chest organs from the clinical report text using a pre-trained large language model, extract organ segmentation masks related to chest organ diseases from a three-dimensional chest CT image dataset using a pre-trained organ model, and generate pseudo-labels in tabular form based on the disease labels and the organ segmentation masks. The pre-training module is used to use the organ segmentation mask as a coarse segmentation label, generate a training dataset based on the coarse segmentation label, the pseudo label and the three-dimensional chest CT image dataset, and train the visual encoder using the training dataset. The input of the visual encoder is a three-dimensional chest CT image, and the output of the three-dimensional chest CT image after decoding is a disease prediction map. A fine-tuning module is used to generate a large-scale chest CT image-text model based on a pre-trained large language model, a visual encoder, and a multimodal projection module. This model is then fine-tuned using a medical image question-and-answer task to obtain the final large-scale chest CT image-text model. Specifically, a three-dimensional chest CT image related to the question-and-answer task is input into the visual encoder. The multimodal projection module projects the disease prediction map output by the visual encoder into visual labels required by the large language model. The large language model then outputs the question-and-answer results based on the visual labels and clinical report text.
10. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method for constructing a large model of chest computed tomography images according to any one of claims 1-8.
Citation Information
Cited By
Experimental record text anomaly classification method and device based on image feature processing
CN121147952A
Method and system for detecting loose strands of power transmission conductor under scarce image sample
CN121599945A
