Method for automatically distinguishing enhanced phase in CT scan based on multi-instance learning attention mechanism

Abstract

The application relates to the technical field of medical imaging, and discloses a method for automatically distinguishing enhanced phases in CT scanning based on an attention mechanism of multi-instance learning. The method extracts CT scanning images and DICOM information from a data set, uniformly processes the image size after HU conversion, truncation and linear scaling; a fixed number of slices are obtained by using direct or indirect random sampling and a cubic interpolation strategy; subsequently, a pre-trained model is used to extract slice features, and a scanning-level feature vector is formed through attention aggregation; finally, the prediction probability of four types of phases is output by a full connection layer and a Softmax module; the model is trained by using cross-entropy loss and an Adam optimizer, the performance is evaluated by using indexes such as accuracy, recall rate and F1-score, a heat map is generated by combining gradient back propagation, the result is directly visualized, and clinical determination is assisted. The application can accurately, quickly and explainably classify CT scanning multi-phase phases in a real clinical environment.

Description

Technical Field

[0001] This invention relates to the field of medical imaging technology, and more specifically, to a method for automatically identifying enhancement phase in CT scans based on an attention mechanism of multi-instance learning. Background Technology

[0002] In the field of medical imaging, especially in the automatic determination of enhancement phase in contrast-enhanced CT (CE-CT) scans, there are many problems that urgently need to be solved.

[0003] With the widespread application of CT and contrast agents in medical imaging, the automatic discrimination of different enhancement phases, such as non-contrast phases, arterial phases, and venous phases, is of great significance for subsequent tasks, including organ segmentation, radiomics feature extraction and analysis, and disease prognosis prediction. However, current CT image metadata often lacks standardized annotations for enhancement phases; for example, DICOM tags are incomplete or non-standard, and even manual input may lead to errors, resulting in low automation levels. Furthermore, distinguishing different enhancement phases faces challenges such as image intensity variations, temporal characteristics, and massive data volumes.

[0004] Traditional methods have several shortcomings. First, labeling is often inconsistent; phase labels in DICOM are frequently entered manually, which is prone to errors or omissions, requiring extensive manual verification and staging. Second, 3D models are computationally intensive; while early 3D convolutional networks such as C3D and 3D-ResNet could process the entire 3D volume data at once, their large parameter count and slow inference speed make them unsuitable for large-scale clinical applications. Third, while deep learning models such as CNNs achieve a certain level of accuracy in classification results, they lack interpretability and cannot intuitively present the key anatomical regions the model focuses on, leading to a lack of physician trust and traceability in clinical applications. Finally, class imbalance is a significant problem; some phases have an excessive amount of data without contrast phases, while "other" phases have less data, which, if left unaddressed, can cause the model to favor the mainstream category.

[0005] Therefore, an innovative and efficient method is urgently needed to solve the above problems. Summary of the Invention

[0006] In the current technological field, automated multi-phase classification methods for computed tomography (CT) face several challenges: 1. Lack of a unified and efficient phase recognition mechanism: Current methods are inefficient when processing complete 3D datasets or require manual phase annotation; in large-scale applications, the flexibility and cost-effectiveness of these methods are unsatisfactory. 2. Insufficient adaptability to data imbalance and incompleteness: In actual hospital scanning processes, some phases may be missing or insufficient in number, making it difficult for traditional methods to adaptively select feature slices, which limits classification accuracy and generalization ability. 3. Weak model interpretability: Radiologists often need to understand how the model makes phase determinations, for example, which blood vessel or organ regions the model focuses on. Without interpretable tools, the model is difficult to gain clinical trust. 4. Impact on downstream tasks: Accurate and efficient phase classification is fundamental for organ segmentation, radiomics feature extraction, and disease prognostic analysis, but current technologies have not fully integrated phase classification with the needs of downstream tasks.

[0007] In view of this, the present invention proposes an attention mechanism based on multi-instance learning for automatic identification of enhancement phase in CT scans. It aims to ensure efficient classification of enhancement phase in computed tomography (CT) scans while taking into account multi-instance learning, sampling strategies, and interpretability, in order to address the diverse scanning situations and data imbalance problems in actual clinical practice, thereby laying a solid foundation for subsequent applications.

[0008] This invention proposes a method for automatically identifying enhancement phases in CT scans based on an attention mechanism of multi-instance learning, comprising:

[0009] A VinDr-Multiphase dataset is formed by acquiring several CT scan records and corresponding slice images of the CT scan records. The metadata of the DICOM format file is extracted, and the pixel values ​​are converted to HU units. At the same time, the images are uniformly scaled to a predetermined size and converted to PNG or NumPy format.

[0010] All scan data are divided into training, validation, and test sets; for each scan data, a fixed number of slices are obtained by using direct random sampling, indirect random sampling, or interpolation strategies.

[0011] Each slice image is input into a pre-trained convolutional neural network to extract convolutional features or spatiotemporal features; using a multi-instance attention aggregation mechanism, the features of each slice within the same scan are weighted and summed to obtain scan-level features;

[0012] The model is trained and evaluated. After the model training and evaluation are completed, the model is used to predict the scanned data of the test set to obtain the prediction results. Pixel-level contribution is generated through gradient backpropagation, and SaliencyMaps or heatmaps are constructed and superimposed on the original image. Finally, the results are output.

[0013] Preferably, the metadata includes Pixel Data, Rescale Slope / Intercept, window width, and / or window level.

[0014] Preferably, when dividing all scan data into training set, validation set and test set, the proportions of training set, validation set and test set are 70%, 15% and 15%, respectively.

[0015] Preferably, the direct random sampling involves randomly selecting 64 slices;

[0016] The indirect random sampling involves first randomly selecting 30% of the slices, and then cropping or supplementing them to make up to 64 slices.

[0017] The interpolation strategy is to generate intermediate slices using a cubic interpolation method.

[0018] Preferably, the convolutional neural network is a two-dimensional convolutional neural network.

[0019] Preferably, when using a multi-instance attention aggregation mechanism to obtain scan-level features by weighted summation of features from each slice within the same scan, the following steps are included:

[0020] Let the features of all slices in the same scan be denoted as {h1, h2, ..., h}. i}, introduce attention weights {a1, ...,a i The scan-level feature H is obtained by weighted summation, and the calculation formula is H = ∑aᵢ·hᵢ.

[0021] Preferably, when training and evaluating the model, the following methods are used: training the model using cross-entropy or weighted binary cross-entropy loss function.

[0022] Preferably, training and evaluating the model includes:

[0023] The Adam optimizer was used, and the training performance was evaluated by precision, recall, F1-score, and AUC. A confusion matrix was also generated to analyze classification performance.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] High efficiency: This invention combines 2D CNN slice-level feature extraction with a multiple instance learning (MIL) mechanism, effectively avoiding the computational redundancy and inference bottlenecks of traditional 3D convolutional models when processing complete sets of CT body data. Through optimized structure and lightweight model design, this invention significantly reduces model parameters and GPU memory usage while ensuring classification accuracy, greatly improving processing speed and resource utilization efficiency in large-scale clinical applications.

[0026] Flexibility: This invention supports multiple slice sampling strategies, including direct random sampling, indirect proportional sampling, and interpolation-enhanced sampling. It can automatically adjust the input method according to the length of the CT scan, slice density, and phase integrity, and has the ability to handle practical problems such as missing enhancement phases and uneven distribution, maintaining high performance even under complex data structures.

[0027] Interpretability: By introducing attention mechanisms and saliency maps, the model can automatically identify and highlight slices or regions that play a key role in classification during inference, such as enhanced features of tissues like the aorta, hepatic hilum, and renal arteries. This allows doctors to intuitively observe the model's focus, thereby improving model verifiability and clinical trust, and driving the shift of AI from "black box" to "white box" applications.

[0028] Accuracy and Robustness: Through a multi-instance attention aggregation mechanism, this invention effectively suppresses the interference of noisy slices, highlighting the key "high-information" slices in classification. Especially when other transitional categories have scarce sample sizes and unstable labels, the model still demonstrates excellent generalization ability and classification robustness. Real-world testing on public datasets such as VinDr-Multiphase yields an F1-score exceeding 0.98, proving its stability and practicality in real-world medical scenarios.

[0029] Scalability for subsequent medical applications: Accurate enhanced phase classification results provide a reliable input prerequisite for downstream tasks such as organ segmentation, radiomics feature extraction and analysis, and disease prognosis prediction.

[0030] In summary, this invention provides an innovative and efficient method for automatically identifying the enhancement phase in contrast-enhanced CT scans, possessing significant clinical value and promising practical applications. It not only improves the accuracy of medical image analysis but also provides physicians with more accurate diagnostic information, thereby enhancing diagnostic efficiency while simultaneously improving the targetedness and effectiveness of treatment plans. Attached Figure Description

[0031] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0032] Figure 1 This is a schematic diagram of the data conversion process of the present invention;

[0033] Figure 2 A schematic diagram illustrating the working principle of Multiple Instance Learning (MIL);

[0034] Figure 3 To validate interpretability on the public dataset VinDr-Multiphase, a Saliency Map visualization example is generated, showing the intensity of attention for key areas using CT slices of the heart in the non-enhanced, venous, and arterial phases and their thermal map coverage.

[0035] Figure 4 To validate interpretability on the public dataset VinDr-Multiphase, a Saliency Map visualization example is generated, showing the intensity of attention for key areas using CT slices of the liver in the non-enhanced, venous, and arterial phases and their heatmap coverage.

[0036] Figure 5 To validate interpretability on the public dataset VinDr-Multiphase, a Saliency Map visualization example is generated, showing the intensity of attention for key areas using CT slices of the kidney in the non-enhanced, venous, and arterial phases and their heatmap coverage.

[0037] Figure 6 To further validate interpretability on the novel dataset HCC-TACE-Seg, which the model has never seen before, a Saliency Map visualization example is generated, showing the intensity of attention to key areas by covering CT slices of the heart during the arterial and venous phases and their heat maps.

[0038] Figure 7 To further validate interpretability on the novel dataset HCC-TACE-Seg, which the model has never seen before, a Saliency Map visualization example is generated, showing the intensity of attention to key areas by covering CT slices of the liver during the arterial and venous phases and their heat maps.

[0039] Figure 8To further validate interpretability on the novel dataset HCC-TACE-Seg, which the model has never seen before, a Saliency Map visualization example is generated, showing the intensity of attention to key areas by covering CT slices of the kidney during the arterial and venous phases with their heatmaps. Detailed Implementation

[0040] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey its scope to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] This invention proposes a contrast-enhanced CT phase classification method that integrates two-dimensional / three-dimensional convolutional neural networks (2D / 3D CNN), multi-instance learning (MIL), and multiple random sampling and interpolation strategies. The specific implementation includes the following steps:

[0042] I. Data Preprocessing

[0043] DICOM parsing: Read the DICOM file of the original CT scan, extract key metadata (Pixel Data, RescaleSlope / Intercept, window width WL, window level WW, etc.), and calculate the HU value of each pixel.

[0044] Image conversion: Convert CT slices to PNG or NumPy format; PNG is suitable for 2D convolutional neural networks, while NumPy can preserve 3D volumetric data for use by 3D convolutional neural networks. Normalize the slice size according to actual needs (e.g., 128×128). Figure 1 This is a schematic diagram of the data conversion process of the present invention.

[0045] II. Data Selection Strategy

[0046] Direct Random Sampling: Randomly selects several slices (e.g., 64 slices) directly from a scan, reducing computational burden.

[0047] Indirect random sampling: First, a portion of the scanned slices is randomly selected (at a certain percentage, such as 30%), and then truncated or amplified to a uniform number.

[0048] Data Interpolation: Linear or cubic interpolation is performed on scans with insufficient slices or inconsistent resolution to enhance data continuity and facilitate the model's capture of phase feature changes.

[0049] III. Multi-instance learning and attention mechanisms

[0050] Treat a scan as a bag, and each slice as an instance within the bag.

[0051] Feature Extraction and Aggregation: A lightweight 2D CNN architecture (such as EfficientNet, ResNet, etc.) is used to extract local features for each slice. Subsequently, the features are aggregated into scan-level representations using two strategies: attention-based pooling, which introduces a learnable attention mechanism to enable the model to focus on more discriminative key slices; and mean pooling, which averages the features of all slices to enhance model stability and reduce the impact of local noise.

[0052] Enhanced interpretability: By generating saliency maps, the key anatomical structures that the model focuses on are visually displayed, thereby helping doctors better understand the model's decision-making process.

[0053] IV. Model Structure and Training

[0054] Training and optimization of multi-structure models:

[0055] Model structure selection:

[0056] 2D CNN: Based on pre-trained models such as ResNet and EfficientNet, it extracts slice features through convolution operations. It is suitable for fine processing of slices, has fast inference speed, and is easy to deploy on a large scale and for real-time applications.

[0057] 3D CNN: Designed specifically for volumetric data, it can learn changes in spatial and temporal dimensions, making it suitable for applications with extremely high accuracy requirements. Although it can process and model spatial and temporal features in one go, its high parameter count and computational complexity mean it will not be adopted in this project.

[0058] Training strategy:

[0059] Loss function: Use cross-entropy or weighted cross-entropy to address the class imbalance problem.

[0060] Optimizer: The Adam optimizer is selected and combined with Cosine Annealing or Warm-up strategies to improve training efficiency.

[0061] Evaluation metrics: The multi-metric evaluation system includes precision, recall, and F1 score to ensure a comprehensive evaluation of model performance.

[0062] The method described above has the following advantages:

[0063] First, it significantly improves the accuracy of phase classification. This invention cleverly integrates the advantages of both two-dimensional and three-dimensional convolutional neural networks, enabling it to capture various phase features in CT scan images more comprehensively and meticulously. This comprehensive feature extraction capability allows the system to classify different phases more accurately and meticulously, thereby greatly improving the accuracy of phase classification.

[0064] Secondly, it significantly enhances the model's generalization ability. This invention employs a combination of multi-instance learning and attention mechanisms, enabling the model to focus more on key slice regions when processing CT scan data, effectively reducing the impact of local noise on classification results. This design allows the model to maintain stable and reliable performance when facing different patients and different scanning conditions.

[0065] Third, it significantly improves computational efficiency. This invention introduces two strategies—direct random sampling and indirect random sampling—to significantly reduce the computational burden while ensuring that model performance is not affected. This optimization enables the model to process large amounts of CT scan data more quickly, improving overall computational efficiency.

[0066] Fourth, it significantly enhances the interpretability of the model. By generating saliency maps, doctors can visually observe the key anatomical structures that the model focuses on when making decisions. This visualization not only helps doctors better understand the model's decision-making process but also contributes to improving the accuracy and reliability of diagnoses, making the clinical application of the model more transparent and credible.

[0067] The technical solution of the present invention will be described below through specific embodiments. Example

[0068] I. Environment and Data Preparation

[0069] This embodiment selects the VinDr-Multiphase Dataset, which contains 1,188 CT scan records, totaling 392,560 slice images. Each scan record is labeled with a phase tag, including non-contrast enhancement, arterial phase, venous phase, and other types.

[0070] The experimental environment was an Ubuntu server equipped with an NVIDIA RTX 4070 GPU, and the model was trained and inferred using Python 3.8.10 and Torch 2.3.0+cu121.

[0071] II. Data Preprocessing

[0072] Extract Pixel Data, Rescale Slope / Intercept, and Window Width / Window Level parameters from the DICOM file, and convert the pixel values ​​to HU units;

[0073] Appropriate truncation and linear scaling were applied to the HU value (window level WL=40, window width WW=400) to enhance the contrast of the organization of interest;

[0074] The data was uniformly scaled to 128×128 PNG or Numpy format, and all scanned data were divided into training set, validation set and test set according to the ratio of 70%, 15% and 15% respectively.

[0075] III. Sampling Strategy Selection

[0076] Direct random sampling strategy: 64 slices are randomly selected from each scan;

[0077] Indirect random sampling strategy: First, randomly select 30% of the slices, and then crop or supplement them to make up to 64 slices;

[0078] Interpolation strategy: For scans with a small number of slices or requiring continuous temporal information, a cubic interpolation method is used to generate intermediate slices.

[0079] Experimental results show that in the application scenario of attention aggregation, the combination of "indirect random sampling of 30%" and cross-entropy loss function has the best effect; while in the "average aggregation" strategy, the "interpolation" method can also achieve high prediction accuracy.

[0080] IV. Network Models and Attention Mechanisms

[0081] Two-dimensional Convolutional Neural Network (2D CNN): This method utilizes the EfficientNet-B2 model pre-trained on ImageNet, removing its final classification layer while retaining its feature extraction capabilities. For each slice, a 128-dimensional deep feature vector is extracted through a series of convolution, activation, and pooling operations.

[0082] Multi-instance attention aggregation mechanism: All slice features from the same scan are denoted as {h1, h2, ..., h...} iWe introduce attention weights {a1, ..., ai} and obtain the scan-level feature representation by weighted summation: H = ∑aᵢ·hᵢ. The attention module is a learnable structure that can dynamically highlight key anatomical regions (such as the abdominal aorta and hepatic hilum), balancing robustness and interpretability.

[0083] V. Training and Assessment

[0084] Loss function: Cross Entropy (CE) or Weighted Binary Cross Entropy (WBCE) are chosen to address the class imbalance problem; experiments show that CE usually achieves a higher F1-score.

[0085] Hyperparameters: Use the Adam optimizer, set the learning rate to 0.001, and adjust the batch size according to the available GPU memory.

[0086] Evaluation metrics: Precision, Recall, F1-score (weighted), and AUC were used; a confusion matrix and heatmap were generated to assess the interpretability of the model.

[0087] VI. Specific Implementation Process

[0088] Scan import and format conversion: The system automatically reads CT scans and completes DICOM parsing, image standardization, and format conversion (see appendix). Figure 1 ).

[0089] Sampling and interpolation processing: Select direct / indirect sampling or interpolation strategies based on the number of scanned slices to form a structurally consistent input sequence (see appendix). Figure 2 ).

[0090] Slice-level feature extraction (2D CNN): Each slice is independently fed into a 2D convolutional network to extract high-dimensional semantic features (see appendix). Figure 2 ).

[0091] Feature aggregation and scan-level classification (MIL + Attention):

[0092] Aggregate all slice features and output the full-scan phase classification probability through an attention mechanism (see attached diagram). Figure 4 ), Softmax outputs four types of phase. (See appendix) Figure 2 ).

[0093] Results output and significance visualization:

[0094] The model output is interpreted and visualized to generate a Saliency Map, which is then overlaid on the original slides to aid clinical interpretation. (See appendix) Figure 3-8 ).

[0095] VII. Summary and Applicability Explanation

[0096] This embodiment details the application process of the method of the present invention in a real-world abdominal contrast-enhanced CT scan multiphase phase recognition task, demonstrating its high operability, reproducibility, and portability. Experimental results show that the method achieves an excellent F1-score exceeding 0.98 on the VinDr-Multiphase dataset.

[0097] The above description is merely one of the preferred embodiments of the present invention. Within the scope of the inventive concept, technical alternatives (such as replacement of the CNN structure, adjustment of the sampling ratio, and expansion of the loss function) should also be considered equivalent embodiments and are all covered within the protection scope of the present invention. The methods and systems provided by the present invention can be directly integrated into medical imaging AI platforms to achieve rapid, accurate, and interpretable classification of enhanced phases in CT scans, providing a solid foundation for automatic segmentation, extraction and analysis of radiomics features, and disease prognosis prediction.

[0098] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for automatically identifying enhancement phases in CT scans based on an attention mechanism using multi-instance learning, characterized in that, include: Acquire several CT scan records and corresponding slice images of several CT scan records to form a VinDr-Multiphase dataset; extract metadata from DICOM format files, convert pixel values ​​to HU units, and uniformly scale the images to a predetermined size and convert them to PNG or NumPy format; All scan data are divided into training, validation, and test sets; for each scan data, a fixed number of slices are obtained by using direct random sampling, indirect random sampling, or interpolation strategies. Each slice image is input into a pre-trained convolutional neural network model to extract convolutional features or spatiotemporal features; By utilizing a multi-instance attention aggregation mechanism, the features of each slice within the same scan are weighted and summed to obtain scan-level features; The model is trained and evaluated. After the model training and evaluation are completed, the model is used to predict the test set scan data to obtain the prediction results. Pixel-level contribution is generated through gradient backpropagation, and SaliencyMaps or heatmaps are constructed and superimposed on the original image. Finally, the results are output. The metadata includes Pixel Data, Rescale Slope / Intercept, window width, and / or window level; The direct random sampling involves randomly selecting 64 slices. The indirect random sampling involves first randomly selecting 30% of the slices, and then cropping or supplementing them to make up to 64 slices. The interpolation strategy is to generate intermediate slices using a cubic interpolation method; The convolutional neural network is a two-dimensional convolutional neural network; When using a multi-instance attention aggregation mechanism to obtain scan-level features by weighted summation of features from each slice within the same scan, the following features are included: All slice features of the same scan are denoted as {h1, h2, ..., hi}. Attention weights {a1, ...,ai} are introduced, and the scan-level features H are obtained by weighted summation. The calculation formula is H = ∑ai·hi. Training and evaluating the model includes: training the model using cross-entropy or weighted binary cross-entropy loss function; Training and evaluating the model includes: The Adam optimizer was used, and the training performance was evaluated by precision, recall, F1-score, and AUC. A confusion matrix was also generated to analyze classification performance.

2. The method for automatically identifying enhancement phase in CT scans based on an attention mechanism using multi-instance learning as described in claim 1, characterized in that, When all scan data is divided into training, validation, and test sets, the proportions of the training, validation, and test sets are 70%, 15%, and 15%, respectively.

Images