Medical image rapid analysis and prediction method based on deep features and multi-instance learning

Abstract

The invention relates to a medical image rapid analysis and prediction method based on depth features and multi-instance learning, belongs to the field of medical artificial intelligence and computer vision, and provides a framework integrating '2.5 D' feature extraction and attention-guided multi-instance learning. The system is converted into an end-to-end trainable system composed of a depth feature extractor, a multi-instance aggregator and a classifier, and only a patient-level label is used for supervision. According to the method, the high-dimensional medical image can be efficiently and accurately analyzed only by using the patient-level weak label which is easy to obtain, and visual interpretation of a decision basis is provided at the same time.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence and computer vision, and relates to a rapid prediction method suitable for medical image big data analysis, especially for efficient analysis methods in scenarios that require processing high-resolution three-dimensional images and have weak patient-level labels, such as the prediction of post-radiotherapy complications, in order to improve the prediction performance, interpretability and computational efficiency of medical image models. Background Technology

[0002] In recent years, radiotherapy has become one of the main treatments for cancer, improving patient survival rates. However, its long-term complications, such as osteoporosis, severely impact the quality of life for survivors. Accurate prediction of individualized complication risks is crucial for adjusting treatment strategies and implementing early interventions. Medical imaging, especially computed tomography (CT), contains rich anatomical and functional information, making it an important data source for building predictive models.

[0003] However, building such predictive models faces significant technical challenges: 1. High annotation costs: Obtaining pixel-level or region-level fine annotations of lesion areas in images requires a significant investment of time by experienced physicians, making it difficult to achieve in large-scale clinical applications. Typically, we can only obtain patient-level labels (such as "whether osteoporosis has occurred"), which is weakly supervised information. 2. Data dimensionality and computational burden: High-resolution 3D medical image data is massive, and directly inputting it into 3D deep learning model training places extremely high demands on computational resources and is prone to overfitting. 3. Model interpretability requirements: Clinical applications require models not only to make predictions but also to indicate the key regions in the image most relevant to the prediction (such as bone regions prone to bone density loss) to assist physicians in decision-making.

[0004] Traditional solutions either employ manually designed image features (radiomics), which suffer from poor repeatability and generalization, or directly use 3D convolutional neural networks, which are computationally inefficient and underutilize weakly labeled data. Therefore, in medical image analysis, developing a method that can fully utilize weakly supervised labels, efficiently process high-dimensional data, and provide interpretability is a core problem that urgently needs to be solved. Summary of the Invention

[0005] To address the technical problems of existing technologies, such as strong dependence on fine annotation, low computational efficiency, and insufficient model interpretability, this invention provides a rapid medical image analysis and prediction method based on deep features and multi-instance learning. Its core objective is to achieve efficient and accurate analysis of high-dimensional medical images using only easily accessible patient-level weak labels, while simultaneously providing visual explanations of the decision-making basis.

[0006] To achieve the above objectives, the overall concept of the technical solution adopted in this invention is as follows: a framework that integrates "2.5D" feature extraction with attention-guided multi-instance learning. This framework transforms the 3D image analysis task into an end-to-end trainable system consisting of a deep feature extractor, a multi-instance aggregator, and a classifier, supervised only by patient-level labels.

[0007] The technical solution of the present invention is described in detail below:

[0008] First, in the data preprocessing stage, this invention proposes an adaptive key slice sampling strategy. By automatically segmenting the target anatomical structure in the 3D image and locating its largest cross-section, a set of adjacent 2D slices is systematically selected. This "2.5D" representation method, while preserving key 3D spatial context information, avoids the huge computational overhead and overfitting risk of processing full 3D volume data, achieving a balance between efficiency and information integrity.

[0009] Secondly, in the feature extraction stage, this invention innovatively employs DenseNet201, pre-trained on ImageNet, as the core feature extractor. Leveraging its densely connected structure to facilitate feature reuse, and through a transfer learning strategy, the weights of its front-end convolutional layers are fixed, with subsequent networks fine-tuned only for medical images. This design encodes each two-dimensional slice into a high-dimensional, semantically rich feature vector (e.g., 1920 dimensions), providing a robust feature foundation for subsequent analysis.

[0010] Furthermore, in the information aggregation and decision-making stage, this invention introduces a multi-instance learning paradigm based on an attention mechanism. This stage is the core innovation of the entire method: 1) Weakly supervised learning: Treating all slice features of a single patient as a "package," with each slice as an "instance," the model is trained using only patient-level diagnostic labels, perfectly adapting to the current state of clinical data labeling. 2) Attention guidance: Through a learnable attention pooling layer, the model dynamically calculates an importance weight (score) for each slice feature. This weight reflects the contribution of the slice to the final disease diagnosis. 3) Feature aggregation: All slice features are weighted and summed according to their attention weights to generate a unified, patient-level comprehensive feature representation. 4) Classification prediction: This comprehensive feature is finally fed into a classifier, outputting a specific disease risk prediction probability.

[0011] Finally, in the results output and system maintenance phase, this invention not only provides prediction results but also generates a visual heatmap based on the aforementioned attention weights, intuitively highlighting the image regions with the greatest impact on decision-making, thus meeting the urgent clinical need for model interpretability. Furthermore, the solution incorporates a model iteration and update mechanism, enabling the system to continuously absorb new data through incremental learning, maintaining the advanced nature and adaptability of its predictive performance.

[0012] Based on the above solution, the present invention brings the following significant beneficial effects:

[0013] Efficiently utilizes weakly supervised information to greatly reduce annotation costs: Through a multi-instance learning framework, the model only needs the patient's overall diagnostic labels to be trained, without having to spend huge costs to obtain pixel-level or region-level fine annotations within the image, thus solving the annotation bottleneck problem in large-scale clinical applications.

[0014] Significantly improves computational efficiency and model generalization ability: Employing a "2.5D" processing strategy (i.e., analyzing a set of representative two-dimensional slices instead of the entire three-dimensional volume) avoids the huge computational burden and overfitting risk associated with directly processing three-dimensional data. Simultaneously, by fixing the feature extractor weights of the pre-trained DenseNet201 and only fine-tuning the subsequent aggregation and classification modules through transfer learning, efficient and stable model training can be achieved on limited medical data.

[0015] Providing intuitive interpretability and enhancing clinical credibility and practicality: The model's core—the attention mechanism—calculates an importance score for each input slice. This score can be directly used to generate an attention heatmap, highlighting the anatomical regions on the original image that have the greatest impact on the current prediction. This provides clinicians with clear decision-making support, facilitating the validation of model logic, the discovery of potential radiological biomarkers, and thus aiding in the development of individualized treatment plans.

[0016] Possessing continuous evolution capabilities to ensure long-term system effectiveness: This invention designs a model iterative update mechanism. After system deployment, it can periodically absorb newly added confirmed case data and fine-tune the model through incremental learning. This process enables the model to continuously adapt to changes in data distribution (such as updates to scanning equipment and population differences), achieving incremental performance optimization and ensuring its long-term viability and practicality in real-world clinical environments.

[0017] In summary, this invention, through its ingenious framework design, systematically addresses the three major challenges in medical image analysis: weak annotation, high computational cost, and black-box models. It provides a complete and practical technical solution for achieving accurate, efficient, and reliable intelligent assisted diagnosis. Attached Figure Description

[0018] Figure 1 This is a block diagram of an embodiment of the present invention. Detailed Implementation

[0019] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0020] A rapid analysis and prediction method for medical images based on deep features and multi-instance learning, comprising the following steps:

[0021] A1) Image preprocessing and slice extraction: The three-dimensional medical image of the target patient is read, and the target anatomical structure region is segmented as the Region of Interest (ROI). A set of continuous two-dimensional slices is selected centered on the largest cross-section of this region, forming a slice set. The three-dimensional medical image is a pelvic CT image, the target anatomical structure is the pelvic skeleton, and the slice set contains at least three continuous axial slices centered on the largest cross-section.

[0022] A2) Deep Feature Extraction Based on DenseNet201: Each 2D slice in the slice set is independently input into a DenseNet201 deep convolutional neural network pre-trained on the ImageNet dataset; the fully connected classification layer at the end of the network is removed, and the high-dimensional feature vector output by its global average pooling layer is extracted as the deep feature representation of each slice; after processing all slices of a patient, the output is a set of feature vectors, denoted as... Where k is the number of slices, each This represents the 1920-dimensional depth feature of the i-th slice.

[0023] The DenseNet201 network employs a densely connected structure and is pre-trained on the ImageNet dataset. During feature extraction, the weights of its preceding convolutional layers are fixed, and end-to-end training is performed only on the subsequent attention aggregation and classification modules. This achieves efficient transfer learning and prevents overfitting on small sample medical data.

[0024] A3) Attention-based Multi-Instance Learning Aggregation and Prediction: A single patient is considered a package, and its feature vector set is considered multiple instances within that package; patient-level diagnostic labels serve as the package labels; the instance feature set is input into an attention-based multi-instance learning network. The core of this network is an attention pooling layer, where a learnable parameter matrix V and a vector w are used to represent the features F of each instance. i Calculate an attention score a i All instance features are weighted and summed based on their attention scores to obtain a package-level aggregated feature representation Z. Finally, the aggregated feature Z is input into a fully connected classifier, which outputs the disease prediction probability of the patient.

[0025] The attention score a i The calculation formulas for and the aggregation feature Z are as follows:

[0026]

[0027]

[0028] Where V is the learnable weight matrix and w is the learnable weight vector. This represents the transpose of the weight vector w. For the first The depth feature vector of each slice, where K is the number of slices. Representing the eigenvector transpose, The hyperbolic tangent activation function is used. It is an exponential function.

[0029] A4) Result Interpretation and Model Iteration Update: Risk classification is performed based on the predicted probability; at the same time, an interpretability heatmap is generated based on the attention score aᵢ to identify the key image slice areas that contribute the most to the prediction; after system deployment, newly confirmed case data are collected regularly, and the attention multi-instance learning network and classifier are fine-tuned and updated in an incremental learning manner to achieve continuous optimization and performance improvement of the model.

[0030] The specific method is as follows:

[0031] A4.1: Reading Model and Case Data

[0032] With the continuous operation of the medical system and the continuous accumulation of clinical data, newly generated medical imaging data and corresponding diagnostic results become important bases for validating and optimizing models. First, the system automatically reads the complete dataset of newly added cases, including: the patient's original medical images, segmented regions of interest, extracted deep feature vector sets, clinical feature data, treatment parameters, and final clinical diagnostic labels.

[0033] A4.2: Incremental Training and Model Update

[0034] Based on the newly added case data, the system initiates an incremental training process: using these data as new training samples, while maintaining the original model architecture, the parameters of the multi-instance learning aggregation network and classifier are fine-tuned with a small learning rate, thereby achieving incremental optimization of the model; this process does not destroy the knowledge previously learned by the model, while adapting it to the distribution characteristics of the new data, continuously improving the accuracy and generalization ability of prediction.

[0035] A4.3: Generate interpretability reports and update risk forecasts

[0036] Based on the updated model, the system can re-analyze historical or new cases: for each patient, the model not only outputs the predicted probability of their disease risk, but also automatically generates an interpretable heatmap based on attention weights, visually marking the key imaging areas that contribute the most to the prediction results; this report can be reviewed by clinicians to verify the rationality of the model's decisions, discover new imaging biomarkers, and provide auxiliary basis for adjusting individualized treatment plans.

[0037] Example 1:

[0038] The application process of this method is demonstrated by taking the prediction of osteoporosis after pelvic radiotherapy in cervical cancer patients as an example.

[0039] Step 1: Image preprocessing and key slice extraction

[0040] The system receives baseline pelvic CT 3D images from a cervical cancer patient. First, using a pre-trained U-Net segmentation model, all pelvic skeletal structures (including the hip bones and sacrum) in the image are automatically delineated, forming a 3D region of interest (ROI). Next, the system locates the largest cross-section of this ROI in the axial plane. Using this central layer as a reference, and strictly following a pre-defined sampling protocol, seven adjacent 2D slices are extracted (i.e., layer 0, ±1, ±2, and ±4). This "2.5D" strategy significantly reduces the complexity of subsequent processing while preserving 3D contextual information.

[0041] Step 2: Deep Feature Extraction Based on DenseNet201

[0042] The seven preprocessed 2D slices (uniformly scaled to 224×224 resolution) were then fed one by one into the pre-trained DenseNet201 network. DenseNet201 employs a dense connection pattern, effectively mitigating the vanishing gradient problem and promoting feature reuse, making it particularly suitable for learning rich features from limited medical data. In this invention, we removed the last fully connected layer originally used for ImageNet classification and extracted the output of the global average pooling layer following the last convolutional layer. This layer encodes each slice into a 1920-dimensional high-dimensional feature vector, containing rich image information from low-level edges to high-level semantics. Thus, the patient is digitally represented as a set of feature vectors: ,in .

[0043] Step 3: Multi-instance learning aggregation and risk prediction

[0044] We treat the seven feature vectors of a single patient as a "bag," with each feature vector representing an "instance." The bag label indicates whether the patient was diagnosed with osteoporosis one year later (yes / no). These instances are fed into our designed attention-based multi-instance learning network. The network maps each 1920-dimensional feature Fᵢ to a new representation space through a fully connected layer (parameter V) and a nonlinear transformation (tanh). Subsequently, an attention score aᵢ is computed for each instance using an attention vector w. By superimposing or mapping the attention score aᵢ of each slice back to the original 2D slice image region as a weight for the overall importance of that slice, a visual heatmap is generated. The magnitude of the score aᵢ automatically reflects the importance of that slice in predicting osteoporosis; for example, slices containing major weight-bearing joints (such as the sacroiliac joint) may receive higher weights.

[0045] Subsequently, all instance features are summed in weighted order of score to obtain a comprehensive, patient-level aggregated feature representation Z. Finally, Z is fed into a lightweight classifier (such as a multilayer perceptron with dropout layers), which outputs a predicted probability between 0 and 1, representing the patient's risk of developing osteoporosis.

[0046] Step 4: Interpretable Output and Model Lifecycle Management

[0047] The system not only outputs predicted probabilities (e.g., >0.7 indicates high risk), but more importantly, it can back-map the attention score aᵢ of each slice to the corresponding location in the original CT image, generating an attention heatmap. Doctors can intuitively see which anatomical regions' image features the model's decisions primarily rely on, which greatly enhances clinical trust and may help discover new imaging biomarkers.

[0048] Furthermore, the system of this invention possesses continuous learning capabilities. Once the hospital has accumulated a new batch of patient data that has been followed up and confirmed, the administrator can initiate an incremental learning process. The system will fine-tune the attention aggregation network and classifier with a low learning rate using the new data, without altering the DenseNet201 basic feature extractor. This process allows the model to continuously adapt to changes in data distribution (such as new scanner parameters or population differences), achieving continuous performance evolution without the need for costly retraining.

[0049] Summary of beneficial effects

[0050] Through the above process, this invention successfully achieves: 1) training using easily accessible patient-level labels; 2) efficient and accurate mining of key information in 3D images through the combination of DenseNet201 and attention MIL; 3) providing intuitive visual explanations to meet clinical compliance requirements; and 4) designing a model update mechanism to ensure the long-term effectiveness and practicality of the system in the real world.

Claims

1. A rapid analysis and prediction method for medical images based on deep features and multi-instance learning, characterized in that, Includes the following steps: A1) Image preprocessing and slice extraction: Read the three-dimensional medical images of the target patient and segment the target anatomical structure region as the ROI; select a set of continuous two-dimensional slices centered on the largest cross-section of the region to form a slice set; A2) Deep feature extraction based on DenseNet201: Each two-dimensional slice in the slice set is independently input into a DenseNet201 deep convolutional neural network pre-trained on the ImageNet dataset. Remove the fully connected classification layer at the end of the network and extract the high-dimensional feature vector output from its global average pooling layer as the deep feature representation of each slice; After processing, all slides from a patient are output as a set of feature vectors, denoted as {F1, F2, ..., F...}. k }, where k is the number of slices, and each Fᵢ ∈ R 1920 The 1920-dimensional depth feature represents the i-th slice; A3) Attention-based Multi-Instance Learning Aggregation and Prediction: A single patient is considered a package, and its feature vector set is considered multiple instances within that package; patient-level diagnostic labels serve as the package labels; the instance feature set is input into an attention-based multi-instance learning network. The core of this network is an attention pooling layer, where a learnable parameter matrix V and a vector w are used as the feature F for each instance. i Calculate an attention score a i All instance features are weighted and summed based on their attention scores to obtain a package-level aggregated feature representation Z. Finally, the aggregated feature Z is input into a fully connected classifier, which outputs the disease prediction probability for the patient. A4) Result Interpretation and Model Iteration Update: Risk classification is performed based on the predicted probability; at the same time, an interpretable heatmap is generated based on the attention score aᵢ to identify the key image slice areas that contribute the most to the prediction; after the system is deployed, newly confirmed case data are collected regularly, and the attention multi-instance learning network and classifier are fine-tuned and updated in an incremental learning manner.

2. The method for rapid analysis and prediction of medical images based on deep features and multi-instance learning according to claim 1, characterized in that, In A2), the DenseNet201 network adopts a dense connection structure and is pre-trained on the ImageNet dataset. During feature extraction, the weights of the preceding convolutional layers are fixed, and end-to-end training is performed only on the subsequent attention aggregation and classification modules to achieve transfer learning and prevent overfitting on small sample medical data.

3. The method for rapid analysis and prediction of medical images based on deep features and multi-instance learning according to claim 1, characterized in that, In A3): Attention score a i The calculation formulas for and the aggregation feature Z are as follows: ； ； Where V is the learnable weight matrix and w is the learnable weight vector. This represents the transpose of the weight vector w. For the first The depth feature vector of each slice, where K is the number of slices. Representing the eigenvector transpose, The hyperbolic tangent activation function is used. It is an exponential function.

4. The method for rapid analysis and prediction of medical images based on deep features and multi-instance learning according to claim 1, characterized in that, In A4), the specific method is as follows: A4.1: Reading Model and Case Data With the continuous operation of the medical system and the continuous accumulation of clinical data, newly generated medical imaging data and corresponding diagnostic results become important bases for validating and optimizing models. First, the system automatically reads the complete dataset of newly added cases, including: the patient's original medical images, segmented regions of interest, extracted deep feature vector sets, clinical feature data, treatment parameters, and final clinical diagnostic labels. A4.2: Incremental Training and Model Update Based on the newly added case data, the system initiates an incremental training process: using these data as new training samples, while maintaining the original model architecture, the parameters of the multi-instance learning aggregation network and classifier are fine-tuned at an initial training learning rate of one-tenth to one-hundredth, to achieve incremental optimization of the model; adapting the model to the distribution characteristics of the new data without destroying the knowledge previously learned by the model. A4.3: Generate interpretability reports and update risk forecasts Based on the updated model, the system re-analyzes historical or new cases: for each patient, the model outputs the predicted probability of disease risk, and automatically generates an interpretable heatmap based on attention weights, marking the key imaging regions that contribute the most to the prediction results. The report is provided for clinicians to review, to verify the rationality of the model's decisions, to discover new imaging biomarkers, and to provide auxiliary basis for adjusting individualized treatment plans.

5. The method for rapid analysis and prediction of medical images based on deep features and multi-instance learning according to claim 1, characterized in that, The three-dimensional medical image is a pelvic CT image, the target anatomical structure is the pelvic skeleton, and the slice set contains at least three continuous axial slices centered on the largest cross-section.

6. The application of the rapid medical image analysis and prediction method based on deep features and multi-instance learning according to any one of claims 1-5 in predicting osteoporosis after radiotherapy in cervical cancer patients.

7. The application according to claim 6, characterized in that, The slice set includes a central layer centered on the largest cross-section of the pelvis, and axial slices located in the first layer above, the first layer below, the second layer above, the second layer below, the fourth layer above, and the fourth layer below the central layer.

Images