Three-dimensional medical image classification method based on CT and CBCT cross-modal information fusion

Abstract

The invention provides a CT-CBCT three-dimensional medical image classification method based on a cross-modal difference perception mechanism, and aims to improve the discrimination accuracy and feature consistency in the fusion process of different modal data. The method comprises the following steps: firstly, carrying out resampling, rigid registration, intensity matching and normalization processing on CT and CBCT volume data to obtain network input with a consistent structure; in the feature extraction stage, independent three-dimensional encoders are constructed respectively to carry out multi-scale feature representation on the two modals, and information interaction of the CT and the CBCT on the middle scale is realized through a windowed cross-modal attention mechanism. The difference chart generated based on the attention response is further enhanced through a refining module, so that the network can adaptively sense the structural deviation between modes, and bidirectional feature fusion under difference regulation is realized through a gating fusion mechanism. And then uniformly integrating the multi-scale features by using a feature pyramid network to form high-level representation for classification and discrimination, and finally outputting a target category through a classification head. According to the method, difference perception, cross-modal attention and multi-scale fusion mechanisms are introduced, the classification robustness of CT and CBCT data under the conditions of noise, structural deviation and modal difference is effectively improved, and the method is suitable for various clinical scenes such as medical image typing and focus recognition and has remarkable practical value and application prospects.

Description

Technical Field

[0001] This invention relates to the fields of medical artificial intelligence (Medical AI), deep learning (DL), 3D convolutional neural network (3D-CNN), cross-modal feature modeling, self-attention mechanism, 3D medical image processing, and radiotherapy image registration and error analysis, and is particularly applicable to automated error source classification and clinical decision support based on CT and CBCT multimodal image data. Background Technology

[0002] Radiotherapy is one of the most commonly used and crucial methods in current clinical cancer treatment. Its basic principle is to precisely irradiate the tumor area with high-energy rays, thereby inducing DNA damage in cancer cells and inhibiting their growth. In clinical practice, approximately 70% of cancer patients require radiotherapy, and some cancers can be cured through radiotherapy. However, the efficacy of radiotherapy is highly dependent on the consistency between the treatment irradiation path and the patient's actual anatomical position. Even if the radiotherapy plan is precisely designed using computed tomography (CT) simulations before treatment, there is still a risk that the irradiation location may not perfectly match the planned location due to factors such as patient positioning, organ status, or errors in imaging equipment.

[0003] To improve positioning accuracy, cone-beam CT (CBCT) is often used to acquire the patient's current anatomical structures before each treatment, and the images are then registered and aligned with the planning CT for positioning correction. This process is called radiotherapy positioning. However, due to factors such as changes in patient body shape, organ movement, inconsistent image quality, fixation device deviations, or equipment errors, differences inevitably exist between the planning CT and the treatment CBCT. These differences can further affect the dose distribution to the lesion, thereby reducing treatment accuracy and increasing the risk of collateral damage.

[0004] Current clinical positioning error assessment mainly relies on manual comparison and experience, which has the following shortcomings: 1. High subjectivity: Doctors need to observe the differences in CT / CBCT images layer by layer to determine the source and type of error, which is easily affected by experience differences; 2. Low efficiency: Faced with high-resolution 3D medical images, each positioning assessment is time-consuming and burdensome, which is not conducive to high-throughput treatment processes; 3. Inability to accurately quantify the source of error: Only the overall offset is given, and it is impossible to distinguish the source of positioning error, anatomical changes, artifact interference, or equipment error; 4. Lack of unified intelligent auxiliary tools: Traditional image comparison or simple rigid registration methods cannot reliably analyze local structural changes.

[0005] With the development of medical artificial intelligence technology, researchers have recently begun to explore the use of deep learning methods such as 3D convolutional neural networks (3D-CNN), cross-modal feature fusion, and self-attention mechanisms to automatically analyze CT and CBCT images in order to assist in identifying error types. However, existing methods still have the following problems: 1. Insufficient modeling of cross-modal differences: CT and CBCT imaging mechanisms are different, with significant differences in intensity and artifacts, making it difficult for simple feature stitching to model effective differences; 2. Lack of error source classification ability: Existing models mostly focus on registration residual estimation, which can only output offsets but cannot locate or explain the source of errors; 3. Insufficient utilization of 3D features: Some methods are still based on 2D slices or average projection inputs, failing to achieve joint modeling of global and local information in 3D space.

[0006] Based on the aforementioned background and needs, this invention proposes an intelligent classification method based on the fusion of multimodal three-dimensional medical images and differential response features. This method constructs a dual-path three-dimensional neural network to jointly model cross-modal differential features between planning CT and therapeutic CBCT, and introduces a differential response-based gating fusion mechanism to achieve automatic classification of radiotherapy positioning error sources. Experimental verification shows that the method of this invention can accurately identify various error sources and assist physicians in verification with interpretable atlases, significantly improving the efficiency and reliability of error determination. It can be widely applied to radiotherapy quality control and clinical process optimization. Summary of the Invention

[0007] The purpose of this invention is to provide an intelligent classification method based on the fusion of multimodal three-dimensional medical images and differential response features, to solve the problem of accurately determining the source of error between planning CT and treatment CBCT images in current radiotherapy, especially in real-world scenarios where multiple types of errors are mixed, such as positioning errors, changes in anatomical structures, image artifacts, and equipment geometric deviations. This invention constructs a cross-modal deep neural network to jointly analyze the structural and intensity differences between images before and after radiotherapy, achieving automatic classification of error source types, thereby improving the efficiency of clinicians' judgment and the accuracy of treatment.

[0008] Traditional error analysis methods based on image registration or manual comparison mainly rely on global or local rigid registration residuals, voxel differences, or empirical judgments, making it difficult to effectively distinguish the types of error sources and locate areas of concentrated differences. In addition, existing deep learning models often lack mechanisms for modeling differences between different modalities (CT / CBCT), making it difficult to capture structural features such as imaging artifacts, soft tissue deformation, and equipment deviations, resulting in insufficient interpretability of results and hindering quality control throughout the radiotherapy process.

[0009] To address the aforementioned technical problems, this invention proposes an innovative intelligent classification method, primarily achieved through the following three technical improvements: 1. Dual-path 3D encoding and cross-modal difference feature extraction: This invention employs a dual-path 3D convolutional neural network to extract features from planning CT and therapeutic CBCT images respectively, and obtains their spatial correspondence through a cross-modal attention mechanism to construct a preliminary difference response map based on the registration context, achieving deep modeling of intermodal structural differences; 2. Difference response-guided gated fusion mechanism: This invention proposes a difference response feature enhancement module to address structural differences and imaging artifacts between CT / CBCT. This module utilizes convolution and self-attention structures to generate difference intensity maps, and fuses features of significantly different regions with global skeleton features through a gating mechanism, thereby enhancing the interpretability and robustness of error source discrimination; 3. Multi-scale feature integration and error source classification output: This invention constructs a multi-scale feature fusion structure (FPN) to effectively integrate high-resolution spatial information with deep semantic differences, and finally outputs the error source classification result through global average pooling and fully connected layers. The model is trained using the cross-entropy loss function, which has the advantages of good convergence stability and strong generalization ability.

[0010] In summary, the main technical contents of this invention include:

[0011] (1) A cross-modal attention guidance method based on three-dimensional dual-modal images is proposed to achieve multi-level differential feature extraction;

[0012] (2) Construct a differential response gating network and explicitly guide feature fusion through differential graphs to improve the interpretability and accuracy of classification;

[0013] (3) Based on multi-scale feature aggregation and regression classification output, intelligent classification of the sources of radiotherapy positioning error is completed;

[0014] (4) The model has been verified by actual clinical radiotherapy data and can accurately identify error sources such as positioning error, anatomical changes, image artifacts and equipment deviation, providing an efficient and reliable technical means for radiotherapy quality control. Attached Figure Description

[0015] Figure 1A flowchart illustrating a medical image classification method provided by an embodiment of the present invention is shown.

[0016] Figure 2 This illustration shows a schematic diagram of a cross-modal CT / CBCT feature fusion network structure provided by an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific examples.

[0018] like Figure 1 As shown, the overall process of the intelligent medical image classification method provided in this embodiment of the invention mainly includes three stages. First, data preprocessing is performed on CT and CBCT images to convert the original DICOM data into a unified input format that the network can accept. The preprocessing includes at least resampling, rigid registration, histogram matching, region of interest (ROI) localization and cropping, and intensity normalization. Second, the preprocessed multimodal images are input into a cross-modal feature fusion network for model training. The network includes modules such as dual encoders, cross-modal attention, feature pyramid network (FPN), and global average pooling (GAP) to optimize the model. Finally, the data to be inferred is fed into the trained model, and the corresponding classification results are output.

[0019] First, data preprocessing.

[0020] First, for each case, the corresponding DICOM format files of the CT and CBCT images are read separately and parsed into 3D volume data using a toolkit to meet the input requirements of subsequent deep network processing. Second, to eliminate spatial resolution differences caused by different scanning devices and imaging parameters, the original 3D images are resampled to a consistent isotropic voxel spacing, thus ensuring the consistency of the input data in spatial scale. Subsequently, based on the image orientation matrix and the patient coordinate system, the CBCT images are aligned to the CT image space through 3D rigid registration (including rotation and translation transformations) to address the differences in body position during imaging, thereby achieving spatial consistency of cross-modal images. To address the potential inconsistency in grayscale distribution caused by the different imaging principles of CT and CBCT, histogram matching is performed on the registered CBCT images to ensure that their grayscale probability distribution is consistent with that of the CT images, thereby improving the fusion effect of cross-modal features and the robustness of the network. Intelligent localization of 3D regions of interest (ROI): This invention uses CT images as a reference to extract human body regions through simple thresholding or density segmentation methods and calculates their geometric centroids. Subsequently, a fixed-volume 3D region centered on the centroid is cropped as the region to be processed, ensuring that the depth model focuses on important anatomical structures. Within this ROI, CT images are truncated to unit intervals after window width and level mapping, and then normalized; CBCT images undergo zero-mean unit variance standardization to eliminate intensity biases under different scanning conditions. Finally, the preprocessed CT and CBCT ROI data are saved as .npy format 3D array files, used as standardized inputs for model training and inference.

[0021] Second, network structure.

[0022] The cross-modal 3D classification network consists of a dual-branch 3D encoder, a windowed cross-modal attention module, a difference map refinement and gating fusion unit, a feature pyramid fusion (FPN), and a classification head (GAP+FC). The overall structure is as follows: Figure 2 As shown, the details are as follows.

[0023] (1) Dual-branch 3D encoder

[0024] The network first encodes the volumetric data of CT and CBCT separately. To ensure the alignment consistency of the two modalities in multi-scale space, the two encoders have the same hierarchical structure but do not share weights.

[0025] Let the input CT body data be

[0026]

[0027] CBCT volume data are

[0028]

[0029] After multiple convolutions and downsampling operations, the following results were obtained at scales of 1 / 4, 1 / 8, and 1 / 16:

[0030]

[0031] The multi-scale design of the encoder enables the network to capture local structural information and large-scale spatial dependencies simultaneously, laying the foundation for subsequent cross-modal attention and fusion processes.

[0032] (2) Windowed cross-modal attention mechanism

[0033] To enhance the fine-grained correspondence between CT and CBCT, this invention introduces a windowed cross-modal attention mechanism at the 1 / 8 scale. This mechanism uses CBCT features as the query and CT features as the key and value, establishing a mapping between modalities within a local 3D window.

[0034]

[0035] First, we obtain three sets of features after linear mapping:

[0036] Q = W Q F q K = W K F k V = W V F k

[0037] Calculate the attention weights within each 3D window and perform a weighted sum:

[0038]

[0039] By combining the results of all windows, the cross-modal enhancement features of CT→CBCT can be obtained, and the attention confidence map M can be obtained at the same time. The value of M represents the confidence of the network for each spatial location in cross-modal correspondence.

[0040] (3) Construction and Refinement of Difference Maps

[0041] In the cross-modal attention process, the attention map M contains important information about modal differences. To further extract the structural differences between CT and CBCT, this invention designs a difference map construction and refinement module.

[0042] First, construct a modal difference metric based on M:

[0043] Δ=1-σ(M / τ)

[0044] Where τ is a temperature coefficient used to adjust the difference sensitivity. Subsequently, the difference map Δ undergoes convolution, normalization, and local self-attention processing to obtain a more stable and structurally clear difference response map. The refined difference mask is defined as:

[0045]

[0046] in, This represents the difference refinement network. The mask R will serve as a gating weight in the subsequent fusion stage, guiding the selective mixing of the information flow.

[0047] (4) Dual-branch gating fusion mechanism

[0048] To effectively integrate the complementary information from CT and CBCT at the 1 / 8 scale, this invention proposes a gated fusion mechanism based on difference maps. For any input feature F, the fusion form is as follows:

[0049] G = (1-R)⊙F + R⊙H(F)

[0050] Where ⊙ represents element-wise multiplication, and H(·) is a learnable linear projection. The above formula means that when the difference response is low (small R), the original features are preserved; when the difference response is strong (large R), the projection features are enabled, allowing the network to focus on processing regions with inconsistent modes.

[0051] By performing gating operations on the CT branch and CBCT branch separately, the difference-driven fusion features of the two can be obtained, and cross-modal consistency can be improved.

[0052] (5) Multi-scale Feature Pyramid (FPN) Fusion and Classification Output

[0053] After obtaining the fused 1 / 8 scale features, the network further performs pyramid-style integration at different scales such as 1 / 4, 1 / 8, and 1 / 16. The FPN module achieves mutual complementarity of multi-scale information through upsampling and lateral connections, thereby constructing a high-resolution fused feature map containing rich semantics.

[0054] Let the fused output be:

[0055]

[0056] The final feature map, after global average pooling, is input into the classification head:

[0057]

[0058] The final category prediction result is obtained.

[0059] (6) Model Training

[0060] This invention employs supervised learning for end-to-end network training. Registered CT and CBCT data are used as input, and the network outputs the corresponding category or structure discrimination result. A differentiable supervised loss is calculated based on the deviation between the predicted result and the ground truth annotation, and the network parameters are updated via backpropagation. The training hyperparameters (including learning rate, batch size, etc.) can be flexibly adjusted according to the actual application scenario. During training, the network of this invention can gradually learn the differences and fusion rules between cross-modal data, thereby obtaining stable discrimination capabilities.

Claims

1. A three-dimensional medical image classification method based on CT and CBCT cross-modal information fusion, characterized in that, The method comprises the following steps: Step 1, inputting CT volume data and CBCT volume data into two independent three-dimensional encoders respectively to extract multi-level voxel features of at least three scales; Step 2, at the intermediate scale, using the CBCT features as queries, the CT features as keys and values, and using a windowed cross-modal attention mechanism to interact the features to obtain fused features and an attention response map; Step 3, constructing a difference map based on the attention response map, and enhancing the difference map through a difference refining module to obtain a gating weight for regulating the modal difference; Step 4, using a gating fusion mechanism to weight and fuse the intermediate scale features of CT and CBCT, so that the features of different modalities realize unified expression under the regulation of the difference; Step 5, using a feature pyramid network to perform step-by-step upsampling and integration on the multi-scale features to obtain fused features for classification and discrimination; Step 6, inputting the fused features into a classification head to output a class prediction result corresponding to the target.

2. The method of claim 1, wherein, The windowed cross-modal attention mechanism divides the three-dimensional features into fixed-size three-dimensional windows, and calculates the similarity of query vectors, key vectors and value vectors in each window to realize cross-modal feature association within a local spatial range.

3. The method of claim 1, wherein, The difference refining module includes gradient nonlinear mapping, three-dimensional convolution, normalization and self-attention operation to enhance the representation ability of structural differences between different modalities.

4. The method of claim 1, wherein, The gating fusion mechanism weights and combines the input features according to the gating weight generated by the difference map, so that the regions with larger differences obtain stronger regulation ability, and the regions with smaller differences maintain the original features.

5. The method of claim 1, wherein, The feature pyramid network performs upsampling and step-by-step fusion on the CT features and CBCT features from different scales, so that the final fused features contain multi-scale spatial information.

6. The method of claim 1, wherein, The classification head adopts global pooling combined with a fully connected structure to complete the class discrimination of the three-dimensional fused features.

Images