Pulmonary nodule malignant classification method based on double-view scanning and hypergraph fusion driving

Abstract

The invention discloses a pulmonary nodule malignant classification method based on double-view scanning and hypergraph fusion driving, and the method comprises the steps: firstly collecting and preprocessing a pulmonary nodule CT data set which comprises a public data set and a private data set; the method comprises the following steps: building a DPSHG-VMama model for pulmonary nodule classification, wherein the DPSHG-VMama model comprises a stacked convolution Stem block, a text embedding layer, a collaborative fusion Mama block, a stacked DP-VMama block, a patch merging layer for down-sampling and a feature classifier; the preprocessed image data set serves as input, a DPSHG-VMama model is trained, and network parameters are optimized through iteration; and inputting a to-be-classified region-of-interest image into the trained DPSHG-VMama model, and outputting an accurate pulmonary nodule vicious classification result. According to the method, the pulmonary nodule malignancy degree is predicted by effectively establishing the space-time dynamic model and improving the multi-modal fusion efficiency, so that the classification accuracy is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of spatiotemporal multimodal fusion, axially extended cross-plane scanning, and hypergraph fusion technology, and specifically to a lung nodule malignancy classification method based on dual-view scanning and hypergraph fusion drive. Background Art

[0002] Lung cancer remains the leading cause of cancer-related deaths worldwide, and the significant improvement in five-year survival rates relies primarily on the accurate detection of early-stage lung nodules. Large-scale clinical studies have confirmed that low-dose computed tomography (LDCT) screening can reduce lung cancer mortality by 20% in high-risk populations. This finding emphasizes the critical role of radiomics in accurately assessing the malignant risk of lung nodules. This accurate assessment is crucial to ensure timely treatment of malignant lesions and prevent unnecessary harm caused by overtreatment.

[0003] Dual-Perspective Scanning and Hypergraph Fusion-Driven (DPSHG-VMamba) is an advanced model for predicting the malignancy of pulmonary nodules. This model integrates axial-coronal cross-scanning with a hypergraph fusion mechanism, significantly improving prediction accuracy. DPSHG-VMamba overcomes the imbalance between local and global representations in 3D imaging by decomposing the 3D nodule volume into a spatiotemporal signal sequence along standard diagnostic planes used by radiologists. State-space modeling is employed to capture the morphological dynamics of nodules with linear computational complexity while maintaining anatomical interpretability. Furthermore, a dynamic hypergraph topology is constructed to address the challenges of high-order cross-modal interactions. In this model, the hypergraph explicitly links clinical parameters and imaging radiomic features, enabling the modeling of cross-modal causal chains. This hypergraph structure is integrated into the VMamba hierarchy, facilitating the anatomically layered adaptive fusion of multimodal features. Experimental results show that DPSHG-VMamba significantly improves the prediction accuracy and robustness in predicting the malignancy of lung nodules, bringing broad clinical application prospects for lung nodule classification tasks.

[0004] Previous research, such as CNNs and CNN-Transformers, has attempted to simultaneously capture local microstructure and global spatiotemporal evolution patterns when processing high-dimensional medical temporal data. However, these approaches can lead to inaccuracies in modeling the dynamics of nodule growth. With the increasing number of influencing factors, research has shifted from modeling binary relationships to modeling complex multivariate relationships. Effectively establishing spatiotemporal dynamic models and improving the efficiency of multimodal fusion have become major challenges in predicting the malignancy of pulmonary nodules.

[0005] In general, the current technical problems are as follows:

[0006] 1) The present invention introduces an axial-coronal dual-plane cross-scanning strategy, which uses a state-space model to dynamically model nodule morphology with linear complexity. This effectively captures local microstructural features and global spatiotemporal evolution patterns, while ensuring anatomical interpretability. 2) The present invention develops a dynamic hypergraph topology network to overcome the limitations of binary relationships. By using a hypergrid, the network explicitly connects the nodes of clinical, imaging, and holographic modes, and realizes the adaptive fusion of high-order cross-modal associations in the VMamba hierarchy. 3) The present invention designs a lightweight and efficient computing framework that reduces the complexity of analyzing time series CT data to a linear level. By optimizing axial extension scanning and state-space models, the framework is suitable for real-time processing on standard medical equipment. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and propose a lung nodule malignancy classification method based on dual-view scanning and hypergraph fusion drive. This method predicts the malignancy of lung nodules by effectively establishing a spatiotemporal dynamic model and improving the efficiency of multimodal fusion, thereby significantly improving the classification accuracy.

[0008] In order to achieve the above object, the technical solutions specifically adopted by the present invention are as follows:

[0009] A method for classifying malignant pulmonary nodules based on dual-view scanning and hypergraph fusion drive includes the following steps:

[0010] Step 1: Collect and preprocess lung nodule CT datasets, including public datasets and private datasets;

[0011] Step 2: Build a DPSHG-VMamba model for lung nodule classification, which includes a stacked convolutional Stem block, a text embedding layer, a collaborative fusion Mamba block, a stacked DP-VMamba block, a patch merging layer for downsampling, and a feature classifier.

[0012] Step 3: Use the preprocessed image dataset as input to train the DPSHG-VMamba model and optimize the network parameters through iteration;

[0013] Step 4: Input the image of the region of interest to be classified into the trained DPSHG-VMamba model to output accurate lung nodule malignancy classification results.

[0014] Preferably, the public dataset includes several CT sequences, and the CT sequences are annotated with nodules; the private dataset includes CT scan images and corresponding clinical data, and the CT scan images are annotated with lung nodule regions of interest.

[0015] Preferably, the preprocessing method comprises: extracting a 3D region of interest with a size of 16x64x64 according to the nodule locations of the two data sets, and dividing the public data set into malignant and benign according to subtype categories.

[0016] Preferably, the classification method of the DPSHG-VMamba model includes:

[0017] Stage 1: First, the initial shallow features and semantic vectors are extracted through a dual-plane 2D selective scanning mechanism;

[0018] Then, the initial shallow features are fused with the semantic vectors through the collaborative fusion Mamba block to generate multimodal features;

[0019] Stages 2-4: Multimodal features are taken as input and gradually downsampled through stacked DP-VMamba blocks;

[0020] Stage 5: The features sampled in stage 4 are used as input to output the classification results through the feature classifier.

[0021] Preferably, in the first stage, the dual-plane two-dimensional selective scanning mechanism is implemented as follows:

[0022] Input the original CT image, denoted as Extracting initial shallow features by stacking convolutional Stem blocks

[0023] Input clinical data and convert it into semantic vector T∈R through text embedding layer B×L ;

[0024] The initial shallow features are transformed into With the semantic vector T∈R B×L Fusion generates multimodal features

[0025] Preferably, in the second to fourth stages, the multimodal features As input, it is gradually downsampled by the stacked DP-VMamba blocks from the 2nd to the 4th stage, and features are generated in sequence. and

[0026] Preferably, in the first stage, the method of extracting F0 by stacking convolutional Stem blocks includes:

[0027] First, the original CT image is decomposed into axial planes V axial (d)∈R H×W×C and coronal plane V coronal (w)∈RD×H×C ;

[0028] Sequence generation: The axial plane V axial (d)∈R H×W×C Expand into a one-dimensional sequence along the four diagonal directions, generating four sets of sequences S1, S2, S3, S4∈R N×C , where N = H × W;

[0029] Dynamic adjustment: Apply the S6 block to each group of sequences to adjust the state space model parameters and generate context-aware sequences S′1, S′2, S′3, S′4;

[0030] Feature restoration: restore the context-aware sequence to a two-dimensional feature map I′1,I′2,I′3,I′4∈R by reverse scanning H×W×C , and sum to get the output

[0031] At the same time, the coronal plane V coronal (w)∈R D×H×C Repeat the above operation as input to obtain I″;

[0032] The dual-plane information I′ and I″ are fused through the channel-spatial attention gating module to obtain the initial shallow feature F0.

[0033] Preferably, the channel-spatial attention gating module includes channel attention and spatial attention, wherein the channel attention uses a global average set and a learnable parameter Wc to assign an importance weight α axial =σ(W c [GAP(F axial )]) and α coronal =1-α axial ; The spatial attention generates a mask M through LSTMSA axial ,M coronal ∈R H×W .

[0034] Preferably, in the first stage, the method for implementing the collaborative fusion of the Mamba blocks is:

[0035] Using bidirectional symmetric attention mechanism to process initial shallow features and semantic vector T∈R B×L , the bidirectional symmetric attention mechanism includes text-visual attention and visual-text attention.

[0036] The text-visual attention is calculated as:

[0037]

[0038] in, Represents the semantic vector T obtained by the linear layer, K v ,V v ∈R HWD×d It means that the initial shallow feature F0 is obtained by two different linear layers, P s ∈R HWD×d is a learnable spatial position encoding; d k represents the dimension of vector K;

[0039] The visual-textual attention is calculated as:

[0040]

[0041] in That is the semantic vector T;

[0042] In the final fusion stage, integration is performed through an adaptive mechanism to generate axial or coronal weights based on global feature statistics:

[0043] F1=α·T+(1-α)·F0

[0044] Here, α represents the modal contribution weight derived from global feature statistics.

[0045] Preferably, in the stages 2-4, F1 is taken as input and stacked DP-VMamba blocks including a dual-plane two-dimensional selective scanning mechanism are repeatedly used to obtain F2, F3, and F4 in sequence.

[0046] The present invention has the following characteristics and beneficial effects:

[0047] This paper proposes a pulmonary nodule malignancy classification method based on dual-view scanning and hypergraph fusion (DPSHG-VMamba). Its innovations are reflected in the following aspects: 1) It systematically addresses three major challenges in dynamic modeling and multimodal data fusion of pulmonary nodules: effectively establishing a spatiotemporal dynamic model and improving the efficiency of multimodal fusion. Mainstream 3D convolutional neural networks (CNNs) and CNN-Transformer hybrid architectures struggle to simultaneously capture local microstructure and global spatiotemporal evolution patterns when processing high-dimensional medical temporal data. Existing models are often slow to process continuous multiphase CT images due to their complex computational structure, and their reliance on high-end GPUs limits their deployment on standard clinical equipment. 2) It introduces a 2D sequence modeling method based on a state-space model, SS2D. This adaptive fusion balances the contributions of axial and coronal planes, highlighting local texture in the axial plane and global trends in the coronal plane. This cross-dimensional feature complementarity mechanism not only effectively overcomes the inherent limitations of traditional single-plane analysis but also enables the collaborative representation of fine-grained tissue features and macroscopic anatomical trends. 3) It introduces a Collaborative Mamba module to promote high-level cross-modal interaction. The beneficial effects of this method include: 1) There is an improvement of more than 3% in all indicators, which emphasizes the advantages of the hypergraph structure in modeling the correlation of heterogeneous features. 2) Compared with HGCN, the three-dimensional hypergraph network of the present invention can simultaneously capture the spatial topological structure of imaging features and the hierarchical relationship of text semantics. This improvement increases F1 by 1.2%, indicating that multimodal feature interaction is enhanced by the hypergraph structure. 3) On the unseen CLST dataset, except for AUC, the model of the present invention achieved the best performance in all indicators. 4) Multi-dimensional comprehensive evaluation confirmed that the method of the present invention significantly improved the prediction accuracy and robustness, bringing broad clinical application prospects for the task of lung nodule classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the process of the present invention.

[0049] Figure 2 This is the overall architecture diagram of DPSHG-VMamba of the present invention.

[0050] Figure 3 Schematic diagram of DP-SS2D in the DPSHG-VMamba framework. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0052] A lung nodule malignancy classification method based on dual-view scanning and hypergraph fusion drive, such as Figure 1 As shown, the following steps are included:

[0053] Step 1: Collect and preprocess lung nodule CT datasets, including public datasets and private datasets.

[0054] Specifically, this embodiment utilizes a private dataset and a public dataset. The first dataset, the NLST-smst dataset, originates from the National Lung Screening Trial (NLST) initiated by the National Cancer Institute of the United States and contains physician-annotated region-of-interest (ROI) annotations for lung nodules from 335 subjects. These annotations were rigorously screened, with each participant having at least two longitudinal CT scans and clinical data (age, gender, smoking status, and screening results). The public dataset, CLST, includes 317 CT sequences and 2,295 annotated nodules from 109 patients. Nodules are classified as invasive adenocarcinoma, minimally invasive adenocarcinoma, adenocarcinoma in situ, other malignant subtypes, inflammatory, and other benign subtypes. For CLST, we categorize the subtypes into malignant (invasive adenocarcinoma, minimally invasive adenocarcinoma, adenocarcinoma in situ, and other malignant subtypes) and benign (inflammatory and other benign subtypes), and store the corresponding nodule diameter as a clinical feature. For data partitioning, we divided the NLST-CMST dataset into a training dataset of 268 cases and a test dataset of 67 cases, and used all 30 cases from the filtered CLST dataset for testing. Regarding data preprocessing, to ensure data consistency, we extracted 3D regions of interest (ROIs) of size 16 x 64 x 64 based on the nodule locations in both datasets.

[0055] Step 2: Build a DPSHG-VMamba model for lung nodule classification. The DPSHG-VMamba model includes stacked convolutional Stem blocks, text embedding layers, collaborative fusion Mamba blocks, stacked DP-VMamba blocks, patch merging layers for downsampling, and feature classifiers.

[0056] Specifically, such as Figure 2 As shown in the figure, the constructed DPSHG-VMamba model achieves the final classification through 5 stages:

[0057] Stage 1: First, the initial shallow features and semantic vectors are extracted through a dual-plane 2D selective scanning mechanism;

[0058] Among them, the dual-plane two-dimensional selective scanning mechanism is implemented as follows:

[0059] Input the original CT image, denoted as Extracting initial shallow features by stacking convolutional Stem blocks

[0060] Specifically, such as Figure 3 As shown, first, the original CT image is decomposed into axial plane V axial (d)∈R H×W×C and coronal plane V coronal (w)∈R D×H×C ;

[0061] Sequence generation: The axial plane V axial (d)∈R H×W×c Expand into a one-dimensional sequence along the four diagonal directions, generating four sets of sequences S1, S2, S3, S4∈R N×C , where N = H × W;

[0062] Dynamic adjustment: Apply the S6 block to each group of sequences to adjust the state space model parameters and generate context-aware sequences S′1, S′2, S′3, S′4;

[0063] Feature restoration: restore the context-aware sequence to a two-dimensional feature map I′1,I′2,I′3,I′4∈R by reverse scanning H×W×C , and sum to get the output

[0064] At the same time, the coronal plane V coronal (w)∈R D×H×C Repeat the above operation as input to obtain I″;

[0065] The dual-plane information I′ and I″ are fused through the channel-spatial attention gating module to obtain the initial shallow feature F0.

[0066] The channel-spatial attention gating module includes channel attention and spatial attention. The channel attention uses the global average set and the learnable parameter Wc to assign importance weight α axial =σ(W c [GAP(F axial )]) and α coronal =1-α axial ; The spatial attention generates a mask M through LSTMSA axial ,M coronal ∈R H×W .

[0067] Input clinical data and convert it into semantic vector T∈R through text embedding layer B×L ;

[0068] The initial shallow features are transformed into With the semantic vector T∈R B×L Fusion generates multimodal features

[0069] Specifically, the implementation method of the collaborative integration Mamba block is as follows:

[0070] Using bidirectional symmetric attention mechanism to process initial shallow features and semantic vector T∈R B×L , the bidirectional symmetric attention mechanism includes text-visual attention and visual-text attention.

[0071] The text-visual attention is calculated as:

[0072]

[0073] in, Represents the semantic vector T obtained by the linear layer, K v ,V v ∈R HWD×d It means that the initial shallow feature F0 is obtained by two different linear layers, P s ∈R HWD×d is a learnable spatial position encoding; d k represents the dimension of vector K;

[0074] The visual-textual attention is calculated as:

[0075]

[0076] in That is the semantic vector T;

[0077] In the final fusion stage, integration is performed through an adaptive mechanism to generate axial or coronal weights based on global feature statistics:

[0078] F1=α·T+(1-α)·F0

[0079] Here, α represents the modal contribution weight derived from global feature statistics.

[0080] Stages 2-4: The multimodal feature F1 is taken as input and gradually downsampled by stacked DP-VMamba blocks containing a dual-plane 2D selective scanning mechanism.

[0081] Specifically, the multimodal features As input, it is gradually downsampled by the stacked DP-VMamba blocks from the 2nd to the 4th stage, and features are generated in sequence. and

[0082] Furthermore, the stacked DP-VMamba blocks include layer normalization, SSM blocks and feed-forward networks.

[0083] Input features to the layer normalization, input the output of the layer normalization into the SSM block, then splice the output of the layer normalization with the output of the SSM block, input the spliced ​​result into another layer normalization, input the output of the other layer normalization into the feedforward network, and finally splice the output of the feedforward network with the spliced ​​result again, and finally output the down-sampling result.

[0084] Among them, SSM includes linear layers, depthwise convolutional layers, activation functions, DP-SS2D (dual-plane two-dimensional selective scanning mechanism), layer normalization and linear layers connected in sequence.

[0085] Stage 5: The features sampled in stage 4 are used as input to output the classification results through the feature classifier.

[0086] Specifically, the feature classifier includes adaptive global average pooling and linear layers.

[0087] Step 3: Use the preprocessed image dataset as input to train the DPSHG-VMamba model and optimize the network parameters through iteration.

[0088] Specifically, in this embodiment, random movement and rotation, gamma scaling and brightness adjustment operations are applied to expand the training samples and improve generalization and robustness. It is developed using the PyTorch 2.1.0 framework and enhanced with CUDA 12.1 acceleration and cuDNN8.9 deep neural network library support. The system is deployed on eight NVIDIA TITAN XP 12G GPUs in a distributed data parallel configuration. Initially, we pre-trained the LSTMSA model using binary cross entropy with logarithmic loss function and AdamW optimizer. The learning rate was set to 0.0001 and the pre-training process lasted 100 times. Subsequently, the complete DPSHG-VMamba model was trained under the same hyperparameter conditions.

[0089] Step 4: Input the image of the region of interest to be classified into the trained DPSHG-VMamba model to output accurate lung nodule malignancy classification results.

[0090] Given the limitations of the dataset, in this example, a 5-fold cross-validation approach was used to evaluate the performance of the model. This approach ensures the reliability and generalizability of the results.

[0091] Table 1 is a comparison table of the effects of the technical solution of this embodiment and the existing technical solution

[0092]

[0093] As can be seen from Table 1, in the comparison between DPSHG-VMamba provided in this embodiment and other methods on the NLST-cmst dataset and the CLST dataset, the technical solution provided in this embodiment performed best in the five evaluation methods of Acc, AUC, Pre, Rcc, and F1.

[0094] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for malignant classification of pulmonary nodules based on dual-view scanning and hypergraph fusion, characterized in that: The steps include: Step 1: Collect and preprocess lung nodule CT datasets, including public datasets and private datasets; Step 2: Build a DPSHG-VMamba model for lung nodule classification, which includes a stacked convolutional Stem block, a text embedding layer, a collaborative fusion Mamba block, a stacked DP-VMamba block, a patch merging layer for downsampling, and a feature classifier. Step 3: Use the preprocessed image dataset as input to train the DPSHG-VMamba model and optimize the network parameters through iteration; Step 4: Input the image of the region of interest to be classified into the trained DPSHG-VMamba model to output accurate lung nodule malignancy classification results.

2. The method according to claim 1, characterized in that The public dataset includes several CT sequences, and the CT sequences are annotated with nodules; the private dataset includes CT scan images and corresponding clinical data, and the CT scan images are annotated with pulmonary nodule regions of interest.

3. The method according to claim 1, characterized in that The preprocessing method includes: extracting a 3D region of interest with a size of 16x 64x 64 according to the nodule locations of the two data sets, and classifying the public data set into malignant and benign subtypes.

4. The method according to claim 1, wherein The classification method of the DPSHG-VMamba model includes: Stage 1: First, the initial shallow features and semantic vectors are extracted through a dual-plane 2D selective scanning mechanism; Then, the initial shallow features are fused with the semantic vectors through the collaborative fusion Mamba block to generate multimodal features; Stages 2-4: Multimodal features are taken as input and gradually downsampled through stacked DP-VMamba blocks; Stage 5: The features sampled in stage 4 are used as input to output the classification results through the feature classifier.

5. The method according to claim 4, characterized in that In the first stage, the dual-plane two-dimensional selective scanning mechanism is implemented as follows: Input the original CT image, denoted as Extracting initial shallow features by stacking convolutional Stem blocks Input clinical data and convert it into semantic vector T∈R through text embedding layer B×L ; The initial shallow features are transformed into With the semantic vector T∈R B×L Fusion generates multimodal features 6. The method according to claim 4, characterized in that In the 2nd to 4th stages, the multimodal features As input, it is gradually downsampled by the stacked DP-VMamba blocks from the 2nd to the 4th stage, and features are generated in sequence. and 7. The method according to claim 5, characterized in that In the first stage, the method of extracting F0 by stacking convolutional Stem blocks includes: First, the original CT image is decomposed into axial planes V axial (d)∈R H×W×C and coronal plane V coronal (w)∈R D ×H×C ; Sequence generation: The axial plane V axial (d)∈R H×W×C Expand into a one-dimensional sequence along the four diagonal directions, generating four sets of sequences S1, S2, S3, S4∈R N×C , where N = H × W; Dynamic adjustment: Apply S6 block to each set of sequences to adjust the state space model parameters and generate context-aware sequence S1 ′ ,S2 ′ ,S3 ′ ,S4 ′ ; Feature restoration: restore the context-aware sequence to a two-dimensional feature map I′1,I′2,I′3,I′4∈R by reverse scanning H ×W×C , and sum to get the output At the same time, the coronal plane V coronal (w)∈R D×H×C Repeat the above operation as input to obtain I″; Fusion of dual-plane information via channel-spatial attention gating module ′ and i ″ , and obtain the initial shallow feature F0.

8. The method according to claim 5, characterized in that The channel-spatial attention gating module includes channel attention and spatial attention, and the channel attention uses a global average set and a learnable parameter Wc to assign importance weight α axial =σ(W c [GAP(F axial )]) and α coronal =1-α axial ; The spatial attention generates a mask M through LSTMSA axial ,M coronal ∈R H×W .

9. The method according to claim 5, characterized in that In the first stage, the collaborative integration of Mamba blocks is achieved as follows: Using bidirectional symmetric attention mechanism to process initial shallow features and semantic vector T∈R B×L , the bidirectional symmetric attention mechanism includes text-visual attention and visual-text attention. The text-visual attention is calculated as: in, Represents the semantic vector T obtained by the linear layer, K v ,V v ∈R HWD×d It means that the initial shallow feature F0 is obtained by two different linear layers, P s ∈R HWD×d is a learnable spatial position encoding; d k represents the dimension of vector K; The visual-textual attention is calculated as: in That is the semantic vector T; In the final fusion stage, integration is performed through an adaptive mechanism to generate axial or coronal weights based on global feature statistics: F1=α·T+(1-α)·F0 Here, α represents the modal contribution weight derived from global feature statistics.

10. The method according to claim 9, characterized in that In the described stages 2-4, F1 is taken as input and the stacked DP-VMamba blocks including the dual-plane two-dimensional selective scanning mechanism are repeatedly used to obtain F2, F3, and F4 in sequence.

Images