Lung nodule benign and malignant classification method based on cross-modal spatio-temporal fusion network

Abstract

The application discloses a lung nodule benign and malignant classification method based on a cross-modal spatio-temporal fusion network, first, CT image data is acquired and preprocessed to obtain ROI images; then, a spatial feature extraction module is used to extract spatial features in the ROI images and obtain text features; a time residual fusion module is used to obtain fused CT image spatio-temporal features and clinical text features; the output of the time residual fusion module is input into a cross-modal attention fusion network for preliminary fusion to obtain a final cross-modal attention map; a multi-head self-attention fusion block is applied to further fuse and process the cross-modal attention map to obtain feature representation based on the final fused spatio-temporal data and clinical data; and finally, a FC layer is used to output a benign and malignant classification result. First, features are extracted from follow-up CT image data, spatio-temporal fusion is performed, and then the features are combined with text clinical data, so that the accuracy and robustness of classification are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image morphological operations, and in particular to a method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network. Background Art

[0002] The National Lung Screening Trial (NLST) study showed that regular imaging examinations and pulmonary function assessments can help track changes in nodules and predict their risk of malignancy, thereby increasing the possibility of early detection and significantly improving survival rates. However, existing assessment methods mainly rely on single CT scans, which limits in-depth understanding of nodule progression and may lead to late diagnosis. Regular imaging follow-up helps to dynamically collect data on nodule changes, thereby improving the accuracy of malignant tumor prediction and prevention and improving patient health prognosis.

[0003] However, many early works often rely on single-time-point imaging data for classification and prediction, but this approach cannot fully capture the dynamic information of nodules changing over time. In contrast, using follow-up data for multi-time-point analysis can better reflect the progression of lesions and improve the accuracy and reliability of predictions. Despite this, existing multi-time-point studies have failed to fully integrate clinical data and failed to achieve multimodal fusion, limiting the model's potential in integrating clinical information. Secondly, when combining multimodal and multi-time-point data, many studies use a simple splicing method of the time dimension, failing to fully utilize the rich features of the time dimension, limiting the model's in-depth understanding of temporal changes. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention proposes a method for classifying benign and malignant lung nodules based on a cross-modal spatiotemporal fusion network. First, features are extracted from follow-up CT image data, spatiotemporally fused, and then combined with text clinical data to effectively improve the accuracy and robustness of classification.

[0005] In order to solve the above technical problems, the technical solution of the present invention is:

[0006] A method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network includes the following steps:

[0007] Step 1: Acquire CT image data and preprocess it to obtain ROI image;

[0008] Step 2: A spatial feature extraction module is used to extract spatial features from the ROI image, and the corresponding clinical data is embedded into text features.

[0009] Step 3: The spatial features from different time periods t0 and t1 are input through the temporal residual fusion module, and the input feature data is fused through a multi-level feature fusion strategy; the fused features are restored to resolution through upsampling and integrated through global average pooling; after global average pooling, the multi-scale features are refined through the convolution layer; finally, weighted fusion is performed;

[0010] Step 4: Input the spatiotemporal features of the CT image and the text features after spatiotemporal fusion by the temporal residual fusion module into a cross-modal attention fusion network for preliminary fusion to obtain the final cross-modal attention map;

[0011] Step 5: Apply a multi-head self-attention fusion block to further fuse the cross-modal attention map to obtain the feature representation based on the final fused spatiotemporal data and clinical data;

[0012] Step 6: Finally, based on the feature representation of the final fused spatiotemporal data and clinical data, a FC layer is used to output the benign or malignant classification result.

[0013] Preferably, in step 1, the preprocessing method for the CT image is:

[0014] The HU values ​​of all CT images were adjusted to between [-1200, 600], and then the CT images were cropped into 16*64*64 ROI images according to the nodule location information.

[0015] As a preference, in step 3, when weighted fusion is performed, a learning fusion mechanism is used to drive

[0016] The feature weights α0 and α1 from different time points are dynamically adjusted, and the expressions are as follows:

[0017] STF feat =σ(α0)*F feat +σ(α1)*t1.

[0018] Among them, F feat It is the fusion feature of the spatial features of the two time periods t0 and t1.

[0019] Preferably, the spatial feature extraction module combines 3DResnet and CBAM modules.

[0020] Specifically, the CBAM module is introduced into the residual block of the 3D ResNet. Its channel-wise and spatial-attention mechanisms are used to optimize the convolutional outputs before adding them together with skip connections. This not only retains the 3D ResNet's powerful spatial feature extraction capabilities but also enhances the focus on key feature areas, significantly improving the model's expressiveness, robustness, and classification accuracy.

[0021] Preferably, in step 4, the fusion method of the cross-modal attention fusion network is:

[0022] First, the image feature x is transformed into i and text features y i They are converted to x∈R^(B*D) and y∈R^(B*D), respectively, where B refers to the batch size and D refers to the dimension of the image and text after dimensionality reduction. The input is then reduced in dimension through a fully connected layer, and the reduced features are input into a 1×1 convolutional layer to convert them into three feature spaces. The corresponding weight matrices are generated at the same time. Subsequently, the matching degree between the features is calculated as follows:

[0023]

[0024] Where S = W*H represents the spatial domain size of the follow-up image, β and ρ represent the matching scores of the image and clinical text in the spatial domain, q1, q2, k1 and k2 are the image features x i , the text feature is y i , after the fully connected layer, the matrix required for the subsequent attention mechanism is generated; s i,j Refers to the matching degree between different features of the image and the text, t i,j Refers to the degree of match between different features of the text and the image.

[0025] Preferably, the score β j，i and ρ j，i Used to weight the corresponding feature values ​​and generate the final cross-modal attention map.

[0026] Preferably, steps 2 to 6 constitute a CSF-Net model, and the CSF-Net model is implemented on a device equipped with a GeForceRTX 3090Ti GPU using the torch-2.1.0-cu12.1-cudnn8.9 framework.

[0027] Preferably, when training the CSF-Net model, a cross entropy loss function and an Adam optimizer are used for optimization, the initial learning rate is set to 0.0001, and the total number of training rounds is 200.

[0028] The present invention has the following characteristics and beneficial effects:

[0029] Using this technical solution, we combined follow-up data from multiple time points with clinical data to enhance the ability to predict future malignant transformation of pulmonary nodules. We designed a temporal residual fusion module to effectively combine features from different time points. We also employed cross-modal attention fusion to learn spatiotemporal and clinical features, thereby improving the accuracy and robustness of the model.

[0030] By integrating spatiotemporal features with clinical text information, an efficient cross-modal fusion framework was constructed, which effectively addresses information loss and improves the comprehensiveness and accuracy of feature extraction. Experimental results fully validated the superiority of the proposed method and demonstrated its potential and clinical value in predicting benign and malignant pulmonary nodules.

[0031] This paper proposes an innovative pulmonary nodule prediction network architecture that organically integrates follow-up imaging and clinical data into the training process. By incorporating multimodal data, the model's predictive performance is significantly improved, while also demonstrating a deep understanding of the dynamic changes in pulmonary nodules through the fusion of information from multiple time points. This integration of information not only improves diagnostic accuracy but also provides richer insights into disease trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 Schematic diagram of the network model structure of an embodiment of the present invention.

[0034] Figure 2 for Figure 1 Schematic diagram of the structure of the temporal residual fusion module. DETAILED DESCRIPTION

[0035] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0036] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0037] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0038] The meanings of the special names and English abbreviations involved in the embodiments of the present invention are as follows:

[0039]

[0040] The present invention provides a method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network. To implement the above method in this embodiment, a CSF-Net module is first constructed and trained.

[0041] like Figure 1 As shown in the figure, the CSF-Net module includes a spatial feature extraction module, a temporal residual fusion module, a cross-modal attention fusion network, a multi-head self-attention fusion module and an FC layer.

[0042] Specifically, the spatial feature extraction module combines the 3DResNet and CBAM modules, effectively improving the ability to extract spatial information and enhancing sensitivity to detailed features. On this basis, the temporal residual fusion module captures dynamic changes in the time dimension by fusing features from different time points, thereby better understanding the correlation between temporal features. The cross-modal attention fusion module intelligently integrates the spatiotemporal features of follow-up CT images with clinical text information through a cross-modal attention mechanism, enabling the model to fully explore and fuse the potential correlations between different data sources, thereby improving the collaborative expression capabilities of multimodal data.

[0043] In this embodiment, the CSF-Net model was implemented using the torch-2.1.0-cu12.1-cudnn8.9 framework on a device equipped with a GeForce RTX 3090Ti GPU. During the pre-training phase, the model was optimized using the cross-entropy loss function and the Adam optimizer, with the initial learning rate set to 0.0001 and a total of 200 training rounds. To improve convergence speed and stability, the momentum parameters β1 and β2 of the Adam optimizer were set to 0.5 and 0.999, respectively, and the model parameters were updated every 20 rounds. This optimization strategy effectively addressed the challenges posed by complex data features. In terms of model input, CSF-Net generates a 3D effect by sequentially inputting three sets of stacked 3D slices. During training, the batch size was initially set to 4 to efficiently utilize computing resources.

[0044] Based on the above CSF-Net module, the specific steps are as follows:

[0045] Step 1: Acquire CT image data and preprocess it to obtain ROI image.

[0046] Specifically, the CT image data was preprocessed, and the HU values ​​of all CT images were adjusted to between [-1200, 600]. Then, according to the nodule location information given by the doctor, the CT images were cropped into 16*64*64 3D region of interest (ROI) images.

[0047] In this example, the dataset used is the US National Lung Cancer Study (NLST), a key research program sponsored by the US National Cancer Institute that aims to reduce lung cancer mortality in high-risk individuals through early imaging screening. The dataset contains longitudinal lung CT scans and related clinical information from participants who underwent annual screening for three consecutive years or until a cancer diagnosis was made.

[0048] Because the original data lacked detailed annotation of pulmonary nodule locations, 443 subjects were selected to create a cross-modal spatiotemporal CT dataset, named NLST-cmst. Each participant in this dataset underwent at least two CT scans, and the malignancy of pulmonary nodules was determined using gold-standard pathology, specifically based on lung cancer diagnoses during follow-up. This approach ensured the scientific rigor and reliability of the data. Our rigorous selection process provides valuable clinical insights into the malignant progression of pulmonary nodules, helping to improve early detection and treatment strategies.

[0049] Step 2: For feature extraction of the ROI image, 3DResnet and CBAM modules are used to capture the spatial characteristics of lung nodules in the ROI. For clinical data, word embedding is used to obtain feature representations of text.

[0050] Step 3: In the temporal residual fusion module, spatial features from different time periods t0 and t1 are first input and fused to extract richer contextual information and complex nonlinear relationships. Next, a series of upsampling, pooling, and convolution operations are used to gradually refine the feature representation. Finally, weighted fusion further enhances the model's ability to capture temporal correlations and integrate cross-temporal information, thereby improving the forecast accuracy of time series data.

[0051] Specifically, such as Figure 2 As shown in Figure 1, this module inputs features t0 and t1 from different time periods and employs a multi-level feature fusion strategy to explore the complementarity of multidimensional features. In the high-dimensional feature space, the model captures richer contextual information and complex nonlinear relationships through deep concatenation. The processed features are then upsampled to restore resolution and integrated through global average pooling. Next, convolutional layers refine the multi-scale features, enhancing the model's ability to represent complex data patterns, as shown below:

[0052] F feat =Conv3D(AvgPool(Upsample(t0+t1)))

[0053] Finally, a learnable fusion mechanism dynamically adjusts the feature weights α0 and α1 from different time points. The mechanism is expressed as:

[0054] STF fest =σ(α0)*F feat +σ(α1)*t1

[0055] This mechanism ensures the deep fusion of t1 and the processed temporal features, enhances the ability to capture temporal correlations and integrate information across time, and thus improves the overall prediction performance.

[0056] Step 4: Input the spatiotemporal features of the CT image and the text features after spatiotemporal fusion through the temporal residual fusion module into a cross-modal attention fusion network for preliminary fusion to obtain the final cross-modal attention map.

[0057] Specifically, the input of the cross-modal attention fusion module includes the spatiotemporal features x from the follow-up images i and text features y i First, the input is reduced in dimension through a fully connected layer to simplify the feature space. Next, the reduced features are fed into a 1×1 convolutional layer to transform them into three feature spaces, and the corresponding weight matrices are generated. Subsequently, the matching degree between the features is calculated to accurately capture the correlation between cross-modal information and enhance the multimodal fusion effect:

[0058]

[0059] Where S = W*H represents the spatial domain size of the follow-up image, β and ρ represent the matching scores of the image and clinical text in the spatial domain. j，i and ρ j，i Used to weight the corresponding feature values ​​and generate the final cross-modal attention map.

[0060] Step 5: Apply the multi-head self-attention fusion module to perform pairwise addition on the cross-modal attention maps for further fusion processing to obtain the feature representation based on the final fused spatiotemporal data and clinical data;

[0061] Step 6: Finally, based on the feature representation of the final fused spatiotemporal data and clinical data, a FC layer is used to output the benign or malignant classification result.

[0062] For the classification method provided in this example, the following comparative tests were used to evaluate model performance: (1) pulmonary nodule classification model: prediction using only CT images; (2) multimodal model: prediction using CT images and clinical information; (3) multimodal spatiotemporal network model: prediction using CT images and clinical information via a spatiotemporal network. Their results were compared with those obtained using the method of this example.

[0063] Specifically, the proposed method was comprehensively compared with six existing models: SCANs, which is suitable for diagnosing benign and malignant lung nodules at single or multiple time points; NAS-Lung, which improves model interpretability and classification capabilities based on neural structure and attention mechanism; T-LSTM, which introduces time interval sensitivity to improve prediction accuracy; RadFusion, which combines imaging data and electronic health records to optimize diagnostic results; DeepCAD, which uses a multimodal approach to perform time series prediction of benign and malignant lung nodules; and MCFN, which integrates online data in multiple modules to enhance subtype classification capabilities.

[0064] To ensure a fair comparison, we evaluated the six models and the proposed method using five key metrics: Accuracy (Acc), Precision (Prec), Area Under the Curve (AUC), F1-score (F1), and Recall (Rec). These metrics provide a comprehensive assessment of each model's predictive accuracy, precision, and overall performance.

[0065] Acc Prec F1 AUC Rec SCANs 0.7865 0.7667 0.7077 0.7725 0.6571 NAS-Lung 0.8539 0.8235 0.8116 0.8910 0.8000 T-LSTM 0.7645 0.7012 0.6527 0.7778 0.6000 DeepCAD 0.8590 0.7879 0.8254 0.8990 0.8667 MFCN 0.7949 0.7059 0.7500 0.8903 0.8000 RadFusion 0.7753 0.8026 0.6667 0.7693 0.6000 This embodiment 0.8974 0.8235 0.8750 0.9389 0.9333

[0066] Table 1: Experimental comparison results

[0067] As shown in Table 1, the method of this embodiment performs well on all evaluation metrics, significantly outperforming other models except for Prec. In terms of Acc, AUC, F1, and Rec, the method of this embodiment achieves improvements of at least 0.0384, 0.0496, 0.0399, and 0.0666, respectively, compared to other models.

[0068] While Prec performed comparable to NAS-Lung and T-LSTM, and DeepCAD demonstrated its unique advantages in multimodal and multi-timepoint fusion, the overall results indicate that the method in this example significantly outperforms models that rely solely on a single modality or timepoint in this multimodal and multi-timepoint fusion strategy. These experimental results further validate the effectiveness and clinical applicability of our model in predicting benign and malignant pulmonary nodules.

[0069] In combination with the technical solutions and simulation results provided in this embodiment, it can be seen that the present invention proposes an innovative network architecture for predicting lung nodules, which organically integrates follow-up imaging data and clinical data information into the training process. By introducing multimodal data, the predictive performance of the model is significantly improved, and at the same time, a deep understanding of the dynamic changes of lung nodules is demonstrated in the fusion of information at multiple time points. This integration of information not only optimizes the accuracy of diagnosis, but also provides richer insights into disease development trends. However, the current model has not yet achieved a complete end-to-end training process, so in future work, we will focus on optimizing the end-to-end architecture to improve the applicability of the model and further meet the needs of practical clinical applications.

[0070] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network, characterized in that: The steps include: Step 1: Acquire CT image data and preprocess it to obtain ROI image; Step 2: A spatial feature extraction module is used to extract spatial features from the ROI image, and the corresponding clinical data is embedded into text features. Step 3: The spatial features from different time periods t0 and t1 are input into the temporal residual fusion module and fused using a multi-level feature fusion strategy. The fused features are upsampled to restore the resolution and integrated using global average pooling. After global average pooling, the multi-scale features are refined using a convolutional layer. Finally, weighted fusion is performed to obtain the spatiotemporal features of the CT image. Step 4: Input the spatiotemporal features of the CT image and the text features after spatiotemporal fusion by the temporal residual fusion module into a cross-modal attention fusion network for preliminary fusion to obtain the final cross-modal attention map; Step 5: Apply a multi-head self-attention fusion module to further fuse the cross-modal attention map to obtain the feature representation based on the final fused spatiotemporal data and clinical data; Step 6: Finally, based on the feature representation of the final fused spatiotemporal data and clinical data, a FC layer is used to output the benign or malignant classification result.

2. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 1, characterized in that: In step 1, the preprocessing method for the CT image is: The HU values ​​of all CT images were adjusted to between [-1200, 600], and then the CT images were cropped into 16*64*64 ROI images according to the nodule location information.

3. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 1, characterized in that: In step 3, during weighted fusion, a learning fusion mechanism is used to dynamically adjust the feature weights α0 and α1 from different time points. The expression is as follows: STF feat =σ(α0)*F feat +σ(α1)*t1 Among them, F feat It is the fusion feature of the spatial features of the two time periods t0 and t1.

4. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 1, characterized in that: The spatial feature extraction module combines the 3D Resnet and CBAM modules.

5. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 4, characterized in that: The spatial feature extraction module introduces the CBAM module into the residual block of 3D ResNet, optimizes the convolution output through its channel attention and spatial attention mechanisms, and then adds it with the jump connection.

6. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 1, characterized in that: In step 4, the fusion method of the cross-modal attention fusion network is: First, the image feature x is transformed into i and text features y i They are converted into x∈R^(B*D) and y∈R^(B*D) respectively, where B refers to the batch size and D refers to the dimension of the image and text after dimensionality reduction. The input is reduced in dimension through a fully connected layer, and the reduced features are input into the 1×1 convolution layer respectively, thereby converting them into three feature spaces and generating the corresponding weight matrix. Subsequently, the matching degree between the features is calculated, and the expression is as follows: Where s = W*H represents the spatial domain size of the follow-up image, β and ρ represent the matching scores of the image and clinical text in the spatial domain, q1, q2, k1 and k2 are the image features x i , the text feature is y i , after the fully connected layer, the matrix required for the subsequent attention mechanism is generated; s i,j Refers to the matching degree between different features of the image and the text, t i,j Refers to the degree of match between different features of the text and the image.

7. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 6, characterized in that: The score β j，i and ρ j，i Used to weight the corresponding feature values ​​and generate the final cross-modal attention map.

8. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 7, characterized in that: Steps 2 to 6 constitute the CSF-Net model, which is implemented on a device equipped with a GeForce RTX 3090Ti GPU using the torch-2.1.0-cu12.1-cudnn8.9 framework.

9. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 8, characterized in that: The CSF-Net model was trained using the cross entropy loss function and the Adam optimizer. The initial learning rate was set to 0.0001, and the total number of training rounds was 200.

Images