Pulmonary nodule benign and malignant classification method based on cross-modal space-time fusion network

Abstract

The invention discloses a pulmonary nodule benign and malignant classification method based on a cross-modal space-time fusion network, and the method comprises the steps: firstly obtaining CT image data, and carrying out the preprocessing of the data, and obtaining an ROI image; then, extracting spatial features in the ROI image through a spatial feature extraction module, and obtaining text features; obtaining fused CT image spatial-temporal features and clinical text features through a time residual fusion module; inputting the output passing through the time residual fusion module into a cross-modal attention fusion network, and performing preliminary fusion to obtain a final cross-modal attention graph; a multi-head self-attention fusion block is used to carry out further fusion processing on the cross-modal attention map, and feature representation based on final fusion spatio-temporal data and clinical data is obtained; and finally, outputting a benign and malignant classification result through an FC layer. Firstly, features are extracted from CT image data of follow-up visit, space-time fusion is carried out, then the features are combined with text clinical data, and the classification accuracy and robustness are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image morphological operations, and in particular to a method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network. Background Art

[0002] The National Lung Screening Trial (NLST) study showed that regular imaging examinations and lung function assessments can help track changes in nodules and predict their risk of malignancy, thereby increasing the possibility of early detection and significantly improving survival rates. However, existing evaluation methods mainly rely on a single CT scan, which limits in-depth understanding of nodule progression and may lead to late diagnosis. Regular imaging follow-up can help dynamically collect nodule change data, thereby improving the accuracy of malignant tumor prediction and prevention and improving patient health prognosis.

[0003] However, many early works usually rely on single-time-point imaging data for classification and prediction, but this method cannot fully capture the dynamic information of nodules changing over time. In contrast, using follow-up data for multi-time-point analysis can better reflect the progression of lesions and improve the accuracy and reliability of predictions. Despite this, existing multi-time-point studies have failed to fully integrate clinical data and failed to achieve multimodal fusion, limiting the model's potential in integrating clinical information. Secondly, when combining multimodal and multi-time-point data, many studies use a simple splicing method of the time dimension, failing to fully utilize the rich features of the time dimension, limiting the model's in-depth understanding of temporal changes. Summary of the invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network. First, features are extracted from the follow-up CT image data, and spatiotemporal fusion is performed. Then, the features are combined with text clinical data to effectively improve the accuracy and robustness of classification.

[0005] In order to solve the above technical problems, the technical solution of the present invention is:

[0006] A method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network comprises the following steps:

[0007] Step 1, acquiring CT image data and performing preprocessing to obtain ROI image;

[0008] Step 2: Extract spatial features in the ROI image through a spatial feature extraction module, and obtain text features of the corresponding clinical data through word embedding;

[0009] Step 3: Through the time residual fusion module, the spatial features from different time periods t0 and t1 are input, and the input feature data is fused through a multi-level feature fusion strategy; the fused features are restored to resolution through upsampling and integrated through global average pooling; after global average pooling integration, the multi-scale features are refined through the convolution layer; finally, weighted fusion is performed;

[0010] Step 4: Input the spatiotemporal features of the CT image and the text features after spatiotemporal fusion by the temporal residual fusion module into a cross-modal attention fusion network, perform preliminary fusion and obtain the final cross-modal attention map;

[0011] Step 5: Apply a multi-head self-attention fusion block to further fuse the cross-modal attention map to obtain the feature representation based on the final fused spatiotemporal data and clinical data;

[0012] Step 6: Finally, based on the feature representation of the final fused spatiotemporal data and clinical data, a FC layer is used to output the benign or malignant classification result.

[0013] Preferably, in step 1, the preprocessing method for the CT image is:

[0014] The HU values ​​of all CT images were adjusted to between [-1200, 600], and then the CT images were cropped into 16*64*64 ROI images according to the nodule location information.

[0015] Preferably, in step 3, when weighted fusion is performed, a learning fusion mechanism is used to drive

[0016] The feature weights α0 and α1 from different time points are adjusted dynamically, and the expression is as follows:

[0017] STF feat =σ(α0)*F feat +σ(α1)*t1.

[0018] Among them, F feat It is the fusion feature of the spatial features of the two time periods t0 and t1.

[0019] Preferably, the spatial feature extraction module combines 3DResnet and CBAM modules.

[0020] Specifically, the CBAM module is introduced into the residual block of 3D ResNet, and the convolution output is optimized through its channel attention and spatial attention mechanisms and then added with the jump connection. It not only retains the powerful spatial feature extraction capability of 3D ResNet, but also enhances the focus on key feature areas, thus significantly improving the expressiveness, robustness and classification accuracy of the model.

[0021] Preferably, in step 4, the fusion method of the cross-modal attention fusion network is:

[0022] First, the image feature x is transformed into i and text features y i They are converted to x∈R^(B*D) and y∈R^(B*D), respectively, where B refers to batchsize and D refers to the dimension of image and text after dimensionality reduction. The input is reduced in dimension through a fully connected layer, and the reduced features are input into a 1×1 convolutional layer to be converted into three feature spaces. The corresponding weight matrix is ​​generated at the same time. Then, the matching degree between the features is calculated. The expression is as follows:

[0023]

[0024] Among them, S = W*H represents the spatial domain size of the follow-up image, β and ρ represent the matching scores of the image and clinical text in the spatial domain, q1, q2, k1 and k2 are the image features x i , the text feature is y i , after the fully connected layer, the matrix required for the subsequent attention mechanism is generated; s i,j Refers to the matching degree between different features of the image and the text, t i,j Refers to the degree of match between the various features of the text and the image.

[0025] Preferably, the score β j，i and ρ j，i Used to weight the corresponding feature values ​​and generate the final cross-modal attention map.

[0026] Preferably, steps 2 to 6 constitute a CSF-Net model, and the CSF-Net model is implemented on a device equipped with a GeForceRTX 3090Ti GPU using the torch-2.1.0-cu12.1-cudnn8.9 framework.

[0027] Preferably, when training the CSF-Net model, a cross entropy loss function and an Adam optimizer are used for optimization, an initial learning rate is set to 0.0001, and a total of 200 training rounds are performed.

[0028] The present invention has the following characteristics and beneficial effects:

[0029] The above technical solution is used to combine follow-up data at multiple time points with clinical data to enhance the ability to predict future malignant transformation of pulmonary nodules. A time residual fusion module is designed to effectively combine features at different time points. Cross-modal attention fusion is used to learn spatiotemporal features and clinical features, thereby improving the accuracy and robustness of the model.

[0030] By integrating spatiotemporal features with clinical text information, an efficient cross-modal fusion framework was constructed, which can effectively deal with information loss and improve the comprehensiveness and accuracy of feature extraction. The experimental results fully verified the superiority of the proposed method and proved its application potential and clinical value in the prediction of benign and malignant pulmonary nodules.

[0031] This paper proposes an innovative lung nodule prediction network architecture that organically integrates follow-up imaging data and clinical data information into the training process. By introducing multimodal data, the prediction performance of the model is significantly improved, while a deep understanding of the dynamic changes of lung nodules is demonstrated in the fusion of information at multiple time points. This integration of information not only optimizes the accuracy of diagnosis, but also provides richer insights into disease development trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 Schematic diagram of the network model structure of an embodiment of the present invention.

[0034] Figure 2 for Figure 1 Schematic diagram of the structure of the temporal residual fusion module. DETAILED DESCRIPTION

[0035] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0036] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0037] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0038] The meanings of the special names and English abbreviations involved in the embodiments of the present invention are as follows:

[0039]

[0040] The present invention provides a method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network. To implement the above method in this embodiment, a CSF-Net module is first constructed and trained.

[0041] like Figure 1 As shown in the figure, the CSF-Net module includes a spatial feature extraction module, a temporal residual fusion module, a cross-modal attention fusion network, a multi-head self-attention fusion module and a FC layer.

[0042] Specifically, the spatial feature extraction module combines the 3DResNet and CBAM modules, effectively improving the ability to extract spatial information and enhancing the sensitivity to detail features. On this basis, the temporal residual fusion module captures dynamic changes in the time dimension by fusing features from different time points, thereby better understanding the correlation between temporal features. The cross-modal attention fusion module intelligently integrates the spatiotemporal features of follow-up CT images with clinical text information through a cross-modal attention mechanism, enabling the model to fully explore and fuse the potential correlations between different data sources, thereby improving the collaborative expression capabilities of multimodal data.

[0043] In this embodiment, for the above-mentioned CSF-Net module, on a device equipped with a GeForce RTX 3090Ti GPU, the CSF-Net model is implemented using the torch-2.1.0-cu12.1-cudnn8.9 framework. In the pre-training stage, the model is optimized using the cross entropy loss function and the Adam optimizer, the initial learning rate is set to 0.0001, and the total number of training rounds is 200 rounds. In order to improve the convergence speed and stability, the momentum parameters β1 and β2 of the Adam optimizer are set to 0.5 and 0.999, respectively, and the model parameters are updated every 20 rounds. This optimization strategy effectively responds to the challenges brought by complex data features. In terms of model input, CSF-Net generates a 3D effect by sequentially inputting three sets of stacked 3D slices. During training, the batch size is initially set to 4 to efficiently utilize computing resources.

[0044] Based on the above CSF-Net module, the specific steps are as follows:

[0045] Step 1: Acquire CT image data and perform preprocessing to obtain ROI image.

[0046] Specifically, the CT image data is preprocessed, the HU values ​​of all CT images are adjusted to between [-1200, 600], and then the CT images are cropped into 16*64*64 3D region of interest (ROI) images according to the nodule location information given by the doctor.

[0047] In this embodiment, the dataset used is the US NLST dataset, which is a key research program initiated by the US National Cancer Institute to reduce lung cancer mortality in high-risk populations through early imaging screening. The dataset contains longitudinal lung CT scans and related clinical information of participants, who are screened once a year for three consecutive years or until cancer is diagnosed.

[0048] Due to the lack of detailed annotation of the location of lung nodules in the original data, 443 subjects were screened out to create a cross-modal spatiotemporal CT dataset named NLST-cmst. Each participant in this dataset underwent at least two CT scans, and the malignancy of lung nodules was determined by the pathological gold standard, especially based on the diagnosis of lung cancer during follow-up. This method ensures the scientific rigor and reliability of the data. Our rigorous screening process provides valuable clinical insights into the malignant progression of lung nodules, which helps to improve early detection and treatment strategies.

[0049] Step 2: In the feature extraction of ROI images, 3DResnet and CBAM modules are used to capture the spatial features of lung nodules in ROI. For clinical data, word embedding is used to obtain feature representation of text.

[0050] Step 3: In the temporal residual fusion module, spatial features from different time periods t0 and t1 are first input, and richer contextual information and complex nonlinear relationships are extracted through fusion operations. Then, a series of upsampling, pooling, and convolution operations are used to gradually refine the feature representation. Finally, through weighted fusion, the model's ability to capture temporal correlation and integrate cross-temporal information is further enhanced, thereby improving the prediction accuracy of time series data.

[0051] Specifically, Figure 2 As shown in Figure 1, the module inputs features t0 and t1 from different time periods and adopts a multi-level feature fusion strategy to explore the complementarity of multi-dimensional features. In the high-dimensional feature space, the model captures richer contextual information and complex nonlinear relationships through deep splicing operations. The processed features are then restored to resolution through upsampling and integrated through global average pooling. Next, the convolutional layer refines the multi-scale features to enhance the model's ability to represent complex data patterns, as shown below:

[0052] F feat =Conv3D(AvgPool(Upsample(t0+t1)))

[0053] Finally, a learnable fusion mechanism dynamically adjusts the feature weights α0 and α1 from different time points. The mechanism is expressed as:

[0054] STF fest =σ(α0)*F feat +σ(α1)*t1

[0055] This mechanism ensures the deep fusion of t1 and the processed temporal features, enhancing the ability to capture temporal correlations and integrate information across time, thereby improving the overall prediction performance.

[0056] Step 4: Input the spatiotemporal features of the CT image and the text features after spatiotemporal fusion through the temporal residual fusion module into a cross-modal attention fusion network for preliminary fusion to obtain the final cross-modal attention map.

[0057] Specifically, the input of the cross-modal attention fusion module includes the spatiotemporal features x from the follow-up images i and text features y i First, the input is reduced in dimension through the fully connected layer to simplify the feature space. Then, the reduced features are input into the 1×1 convolutional layer respectively, converted into three feature spaces, and the corresponding weight matrix is ​​generated at the same time. Then, the matching degree between the features is calculated to accurately capture the correlation between cross-modal information and enhance the multimodal fusion effect:

[0058]

[0059] Among them, S = W*H represents the spatial domain size of the follow-up image, and β and ρ represent the matching scores of the image and clinical text in the spatial domain. These scores β j，i and ρ j，i Used to weight the corresponding feature values ​​and generate the final cross-modal attention map.

[0060] Step 5: Apply the multi-head self-attention fusion module to perform pairwise addition of the cross-modal attention maps for further fusion processing to obtain the feature representation based on the final fused spatiotemporal data and clinical data;

[0061] Step 6: Finally, based on the feature representation of the final fused spatiotemporal data and clinical data, a FC layer is used to output the benign or malignant classification result.

[0062] For the classification method provided in this embodiment, the following comparative tests were used to evaluate the model performance: (1) pulmonary nodule classification model: prediction was performed using only CT images; (2) multimodal model: prediction was performed using CT images and clinical information; (3) multimodal spatiotemporal network model: prediction was performed using CT images and clinical information through a spatiotemporal network. Their results were compared with the results obtained using the method of this embodiment.

[0063] Specifically, the proposed method was comprehensively compared with six existing models: including SCANs, which is suitable for the diagnosis of benign and malignant lung nodules at a single time point or multiple time points; NAS-Lung, which improves the model's interpretability and classification capabilities based on neural structure and attention mechanism; T-LSTM, which introduces time interval sensitivity to improve prediction accuracy; RadFusion, which combines imaging data and electronic health records to optimize diagnostic results; DeepCAD, which uses a multimodal approach to perform time series prediction of benign and malignant lung nodules; and MCFN, which integrates online data in multiple modules to enhance subtype classification capabilities.

[0064] To ensure a fair comparison, we use five key indicators to evaluate the above six models and the proposed method: accuracy (Acc), precision (Prec), area under the curve (AUC), F1 value (F1) and recall (Rec). These indicators provide a comprehensive evaluation of the prediction accuracy, precision and overall performance of each model.

[0065] Acc Prec F1 AUC Rec SCANs 0.7865 0.7667 0.7077 0.7725 0.6571 NAS-Lung 0.8539 0.8235 0.8116 0.8910 0.8000 T-LSTM 0.7645 0.7012 0.6527 0.7778 0.6000 DeepCAD 0.8590 0.7879 0.8254 0.8990 0.8667 MFCN 0.7949 0.7059 0.7500 0.8903 0.8000 RadFusion 0.7753 0.8026 0.6667 0.7693 0.6000 This embodiment 0.8974 0.8235 0.8750 0.9389 0.9333

[0066] Table 1: Experimental comparison results

[0067] It can be seen from Table 1 that the method of this embodiment performs well in all evaluation indicators, and is significantly better than other models except Prec. In terms of Acc, AUC, F1 and Rec, the method of this embodiment achieves at least 0.0384, 0.0496, 0.0399 and 0.0666 improvements over other models, respectively.

[0068] Although Prec performs comparable to NAS-Lung and T-LSTM, and DeepCAD also demonstrates its unique advantages in multi-modality and multi-time point fusion, the overall results show that the method of this embodiment is significantly better than the model that only relies on a single modality or a single time point in multi-modality and multi-time point fusion strategy. These experimental results further verify the effectiveness and clinical applicability of our model in predicting benign and malignant pulmonary nodules.

[0069] Combined with the technical solutions and simulation results provided in this embodiment, it can be seen that the present invention proposes an innovative lung nodule prediction network architecture, which organically integrates follow-up imaging data and clinical data information into the training process. By introducing multimodal data, the predictive performance of the model is significantly improved, and at the same time, a deep understanding of the dynamic changes of lung nodules is demonstrated in the fusion of information at multiple time points. The integration of this information not only optimizes the accuracy of diagnosis, but also provides richer insights into disease development trends. However, the current model has not yet achieved a complete end-to-end training process, so in future work, we will focus on optimizing the end-to-end architecture to improve the applicability of the model and further meet the needs of practical clinical applications.

[0070] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network, characterized in that: The steps include: Step 1, acquiring CT image data and performing preprocessing to obtain ROI image; Step 2: Extract spatial features in the ROI image through a spatial feature extraction module, and obtain text features of the corresponding clinical data through word embedding; Step 3: Through the time residual fusion module, the spatial features from different time periods t0 and t1 are input, and the input feature data is fused through a multi-level feature fusion strategy; the fused features are restored to resolution through upsampling and integrated through global average pooling; after global average pooling integration, the multi-scale features are refined through the convolution layer; finally, weighted fusion is performed to obtain the spatiotemporal features of the CT image; Step 4: Input the spatiotemporal features of the CT image and the text features after spatiotemporal fusion by the temporal residual fusion module into a cross-modal attention fusion network, perform preliminary fusion and obtain the final cross-modal attention map; Step 5: Apply a multi-head self-attention fusion module to further fuse the cross-modal attention map to obtain the feature representation based on the final fused spatiotemporal data and clinical data; Step 6: Finally, based on the feature representation of the final fused spatiotemporal data and clinical data, a FC layer is used to output the benign or malignant classification result.

2. According to claim 1, a method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network is characterized in that: In step 1, the preprocessing method for the CT image is: The HU values ​​of all CT images were adjusted to between [-1200, 600], and then the CT images were cropped into 16*64*64 ROI images according to the nodule location information.

3. According to claim 1, a method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network is characterized in that: In step 3, during weighted fusion, a learning fusion mechanism is used to dynamically adjust the feature weights α0 and α1 from different time points. The expression is as follows: STF feat =σ(α0)*F feat +σ(α1)*t1 Among them, F feat It is the fusion feature of the spatial features of the two time periods t0 and t1.

4. According to claim 1, a method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network is characterized in that: The spatial feature extraction module combines the 3D Resnet and CBAM modules.

5. According to claim 4, a method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network is characterized in that: The spatial feature extraction module introduces the CBAM module into the residual block of 3D ResNet, optimizes the convolution output through its channel attention and spatial attention mechanisms, and then adds it with the jump connection.

6. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 1, characterized in that: In step 4, the fusion method of the cross-modal attention fusion network is: First, the image feature x is transformed into i and text features y i They are converted to x∈R^(B*D) and y∈R^(B*D) respectively, where B refers to batchsize and D refers to the dimension of image and text after dimensionality reduction. The input is reduced in dimension through a fully connected layer, and the reduced features are input into a 1×1 convolutional layer respectively, thereby converting them into three feature spaces and generating corresponding weight matrices. Subsequently, the matching degree between the features is calculated, and the expression is as follows: Where s = W*H represents the spatial domain size of the follow-up image, β and ρ represent the matching scores of the image and clinical text in the spatial domain, q1, q2, k1 and k2 are the image features x i , the text feature is y i , after the fully connected layer, the matrix required for the subsequent attention mechanism is generated; s i,j Refers to the matching degree between different features of the image and the text, t i,j Refers to the degree of match between the various features of the text and the image.

7. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 6, characterized in that: The score β j，i and ρ j，i Used to weight the corresponding feature values ​​and generate the final cross-modal attention map.

8. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 7, characterized in that: Steps 2 to 6 constitute the CSF-Net model, which is implemented on a device equipped with a GeForce RTX 3090Ti GPU using the torch-2.1.0-cu12.1-cudnn8.9 framework.

9. The method for classifying benign and malignant pulmonary nodules based on a cross-modal spatiotemporal fusion network according to claim 8, characterized in that: When the CSF-Net model is trained, the cross entropy loss function and the Adam optimizer are used for optimization, the initial learning rate is set to 0.0001, and the total number of training rounds is 200.

Images