Multi-modal space-time network model for pulmonary nodule segmentation and segmentation method thereof

Abstract

The invention discloses a multi-modal space-time network model for pulmonary nodule segmentation, and the model comprises a multi-modal feature extraction module which is used for processing CT image data at multiple time points and corresponding clinical text data to obtain high-dimensional image features and text feature vectors; the Vi T STUnet backbone network comprises an encoder, a space-time dependence module and a decoder, wherein the encoder is composed of a plurality of layers of Transform blocks; the cross-modal space-time fusion module is used for fusing image feature differences of multiple time points and clinical text features; and the classification module outputs benign and malignant classification results of the pulmonary nodules based on the fusion features. The invention provides an end-to-end multi-mode and multi-task space-time classification network (MST-Net). According to the network, multi-time-point CT images and clinical data are closely fused through a deep coupling mechanism, and the long-term evolution trend of nodules is captured.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image morphology, and in particular to a multimodal spatiotemporal network model for pulmonary nodule segmentation and a segmentation method thereof. Background Art

[0002] Lung cancer originates from lung cells. It is a rapidly progressing malignant tumor that poses a serious threat to global public health security. According to statistics from the World Health Organization and the American Cancer Society, there will be approximately 2.5 million newly diagnosed cases of lung cancer in 2022 worldwide, accounting for 12.4% of all newly diagnosed cancers, and approximately 1.8 million lung cancer-related deaths, accounting for 18.7% of all cancer deaths. As a common precursor lesion of lung cancer, pulmonary nodules are at risk of transforming into malignant tumors if they are not monitored in a timely and continuous manner. Studies of the National Lung Screening Trial (NLST) in the United States have shown that regular imaging examinations combined with pulmonary function assessments can help track changes in nodules, thereby improving early diagnosis rates and survival rates. However, existing clinical evaluation methods mainly rely on a single CT scan, which is difficult to fully reveal the full picture of the dynamic evolution of pulmonary nodules and poses potential risks for late diagnosis.

[0003] Most of the early related studies were based on classification and prediction based on imaging data at a single time point, which was unable to capture the dynamic changes of nodules over time. To make up for this shortcoming, some methods have tried to introduce multi-time point follow-up data and time information, such as using the Leaky Noisy OR network in a 3D deep learning framework. However, its linear structure makes it difficult to fully characterize the nonlinear spatiotemporal evolution of lesions, resulting in inaccurate detection of subtle changes. On the other hand, although some studies have attempted to integrate imaging, clinical indicators and radiological features to achieve multimodal fusion, due to the lack of precise lesion segmentation guidance or the use of simple time dimension splicing methods, they have failed to fully explore the potential information in multiple spatiotemporal features, limiting the performance of the model in the comprehensive utilization of global and local information. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology, propose a multimodal spatiotemporal network model and its segmentation method for pulmonary nodule segmentation, and propose an end-to-end multimodal multi-task spatiotemporal classification network (MST-Net). The network tightly integrates multi-time point CT images with clinical data through a deep coupling mechanism to capture the long-term evolution trend of nodules; introduces a lightweight spatiotemporal dependency module into the segmentation network to optimize the extraction of local features of small and fuzzy nodules; at the same time, designs a segmentation-guided classification network to achieve adaptive integration of local details and global context; in addition, constructs a cross-modal spatiotemporal fusion module to effectively capture the nonlinear evolution characteristics of nodules and cope with the dynamic changes of cross-modal and multi-time scale data distribution. The above technical solution significantly improves the classification accuracy of the risk of malignant transformation of pulmonary nodules, and provides strong technical support for early intervention and precision treatment.

[0005] In order to achieve the above object, the technical solutions specifically adopted by the present invention are as follows:

[0006] A multimodal spatiotemporal network model for pulmonary nodule segmentation, including:

[0007] (a) A multimodal feature extraction module, which processes CT image data at multiple time points and corresponding clinical text data to obtain high-dimensional image features and text feature vectors;

[0008] (b) The ViT STUnet backbone network, including an encoder consisting of multiple layers of Transformer blocks, a spatiotemporal dependency module, and a decoder, where:

[0009] The encoder receives the high-dimensional image features and extracts multi-scale features;

[0010] The spatiotemporal dependency module models the spatiotemporal dependency of features at multiple time points through local spatial convolution and global temporal self-attention mechanism to obtain spatiotemporal fusion features.

[0011] The decoder maps the spatiotemporal fusion features to the input image resolution and outputs the lung nodule segmentation result;

[0012] (c) Cross-modal spatiotemporal fusion module, used to fuse image feature differences at multiple time points with clinical text features, including:

[0013] Calculate the difference ΔZ of image features at different time points to generate dynamic evolution features;

[0014] Through the cross-modal attention mechanism, the dynamic evolution features are aligned with the text feature vector to generate fusion features;

[0015] (d) A classification module outputs a benign or malignant classification result of the lung nodules based on the fusion features.

[0016] Preferably, the multimodal feature extraction module performs 3D block processing on the CT image data to generate 3DPatches, extracts high-dimensional image features through the Patch Embedding module, and generates text feature vectors for the clinical text data through a word embedding method.

[0017] Preferably, the specific implementation of the spatiotemporal dependency module includes:

[0018] (a) Perform 3D convolution on the input features to extract local spatial features;

[0019] (b) Expanding the local spatial features into a time series along the depth axis and modeling the long-term dependencies across slices through a multi-head self-attention mechanism;

[0020] (c) Combine local spatial features with global temporal features through residual connections and output spatiotemporal fusion features.

[0021] Preferably, the cross-modal attention mechanism in the cross-modal spatiotemporal fusion module is specifically:

[0022] (a) Using text feature vector as query Q and dynamically evolving features as key K and value V;

[0023] (b) Calculate the fused features through the attention weights normalized by Softmax.

[0024] Preferably, the preprocessing step of the multimodal feature extraction module includes:

[0025] (a) The HU value of CT image data is normalized to the range of [-1200, 600];

[0026] (b) Cropping the CT image into a 16×512×512 3D subimage.

[0027] Preferably, the decoder of the ViT STUnet backbone network adopts a multi-stage upsampling structure and retains multi-scale detail information through a skip connection with the encoder.

[0028] Preferably, the classification module is trained using a cross entropy loss function and combined with the Dice loss of the segmentation task to perform multi-task joint optimization.

[0029] The present invention also provides a lung nodule segmentation method, comprising the following steps:

[0030] (a) Acquire CT image data and corresponding clinical text data at multiple time points;

[0031] (b) The acquired data is preprocessed, the HU value of the CT image data is normalized to the range of [-1200, 600], and the CT image is cropped into a 16×512×512 3D sub-image;

[0032] (c) Processing the data using the multimodal spatiotemporal network model described in any one of claims 1 to 6 to generate lung nodule segmentation results and benign and malignant classification results.

[0033] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the pulmonary nodule segmentation method when executing the computer program.

[0034] The present invention also provides a computer-readable storage medium storing a computer program, which implements the pulmonary nodule segmentation method when executed by a processor.

[0035] The present invention has the following characteristics and beneficial effects:

[0036] Through a deep coupling mechanism, multi-time point CT images are tightly integrated with clinical data to capture the long-term evolution trend of nodules; a lightweight spatiotemporal dependency module is introduced into the segmentation network to optimize the extraction of local features of small and fuzzy nodules; at the same time, a segmentation-guided classification network is designed to achieve adaptive integration of local details and global context; in addition, a cross-modal spatiotemporal fusion module is constructed to effectively capture the nonlinear evolution characteristics of nodules and cope with the dynamic changes in cross-modal and multi-time scale data distribution. The above technical solutions significantly improve the classification accuracy of the risk of malignant transformation of lung nodules, providing strong technical support for early intervention and precision treatment.

[0037] This method first detects the location of pulmonary nodules, extracts multi-temporal features, and integrates clinical information to construct an efficient cross-modal feature fusion framework. This effectively overcomes the problem of missing information, improves the comprehensiveness and accuracy of feature extraction, and better aligns with the actual clinical diagnostic process. Experimental results fully validate the superiority of this method, demonstrating its broad application prospects and significant clinical value in predicting benign and malignant pulmonary nodules. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a principle block diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0040] Example 1

[0041] This embodiment provides a multimodal spatiotemporal network model for pulmonary nodule segmentation. Figure 1 As shown, the following steps are included:

[0042] (a) A multimodal feature extraction module, which processes CT image data at multiple time points and corresponding clinical text data to obtain high-dimensional image features and text feature vectors;

[0043] The multimodal feature extraction module performs 3D block processing on the CT image data to generate 3DPatches, extracts high-dimensional image features through the Patch Embedding module, and generates text feature vectors for the clinical text data through the word embedding method;

[0044] (b) The ViT STUnet backbone network includes an encoder composed of multi-layer Transformer blocks, a spatiotemporal dependency module, and a decoder. The decoder of the ViT STUnet backbone network adopts a multi-level upsampling structure and retains multi-scale detail information through skip connections with the encoder.

[0045] in:

[0046] The encoder receives the high-dimensional image features and extracts multi-scale features;

[0047] The spatiotemporal dependency module models the spatiotemporal dependency of features at multiple time points through local spatial convolution and global temporal self-attention mechanism to obtain spatiotemporal fusion features.

[0048] Specifically, the specific implementation of the spatiotemporal dependency module includes:

[0049] First, perform 3D convolution operation on the input features to extract local spatial features;

[0050] Then, the local spatial features are expanded into a time series along the depth axis, and the long-term dependencies across slices are modeled through a multi-head self-attention mechanism;

[0051] Finally, the local spatial features are combined with the global temporal features through residual connections to output spatiotemporal fusion features.

[0052] The decoder maps the spatiotemporal fusion features to the input image resolution and outputs the lung nodule segmentation result;

[0053] (c) Cross-modal spatiotemporal fusion module, used to fuse image feature differences at multiple time points with clinical text features, including:

[0054] Calculate the difference ΔZ of image features at different time points to generate dynamic evolution features;

[0055] Through the cross-modal attention mechanism, the dynamic evolution features are aligned with the text feature vector to generate fusion features;

[0056] The cross-modal attention mechanism in the cross-modal spatiotemporal fusion module is specifically as follows: the text feature vector is used as the query Q, the dynamic evolution feature is used as the key K and the value V; and the fusion feature is calculated through the Softmax normalized attention weight.

[0057] (d) A classification module outputs a benign or malignant classification result of the lung nodules based on the fusion features.

[0058] The classification module is trained using a cross entropy loss function and combined with the Dice loss of the segmentation task for multi-task joint optimization.

[0059] Specifically, the model architecture consists of four main modules: multimodal feature extraction, the ViT STUnet backbone, a cross-modal spatiotemporal prediction module, and a classification module. These components work together through cross-modal attention, spatiotemporal feature fusion, and a multi-scale decoding mechanism to ensure high segmentation accuracy and significantly enhance the prediction of lesion malignancy and other clinical indicators.

[0060] Furthermore, the multimodal feature extraction module uses different methods to process different modal data. For image features, the 3D Patches blocking strategy is adopted in this embodiment to enhance the model's sensitivity to lung nodules, thereby more accurately detecting lesions. For clinical features, the feature representation of the text is obtained through the word embedding method. The ViT STUnet backbone is composed of a temporal dependency module (STDM) and an encoder composed of 12 layers of Transformer blocks. This module enables the model to focus on learning local spatial patterns by enhancing the ability to capture spatial details of a single slice, and to model long-term dependencies across slices, thereby achieving effective fusion of multi-scale features and spatiotemporal information. The cross-modal spatiotemporal prediction module receives features at different time points and feature representations of clinical texts, and obtains the change results by subtracting features at different time points, and finally inputs them into the CMAF module for cross-modal fusion.

[0061] ViT STUnet is shown in part b of the figure. This part consists of a transformer block, a spatiotemporal dependency module (STDM), and a decoder. The transformer block receives the image features after patching and outputs multi-scale features of different layers. The features of different scales are then input into the STDM module. Let the features of different scales be X, W 3d Refers to 3D convolution weights with a kernel size of (1×3×3) and applies zero padding to maintain the spatial dimension, the output is recorded as Z spa :

[0062] Z spa =GELU(BN(Conv3D(X;W 3d )))

[0063] By decomposing the spatiotemporal dimensions, the model focuses on learning local spatial patterns. The spatial features are then unfolded into a time series along the depth axis, treating each depth slice d∈D at each spatial position (h, w) as an independent time sample, forming the following spatiotemporal joint representation:

[0064] Z seq =Reshape(Z spa )∈R (B*H*W)*D*C

[0065] A multi-head self-attention mechanism is adopted along the depth to model the long-term dependencies across slices. The query matrix Q, key matrix K and value matrix V are obtained by Z seq The output is derived from the linear projection of temp :

[0066] Z temp =MSA(Q,K,V)=Concat(head1,…,head h )W 0

[0067] head i =Attention(QW i Q ,KW i K ,VW i V )

[0068] To prevent the degradation of deep networks, cross-branch residual connections are used to innovatively combine lightweight local spatial feature extraction with global temporal dependency learning:

[0069] Z out =LayerNorm(Z spa +δ*Z temp )

[0070] After aggregating multi-scale features and spatiotemporal information, the decoder module maps the fused high-dimensional features back to a spatial resolution similar to the input image to generate segmentation results. Like the traditional U-Net architecture, the decoder uses multi-level upsampling or deconvolution while preserving details through corresponding connections with the encoder.

[0071] In the Cross-Modal Spatio-Temporal Fusion (CMSTF) module, this embodiment aims to utilize imaging and clinical data at multiple time points to capture the dynamic evolution of lesions. This approach improves the accuracy of classification and progression prediction. Initially, the module processes visual features from time points t0 and t1, represented as and And calculate the difference between them as follows:

[0072]

[0073] This method not only preserves the original information at each time point, but also explicitly captures the changing trend through ΔZ, providing the necessary dynamic context for the subsequent fusion process. and ΔZ features to form a more comprehensive joint visual representation Z fuse At the same time, the text or clinical prior information is processed to generate the text feature vector Z text In order to achieve cross-modal fusion, this module uses the cross-modal attention mechanism to text and Z fuse In this process, the query Q comes from Z text , and the key K and value V are both derived from Z fuse The attention mechanism is calculated as follows:

[0074]

[0075] The final fused output feature Z cross It fuses the visual information from t0 and t1, their dynamic changes, and text-based prior information. This comprehensive integration provides robust spatiotemporal support for subsequent classification or progress prediction tasks.

[0076] Example 2

[0077] This embodiment provides a lung nodule segmentation method, comprising the following steps:

[0078] (a) Acquire CT image data and corresponding clinical text data at multiple time points;

[0079] The dataset used in this example is the US National Lung Cancer Study (NLST), a key research program sponsored by the US National Cancer Institute that aims to reduce lung cancer mortality in high-risk individuals through early imaging screening. The dataset contains longitudinal lung CT scans and associated clinical information from participants who underwent annual screening for three consecutive years or until a cancer diagnosis was made.

[0080] Because the original data lacked detailed annotation of pulmonary nodule locations, 150 subjects were selected to create a cross-modal spatiotemporal CT dataset. Each participant in this dataset underwent at least two CT scans, and the malignancy of pulmonary nodules was determined using gold-standard pathology, specifically based on lung cancer diagnoses during follow-up. This approach ensured the scientific rigor and reliability of the data. Our rigorous selection process provides valuable clinical insights into the malignant progression of pulmonary nodules, helping to improve early detection and treatment strategies.

[0081] (b) The acquired data was preprocessed, the HU value of the CT image data was normalized to the range of [-1200, 600], and the CT image was cropped into a 16×512×512 3D sub-image; in order to minimize memory usage, the data was preprocessed into a 16×512×512 format, and nodule coordinates were not provided in the experiment.

[0082] (c) constructing and training a multimodal spatiotemporal network model for pulmonary nodule segmentation provided in Example 1;

[0083] For the segmentation task, the model first detects the nodule location and then performs accurate segmentation, achieving simultaneous detection and segmentation. The framework runs on a GeForce RTX A6000 (48GB) using torch-1.8.1-cu11.1-cudn8, and the performance is evaluated by five-fold cross-validation. During the pre-training phase, the dataset is divided into training and test sets in a 4:1 ratio. We use Dice loss for segmentation and cross entropy loss for classification, and optimize with the Adam optimizer (initial learning rate 0.0001) within 90 cycles. In addition, MST-Net adopts a 2.5D strategy to process three consecutive slices in the 3D volume to generate a segmentation of the central slice. Experiments have verified the efficiency and accuracy of this method in lung nodule analysis.

[0084] In this example, the following comparative tests were used to evaluate model performance: (1) a lung nodule classification model: predictions were made using only CT images; (2) a multimodal model: predictions were made using CT images and clinical information; (3) a multimodal spatiotemporal network model: predictions were made using CT images and clinical information via a spatiotemporal network. (4) a multimodal segmentation-assisted classification spatiotemporal network model: CT images and clinical information were used to locate lung nodules via a segmentation network, and then input into the spatiotemporal network for prediction. Their results were compared with those obtained using our method.

[0085] (d) Applying a pre-trained multimodal spatiotemporal network model for lung nodule segmentation to process the data, and generating lung nodule segmentation results and benign and malignant classification results.

[0086] Finally, to demonstrate the effectiveness of this example, we compared it with six leading methods, including three classification methods (Naslung, Scan, and DeepCAD), one segmentation method (UnetR), and two models that combine segmentation and classification (MLTN and ACSNet). We evaluated the experiments using seven metrics: Dice, Jaccard, HD95, and ASD for segmentation tasks, and accuracy, precision, and F1 score for classification tasks (see Table 1 for details).

[0087] Results show that MST-Net performs well on both segmentation and classification tasks. For segmentation, the model achieved 46.09% and 27.39% improvements in Dice coefficient compared to MLTN and ACSNet, respectively, and slightly outperformed the segmentation-specific UnetR method, demonstrating its ability to accurately define lesion boundaries. For classification, MST-Net significantly surpassed all compared methods, with improvements in accuracy, precision, and F1 score of at least 2.93%, 2.24%, and 1.87%, respectively. In summary, the MST-Net model, jointly trained with multimodal data and spatiotemporal information, demonstrated superior performance in lung nodule segmentation and prognosis tasks compared to models trained using only a single modality, a single time point, or a single plane of data.

[0088] As can be seen, the present invention proposes an end-to-end lung nodule segmentation and diagnosis network architecture that organically integrates follow-up data and clinical data into the training process. This network simulates the doctor's diagnosis and treatment process, first using a segmentation network to locate lung nodules, and then inputting features extracted from different time periods into a multimodal spatiotemporal fusion module. By introducing multimodal data, this method not only significantly improves prediction performance but also deeply captures the dynamic changes of lung nodules, optimizing diagnostic accuracy while providing rich information on disease development trends.

[0089] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal spatiotemporal network model for pulmonary nodule segmentation, characterized by: include: (a) A multimodal feature extraction module, which processes CT image data at multiple time points and corresponding clinical text data to obtain high-dimensional image features and text feature vectors; (b) The ViT STUnet backbone network, including an encoder consisting of multiple layers of Transformer blocks, a spatiotemporal dependency module, and a decoder, where: The encoder receives the high-dimensional image features and extracts multi-scale features; The spatiotemporal dependency module models the spatiotemporal dependency of features at multiple time points through local spatial convolution and global temporal self-attention mechanism to obtain spatiotemporal fusion features. The decoder maps the spatiotemporal fusion features to the input image resolution and outputs the lung nodule segmentation result; (c) Cross-modal spatiotemporal fusion module, used to fuse image feature differences at multiple time points with clinical text features, including: Calculate the difference ΔZ of image features at different time points to generate dynamic evolution features; Through the cross-modal attention mechanism, the dynamic evolution features are aligned with the text feature vector to generate fusion features; (d) A classification module outputs a benign or malignant classification result of the lung nodules based on the fusion features.

2. The multimodal spatiotemporal network model according to claim 1, characterized in that: The multimodal feature extraction module performs 3D block processing on the CT image data to generate 3D Patches, extracts high-dimensional image features through the Patch Embedding module, and generates text feature vectors for the clinical text data through a word embedding method.

3. The multimodal spatiotemporal network model according to claim 1, characterized in that: The specific implementation of the spatiotemporal dependency module includes: (a) Perform 3D convolution on the input features to extract local spatial features; (b) Expanding the local spatial features into a time series along the depth axis and modeling the long-term dependencies across slices through a multi-head self-attention mechanism; (c) Combine local spatial features with global temporal features through residual connections and output spatiotemporal fusion features.

4. The multimodal spatiotemporal network model according to claim 1, characterized in that: The cross-modal attention mechanism in the cross-modal spatiotemporal fusion module is specifically as follows: (a) Using text feature vector as query Q and dynamically evolving features as key K and value V; (b) Calculate the fused features through the attention weights normalized by Softmax.

5. The multimodal spatiotemporal network model according to claim 1, characterized in that: The preprocessing steps of the multimodal feature extraction module include: (a) The HU value of CT image data is normalized to the range of [-1200, 600]; (b) Cropping the CT image into a 16×512×512 3D subimage.

6. The multimodal spatiotemporal network model according to claim 1, characterized in that: The decoder of the ViT STUnet backbone network adopts a multi-level upsampling structure and retains multi-scale detail information through skip connections with the encoder.

7. The multimodal spatiotemporal network model according to claim 1, characterized in that: The classification module is trained using a cross entropy loss function and combined with the Dice loss of the segmentation task for multi-task joint optimization.

8. A lung nodule segmentation method, characterized in that: The following steps are involved: (a) Acquire CT image data and corresponding clinical text data at multiple time points; (b) The acquired data is preprocessed, the HU value of the CT image data is normalized to the range of [-1200, 600], and the CT image is cropped into a 16×512×512 3D sub-image; (c) Processing the data using the multimodal spatiotemporal network model described in any one of claims 1 to 7 to generate lung nodule segmentation results and benign and malignant classification results.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the pulmonary nodule segmentation method according to claim 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the pulmonary nodule segmentation method according to claim 8 is implemented.

Images