Lung cancer high-risk factor classification method based on multi-modal fusion

Abstract

The invention provides a lung cancer high-risk factor classification method based on multi-modal fusion, which is an early-stage lung cancer high-risk factor prediction method based on deep learning and multi-modal data, and belongs to the field of medical image processing (G06T) and medical diagnosis (A61B). The method is particularly related to early diagnosis of lung cancer by using an artificial intelligence technology. The objective of the invention is to analyze multi-modal medical data through a deep learning model to identify and predict early-stage lung cancer high-risk factors. Experiments prove that the multi-modal method conforms to the NCCN guide. Preoperative plans and auxiliary treatment decisions can be assisted, and finally personalized patient nursing is facilitated.

Description

Technical Field

[0001] The present invention is a method for predicting high-risk factors of early lung cancer based on deep learning and multimodal data, belonging to the fields of medical image processing (G06T) and medical diagnosis (A61B). Background Art

[0003] Lung cancer includes non-small cell lung cancer (NSCLC) and small cell lung cancer (SCLC), of which NSCLC accounts for about 85% and SCLC accounts for about 15%. SCLC patients have a very poor prognosis, and two-thirds of patients already have distant metastatic disease at the time of initial diagnosis. The survival rate of NSCLC patients has improved in recent years, which is related to recent advances in screening, minimally invasive techniques for diagnosis and treatment, radiotherapy (RT) including stereotactic ablative radiotherapy (SABR), and new targeted therapies and immunotherapy. Overall, most patients with advanced (stage IV) lung cancer die within 5 years of diagnosis, while the survival rate of patients with early (stage IA) lung cancer is more than 75% within 5 years. For patients, early detection, early diagnosis, and early treatment can bring better prognosis to patients with lung cancer. Therefore, research on early lung cancer is particularly important.

[0004] According to the NCCN Clinical Practice Guidelines for Oncology, radical surgical resection is the recommended preferred local treatment for patients with early lung cancer. However, some patients still face secondary surgery or postoperative adjuvant therapy due to high-risk factors (including poorly differentiated tumors [excluding well-differentiated neuroendocrine tumors], vascular invasion, visceral pleural invasion, unclear lymph node status, etc.). These high-risk factors are related to the aggressiveness and metastatic tendency of tumors. Therefore, using preoperative low-dose computed tomography (LDCT) imaging data to predict the patient's possible high-risk factors, guide thoracic surgeons to choose the scope of lung resection and surgical method, and whether preoperative (neoadjuvant) chemotherapy or radiotherapy is needed to shrink the tumor, which is of great significance for patients with early lung cancer.

[0005] In clinical treatment, the discovery of these high-risk factors for lung cancer is usually obtained through pathological observation and cannot be reflected in chest CT images or laboratory tests. Therefore, to date, there has been no major breakthrough in the diagnosis of high-risk factors for lung cancer using clinical data. However, thanks to the rapid development of artificial intelligence technology in recent years and its excellent performance in microscopic observation, it has become possible to discover those "invisible" factors through artificial intelligence. Summary of the invention

[0006] The purpose of this invention is to predict in advance whether lung cancer patients have high-risk factors based on deep learning and machine learning theories and methods, when this cannot be achieved clinically, so as to improve the patient's prognosis.

[0007] The present invention finally incorporated multimodal data (including chest CT, medical records and laboratory test data) of 1,113 patients collected from 2019 to 2022. A deep learning method based on fast Fourier transform and multi-head self-attention mechanism is proposed to predict high-risk factors for early lung cancer. This method is a multi-label classification method. Through comprehensive analysis of multimodal data, the patient's performance in various high-risk situations (whether there is a poorly differentiated tumor, whether there is vascular invasion, whether there is visceral pleural invasion, whether there is wedge resection) can be obtained.

[0008] The proposed method has excellent performance, with an AUC (area under the ROC curve) of 70%. We conducted sufficient comparative experiments and ablation experiments to prove the effectiveness of our method. Our method provides an intelligent multimodal solution for the prediction of high-risk factors for early lung cancer, which may serve as an intelligent decision support system for clinicians, and ultimately promote the formulation of rapid and accurate treatment strategies for early lung cancer.

[0009] The present invention comprises the following steps,

[0010] Step 1: Build training data:

[0011] 1. Through exclusion screening, a total of 1,113 patient data were collected, including 393 lung cancer patients with high-risk factors and 720 lung cancer patients without high-risk factors. The input data for each patient included 16 chest CT images, a set of medical records, and a set of laboratory test data. Exclusion criteria included: 1) patients with non-lung cancer stage I or II; 2) patients who did not undergo chest LDCT examination, laboratory examination, or pathological examination; 3) patients who lacked a final diagnosis of any of the four high-risk factors.

[0012] 2. In this study, patients with lung cancer with high-risk factors were defined as those with one or more of the following conditions: poorly differentiated tumors, vascular invasion, visceral pleural involvement, or wedge resection. Finally, 393 patients with lung cancer with high-risk factors and 720 patients with lung cancer without high-risk factors were included in the study. Among the 393 high-risk patients, 282 underwent wedge resection, 109 had poorly differentiated tumors, 75 had visceral pleural involvement, and 39 had vascular invasion. We collected demographic characteristics, medical history, and laboratory test results from the patients' medical records.

[0013] 3. Four radiologists reviewed all chest CT images to ensure image quality and clarity and mark the location of lesions. According to the radiologists' markings, 16 spatially continuous chest CT images (continuous CT plain scan images at 1.25 mm intervals) were selected for each patient. Since this study targets patients with early-stage lung cancer with relatively small lesion areas, for patients with less than 16 CT images containing lesion areas, we selected a total of 16 images centered on the lesion area as model input. Medical record data include gender, age, smoking history, drinking history, history of lung disease, history of malignant tumors, family history, size of lung nodules, and location of lung nodules. We processed the size and location of lung nodules to comprehensively characterize lung information. The final medical record data containing 11 variables were generated. Laboratory test data include six variables: white blood cells (WBC), granulocyte ratio (GR), carcinoembryonic antigen (CEA), cytokeratin 19 fragment (CY), neuron-specific enolase (NSE), and pro-gastrin releasing peptide (Pro-GRP). These data were included based on previous studies revealing its association with cancer.

[0014] 4. To ensure a balanced distribution among different categories, we randomly divided the dataset into training and test sets in a ratio of 8:2, and used five-fold cross validation for training and testing. The training set included 225 patients with wedge resection, 87 patients with poorly differentiated tumors, 60 patients with pleural invasion, and 31 patients with vascular invasion. The test set included 57 patients with wedge resection, 22 patients with poorly differentiated tumors, 15 patients with pleural invasion, and 8 patients with vascular invasion.

[0015] Step 2: Design and train a multimodal model:

[0016] 1. In order to integrate different forms of medical data, the model proposed in this invention consists of three parts, which enables it to adapt to and effectively fuse various data modalities: data mapping, feature extraction, and modality fusion. As a general method, these three parts are plug-and-play modules that can be switched freely to meet different task requirements. In addition, considering the optimization workflow, the model training process is carried out in an end-to-end manner without the need to independently optimize each component. The data is processed and mapped to the input format. Then, the data passes through the feature extraction module of each modality to obtain a high-dimensional semantic feature representation. Next, the feature fusion module performs semantic feature integration analysis. ;

[0017] 2. Feature extraction, visual embedding. To effectively extract semantic information from CT images, we use the encoder part of the Swin Transformer V2 model (Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., Wei, F., Guo, B.: Swin transformer v2: Scaling up capacity and resolution. In: 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11999–12009 (2022)) as the backbone for image feature extraction. We collect patients' medical records and laboratory test data and organize them into a structured and standardized data format. Medical records include discrete values ​​(such as gender and smoking history) and continuous values ​​(such as lung nodule size). The laboratory test data consists of continuous real numbers. In order to enhance the model's fitting and generalization capabilities and speed up the convergence of the training process, we preprocessed both discrete and continuous data. For discrete values, we used one-hot encoding. For continuous data, we performed min-max normalization on each feature column and standardized it using the maximum and minimum values ​​of each feature. Subsequently, we concatenated the medical records and laboratory test data. Through the feature adapter, we aligned the feature records of the medical records and the laboratory test data to have the same length as the image features. ;

[0018] 3. Missing data is common in medical datasets. In this paper, we excluded patients who lacked chest CT images, so missing data only occurred in medical records and laboratory test data. To solve this problem, we used polynomial interpolation to handle missing data.

[0019] 4. Multimodal fusion:

[0020] (1. We use an intermediate fusion strategy to fuse the high-dimensional features of each modality in the feature space, and then use a neural network to learn and predict the fused features. We propose a feature fusion method that combines linear attention and fast Fourier transform (FFT). The multimodal features are concatenated and then first input into the linear attention module to discover the dependencies between the features. Then, the output of the linear attention module is input into the fast Fourier transform module to transform the spatial domain of the features into the frequency domain, thereby discovering the fused frequency domain information of the features.

[0021] By using the linear attention mechanism (Linear Attention with Double Softmax), the time and space complexity is O(n) compared to the traditional Attention (Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, AN, Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30) 2 ), which uses the joint law of matrix multiplication and matrix transformation to reduce the time and space complexity to O(n). The specific equation can be expressed as:

[0022]

[0023] where Q,K∈R n×d , which is the same as the definition of traditional Attention. Consider Q, K as n×d vectors, and then cluster them into m categories to obtain a matrix consisting of m cluster centers. Represents the transpose of a matrix. Indicates that when the matrix When the matrix is ​​invertible, the inverse of the matrix is When not invertible, the pseudo-inverse of the matrix.

[0024] (2. Fast Fourier Transform. Discrete Fourier Transform (DFT) plays a very important role in the field of digital signal processing. The fast Fourier transform algorithm uses the symmetry and periodicity of the signal to reduce the computational complexity of DFT from O(n 2 ) is reduced to O(nlogn). The inverse DFT transform is similar in form to the DFT and can also be efficiently computed via the inverse fast Fourier transform (IFFT). Our implementation is similar to the given feature x∈R H×W×D , we first perform a 2D FFT on the spatial dimension to transform it into the frequency domain:

[0025]

[0026] where F[·] is the 2D FFT. Note that X is a complex tensor representing the frequency spectrum of x. We can then H×W×D Multiply to adjust the spectrum of x:

[0027]

[0028] where ⊙ is the product of the elements (also called the Hadamard product). The filter K has the same dimensions as X and is therefore called a global filter. X can represent any filter in the frequency domain. Finally, we use IFFT to transform the modulation spectrum X back to the spatial domain and update the features:

[0029]

[0030] (3. Perform deep feature compression and linear regression on the data after FFT operation, and operate the four variables with different multi-layer perceptrons respectively.

[0031] Step 3: Set training parameters

[0032] 1. Use the Adam optimizer, set the learning rate and the maximum number of iterations. The training process includes forward propagation to calculate the loss function, and back propagation to update the weights of the neural network. Training continues until the preset convergence condition is reached.

[0033] 2. After the training is completed, the final trained model parameters are saved for subsequent actual pathological automatic diagnosis and classification tasks. The feature vector corresponds to the label, that is, it can be diagnosed whether there is wedge resection, poor differentiation, vascular invasion, and pleural invasion. The labels are 0 and 1 respectively in the data;

[0034] The present invention seeks to use advances in artificial intelligence and deep learning to enhance the detection and treatment of high-risk factors for early lung cancer. Although there are clinical challenges in determining the association between these data and high-risk factors for lung cancer, the present method has achieved promising results with an AUC of 70%.

[0035] In addition, the proposed method highlights the potential of combining multiple data modalities to improve diagnostic performance. Compared with single-modality methods, multimodal models integrating lung CT images, medical record text, and laboratory test data achieved higher AUC and accuracy. This suggests that integrating different types of data can provide a more reliable and accurate assessment of high-risk factors for lung cancer.

[0036] The present invention solves the problem of how to reveal potential associations and "invisible" features in multimodal transport by designing different feature extractors to capture the features of each modality separately. These features are then fused using Fast Fourier Transform (FFT) and multi-head self-attention, so that frequency domain features and attention features associated with high-risk factors in lung cancer can be discovered.

[0037] The classification method provided by the present invention is entirely executed by a computer program. The computer program is executed to process data, train the designed multimodal fusion model, and execute the trained multimodal fusion model by a computer to process lung CT images, medical record texts, and laboratory test data to obtain classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The overall process of the multimodal fusion classification method is demonstrated, from data input to designing training models, performing multimodal fusion, to visualization and statistical analysis;

[0039] Figure 2 Shows the structure diagram of the multimodal fusion model

[0040] Figure 3 Demonstrates the steps for data screening in the early stages of an experiment DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the implementation modes and the accompanying drawings.

[0042] The specific processing steps are as follows:

[0043] Step 1: Build training data:

[0044] 1. As Figure 3 As shown, we initially collected 4351 patients before applying strict exclusion criteria. The exclusion criteria include: 1) patients with non-lung cancer stage I or II; 2) patients who did not undergo chest LDCT examination, laboratory tests or pathological examinations; 3) patients who lacked a final diagnosis of any of the four high-risk factors. In this study, patients with lung cancer with high-risk factors were defined as patients with one or more of the following conditions: wedge resection, poor differentiation, vascular invasion, and pleural invasion, that is, four labels, and each label was classified into two categories. These four labels are also the prediction result categories to be output by the multimodal model of the present invention. Finally, 393 patients with lung cancer with high-risk factors and 720 patients with lung cancer without high-risk factors were included in the study.

[0045] 2. Among 393 high-risk patients, 282 underwent wedge resection, 109 had poorly differentiated tumors, 75 had visceral pleural involvement, and 39 had vascular invasion. We collected demographic characteristics, medical history, and laboratory test results from the patients' medical records.

[0046] 3. Four radiologists reviewed all chest CT images to ensure image quality and clarity and mark the location of lesions. According to the radiologists' markings, 16 spatially continuous chest CT images were selected for each patient. Since this study targets patients with early-stage lung cancer with relatively small lesion areas, for patients with less than 16 CT images containing lesion areas, we centered the lesion area and selected a total of 16 images as model input. Medical record data included gender, age, smoking history, drinking history, history of lung disease, history of malignant tumors, family history, size of lung nodules, and location of lung nodules. We processed lung nodule size and location of lung nodules into four variables to comprehensively characterize lung information. Nodules, resulting in final medical record data containing 11 variables. Laboratory test data included six variables: white blood cells (WBC), granulocyte ratio (GR), carcinoembryonic antigen (CEA), cytokeratin 19 fragment (CY), neuron-specific enolase (NSE), and pro-gastrin releasing peptide (Pro-GRP). These data were included based on previous studies that disclosed their association with cancer.

[0047] 4. To ensure a balanced distribution among different categories, we randomly divided the dataset into training and test sets in a ratio of 8:2, and used five-fold cross validation for training and testing. The training set included 225 patients with wedge resection, 87 patients with poorly differentiated tumors, 60 patients with pleural invasion, and 31 patients with vascular invasion. The test set included 57 patients with wedge resection, 22 patients with poorly differentiated tumors, 15 patients with pleural invasion, and 8 patients with vascular invasion.

[0048] Step 2: Design and train a multimodal model:

[0049] 1. In order to integrate different forms of medical data, the model we proposed consists of three parts, which enables it to adapt to and effectively fuse various data modalities: data mapping, feature extraction, and modality fusion. As a general approach, these three parts are plug-and-play modules that can be switched freely to meet different task requirements. In addition, considering the optimization workflow, the model training process is carried out in an end-to-end manner without the need to independently optimize each component. The data is processed and mapped to the input format. Then, the data passes through the feature extraction module of each modality to obtain a high-dimensional semantic feature representation. Next, the feature fusion module performs semantic feature integration analysis;

[0050] 2. In order to effectively extract semantic information from CT images, we use the encoder part of the Swin Transformer V2 model (Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., Wei, F., Guo, B.: Swin transformer v2: Scaling up capacity and resolution. In: 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11999–12009 (2022)) as the backbone of image feature extraction, that is, Figure 2 The image encoder shown in Figure 1. This model adopts a variant of the Transformer architecture to capture local and global features of the image through layering and shifting mechanisms. The Swin Transformer V2 model uses a self-attention mechanism to handle long-range dependencies in the image and maintains sensitivity to the order of elements in the image through position encoding. It gradually extracts and refines features through multiple encoder layers, each of which includes a multi-head self-attention module and a feedforward network to achieve efficient feature learning and representation. In addition, the model is designed with an extensible architecture that allows it to adapt to different tasks and computing resource requirements by increasing the number of layers and heads. Swin Transformer V2 also specifically introduces a shifting window mechanism that periodically moves the processing window to obtain a more comprehensive image context, thereby enhancing the model's ability to capture information at different locations in the image. This design not only improves the model's sensitivity to multi-scale and multi-angle features, but also makes the model more computationally efficient when processing high-resolution images.

[0051] The image encoder pseudo code is as follows:

[0052]

[0053] The input feature vectors from medical record data and laboratory test data are respectively divided into three different weight vector matrices ω Q ,ω K ,ω V , multiplied together to form Q, K, V, where ω Q ,ω K ,ω V is a fixed constant. In the text encoder, Q and K are first multiplied, and then V is convolved through a linear layer and combined with the value that has passed through the previous linear layer.

[0054] Through the image encoder and text encoder (Encoder), the image and text features are converted into fixed-length encoding vectors, which facilitates subsequent multimodal fusion and various processing.

[0055] 3. We collect patients’ medical records and laboratory test data and organize them into a structured and standardized data format, namely Figure 2 Normalization operation in the process. Medical records include discrete values ​​(such as gender and smoking history) and continuous values ​​(such as lung nodule size). Laboratory test data consists of continuous real numbers. In order to enhance the fitting and generalization ability of the model and accelerate the convergence of the training process, we preprocessed both discrete data and continuous data. For discrete values, we used one-hot encoding. For continuous data, we performed min-max normalization on each feature column and standardized it using the maximum and minimum values ​​of each feature. Subsequently, we connected the medical records and laboratory test data. Through the feature adapter, we aligned the feature records of the medical records and the laboratory test data to have the same length as the image features.

[0056] 4. Missing data is common in medical datasets. In this paper, we excluded patients who lacked chest CT images, so missing data only occurred in medical records and laboratory test data. To solve this problem, we used polynomial interpolation to handle missing data.

[0057] 5. Combine the case data and laboratory test data and then combine them with the image data. The three types of data are combined to form the following Figure 2 The feature vector shown is used for subsequent LADS operations.

[0058] 6. Multimodal fusion:

[0059] (1. We use an intermediate fusion strategy to fuse the high-dimensional features of each modality in the feature space, and then use a neural network to learn and predict the fused features. Figure 2 As shown in Figure 2, we propose a feature fusion method that combines linear attention and fast Fourier transform (FFT). The multimodal features are concatenated and then first input into LADS to discover the dependencies between features. Then, the output of the linear attention module is input into the fast Fourier transform module to transform the spatial domain of the features into the frequency domain, thereby discovering the fused frequency domain information of the features.

[0060] The time and space complexity of traditional Attention is O(n 2 ), and use the linear attention mechanism (LinearAttention with Double Softmax), by utilizing the joint law of matrix multiplication and matrix transformation, the time and space complexity is reduced to O(n). The specific equation can be expressed as:

[0061]

[0062] where Q,K∈R n×d , which is the same as the definition in traditional Attention. Q (Query) is the query vector, which represents the information you want to find in other elements (i.e. Key). K (Key) is the key vector, which represents the "identity" or feature of each element in the sequence, and is used to match the query vector (Query). The operation is the same as before. The feature vector at this time is multiplied by the weight vector matrix ω Q ,ω K , get Q, K, treat Q, K as n×d vectors, and then cluster them into m categories to get a matrix consisting of m cluster centers Represents the transpose of a matrix. Indicates that when the matrix When the matrix is ​​invertible, the inverse of the matrix is When not invertible, the pseudo-inverse of the matrix.

[0063] (2. Fast Fourier Transform. Discrete Fourier Transform (DFT) plays a very important role in the field of digital signal processing. The fast Fourier transform algorithm uses the symmetry and periodicity of the signal to reduce the computational complexity of DFT from O(n 2 ) is reduced to O(nlogn). The inverse DFT transform is similar to the DFT and can also be efficiently computed via the inverse fast Fourier transform (IFFT). Our implementation sets the LADS-processed feature to x, where x∈R H×W×D , where H is the height, W is the width, and D is the depth, we first perform a 2D FFT on the spatial dimension to transform it into the frequency domain:

[0064]

[0065] where F[·] is the 2D FFT. X is a complex tensor representing the spectrum of x, i.e. the processed form of feature x. We can then h×w×D , where h is the height, w is the width, D is the depth, and C is a complex number, which is multiplied to adjust the spectrum of x:

[0066]

[0067] Where ⊙ is the product of elements (also called Hadamard product). The filter K has the same depth as X (D is the same), so it is called a global filter. Its code is as follows:

[0068] self.complex_weight=nn.Parameter(torch.randn(h,w,dim,2,dtype=torch.float32)*0.02)

[0069] Among them, h=4 and w=3 are fixed values.

[0070] Finally, we use IFFT to transform the modulation spectrum X back to the spatial domain to update the features, that is, update the feature x:

[0071]

[0072] (3. Perform deep feature compression and linear regression on the data after FFT operation. The feature compression pseudo code is as follows:

[0073]

[0074] The input features are mapped to a lower dimensional space through a linear layer. Subsequently, the Tanh activated features are mapped into weight vectors through another linear layer. These weights are normalized by the Softmax function and multiplied with the concatenated features to obtain weighted features.

[0075] (4. The four categories (wedge resection, poor differentiation, vascular invasion, pleural invasion) are identified by different multi-layer perceptrons, that is, Figure 2 Each variable is classified into 0-1 to indicate whether the patient has the disease under the label variable.

[0076] Step 3: Set training parameters

[0077] 1. Use the Adam optimizer, set the learning rate and the maximum number of iterations. The training process includes forward propagation to calculate the loss function, and back propagation to update the weights of the neural network. Training continues until the preset convergence condition is reached.

[0078] 2. After the training is completed, the final trained model parameters are saved for subsequent actual pathological automatic diagnosis and classification tasks. Using the feature vector corresponding to the label, it can be identified whether it belongs to the four categories of wedge resection, poor differentiation, vascular invasion, and pleural invasion. The labels are 0 and 1 respectively in the data;

[0079] Compared with other AI-based lung cancer risk factor classification methods (such as visceral pleural involvement and unknown lymph node status), the present invention can simultaneously predict multiple high-risk factors. In addition, the research cohort of the present invention is larger and more comprehensive. This is the first multi-label study on lung cancer risk factors, with a cohort of more than one thousand cases. We calculated AUC, sensitivity, specificity, and accuracy. The 95% confidence intervals for sensitivity and specificity were calculated using the "exact" Clopper-Pearson confidence interval. The 95% confidence interval for AUC was obtained using the Delong method (DeLong, ER, DeLong, DM, Clarke-Pearson, DL: Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 44 (3), 837–845). When comparing AUCs, the P value was also calculated by the Delong test. The ROC curve was generated by changing the threshold for the model output prediction. The threshold is used to binarize the real output of the model. Different thresholds may result in different binary model predictions for each image, leading to different sensitivity and specificity values ​​on the test dataset for plotting the ROC curve.

[0080] There are some limitations to the present invention. First, the sensitivity of multimodal models is relatively low and the false negative rate is high. This is particularly problematic in cancer diagnosis, where missing high-risk cases can have serious consequences. Second, deep learning is a data-driven analysis technique that lacks transparent reasoning explanations. To alleviate this problem, the present invention uses visualization techniques and conducts comprehensive ablation experiments. However, these methods all interpret from the results rather than analyzing the internal reasoning process of the model.

[0081] In summary, the multimodal approach is consistent with the NCCN guidelines, which emphasize the complexity of treatment decisions for lung cancer patients and the importance of considering a range of high-risk factors. By providing a comprehensive risk assessment, the present invention can assist in preoperative planning and adjuvant treatment decisions, ultimately contributing to personalized patient care. In summary, the present invention adds to the growing evidence that deep learning models, especially those that utilize multimodal data, play an important role in the future of oncology diagnostics.

Claims

1. A classification method for high-risk factors of lung cancer based on multimodal fusion, characterized in that: The following steps are involved: Step 1: Build a training dataset:

1. The collected patient data are divided into lung cancer patients with high-risk factors and lung cancer patients without high-risk factors; Among them, patients with high-risk lung cancer are defined as having one or more of the following: wedge resection, poor differentiation, vascular invasion, and pleural invasion. These four variables are set as four labels, and these four labels are classified into 0-1 binary categories; 2. Screen out chest CT images of patients with clearly marked lesion locations; select 16 consecutive chest CT images for each patient based on the markings, with the lesion area as the center; medical record data include gender, age, smoking history, drinking history, history of lung disease, history of malignant tumors, family history, size of lung nodules, and location of lung nodules; 3. Laboratory test data included six variables: white blood cells (WBC), granulocyte ratio (GR), carcinoembryonic antigen (CEA), cytokeratin 19 fragment (CY), neuron-specific enolase (NSE), and progastrin-releasing peptide (Pro-GRP); 4. The training data set was randomly divided into training set and test set in a ratio of 8:2, and five-fold cross validation was used for training and testing; Step 2: Design and train a multimodal model:

1. The multimodal model includes a data mapping module, a feature extraction module and a modality fusion module; 2. The data mapping module is used to normalize the case data and laboratory test data respectively, and then combine them as the input of the text encoder, use polynomial interpolation to handle missing data, and use the CT image as the input of the image encoder; 3. Feature extraction module, which is used to convert case data and laboratory test data into feature vectors through text encoders, and combine them with the feature vectors converted from CT image data through image encoders as input to the LADS module; Among them, the encoder part of the Swin Transformer V2 model is used as the backbone network of the image encoder; 4. The modal fusion module uses an intermediate fusion strategy to fuse the high-dimensional features of each modality in the feature space, and then uses a neural network to learn and predict the fused features. The features output by the feature extraction module are first input into the LADS module, and then the fast Fourier transform is used. First, 2DFFT is used in the spatial dimension, and then the filter is used for Hadamard product, and finally the fast inverse Fourier transform is used to process the features; and the entire process from LADS input to fast inverse Fourier transform output is stacked; 5. For the features processed by the modal fusion module, deep feature compression and linear regression are performed. After output, the four categories to be identified are processed by multi-layer perceptrons (MLP1, MLP2, MLP3, MLP4) and 0-1 binary classification is performed respectively; 6. Train the multimodal model with the training data set and save the final trained model parameters to obtain the trained multimodal model, which is used to receive actual chest CT images, laboratory test data, and medical record data and output classification results.

2. The method according to claim 1, characterized in that LADS uses the joint law of matrix multiplication and matrix transformation to reduce the time and space complexity to O(n). The specific equation can be expressed as: where Q,K∈R n×d , Q (Query) is the query vector, which represents the information you want to find in other elements. K is the key vector, which represents the "identity" or characteristics of each element in the sequence, and is used to match the query vector (Query). Q and K are generated according to the input of the LADS module. Q and K are regarded as n×d vectors, and then clustered into m categories to obtain a matrix composed of m cluster centers. represents the transpose of a matrix, Indicates that when the matrix When the matrix is ​​invertible, the inverse of the matrix is When not invertible, the pseudo-inverse of the matrix.

3. The method according to claim 2, characterized in that: Performing a 2D FFT on the spatial dimensions transforms it into the frequency domain: where F[·] is the 2D FFT, X is a complex tensor representing the spectrum of c, and the LADS-processed feature is set to x, where x∈R H×W×D , where H is the height, W is the width, and D is the depth; Then, by combining X with a learnable filter K∈C H×W×D Multiply to adjust the spectrum of x: Where ⊙ is the product of the elements, also called the Hadamard product, and the filter K has the same dimension as X, which is called a global filter; Finally, the modulation spectrum X is transformed back to the spatial domain using IFFT and the feature x is updated:

4. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

Images