Intelligent prediction system for lung cancer metastasis based on GCAVE-GAN and multimodal fusion

Abstract

This invention discloses an intelligent prediction system for lung cancer metastasis based on GCAVE-GAN and multimodal fusion. The system includes an image data augmentation module (GCAVE-GAN), an image feature fusion module, a gene feature extraction module, a multimodal fusion module (CFF), and a classifier module. The image data augmentation module uses a data augmentation network that measures the characteristic features of image-gene association. The gene data processing module uses feature selection and channel convolution techniques to extract gene markers related to lung cancer metastasis. The multimodal fusion module accurately captures the correlation between image and gene data through an adaptive multi-feature cross-attention mechanism, achieving deep information fusion and optimizing the prediction of lung cancer metastasis risk. This invention is particularly suitable for handling situations where the number of lung cancer images is insufficient, enabling rapid and stable assessment of lung cancer metastasis risk and significantly improving the accuracy and efficiency of the prediction model.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical image analysis, specifically a system for predicting lung cancer metastasis using a GCAVE-GAN (Generative Adversarial Network) and multimodal data fusion technology. Background Technology

[0002] Lung cancer is one of the deadliest malignant tumors worldwide. Its metastatic nature makes early identification and accurate prediction particularly important, as it has a decisive impact on improving patient survival rates and quality of life. Currently, the prediction of lung cancer metastasis mainly relies on traditional imaging examinations and biomarker analysis. However, these methods face many challenges, especially in terms of accuracy and early detection capabilities, often limited by the technology and methods themselves, leading to frequent misdiagnosis and missed diagnosis. With the development of technology, particularly the advancements in genomics and medical image processing, researchers have begun to explore how to improve the accuracy of lung cancer metastasis prediction by fusing these multimodal data. Nevertheless, challenges such as sample size limitations and heterogeneity of data types remain, requiring innovative solutions.

[0003] Existing research has confirmed the remarkable potential of deep learning techniques, especially deep convolutional neural networks (DCNNs), in handling such problems. For example, some studies have achieved highly accurate predictions of lymph node metastasis by designing 3D multi-scale DCNN models to analyze lung CT scan data. Simultaneously, mask-guided dual-stream attention networks have effectively predicted distant metastases in lung cancer. These techniques, based on lung images, focus on the local analysis of tumor features. However, current methods have significant limitations; they often focus only on local tumor features in CT images, neglecting global and tumor marginal contextual information that provides more information about metastatic potential. In fact, non-small cell lung cancer tumors may exhibit a range of irregular texture features, such as pleural invasion, bronchial interruption, and vascular attachment, all of which are important clues for predicting tumor behavior. Furthermore, the molecular genetic characteristics of tumors, such as gene expression levels, also provide valuable information for tumor prediction. Combining these genetic data with imaging features can provide a comprehensive tumor assessment from both microscopic and macroscopic perspectives. Especially in the area of ​​multimodal data fusion, novel deep learning frameworks, such as deep orthogonal fusion (DOF) models, have been shown to intelligently combine radiological, pathological, and genomic features, improving the complementarity and accuracy of predictive models. Addressing the challenges of structural consistency and modal balance in multimodal data fusion, this invention proposes an intelligent lung cancer metastasis prediction method based on GCAVE-GAN and multimodal fusion, fusing CT images and genetic data to predict the risk of lung cancer metastasis. This method not only overcomes the shortcomings of traditional methods in handling modal heterogeneity and data inconsistency but also provides a novel and more accurate multimodal fusion lung cancer metastasis prediction model. This technology enables more effective diagnosis and prediction of lung cancer metastasis, ultimately improving patient treatment outcomes and survival expectations. Summary of the Invention

[0004] Based on the above ideas, this invention proposes an intelligent prediction method for lung cancer metastasis based on GCAVE-GAN and multimodal fusion. The goal of this method is to improve the accuracy of lung cancer metastasis risk prediction by developing a novel model that integrates multimodal data and employs a novel feature extraction and fusion mechanism. First, this method utilizes GCAVE-GAN to generate more high-quality, highly realistic medical images of lung cancer metastases, thereby improving the model's training effect. Simultaneously, by combining multimodal information from patient imaging and genetic data, and through comprehensive analysis using deep learning algorithms, a comprehensive prediction model is formed, capable of effectively predicting lung cancer metastasis.

[0005] To achieve the above objectives, the following technical solution was adopted:

[0006] The image data augmentation module GCAVE-GAN is used to augment tumor images. This module can explore the complementary relationship between images and genes, link tumor imaging features with potential tumor genetic information and other data, reveal potential correlations with lung cancer metastasis, generate high-quality tumor images with a small sample size, and maintain the richness of genetic information in the images.

[0007] Image feature fusion module: This module includes a dual-path encoder and a boundary-guided contextual attention part. The purpose of each path encoder is to more accurately obtain specific features corresponding to the input data.

[0008] Gene feature extraction module: Due to the small sample size and high dimensionality of gene data from lung cancer patients, and the fact that high-dimensional data is rich in redundant features, the Adaboost algorithm is used to select features from the gene data. At the same time, there are significant differences in gene data among different patients, and treating the entire gene data as a matrix for convolution would result in a large error. Therefore, a channel-level convolution mechanism is used to extract the gene data of each patient independently.

[0009] Multimodal fusion module CFF: To address the inconsistency of data from different modalities, a cross-modal feature fusion (CFF) module based on channel spatial attention was designed to extract features from calibrated image features and non-image features.

[0010] Classifier module: The features output by the multimodal fusion module are processed by the classifier to obtain accurate lung cancer metastasis classification and prediction results.

[0011] The image data augmentation module primarily performs tasks such as gene-to-image conversion and cross-modal medical transformation. GCAVE-GAN provides an effective solution, characterized by its ability to introduce additional genomic conditional information during the generation process, thereby achieving more genetically informed image generation. The main steps are as follows: 1) Encoder E projects the tumor image onto the latent space z; 2) Generator G converts the latent vector z into a tumor image; 3) Discriminator D determines whether the tumor image is real or fake; 4) Classifier C projects the tumor image onto the gene space. The input is the tumor region image and gene data, and the output is the augmented tumor image.

[0012] The average feature matching objective is also applied to the loss function to achieve a similar purpose. The encoder outputs the mean and covariance of the latent vectors. Then, the Kullback-Leibler (KL) loss is used to approximate the distribution of the latent space to a Gaussian distribution. The KL loss can be expressed as:

[0013]

[0014] Where μ is the mean and ε is the covariance.

[0015] The generator aims to capture the distribution characteristics of the data, while the discriminator determines whether a sample is drawn directly from real data or generated by the generator. Both networks may employ non-linear mapping functions, similar to a multilayer perceptron structure. During training, the entire system adjusts the parameters of these two models by optimizing the objective function. A condition is given during the generation process, leading to a certain output; the objective function is as follows:

[0016]

[0017] To effectively train the model and enhance its ability to distinguish between real and synthetic tumors, the discriminator employs the following method of minimizing the loss function:

[0018]

[0019] To address the issue of gradient instability of G during training, this paper uses average feature matching for the target:

[0020]

[0021] L GD As a perceived loss, where f D (x) represents a feature on the first fully connected layer of the discriminator.

[0022] Since the encoded gene data reveals overexpression or underexpression trends of a certain group of genes, logarithmic loss function was not chosen for fitting. Instead, mean squared error loss was used to process these data.

[0023]

[0024] To ensure that the tumor image generated by G belongs to the corresponding category c, the function needs to be minimized:

[0025]

[0026] Then, the latent vector z can be sampled as z = μ + r⊙exp(ε), where r ~ N(0,I) is a random vector, and ⊙ denotes element-wise multiplication. After inputting z into the generator G, we can obtain the synthesized tumor. The generator loss is calculated as follows:

[0027]

[0028] Where f D and f C These are the features of the first FC layer of discriminator D and classifier C, respectively.

[0029] The image feature fusion module is primarily designed to more accurately grasp the contextual regions of the tumor and the information provided by the tumor mask. This module includes a dual-path encoder and a boundary-guided contextual attention component. This design aims to fully utilize the rich information surrounding the tumor and the morphological features of the tumor mask, such as its shape, boundaries, and location.

[0030] In the dual-path encoder, the GCAVE-GAN method is first used to encode the tumor image after enhancement and the mask image in a dual-branch manner, and then reassembles them into 3D blocks for depth analysis. Feature extraction of the tumor region uses the 3D-Unet model, operating at two different scales (1.5× and 2.0×) to capture more details. For the mask branch, 3D-Unet is also used to extract boundary detail features.

[0031] The context features, tumor features, and boundary features extracted by the dual-path encoder are denoted as M. c M t and M b Feature aggregation is achieved by guiding the contextual attention component through boundaries.

[0032] First, extract context region features and tumor region features from the feature map obtained from the image branch, denoted as M. c and M t Extracting boundary features from mask branches, denoted as M b Employ an attention mechanism and Obtain the updated context features M c Boundary features M b ′, where α c α t The image is encoded to obtain contextual attention and tumor attention, α b Boundary attention is obtained after the mask passes through the encoder.

[0033]

[0034] To account for the different impacts of tumor context and boundary information on classification results, the proposed network enables the tumor to be classified in W... c +W b Adaptive learning W under the condition of =1 c and W b These represent the weights used in the fusion process, and the specific calculation process is as follows:

[0035]

[0036] Where F represents the operation after the two fully connected layers and the softmax layer, element-wise multiplication

[0037] The gene feature extraction module first preprocesses the sample data using Z-score normalization, using the expression value of each gene as a feature and the sample label as the target variable of the classifier. Then, it trains the classifier using weighted methods to focus more on samples that are frequently misclassified. Finally, it iterates multiple times to select the optimal subset of genes, and inputs the expression values ​​of these genes as the final features into the model for feature extraction. Considering that the Adaboost algorithm gradually reduces the error rate in each iteration, it tends to combine multiple classifiers to improve classification accuracy and weights the training samples to reduce the impact of outliers, selecting the features with the highest discriminative power to improve robustness. Since the Adaboost model uses the Gini coefficient in the CART classification tree to select the optimal splitting attribute, the Gini coefficient can be used to measure the importance of features. To reduce the complexity of the model, feature selection can be achieved by inputting the original feature set into the Adaboost model, calculating the feature importance based on the Gini coefficient, and ranking the features according to their importance. Specifically, the importance of each feature on each tree is calculated and a weighted average is taken to obtain the overall importance of the feature. The calculation formula is as follows:

[0038] F imp (j)=ω k ×IMP j

[0039] As can be seen from the above formula, the higher the feature importance, the greater its contribution to the model, and therefore the better the model performance.

[0040] Where ωk represents the weight of the weak learner, and IMPj is the importance of the features in a single tree.

[0041]

[0042] In the formula, L⁻¹ represents the number of non-leaf nodes in the tree, and ΔR represents the reduction in weighted impurity of a non-leaf node during splitting. The larger the reduction, the higher the importance of the feature. The formula for calculating the weighted impurity of node i is R. i =P i ×E i , where P i Let E be the node probability. i To measure the impurity of a node, this paper uses the Gini coefficient (Gini(i)) as follows:

[0043]

[0044] A channel-level convolution mechanism is employed to independently extract the genetic data of each patient. This mechanism ensures that each convolutional kernel can only convolve data from one patient, allowing the designed network to independently extract the desired features from multiple patients without interference from other patients. In channel-oriented convolution operations, only one convolutional kernel is used for each channel. The genetic data of patient N is denoted as... Channel convolution kernel is denoted as Both have the same number of channels, and the feature map generated after the gene data is processed by the convolution kernel is denoted as... And perform the operation according to the following formula, where This is represented as a convolution operation. By convolving each patient's genetic data with a separate channel, the features of each patient do not interfere with each other. After multiple layers of channel convolution, higher-level feature maps are learned, which reduces the number of parameters for network training, enables efficient extraction of useful features, and greatly improves network efficiency.

[0045]

[0046] The multimodal fusion module (CFF) extracts features from both image and non-image features. First, it performs global pooling and average pooling on both types of features along the channel dimension to extract more comprehensive features and focus on learning global information. Then, the four features are concatenated and fed into an MLP to obtain different attention values, which are mapped to [0,1] through activation functions. The resulting new feature weights after the segmentation operation are denoted as follows: and Image and non-image features that have undergone channel attention are denoted as F'. image and F' non-image The overall process is as follows:

[0047]

[0048] Where σ represents the sigmoid function, This indicates element-wise multiplication.

[0049] Next, a spatial attention mechanism is used to learn more local information, first by... image and F' non-image After concatenation, the data passes through a convolutional layer, a ReLU activation function, and another convolutional layer to generate global features F. Then, a split operation is performed to generate new spatial features with weights denoted as F. and The obtained spatial feature weights Image features F' after channel attention image Element-wise multiplication yields F” image The non-image features F' that have undergone channel attentionnon-image Spatial feature weights of image features Element-wise multiplication yields F” non-image The final aggregation generates F. fuse The overall process for predicting lung cancer metastasis is defined as follows:

[0050]

[0051] in Indicates series connection. This represents element-wise multiplication, and the final output is denoted as F. fuse .

[0052] The classifier module consists of three fully connected layers and a softmax function. The loss function is defined as cross-entropy loss, as shown in the following formula:

[0053]

[0054] Where y i The label p represents the sample. i This represents the probability that a sample is predicted to be of the positive class.

[0055] During the training process of this algorithm, we use gradient descent to calculate the minimum value of the loss function. This method gradually adjusts and optimizes the model parameters based on the obtained gradient direction. As training progresses, the model gradually converges, eventually achieving a relatively ideal classification prediction effect, significantly improving the accuracy and reliability of classification.

[0056] Beneficial effects of the present invention

[0057] This invention provides an intelligent prediction method for lung cancer metastasis based on GCAVE-GAN and multimodal fusion. This method effectively combines cutting-edge deep learning technology and medical imaging analysis, generating high-quality tumor images through a generative adversarial network (GCAVE-GAN) and performing multimodal fusion analysis with patient biomarker data. This method can achieve accurate prediction of lung cancer metastasis, significantly improving the accuracy and reliability of prediction. First, this invention effectively enhances the quantity and quality of lung medical image data through GCAVE-GAN technology, enabling the model to be trained on richer datasets and improving the model's ability to learn lung cancer metastasis characteristics. Second, through multimodal data fusion technology, this invention combines image data and genetic data, performing comprehensive analysis through a deep learning model, further enhancing the overall judgment power of the prediction model. Furthermore, this method can not only be used for preclinical lung cancer metastasis risk assessment but also assist doctors in developing more personalized treatment plans, thereby improving treatment outcomes. Due to its efficient processing flow and excellent predictive performance, this method is expected to be widely applied in the diagnosis and treatment of lung cancer, and is of great significance for improving the survival rate and quality of life of lung cancer patients. Ultimately, the implementation of this invention will reduce medical costs, improve diagnostic and treatment efficiency, and bring innovative technological solutions to the medical industry, promoting the development of medical imaging and artificial intelligence technologies.

[0058] The intelligent prediction method for lung cancer metastasis based on GCAVE-GAN and multimodal fusion has a classification accuracy of 96.88%, an AUC of 97.12%, and a response time of less than or equal to 1.5 seconds to complete a classification task. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the overall process of the intelligent prediction method for lung cancer metastasis based on GCAVE-GAN and multimodal fusion of the present invention.

[0060] Figure 2 This is the structural framework of the GCAVE-GAN image data enhancement module of the present invention;

[0061] Figure 3 This is a schematic diagram illustrating the operation of the image feature fusion module of the present invention;

[0062] Figure 4 This is the structural framework of the gene feature extraction module of the present invention;

[0063] Figure 5 This is a detailed flowchart of the multimodal fusion module CFF of the present invention;

[0064] Figure 6 This is a schematic diagram illustrating the image enhancement effect of the GCAVE-GAN method of the present invention;

[0065] Figure 7 The classification results of the intelligent prediction method for lung cancer metastasis based on GCAVE-GAN and multimodal fusion in this invention are visualized.

[0066] Figure 8 The ROC curve of the intelligent prediction method for lung cancer metastasis based on GCAVE-GAN and multimodal fusion in this invention is compared with four other methods.

[0067] Figure 9 This is a graph showing the detection rate of the intelligent prediction method for lung cancer metastasis based on GCAVE-GAN and multimodal fusion in this invention compared with four other methods. Detailed Implementation

[0068] The present invention will be described in detail below with reference to specific embodiments.

[0069] This invention uses CT image data and mRNASeq data of gene expression levels from specific lung cancer patients, annotated by professional physicians, for testing. The entire process is as follows: Figure 1 As shown, firstly, GCAVE-GAN is used to process lung CT scan images to generate lung cancer tumor images containing genetic information; then, image features are extracted using an image feature fusion module; next, these image features are fused with genetic data in a multimodal manner to extract deeper features; finally, a developed deep learning model is used for training and validation. Validation results show that the prediction accuracy and sensitivity of this invention are significantly higher than traditional methods, effectively improving the diagnostic and prognostic assessment capabilities for lung cancer metastasis.

[0070] A1: First, use as... Figure 2 The method shown is based on GCAVE-GAN (Gene-Guided Conditional Generative Adversarial Network) for tumor image generation and enhancement. In the experiment, each patient's CT slice contained 512×512 pixels. First, the lung tumor region was cropped according to the ROI mask marked by professional physicians, and the corresponding gene data was found for the experiment.

[0071] A2: First, the original tumor image is projected into a latent space z using encoder E. This process is achieved through a deep learning model (convolutional neural network) trained to capture key features and attributes of the tumor image. In this way, the encoder extracts the core information from the image and transforms it into a latent vector z, which serves as the input to the generator. The encoder outputs the mean and covariance of the latent vector. Then, the Kullback-Leibler (KL) loss is used to approximate the distribution of the latent space to a Gaussian distribution. The KL loss is represented as follows, where μ is the mean and ε is the covariance:

[0072]

[0073] A3: Then, the generator G primarily performs the transformation from latent vectors to tumor images. It takes a latent vector z from the encoder, captures the distribution characteristics of the data, and generates a new tumor image based on this vector. The generator is a deconvolutional neural network that can progressively convert latent vectors into image data. In this process, the generator attempts to create images that are visually indistinguishable from real tumor images. These images are richer and more precise in detail, reflecting the tumor characteristics encoded in the genetic data.

[0074] A4: The discriminator D's task is to evaluate whether the generated tumor images are realistic enough, that is, to determine whether these images are real or generated by the generator. The discriminator is also a convolutional neural network, which makes its judgment by analyzing the details, texture, and other visual features of the image. The training goal of the discriminator is to accurately distinguish between real and generated images, thereby prompting the generator to produce higher-quality images.

[0075] A5: Both networks may employ non-linear mapping functions, similar to a multilayer perceptron structure. During training, the entire system adjusts the parameters of these two models by optimizing the objective function. A condition is given during the generation process, leading to a certain output; the objective function is as follows:

[0076]

[0077] A6: Classifier C projects the generated tumor image into the genetic space, a step that is one of the core innovations of the GCAVE-GAN method. The classifier analyzes the tumor image and attempts to predict associated gene expression patterns or mutation information. This can be achieved by directly extracting features from the image and associating them with genetic data. The classifier's output helps to understand the relationship between image features and gene expression, which in turn guides the generator to create more accurate and personalized tumor images.

[0078] A7: To effectively train the model and enhance its ability to distinguish between real and synthetic tumors, the discriminator employs the following method of minimizing the loss function:

[0079]

[0080] A8: To address the issue of gradient instability of G during training, average feature matching is used for the target:

[0081]

[0082] L GD As a perceived loss, where f D (x) represents a feature on the first fully connected layer of the discriminator.

[0083] A9: Because the encoded gene data reveals overexpression or underexpression trends of a certain group of genes, we chose not to use the logarithmic loss function for fitting. Instead, we used the mean squared error loss to process these data.

[0084]

[0085] A10: To ensure that the tumor image generated by G belongs to the corresponding category c, the function needs to be minimized:

[0086]

[0087] A11: Then, the latent vector z can be sampled as z = μ + r⊙exp(ε), where r ~ N(0,I) is a random vector, and ⊙ denotes element-wise multiplication. After inputting z into the generator G, we can obtain the synthesized tumor. The generator loss is calculated as follows, where f D and f C These are the features of the first FC layer of discriminator D and classifier C, respectively.

[0088]

[0089] A12: Through the above steps, the system finally outputs an enhanced tumor image. This enhanced image is not only more detailed and accurate visually, but also reflects the true characteristics of the tumor at the molecular level.

[0090] B1: Utilizing the aforementioned image feature fusion module (such as...) Figure 3 As shown, this module (including a dual-path encoder and a boundary-guided contextual attention component) is designed to better extract contextual region features and tumor mask features from the tumor. This approach aims to fully utilize the rich information surrounding the tumor region and the morphological features of the tumor mask, such as shape, boundaries, and location.

[0091] B2: In the dual-path encoder, the tumor image after enhancement and the mask image are first encoded using the method described in A1, and then reassembled into 3D blocks for depth analysis. Feature extraction of the tumor region uses the 3D-Unet model, operating at two different scales (1.5× and 2.0×) to capture more details. For the mask branch, 3D-Unet is also used to extract boundary detail features.

[0092] B3: The context features, tumor features, and boundary features extracted by the dual-path encoder are represented as M, respectively. c M t and M b Feature aggregation is achieved by guiding the contextual attention component through boundaries.

[0093] B4: First, extract the context region features and tumor region features from the feature map obtained from the image branch, denoted as M. c and M t Extracting boundary features from mask branches, denoted as M b Employ an attention mechanism and Obtain the updated context features M c Boundary features M b ′, where α c α t The image is encoded to obtain contextual attention and tumor attention, α b Boundary attention is obtained after the mask passes through the encoder.

[0094]

[0095] B5: To account for the different impacts of tumor context and boundary information on classification results, the proposed network enables the tumor to be classified in W... c +W b Adaptive learning W under the condition of =1 c and W b These represent the weights used in the fusion process, and the specific calculation process is as follows, where F represents the operation after the two fully connected layers and softmax. Element-wise multiplication:

[0096]

[0097] C1: Next, feature extraction is performed on the gene data. First, Z-score normalization is applied to the gene sample data to ensure that the expression values ​​of each gene have a uniform proportion and distribution, enabling more effective feature selection. This step standardizes the data, reduces the bias between features of different magnitudes, and provides a stable input foundation for subsequent classifier training.

[0098] C2: Next, the AdaBoost algorithm is used to train the classifier. AdaBoost is an ensemble learning method that combines multiple weak classifiers to build a strong classifier. In each iteration, AdaBoost increases the weight of samples misclassified in the previous round and decreases the weight of correctly classified samples, thus making the classifier focus more on those difficult-to-classify samples in subsequent iterations. This method helps improve the overall classification accuracy of the model and reduces the impact of outliers.

[0099] C3: During Adaboost training, the Gini coefficient is used to measure feature importance, so it's employed to evaluate and select features in the CART classification tree. The Gini coefficient measures data impurity; a node's Gini impurity indicates its uncertainty. Each time a node splits, the reduction in impurity (ΔR) is calculated; the larger the reduction, the more important the feature. By calculating the reduction in impurity for each feature in each tree and taking a weighted average, the overall importance of each feature can be assessed. To reduce model complexity, feature selection can be achieved by inputting the original feature set into the Adaboost model, calculating feature importance based on the Gini coefficient, and then ranking features according to their importance. Specifically, the importance of each feature in each tree is calculated and averaged to obtain the overall importance of that feature, as shown in the following formula:

[0100] F imp (j)=ω k ×IMP j

[0101] C4: As can be seen from the above formula, the higher the feature importance, the greater its contribution to the model, and therefore the better the model performance. Here, ωk represents the weight of the weak learner, and IMPj is the importance of the features in a single tree.

[0102]

[0103] C5: In the formula, L-1 represents the number of non-leaf nodes in the tree, and ΔR represents the reduction in weighted impurity of a non-leaf node during splitting. The larger the reduction, the higher the importance of the feature. The formula for calculating the weighted impurity of node i is R. i =P i ×E i , where P i Let E be the node probability. i To measure the impurity of a node, this paper uses the Gini coefficient (Gini(i)) as follows:

[0104]

[0105] C6: To extract individual patient genetic features without interference from other patient data, a channel-level convolution mechanism is introduced. Under this mechanism, each patient's genetic data is processed through an independent convolution kernel, ensuring the independence of feature extraction. This method can effectively extract features independently from each patient's data, avoiding mutual interference between data. Through multi-level channel convolution, the network can learn higher-level feature representations from genetic data, while reducing the number of model parameters and improving network efficiency and performance. In channel-oriented convolution operations, only one convolution kernel is used for each channel. The genetic data of patient N is denoted as... Channel convolution kernel is denoted as Both have the same number of channels, and the feature map generated after the gene data is processed by the convolution kernel is denoted as... And perform the operation according to the following formula, where This is represented as a convolution operation. By convolving each patient's genetic data with a separate channel, the features of each patient do not interfere with each other. After multiple layers of channel convolution, higher-level feature maps are learned, which reduces the number of parameters for network training, enables efficient extraction of useful features, and greatly improves network efficiency.

[0106]

[0107] C7: In terms of model training and feature optimization, after multiple iterations, the best-performing subset of genes is selected as the final feature input to the model. These optimized features can better reflect the relationship between gene data and the target variable (lung cancer metastasis status), thereby improving the model's prediction accuracy and robustness.

[0108] D1: Next, the multimodal fusion module (CFF) is used to fuse image features and non-image features. First, global pooling and average pooling operations are performed on both image and non-image features. The purpose of this step is to extract more comprehensive features along the channel dimension and focus on capturing global contextual information. Global pooling helps extract information from the entire feature map, while average pooling reduces the feature dimensionality while preserving important feature information.

[0109] D2: Next, the four feature vectors obtained through global pooling and average pooling are concatenated and merged into a single comprehensive feature representation. This merged feature vector is then input into a multilayer perceptron (MLP) to obtain different attention weights. These attention weights are mapped to the [0,1] interval through an activation function (such as the sigmoid function σ), generating channel attention weights for image features and non-image features, denoted as follows: and The image features and non-image features obtained after channel attention weighting are denoted as F', respectively.image and F' non-image The overall process is as follows:

[0110]

[0111] Where σ represents the sigmoid function, This indicates element-wise multiplication.

[0112] D3: Subsequently, the module introduces a spatial attention mechanism to further learn local information. First, the image features F', which have undergone channel attention weighting, are... image Non-image features F' non-image The concatenation is performed, and then passed through a convolutional layer, a ReLU activation function, and another convolutional layer to generate global features F. Then, a split operation is performed to produce new spatial features. and

[0113] D4: Spatial Feature Weights Image features F' after channel attention image Element-wise multiplication yields F” image Similarly, the non-image features F' that have undergone channel attention are... non-image Spatial feature weights of image features Element-wise multiplication yields F” non-image This step further refines the representation of the two features by combining channel attention and spatial attention. Finally, F”image and F”non-image are aggregated to generate F. fuse For lung cancer metastasis prediction, this final fusion feature integrates global and local information from both image and non-image sources, providing a richer and more accurate feature representation for lung cancer metastasis prediction. The overall process is defined as follows:

[0114]

[0115] in Indicates series connection. This represents element-wise multiplication, and the final output is denoted as F. fuse .

[0116] D5: Overall, the CFF module achieves effective integration of image and non-image features through a carefully designed process that combines attention mechanisms and feature fusion strategies. This approach holds promise for significantly improving the performance of lung cancer metastasis prediction and providing physicians with more accurate diagnostic information.

[0117] E1: To build an effective lung cancer metastasis prediction model, we adopted a classifier module consisting of three fully connected layers (FC) and a softmax function. The following is a detailed description of the steps and optimization process of this module:

[0118] E2: Fully connected layers are a common structure in deep learning, connecting the output of the previous layer to each neuron in the current layer. In this classifier module, three such fully connected layers are used to enhance the model's learning and representation capabilities. Each layer is connected to the output of the previous layer via weights and biases, and uses an activation function (such as ReLU) to add non-linearity, helping the model capture more complex features. At the output of the last fully connected layer, we apply the softmax function to transform the model's output, resulting in a probability distribution. The softmax function converts any real number to a value between 0 and 1, ensuring that the sum of all output values ​​is 1. This allows the model's output to be directly interpreted as the predicted probabilities for each class.

[0119] E3: The training objective of the model is to minimize the loss function. This model uses the cross-entropy loss function, as shown in the following formula:

[0120]

[0121] Where y i The label p represents the sample. i This represents the probability that a sample is predicted to be of the positive class.

[0122] E4: During the training process of this algorithm, we use gradient descent to calculate the minimum value of the loss function. This method gradually adjusts and optimizes the model parameters based on the obtained gradient direction. As training progresses, the model parameters are gradually adjusted through continuous iterative optimization. During training, the value of the loss function gradually decreases, and the accuracy of the model's predictions gradually improves. Ultimately, the model converges to a relatively stable state, at which point it can provide a more ideal classification prediction effect. Through this series of optimization steps, the classifier module can significantly improve the accuracy and reliability of classification tasks, and has broad application potential in various application scenarios.

[0123] Figure 6This diagram illustrates the image enhancement effect of the GCAVE-GAN method of this invention. Compared with other generative network methods, GCAVE-GAN outperforms the Conditional Variational Autoencoder Generative Adversarial Network (CVAE-GAN) and the Conditional Adversarial Network (CGAN). The core innovation of this invention lies in introducing genomic data as auxiliary conditional information, a strategy that significantly enhances the model's ability to generate images. By associating genomic data with images, GCAVE-GAN can gain a deeper understanding and capture subtle features of the image during the generation process. When generating CT images describing tumor features, the model can accurately encode tumor details using genomic information, generating more accurate and vivid images. This method not only improves the quality of generated images but also increases their diversity and accuracy. Especially in the challenging task of predicting lung cancer metastasis, GCAVE-GAN plays a crucial role through image enhancement techniques.

[0124] Figure 7 This is a visualization of the classification results of the intelligent prediction method for lung cancer metastasis based on GCAVE-GAN and multimodal fusion in this invention; when the image only uses boundary attention (e.g.) Figure 7 (b) The network focuses on both the morphological information of the nodule itself (outline and location, etc.) and some features of its boundaries. However, sometimes it focuses too much on boundary information, thus ignoring the contextual features around the tumor that are closely related to lung cancer metastasis, affecting the final prediction results. Compared with using only boundary attention, when contextual information is introduced (e.g., Figure 7 (c) As shown in the present invention, the image feature extraction module will give greater weight to the tumor boundary and its surrounding area rather than to other irrelevant information in the entire lung parenchyma. This will allow it to learn the tumor features and the rich semantic information surrounding them. When there is no pleural invasion or vascular attachment (e.g., ...), this will be particularly relevant when there is no pleural invasion or vascular attachment (e.g., ...). Figure 7 Even irrelevant contextual features (such as those in the fifth and tenth columns) will not interfere with experimental judgments. This fully demonstrates the importance of the module proposed in this invention, which can greatly improve the accuracy of predicting the risk of metastasis in non-small cell lung cancer.

[0125] Figure 8The image shows the ROC curves of the intelligent lung cancer metastasis prediction method based on GCAVE-GAN and multimodal fusion of this invention, compared with four other methods. Method 1 uses only single image data for lung cancer metastasis prediction; Method 2 uses single gene data; Method 3 uses image-gene tandem to predict lung cancer metastasis; Method 4 incorporates the multimodal fusion method of this invention to perform image-gene fusion for lung cancer metastasis prediction, but does not include an image data enhancement module. This invention, using the intelligent lung cancer metastasis prediction method based on GCAVE-GAN and multimodal fusion, achieved an excellent AUC of 97.12%. Compared with existing models, this invention demonstrates significant improvements and superior overall performance, with higher accuracy, sensitivity, and specificity. This is attributed to the innovative image enhancement, feature extraction, and multimodal fusion strategies of this invention, which show significant advantages in lung cancer metastasis risk prediction, revealing the indispensability of multimodal approaches in metastasis risk prediction tasks.

[0126] Figure 9 This is a comparison chart of the detection rates of the intelligent prediction method for lung cancer metastasis based on GCAVE-GAN and multimodal fusion in this invention with four other methods. Methods 1-4 are described as follows. Figure 8 As can be seen, compared with other methods, the system proposed in this invention has the highest detection rate, and the average detection rate is still higher than the other four models as the number of image sequences increases.

[0127] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A lung cancer metastasis intelligent prediction system based on GCAVE-GAN and multimodal fusion, characterized in that, It includes the image data enhancement module GCAVE-GAN, the image feature fusion module, the gene feature extraction module, the multimodal fusion module CFF, and the classifier module; The image data augmentation module GCAVE-GAN is used to augment tumor images. This module can mine the complementary relationship between images and genes, link tumor imaging features with potential tumor genetic information data, reveal potential correlations with lung cancer metastasis, generate high-quality tumor images with a small sample size, and maintain the richness of genetic information in the images. The image data augmentation module realizes gene-to-image and cross-modal medical conversion tasks. GCAVE-GAN includes the following steps: 1) Encoder E, which projects the tumor image onto the latent space z; 2) Generator G, which converts the latent vector z into a tumor; 3) Discriminator D, which determines whether the tumor image is real or fake; 4) Classifier C, which projects the tumor image onto the gene space. The input is a tumor region image and gene data, and the output is the augmented tumor image. Image feature fusion module: This module includes a dual-path encoder and a boundary-guided contextual attention part. The purpose of each path encoder is to more accurately obtain specific features corresponding to the input data. Gene feature extraction module: Due to the small sample size and high dimensionality of gene data from lung cancer patients, and the fact that high-dimensional data is rich in redundant features, the Adaboost algorithm is used to select features from the gene data. At the same time, there are significant differences in gene data among different patients. Treating the overall gene data as a matrix for convolution would result in a large error. Therefore, a channel-level convolution mechanism is used to extract the gene data of each patient independently. Multimodal fusion module (CFF): To address the inconsistency between different modal data, a cross-modal feature fusion module based on channel spatial attention was designed to extract features from calibrated image features and non-image features. Classifier module: The features output by the multimodal fusion module are processed by the classifier to obtain accurate lung cancer metastasis classification and prediction results.

2. The intelligent prediction system for lung cancer metastasis based on GCAVE-GAN and multimodal fusion according to claim 1, characterized in that, First, the original tumor image is projected into a latent space z using an encoder E. This process is implemented using a deep learning model trained to capture key features and attributes of the tumor image. In this way, the encoder can extract core information from the image and transform it into a latent vector z, which will serve as the input to the generator. The encoder outputs the mean and covariance of the latent vector. Then, the Kullback-Leibler (KL) loss is used to approximate the distribution of the latent space to a Gaussian distribution. The KL loss can be expressed as: Where μ is the mean and ε is the covariance; The generator aims to capture the distribution characteristics of the data, while the discriminator aims to determine whether a sample is drawn directly from real data or generated by the generator; both networks employ non-linear mapping functions; during training, the entire system adjusts the parameters of these two models by optimizing the objective function; a condition is given during the generation process, leading to a certain output, and the objective function is as follows: To effectively train the model and enhance its ability to distinguish between real and synthetic tumors, the discriminator employs the following method of minimizing the loss function: To address the issue of gradient instability of G during training, average feature matching is used to match the target: L GD As a perceived loss, where f D (x) represents a feature on the first fully connected layer of the discriminator; Since the encoded gene data reveals overexpression or underexpression trends of a certain group of genes, we chose not to use the logarithmic loss function for fitting; instead, we used the mean squared error loss to process these data. To ensure that the tumor image generated by G belongs to the corresponding category c, the function needs to be minimized: Then, the latent vector z can be sampled as z = μ + r⊙exp(ε), where r ~ N(0,I) is a random vector, and ⊙ denotes element-wise multiplication; after inputting z into the generator G, the synthesized tumor is obtained. The generator loss is calculated as follows: Where f D and f C These are the features of the first FC layer of discriminator D and classifier C, respectively.

3. The intelligent prediction system for lung cancer metastasis based on GCAVE-GAN and multimodal fusion according to claim 1, characterized in that, The image feature fusion module is designed to more accurately grasp the contextual regions of the tumor and the information provided by the tumor mask. This module includes a dual-path encoder and a boundary-guided contextual attention component. The aim is to fully utilize the rich information in the area surrounding the tumor and the morphological features of the tumor mask; In the dual-path encoder, the GCAVE-GAN method is first used to encode the tumor image and the mask image in a dual-branch manner, and then reassemble them into 3D blocks for depth analysis. Feature extraction of the tumor region uses the 3D-Unet model, which operates at two different scales of 1.5× and 2.0× to capture more details. For the mask branch, 3D-Unet is also used to extract boundary detail features. The context features, tumor features, and boundary features extracted by the dual-path encoder are denoted as M. c M t and M b Feature aggregation is achieved by guiding the contextual attention component through boundaries. First, extract context region features and tumor region features from the feature map obtained from the image branch, denoted as M. c and M t Extracting boundary features from mask branches, denoted as M b Employ an attention mechanism and Obtain the updated context features M c Boundary features M b ′, where α c α t The image is encoded to obtain contextual attention and tumor attention, α b Boundary attention is obtained after the mask passes through the encoder; To account for the different impacts of tumor context and boundary information on classification results, the proposed network enables the tumor to be classified in W... c +W b Adaptive learning W under the condition of =1 c and W b These represent the weights used in the fusion process, and the specific calculation process is as follows: Where F represents the operation after two fully connected layers and softmax, and ⊙ represents element-wise multiplication.

4. The intelligent prediction system for lung cancer metastasis based on GCAVE-GAN and multimodal fusion according to claim 1, characterized in that, The gene feature extraction module first preprocesses the sample data using Z-score normalization, using the expression value of each gene as a feature and the sample label as the target variable of the classifier. Then, it trains the classifier using weighted methods to focus more on samples that are frequently misclassified. Finally, it iterates multiple times to select the optimal gene subset, and uses the expression values ​​of these genes as the final features input into the model for feature extraction. Considering that the Adaboost algorithm gradually reduces the error rate in each iteration, it tends to combine multiple classifiers to improve classification accuracy and weights the training samples to reduce the impact of outliers, selecting the features with the highest discriminative power to improve robustness. Since the Adaboost model uses the Gini coefficient in the CART classification tree to select the optimal splitting attribute, the Gini coefficient can be used to measure the importance of features. To reduce model complexity, feature selection is achieved by inputting the original feature set into the Adaboost model, calculating the feature importance based on the Gini coefficient, and ranking the features according to their importance. Specifically, the importance of each feature on each tree is calculated and weighted to obtain the overall importance of the feature, as shown in the following formula: F imp (j)=ω k ×IMP j As can be seen from the above formula, the higher the feature importance, the greater its contribution to the model, and therefore the better the model performance; where ω k IMP represents the weights of the weak learner. j The importance of features in a single tree; In the formula, L-1 represents the number of non-leaf nodes in the tree, and ΔR represents the reduction in weighted impurity of non-leaf nodes during splitting. The larger the reduction value, the higher the importance of the feature. The formula for calculating the weighted impurity of node i is R. i =P i ×E i , where P i Let E be the node probability. i The impurity of node i is measured by the Gini coefficient Gini(i) as follows: The network employs a channel-level convolution mechanism to independently extract the genetic data of each patient. This mechanism ensures that each convolutional kernel can only convolve data from one patient at a time, allowing the network to independently extract the desired features from multiple patients without interference from others. In channel-oriented convolution operations, only one convolutional kernel is used for each channel. The genetic data of patient N is denoted as... Channel convolution kernel is denoted as Both have the same number of channels, and the feature map generated after the gene data is processed by the convolution kernel is denoted as... And perform the operation according to the following formula, where This is represented as a convolution operation; by convolving each patient's genetic data with a separate channel, the features of each patient do not interfere with each other; After multi-channel convolution, higher-level feature maps are learned, which reduces the number of parameters for network training, enables efficient extraction of useful features, and greatly improves network efficiency.

5. The intelligent prediction system for lung cancer metastasis based on GCAVE-GAN and multimodal fusion according to claim 1, characterized in that, The multimodal fusion module is designed to address the inconsistency of data across different modalities. It employs a channel-space attention-based cross-modal feature fusion (CFF) module to extract features from both image and non-image features. First, both types of features are globally pooled and average pooled along the channel dimension to extract more comprehensive features and focus on learning global information. Then, the four features are concatenated and fed into an MLP to obtain different attention values, which are mapped to [0,1] using activation functions. After segmentation, new feature weights are obtained, denoted as... and Image and non-image features after channel attention are denoted as F'image and F''. non-image The overall process is as follows: Where σ represents the sigmoid function, and ⊙ represents element-wise multiplication; Next, a spatial attention mechanism is used to learn more local information. First, F'image and F' non-imag After concatenation, the features are sequentially passed through convolutional layers, ReLU activation function, and another convolutional layer to generate global features F. Then, a split operation is performed to generate new spatial features with weights denoted as... and The obtained spatial feature weights The image features F'image after channel attention are multiplied element-wise to obtain F”image; the non-image features F' after channel attention are multiplied element-wise. non-image Spatial feature weights of image features Element-wise multiplication yields F” non-image Finally, F is generated through aggregation. fuse The overall process for predicting lung cancer metastasis is defined as follows: in The symbol represents concatenation, ⊙ represents element-wise multiplication, and the final output is denoted as F. fuse .

6. The intelligent prediction system for lung cancer metastasis based on GCAVE-GAN and multimodal fusion according to claim 1, characterized in that, The classifier module consists of three fully connected layers and a softmax function. The loss function is defined as cross-entropy loss, as shown in the following formula: Where y i The label p represents the sample. i This indicates the probability that a sample is predicted to be of the positive class. During the training process of this algorithm, the gradient descent method is used to calculate the minimum value of the loss function; This method gradually adjusts and optimizes the model parameters based on the obtained gradient direction; as training progresses, the model gradually converges and eventually achieves a relatively ideal classification prediction effect, significantly improving the accuracy and reliability of classification.

Images