Lung cancer early diagnosis system and method based on multi-mode deep learning

Abstract

The invention relates to the technical field of medical information, in particular to a lung cancer early diagnosis system and method based on multi-modal deep learning, and the system comprises a data acquisition module, an image preprocessing module, a feature extraction module electrically connected with the data acquisition module, and a data processing module electrically connected with the image preprocessing module. The system comprises a feature extraction module, a data preprocessing module electrically connected with the feature extraction module, a deep learning module electrically connected with the data preprocessing module, and a diagnosis result output module electrically connected with the deep learning module and used for receiving abstract features of a 3DResNet52 coding network and classification results of a 3DResNet34 decoding network; and generating a lung cancer early diagnosis report based on the abstract features and the classification result. Through deep fusion of multi-modal data, an innovative feature extraction technology and an advanced deep learning architecture, early-stage high-precision diagnosis of lung cancer is realized.

Description

Technical Field

[0001] The present invention relates to the field of medical information technology, and in particular to an early diagnosis system and method for lung cancer based on multimodal deep learning. Background Art

[0002] With the rapid development of medical imaging technology and gene sequencing technology, multimodal data plays an increasingly important role in the early diagnosis of lung cancer. Traditional lung cancer diagnosis methods mainly rely on single-modality imaging examinations, such as CT or PET scans. Although this method can detect lung abnormalities to a certain extent, it is often difficult to accurately distinguish early lung cancer from benign lesions, resulting in a high misdiagnosis rate.

[0003] In recent years, researchers have begun to try to combine CT and PET images for analysis to improve the accuracy of diagnosis. This method improves the sensitivity and specificity of diagnosis to a certain extent by fusing the anatomical information of CT and the functional metabolic information of PET. However, this simple image fusion method still has limitations, especially when dealing with complex early lung cancer cases, it is difficult to fully utilize the complementary information of the two imaging modalities.

[0004] With the deepening of genomic research, researchers have found that certain gene expression features are closely related to the occurrence and development of lung cancer. Therefore, some studies have begun to try to combine gene expression data with imaging data in order to obtain more comprehensive diagnostic information. However, how to effectively integrate these heterogeneous data and extract meaningful features from them is still a huge challenge.

[0005] In terms of data analysis, traditional machine learning methods, such as support vector machines (SVM) and random forests, have been applied to lung cancer diagnosis. These methods perform well when processing relatively simple data, but their performance is often limited when faced with high-dimensional, multimodal complex data. In particular, these methods are unable to extract and fuse deep features of data of different modalities.

[0006] Recently, deep learning technology has made breakthrough progress in the field of medical image analysis. Some researchers have begun to try to use convolutional neural networks (CNN) to analyze CT or PET images and have achieved certain results. However, most of these methods are still limited to the analysis of single-modality data and fail to fully utilize the advantages of multimodal data. In addition, existing deep learning models are often designed for two-dimensional images and are difficult to effectively process three-dimensional medical imaging data, resulting in some important spatial information being ignored.

[0007] Another key problem is that most existing methods use "black box" end-to-end learning and lack the explanation and analysis of intermediate features. Although this method may achieve high accuracy on certain data sets, its diagnostic process and basis are difficult to explain, which is not conducive to doctors' understanding and acceptance. In high-risk fields such as medical diagnosis, explainability is crucial.

[0008] In summary, the existing methods for early diagnosis of lung cancer still have significant deficiencies in multimodal data fusion, three-dimensional medical image processing, deep feature extraction, and model interpretability, and are unable to meet the urgent needs for accurate diagnosis of early lung cancer in clinical practice. Summary of the invention

[0009] The present invention aims to solve the above technical problems and provide a system and method for early diagnosis of lung cancer based on multimodal deep learning. The system innovatively integrates CT images, PET images and gene expression data, and adopts advanced 3D convolutional neural networks and multi-scale feature extraction technology to achieve high-precision diagnosis of early lung cancer.

[0010] The present invention proposes an early diagnosis system for lung cancer based on multimodal deep learning, comprising:

[0011] Data acquisition module, used to:

[0012] Obtain CT images, PET images, and gene expression data from patients;

[0013] Extracting lung three-dimensional volume data and lung two-dimensional slice data from the CT image and the PET image;

[0014] An image preprocessing module is electrically connected to the data acquisition module and is used to:

[0015] Receiving the CT image and the PET image;

[0016] Extracting lung nodule pixels in the CT image by threshold segmentation, and calculating the coordinates of the lung nodule pixels;

[0017] Based on the coordinates of the pulmonary nodule pixels, extracting corresponding pixels in the PET image;

[0018] Constructing a spherical ROI region surrounding the pulmonary nodule;

[0019] Multiple spheres were randomly selected in the non-lesion area as control ROI regions;

[0020] A feature extraction module is electrically connected to the image preprocessing module and is used to:

[0021] Extracting PET concentration and CTHU value in the ROI region and the control ROI region;

[0022] Calculating the ratio of the PET concentration to the CTHU value;

[0023] Based on the ratio, the tumor volume is calculated;

[0024] Extract the long-axis and short-axis information of the tumor;

[0025] Calculating a ratio of the PET concentration to the CTHU value that is greater than 1;

[0026] The ratio of the long axis, short axis and volume of the tumor was calculated to obtain the tumor characteristics;

[0027] Extracting corresponding gene expression data in the ROI region to obtain gene features;

[0028] A data preprocessing module is electrically connected to the feature extraction module and is used to:

[0029] receiving the tumor signature and the gene signature;

[0030] normalizing the tumor features and the gene features;

[0031] A deep learning module is electrically connected to the data preprocessing module and is used to:

[0032] Receiving the normalized tumor feature and the gene feature;

[0033] Based on a multi-scale and multi-view 3D convolutional neural network, feature fusion of the tumor features and the gene features;

[0034] Use 3DResNet52 encoding network to extract abstract features of multimodal data;

[0035] The 3DResNet34 decoding network was used to generate classification results for various lung cancer tumors;

[0036] A diagnosis result output module is electrically connected to the deep learning module and is used to:

[0037] Receiving the abstract features of the 3DResNet52 encoding network and the classification results of the 3DResNet34 decoding network;

[0038] An early diagnosis report of lung cancer is generated based on the abstract features and the classification results.

[0039] Preferably, the image preprocessing module comprises:

[0040] Spherical ROI construction unit, used for:

[0041] In the CT image, a surrounding sphere is constructed with the CT lesion as the sphere center;

[0042] Calculating the three-dimensional coordinates of the enclosing sphere;

[0043] Compare ROI building blocks for:

[0044] Randomly select 3 to 5 points in the non-lesion area of ​​the CT image as the center of the sphere;

[0045] constructing a control sphere with the same area as the enclosing sphere;

[0046] The three-dimensional coordinates of the control sphere are calculated.

[0047] Preferably, the feature extraction module comprises:

[0048] PET concentration extraction unit for:

[0049] In the PET image ROI area, the mean of the pixels is calculated to obtain the mean pixel;

[0050] Calculate the 3D distance and sort it from small to large to get the 10 pixel points with the smallest distance;

[0051] The average value of the 10 pixels is used as a new mean pixel;

[0052] The pixel value of the new mean pixel point is set as the CTHU value;

[0053] Calculate the PET concentration according to the total number of pixels in the ROI region image of the PET image and the CTHU value;

[0054] Tumor volume calculation unit for:

[0055] Get the total number of pixels N in the ROI area of ​​the image, where N = N x ×N y ×N z , N x 、N y 、N z The number of pixels on the X-axis, Y-axis, and Z-axis respectively; through the formula V = N × R × Voxel - size to calculate the tumor volume, where R is the ratio of the PET concentration to CTHU greater than 1, and Voxel_size is the voxel size.

[0056] Preferably, the deep learning module comprises:

[0057] Multi-scale feature extraction unit, used for:

[0058] Use 3D convolution kernels of different scales to extract local features from CT images, PET images, and genetic data;

[0059] Use 3D convolution kernels from different perspectives to extract global features from CT images, PET images, and genetic data;

[0060] Feature fusion unit, used for:

[0061] Performing weighted fusion on the local features and the global features;

[0062] Generate multimodal fusion features;

[0063] 3DResNet52 encoding unit, used for:

[0064] Receiving the multimodal fusion feature;

[0065] Feature encoding is performed through 5 convolutional unit modules, each of which includes a batch normalization layer, a 3DResNet group, and a maximum pooling layer;

[0066] Output the encoded abstract features;

[0067] 3DResNet34 decoding unit, used for:

[0068] receiving the encoded abstract features;

[0069] Feature decoding is performed through 4 convolutional unit modules, each of which includes a batch normalization layer, a 3DResNet group, and a deconvolution layer;

[0070] Output the classification results of various lung cancer tumors.

[0071] Preferably, the deep learning module further includes:

[0072] LSTM learning unit, used for:

[0073] Receiving the abstract features of the 3DResNet52 encoding unit and the classification results of the 3DResNet34 decoding unit;

[0074] Using an LSTM network to perform time series learning on the abstract features and the classification results;

[0075] Capture the temporal evolution of tumor characteristics;

[0076] Loss function calculation unit, used for:

[0077] Based on the classification result and the true label, calculate the cross entropy loss;

[0078] Optimization unit for:

[0079] Using Adam optimizer to optimize the parameters of the deep learning module;

[0080] Update the network weights via backpropagation.

[0081] As a preference, it also includes:

[0082] A data management module is electrically connected to the data acquisition module and is used to:

[0083] Store CT images, PET images and corresponding gene information obtained from the LIDC database;

[0084] The samples are divided into training set, validation set and test set in a ratio of 6:2:2;

[0085] A model training module, electrically connected to the data management module and the deep learning module, is used to:

[0086] Using the training set to train the deep learning module;

[0087] Using the validation set to adjust the trained model;

[0088] The model performance was evaluated using the test set.

[0089] Preferably, the feature extraction module further comprises:

[0090] Texture feature extraction unit, used for:

[0091] Extracting texture features from the CT image and the PET image;

[0092] Converting the texture features from the spatial domain to the frequency domain through a two-dimensional discrete cosine transform;

[0093] Get the frequency coefficient;

[0094] Graph Convolutional Network units, used for:

[0095] receiving the frequency coefficient;

[0096] Processed through multi-layer graph convolution modules;

[0097] Calculate attention coefficient and attention score;

[0098] Perform attention weighting on features;

[0099] The adjacency features are obtained through the ReLU activation function.

[0100] Preferably, the deep learning module adopts a BiSeNet fusion convolutional neural network structure, including: a spatial path for:

[0101] Fast downsampling to obtain spatial information;

[0102] The context path is used to:

[0103] Expand the receptive field through cascaded global average pooling;

[0104] Feature fusion module, used for:

[0105] Fuse the features of spatial path and context path;

[0106] Generate the final feature representation.

[0107] Preferably, the feature extraction module further comprises:

[0108] Gene clustering unit, used for:

[0109] performing cluster analysis on the gene expression data;

[0110] Calculate similarity measures for gene sets;

[0111] The gene expression data were clustered using the k-means method;

[0112] The gene feature clustering results were obtained.

[0113] The early diagnosis method of lung cancer based on multimodal deep learning includes the following steps:

[0114] S1, obtain the patient's CT images, PET images and gene expression data;

[0115] S2, preprocessing the CT image and the PET image, including:

[0116] Extract lung nodule pixels and calculate coordinates;

[0117] Construct a spherical ROI region surrounding the lung nodule;

[0118] Select a control ROI region in the non-lesion area;

[0119] S3, extract features, including:

[0120] The ratio of PET concentration to CTHU value was calculated;

[0121] Calculate tumor volume, long-axis, and short-axis information;

[0122] Extract genetic features;

[0123] S4, normalize the extracted features;

[0124] S5, uses multi-scale and multi-view 3D convolutional neural network for feature fusion;

[0125] S6, extract abstract features using 3DResNet52 encoding network;

[0126] S7, lung cancer tumor classification results were generated using the 3DResNet34 decoding network;

[0127] S8, uses LSTM to learn features in time series;

[0128] S9, model training and optimization based on cross entropy loss function and Adam optimizer;

[0129] S10, generate an early diagnosis report of lung cancer.

[0130] The beneficial effects of the present invention are reflected in multiple aspects. First, from a macroscopic perspective, the present invention establishes a complete multimodal early diagnosis framework for lung cancer, organically combining imaging examinations with genomic analysis, significantly improving the comprehensiveness and accuracy of diagnosis. This multimodal fusion method can not only capture the morphological characteristics of lung lesions, but also reflect their functional metabolic status and genetic background, providing a solid foundation for accurate diagnosis.

[0131] Secondly, at the data processing level, the present invention innovatively introduces spherical ROI construction technology and control ROI selection method. This method can not only accurately locate and analyze suspicious lesions, but also provide a control reference for healthy tissues, greatly improving the pertinence and comparability of feature extraction. Especially when dealing with early small lesions, this method shows significant advantages and can effectively reduce false positive results.

[0132] In terms of feature extraction, the present invention uses a multi-scale and multi-perspective 3D convolutional neural network, which can simultaneously capture local detail features and global context information. This design makes full use of the three-dimensional characteristics of medical images and can more comprehensively analyze the spatial structure and growth pattern of tumors compared to traditional 2D methods. In addition, by introducing LSTM for time series learning, the present invention can also effectively capture the dynamic changes of tumor characteristics, providing an important basis for early diagnosis and prognosis evaluation.

[0133] In terms of model structure, the present invention adopts a combination of 3DResNet52 encoding network and 3DResNet34 decoding network. This deep residual structure can not only extract richer features, but also effectively alleviate the gradient vanishing problem in deep network training, and improve the learning ability and generalization performance of the model. Especially when processing high-dimensional, high-complexity multimodal medical data, this structure shows excellent performance.

[0134] It is worth noting that while ensuring high accuracy, the present invention also focuses on the interpretability of the model. By introducing a feature fusion module based on the attention mechanism, the system can automatically identify and highlight important diagnostic features, making the diagnosis process more transparent and understandable. This feature not only helps doctors understand and verify the diagnosis results, but also provides an important reference for the subsequent formulation of personalized treatment plans.

[0135] In addition, the system of the present invention has good scalability and adaptability. Through modular design, the system can easily integrate new imaging modalities or genetic features to adapt to the evolving medical technology. At the same time, the system's training strategy and optimization method are also very flexible and can be adjusted according to different clinical needs and data characteristics.

[0136] In general, the present invention achieves high-precision early diagnosis of lung cancer through deep fusion of multimodal data, innovative feature extraction technology and advanced deep learning architecture. It not only overcomes many problems existing in the existing technology, but also makes significant progress in diagnostic accuracy, interpretability and clinical practicality, opening up a new way for early detection and precision treatment of lung cancer. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] Figure 1 It is a logic block diagram of the whole system of the present invention.

[0138] Figure 2 It is a logic block diagram of the image preprocessing module of the present invention.

[0139] Figure 3 It is a logic block diagram of the feature extraction module of the present invention.

[0140] Figure 4 This is a logical block diagram of the deep learning module of the present invention. DETAILED DESCRIPTION

[0141] The present invention provides a system and method for early diagnosis of lung cancer based on multimodal deep learning. The system achieves early and accurate diagnosis of lung cancer by integrating multimodal medical images and gene data and using advanced deep learning technology.

[0142] like Figure 1-4 As shown, the lung cancer early diagnosis system of the present invention includes a data acquisition module 1, an image preprocessing module 2, a feature extraction module 3, a data preprocessing module 4, a deep learning module 5 and a diagnosis result output module 6. These modules work together to form a complete diagnosis process.

[0143] The data acquisition module 1 is used to acquire the patient's CT images, PET images and gene expression data. Preferably, the CT and PET images include lung three-dimensional volume data and lung two-dimensional slice data. The combination of such multimodal data can provide more comprehensive lesion information, which helps to improve the accuracy of diagnosis. For example, CT images can provide clear anatomical structure information, while PET images can reflect the metabolic activity of tissues. The combination of the two can better identify and locate early lung cancer lesions.

[0144] The image preprocessing module 2 is electrically connected to the data acquisition module 1 and is used to preprocess the acquired image. The module first extracts the pulmonary nodule pixels in the CT image by threshold segmentation and calculates the coordinates of these pixels. In one embodiment of the present invention, the threshold can be set between -450HU and -600HU, which is usually able to effectively segment the pulmonary nodules. Then, based on these coordinates, the corresponding pixels are extracted in the PET image.

[0145] The image preprocessing module 2 also includes a sphere ROI construction unit 21 and a control ROI construction unit 22. The sphere ROI construction unit 21 constructs an enclosing sphere with the CT lesion as the center in the CT image, and calculates the three-dimensional coordinates of the sphere. The control ROI construction unit 22 randomly selects 3 to 5 points as the center in the non-lesion area of ​​the CT image, constructs a control sphere with the same area as the enclosing sphere, and calculates the three-dimensional coordinates of these control spheres. This innovative ROI extraction method can not only accurately locate the lesion area, but also provide a control for healthy tissue, which helps to better distinguish the characteristics of normal tissue and diseased tissue.

[0146] The feature extraction module 3 is electrically connected to the image preprocessing module 2 and is used to extract various features from the ROI area and the control ROI area. The module first extracts the PET concentration and the CTHU value, and then calculates the ratio of the two. This ratio is an important indicator because it combines the functional information of PET and the anatomical information of CT, which can more accurately characterize the nature of the lesion.

[0147] The feature extraction module 3 also includes a PET concentration extraction unit 31 and a tumor volume calculation unit 32. The PET concentration extraction unit 31 uses an innovative method to extract the PET concentration: first, the mean of the pixels in the PET image ROI area is calculated to obtain the mean pixel; then the 3D distance is calculated and sorted from small to large distance to obtain 10 pixels with smaller distance; then the average of these 10 pixels is used as the new mean pixel; finally, the pixel value of the new mean pixel is set as the CTHU value, and the PET concentration is calculated based on the total number of pixels and the CTHU value of the PET image ROI area image. This method can obtain the representative PET concentration value of the ROI area more stably and accurately.

[0148] The tumor volume calculation unit 32 calculates the tumor volume using the following formula:

[0149] V = N × R × Voxel_size,

[0150] Where V is the tumor volume, N is the total number of pixels in the ROI area image (N = N x ×N y ×N z , N x, N y ,N z are the number of pixels on the X-axis, Y-axis, and Z-axis respectively), R is the ratio of the PET concentration to CTHU that is greater than 1, and Voxel_size is the voxel size. This volume-based tumor feature extraction method can better describe the three-dimensional morphology and growth characteristics of the tumor.

[0151] Feature extraction module 3 also extracts the long and short axis information of the tumor, calculates the ratio of the PET concentration to the CTHU value greater than 1, and the ratio of the long axis, short axis and volume of the tumor, so as to obtain comprehensive tumor characteristics. In addition, the module also extracts the corresponding gene expression data in the ROI area to obtain gene features. This combination of imaging omics and genomics can explore the relationship between imaging features and gene expression, providing more information for accurate diagnosis.

[0152] The data preprocessing module 4 is electrically connected to the feature extraction module 3 and is used to normalize the extracted tumor features and gene features. Normalization can unify features of different scales to the same level, which is helpful for the subsequent training and prediction of the deep learning model.

[0153] The deep learning module 5 is electrically connected to the data preprocessing module 4 and is the core component of the system of the present invention. The module receives normalized tumor features and gene features, and fuses these features based on a multi-scale and multi-view 3D convolutional neural network. In a preferred embodiment of the present invention, the deep learning module 5 includes a multi-scale feature extraction unit 51, a feature fusion unit 52, a 3DResNet52 encoding unit 53, and a 3DResNet34 decoding unit 54.

[0154] The multi-scale feature extraction unit 51 uses 3D convolution kernels of different scales to extract local features from CT images, PET images, and genetic data, and uses 3D convolution kernels of different perspectives to extract global features. This multi-scale and multi-perspective design can capture information of different scales and angles, thereby comprehensively analyzing complex lung imaging data. For example, a smaller convolution kernel (such as 3x3x3) can capture local details, while a larger convolution kernel (such as 7x7x7) can obtain a wider range of contextual information.

[0155] The feature fusion unit 52 performs weighted fusion on the extracted local features and global features to generate multimodal fusion features. Preferably, the present invention adopts an attention mechanism to perform feature weighting, which can adaptively adjust the importance of different features, thereby improving the representativeness and discrimination ability of the fusion features.

[0156] The 3DResNet52 encoding unit 53 receives the multimodal fusion features and performs feature encoding through 5 convolution unit modules. Each convolution unit includes a batch normalization layer, a 3DResNet group and a maximum pooling layer. This deep residual structure can effectively alleviate the gradient vanishing problem and improve the training effect and feature extraction capability of the network. In one embodiment of the present invention, the convolution kernel size of the 3DResNet group can be set to 3x3x3 with a step size of 1, and the pooling kernel size of the maximum pooling layer can be set to 2x2x2 with a step size of 2. This setting can gradually reduce the spatial dimension of the feature map while maintaining the feature extraction capability and improve computational efficiency.

[0157] The 3DResNet34 decoding unit 54 receives the encoded abstract features and performs feature decoding through 4 convolution unit modules. Each convolution unit includes a batch normalization layer, a 3DResNet group and a deconvolution layer. The deconvolution layer can gradually restore the spatial dimension of the feature map and finally output the classification results of multiple lung cancer tumors. Preferably, the convolution kernel size of the deconvolution layer can be set to 4x4x4 with a step size of 2, so that the spatial dimension of the feature map can be doubled in each operation.

[0158] The system of the present invention also introduces an LSTM learning unit 55, a loss function calculation unit 56 and an optimization unit 57 in the deep learning module 5, which further improves the performance and training effect of the model. The LSTM learning unit 55 receives the abstract features of the 3DResNet52 encoding unit and the classification results of the 3DResNet34 decoding unit, and uses the LSTM network to perform time series learning on these features and results to capture the time evolution law of tumor characteristics. This design can make full use of the patient's historical examination data and improve the accuracy and reliability of diagnosis.

[0159] The loss function calculation unit 56 calculates the cross entropy loss based on the classification result and the true label. Preferably, the present invention adopts a weighted cross entropy loss function to solve the possible class imbalance problem. The loss function can be expressed as:

[0160]

[0161] Among them, C is the number of categories, w i is the weight of the i-th class, y i is the true label, is the prediction probability. The optimization unit 57 uses the Adam optimizer to optimize the parameters of the deep learning module and updates the network weights by back propagation. The Adam optimizer combines the advantages of the momentum method and the adaptive learning rate method, and can converge to the global optimal solution faster. Preferably, the present invention sets the initial learning rate to 0.001, β1 to 0.9, β2 to 0.999, and ∈ to 1e-8. These parameter settings can usually achieve good results in various deep learning tasks.

[0162] In order to further improve the practicality and reliability of the system, the present invention also includes a data management module 7 and a model training module 8. The data management module 7 is electrically connected to the data acquisition module 1 and is used to store CT images, PET images and corresponding gene information acquired from the LIDC database. Preferably, the present invention divides the samples into a training set, a validation set and a test set in a ratio of 6:2:2. This division ratio can ensure that there is enough data for model training, while retaining enough data for model verification and final performance evaluation.

[0163] The model training module 8 is electrically connected to the data management module 7 and the deep learning module 5, and is responsible for the training, adjustment and evaluation process of the model. The module first uses the training set to train the deep learning module, then uses the validation set to adjust the trained model, and finally uses the test set to evaluate the model performance. In one embodiment of the present invention, an early stopping strategy is used to prevent overfitting, that is, when the performance on the validation set has not improved for 5 consecutive epochs, the training process is stopped. In addition, the present invention also adopts a learning rate decay strategy, reducing the learning rate to 0.1 times the original every 10 epochs to help the model better converge to the local optimal solution.

[0164] The deep learning module 5 of the present invention adopts the BiSeNet fusion convolutional neural network structure, which is designed to balance the extraction of spatial information and contextual information, thereby achieving more accurate early diagnosis of lung cancer. The BiSeNet structure includes a spatial path, a contextual path, and a feature fusion module, and these three parts work together to effectively improve the performance of the model.

[0165] The spatial path is mainly responsible for fast downsampling to obtain spatial information. In a preferred embodiment of the present invention, the spatial path contains three convolutional layers, each followed by a batch normalization layer and a ReLU activation function. The convolution kernel size is set to 3x3 with a step size of 2, which can halve the spatial size of the feature map at each layer while retaining important spatial detail information. For example, for an image with an input size of 224x224x3, after three layers of convolution, the feature map size becomes 28x28x64.

[0166] The context path expands the receptive field through cascaded global average pooling. The present invention uses a variant structure of ResNet in the context path, which includes multiple residual blocks. Each residual block contains two 3x3 convolutional layers, followed by a batch normalization layer and a ReLU activation function. Between residual blocks, convolution with a step size of 2 is used for downsampling. This design can obtain a wider range of contextual information while maintaining computational efficiency. Preferably, the present invention adds a global average pooling layer at the end of the context path to compress the feature map into a 1x1xC vector, where C is the number of channels.

[0167] The feature fusion module is a key part of the BiSeNet structure, which is used to fuse the features of the spatial path and the context path to generate the final feature representation. The present invention uses an attention mechanism to adaptively adjust the importance of the two path features. Specifically, the feature fusion module first upsamples the features of the context path so that they have the same spatial dimension as the features of the spatial path. Then, attention weights are generated through 1x1 convolution and sigmoid activation function, and these weights are applied to the fused features. This approach allows the model to learn to flexibly focus on spatial details or global context in different areas.

[0168] In order to further improve the effect of feature extraction, the feature extraction module 3 of the present invention also includes a texture feature extraction unit 33 and a graph convolution network unit 34. The texture feature extraction unit 33 extracts texture features from CT images and PET images. These features can reflect the microscopic structural information of the tissue and are of great significance for the detection of early lung cancer. Preferably, the present invention uses a gray level co-occurrence matrix (GLCM) method to extract texture features, including statistics such as contrast, correlation, energy and homogeneity.

[0169] After extracting texture features, the present invention innovatively introduces a two-dimensional discrete cosine transform (DCT) to convert features from the spatial domain to the frequency domain. The DCT transform can compress image information into a few DCT coefficients, which is beneficial for subsequent feature processing. The frequency coefficients obtained after the transform are input into the graph convolutional network unit 34 for further processing.

[0170] The graph convolution network unit 34 adopts a multi-layer graph convolution module structure to capture the spatial relationship between features. In one embodiment of the present invention, the graph convolution network includes three layers of graph convolution layers, each followed by a batch normalization layer and a ReLU activation function. The graph convolution operation can be expressed as:

[0171]

[0172] Among them, H (l) represents the node feature matrix of the lth layer, A is the adjacency matrix, D is the degree matrix, W (l)is a learnable weight matrix, and σ is an activation function. After the output of the graph convolutional network is average pooled and reduced in dimension, it is input into the attention calculation module. The attention calculation module uses a soft attention mechanism to calculate the importance score of each node. The calculation formula of the attention score is as follows:

[0173] e i =v T tanh(Wh i +b),

[0174]

[0175] Among them, h i is the feature vector of node i, W and b are learnable parameters, and v is the attention vector. Here, e i represents the unnormalized attention score of node i, and a i is the attention weight normalized by the softmax function, which indicates the relative importance of node i among all nodes. This mechanism allows the model to focus on more important nodes when processing information.

[0176] Finally, the graph convolutional network unit 34 performs attention weighting on the features and obtains the final adjacent features through the ReLU activation function. This feature extraction method based on graph convolution and attention mechanism can effectively capture the local structural information and global context information in lung images and improve the expressiveness of features.

[0177] The present invention also introduces a gene clustering unit 35 in the feature extraction module 3 for clustering the gene expression data. First, the gene clustering unit 35 calculates the similarity measure of the gene set, preferably using the Pearson correlation coefficient as the similarity measure indicator. Then, the gene expression data is clustered using the k-means method to obtain the gene feature clustering result. The objective function of the k-means algorithm can be expressed as:

[0178]

[0179] Among them, k is the number of clusters, C i is the i-th cluster, μ iis the center of the ith cluster. Preferably, the present invention determines the optimal number of clusters k by the elbow rule, that is, by plotting the sum of squared errors (SSE, Sum of Squared Errors) curve under different k values, and selecting the k value at the inflection point of the curve. The elbow rule refers to the point where the decrease in SSE begins to decrease significantly when a cluster is added. This point is like the elbow of an arm, hence the name. After this point, even if the number of clusters is increased, the reduction in SSE becomes very limited, indicating that additional clusters no longer provide substantial information gain. Therefore, the elbow rule helps avoid overfitting and ensures that the model has good generalization ability.

[0180] Through gene clustering, the present invention can group genes with similar expression patterns together, thereby reducing data dimensions and extracting more representative gene features. This method can not only reduce computational complexity, but also discover potential gene function modules, providing more valuable biological information for subsequent lung cancer diagnosis.

[0181] Finally, the present invention proposes a method for early diagnosis of lung cancer based on multimodal deep learning, which corresponds to the above system and includes the following steps:

[0182] S1, obtain the patient's CT images, PET images and gene expression data;

[0183] S2, preprocessing the CT images and PET images, including extracting the pixels of the lung nodules and calculating the coordinates, constructing a spherical ROI region surrounding the lung nodules, and selecting a control ROI region in the non-lesion area;

[0184] S3, feature extraction, including calculating the ratio of PET concentration to CTHU value, calculating tumor volume, long axis and short axis information, and extracting gene features;

[0185] S4, normalize the extracted features;

[0186] S5, uses multi-scale and multi-view 3D convolutional neural network for feature fusion;

[0187] S6, extract abstract features using 3DResNet52 encoding network;

[0188] S7, lung cancer tumor classification results were generated using the 3DResNet34 decoding network;

[0189] S8, uses LSTM to learn features in time series;

[0190] S9, model training and optimization based on cross entropy loss function and Adam optimizer;

[0191] S10, generate an early diagnosis report of lung cancer.

[0192] This method achieves early and accurate diagnosis of lung cancer through multimodal data fusion, innovative feature extraction technology and advanced deep learning architecture. Compared with traditional methods, the method of the present invention can more comprehensively analyze the imaging and genomic characteristics of patients and improve the accuracy and reliability of diagnosis. In addition, the method of the present invention has good scalability and adaptability, and the parameters and algorithms of each step can be flexibly adjusted according to actual needs to adapt to different clinical scenarios.

[0193] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An early diagnosis system for lung cancer based on multimodal deep learning, characterized in that: include: Data acquisition module, used to: Obtain CT images, PET images, and gene expression data from patients; Extracting lung three-dimensional volume data and lung two-dimensional slice data from the CT image and the PET image; An image preprocessing module is electrically connected to the data acquisition module and is used to: Receiving the CT image and the PET image; Extracting lung nodule pixels in the CT image by threshold segmentation, and calculating the coordinates of the lung nodule pixels; Based on the coordinates of the pulmonary nodule pixels, extracting corresponding pixels in the PET image; Constructing a spherical ROI region surrounding the pulmonary nodule; Multiple spheres were randomly selected in the non-lesion area as control ROI regions; A feature extraction module is electrically connected to the image preprocessing module and is used to: Extracting PET concentration and CTHU value in the ROI region and the control ROI region; Calculating the ratio of the PET concentration to the CTHU value; Based on the ratio, the tumor volume is calculated; Extract the long-axis and short-axis information of the tumor; Calculating a ratio of the PET concentration to the CTHU value that is greater than 1; The ratio of the long axis, short axis and volume of the tumor was calculated to obtain the tumor characteristics; Extracting corresponding gene expression data in the ROI region to obtain gene features; A data preprocessing module is electrically connected to the feature extraction module and is used to: receiving the tumor signature and the gene signature; normalizing the tumor features and the gene features; A deep learning module is electrically connected to the data preprocessing module and is used to: Receiving the normalized tumor feature and the gene feature; Based on a multi-scale and multi-view 3D convolutional neural network, feature fusion of the tumor features and the gene features; Use 3DResNet52 encoding network to extract abstract features of multimodal data; The 3DResNet34 decoding network was used to generate classification results for various lung cancer tumors; A diagnosis result output module is electrically connected to the deep learning module and is used to: Receiving the abstract features of the 3DResNet52 encoding network and the classification results of the 3DResNet34 decoding network; An early diagnosis report of lung cancer is generated based on the abstract features and the classification results.

2. The early diagnosis system for lung cancer based on multimodal deep learning according to claim 1, characterized in that: The image preprocessing module comprises: Spherical ROI construction unit for: In the CT image, a surrounding sphere is constructed with the CT lesion as the sphere center; Calculating the three-dimensional coordinates of the enclosing sphere; Compare ROI building blocks for: Randomly select 3 to 5 points in the non-lesion area of ​​the CT image as the center of the sphere; constructing a control sphere with the same area as the enclosing sphere; The three-dimensional coordinates of the control sphere are calculated.

3. The early diagnosis system for lung cancer based on multimodal deep learning according to claim 1, characterized in that: The feature extraction module comprises: PET concentration extraction unit for: In the PET image ROI area, the mean of the pixels is calculated to obtain the mean pixel; Calculate the 3D distance and sort it from small to large to get the 10 pixel points with the smallest distance; The average value of the 10 pixels is used as a new mean pixel; The pixel value of the new mean pixel point is set as the CTHU value; Calculate the PET concentration according to the total number of pixels in the ROI region image of the PET image and the CTHU value; Tumor volume calculation unit for: Get the total number of pixels N in the ROI area of ​​the image, where N = N x ×N y ×N z , N x 、N y 、N z are the number of pixels on the X-axis, Y-axis, and Z-axis respectively; the tumor volume is calculated by the formula V=N×R×Voxel_size, where R is the ratio of the PET concentration to CTHU greater than 1, and Voxel_size is the voxel size.

4. The early diagnosis system for lung cancer based on multimodal deep learning according to claim 1, characterized in that: The deep learning module includes: Multi-scale feature extraction unit, used for: Use 3D convolution kernels of different scales to extract local features from CT images, PET images, and genetic data; Use 3D convolution kernels from different perspectives to extract global features from CT images, PET images, and genetic data; Feature fusion unit, used for: Performing weighted fusion on the local features and the global features; Generate multimodal fusion features; 3DResNet52 encoding unit, used for: Receiving the multimodal fusion feature; Feature encoding is performed through 5 convolutional unit modules, each of which includes a batch normalization layer, a 3DResNet group, and a maximum pooling layer; Output the encoded abstract features; 3DResNet34 decoding unit, used for: receiving the encoded abstract features; Feature decoding is performed through 4 convolutional unit modules, each of which includes a batch normalization layer, a 3DResNet group, and a deconvolution layer; Output the classification results of various lung cancer tumors.

5. The early diagnosis system for lung cancer based on multimodal deep learning according to claim 4, characterized in that: The deep learning module also includes: LSTM learning unit, used for: Receiving the abstract features of the 3DResNet52 encoding unit and the classification results of the 3DResNet34 decoding unit; Using an LSTM network to perform time series learning on the abstract features and the classification results; Capture the temporal evolution of tumor characteristics; Loss function calculation unit, used for: Based on the classification result and the true label, calculate the cross entropy loss; Optimization unit for: Using Adam optimizer to optimize the parameters of the deep learning module; Update the network weights via backpropagation.

6. The early diagnosis system for lung cancer based on multimodal deep learning according to claim 1, characterized in that: Also includes: A data management module is electrically connected to the data acquisition module and is used to: Store CT images, PET images and corresponding gene information obtained from the LIDC database; The samples are divided into training set, validation set and test set in a ratio of 6:2:2; A model training module, electrically connected to the data management module and the deep learning module, is used to: Using the training set to train the deep learning module; Using the validation set to adjust the trained model; The model performance was evaluated using the test set.

7. The early diagnosis system for lung cancer based on multimodal deep learning according to claim 1, characterized in that: The feature extraction module also includes: Texture feature extraction unit, used for: Extracting texture features from the CT image and the PET image; Converting the texture features from the spatial domain to the frequency domain through a two-dimensional discrete cosine transform; Get the frequency coefficient; Graph Convolutional Network units, used for: receiving the frequency coefficient; Processed through multi-layer graph convolution modules; Calculate attention coefficient and attention score; Perform attention weighting on features; The adjacency features are obtained through the ReLU activation function.

8. The early diagnosis system for lung cancer based on multimodal deep learning according to claim 1, characterized in that: The deep learning module adopts the BiSeNet fusion convolutional neural network structure, including: Spatial paths, used to: Fast downsampling to obtain spatial information; The context path is used to: Expand the receptive field through cascaded global average pooling; Feature fusion module, used for: Fuse the features of spatial path and context path; Generate the final feature representation.

9. The early diagnosis system for lung cancer based on multimodal deep learning according to claim 1, characterized in that: The feature extraction module also includes: Gene clustering unit, used for: performing cluster analysis on the gene expression data; Calculate similarity measures for gene sets; The gene expression data were clustered using the k-means method; The gene feature clustering results were obtained.

10. A method for early diagnosis of lung cancer based on multimodal deep learning, using the system according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1, obtain the patient's CT images, PET images and gene expression data; S2, preprocessing the CT image and the PET image, including: Extract lung nodule pixels and calculate coordinates; Construct a spherical ROI region surrounding the lung nodule; Select a control ROI region in the non-lesion area; S3, extract features, including: The ratio of PET concentration to CTHU value was calculated; Calculate tumor volume, long-axis, and short-axis information; Extract genetic features; S4, normalize the extracted features; S5, uses multi-scale and multi-view 3D convolutional neural network for feature fusion; S6, extract abstract features using 3DResNet52 encoding network; S7, lung cancer tumor classification results were generated using the 3DResNet34 decoding network; S8, uses LSTM to perform temporal learning on features; S9, model training and optimization based on cross entropy loss function and Adam optimizer; S10, generating an early diagnosis report of lung cancer.

Images