Lung cancer diagnosis method and equipment

Abstract

The invention relates to the technical field of medical artificial intelligence, in particular to a lung cancer diagnosis method, which comprises the following steps: data acquisition and preprocessing: acquiring CT image data and text data, and preprocessing the data, including image normalization, text standardization and weighting processing; image feature extraction: based on a pre-trained ResNet50 network, deep features of the CT image are extracted, and key features are enhanced through an attention mechanism; text feature processing: performing feature extraction on the preprocessed text data to generate a text feature vector; multi-modal feature fusion: fusing the image features and the text features, and extracting fusion features through a full connection layer and a regularization layer; and classification: model integration. Through the method, the contribution of each index to the classification result is clearer, so that the prediction stability and generalization ability are improved, the accuracy rate, the precision rate, the recall rate, the F1 score and the discrimination ability of the model are comprehensively evaluated, and the model effect is conveniently and objectively judged.

Description

Technical Field

[0001] The present invention relates to the technical field of medical artificial intelligence, and particularly to a method for diagnosing lung cancer. Background Art

[0002] Lung cancer has become the most dangerous type of all cancers due to its highest mortality rate, and early detection can save many lives. [1] Among cancers, lung cancer has become the second most common cancer after prostate cancer in men and breast cancer in women [2]. The International Agency for Research on Cancer (IARC) estimated the number of lung cancers in the United States in 2020 as follows [3]: approximately 235,760 new cases of lung cancer (119,100 in men and 116,660 in women), and approximately 131,880 people died from lung cancer (69,410 in men and 62,470 in women). Accordingly, lung cancer remains the leading cause of cancer death globally, and delayed diagnosis is the main factor leading to adverse outcomes and remains a key challenge that needs to be overcome. Although the debate around cancer screening for asymptomatic high-risk populations continues, rapid access to relevant diagnostic tests is still very necessary [4].

[0003] In reality, the diagnosis of lung cancer by actual doctors is not only based on CT images, but is a comprehensive judgment made by integrating various test indicators. These test indicators respectively reflect the body functions and health conditions from various aspects and have important guiding and reference roles for identification and diagnosis. As will be mentioned below, neuron-specific enolase (NSE) in the nuclear protein tumor chip detection is one of the enolases participating in the glycolysis pathway and exists in nerve tissues and neuroendocrine tissues. It has been found that there is overexpression of NSE in tumors related to the origin of neuroendocrine tissues, especially in small cell lung cancer (SCLC), resulting in a significant increase in serum NSE. Obviously, this indicator will become a good specific factor indicating lung cancer.

[0004] This part of information has not received attention or has not been fully comprehensively investigated and utilized in various lung cancer diagnosis models. Based on this, a comprehensive diagnostic method integrating multi-modal information may be a necessary and beneficial attempt, and a good multi-modal model will be closer to the actual diagnosis process and obtain better results.

[0005] The treatment of lung cancer depends on the type and stage of the cancer found in the patient [5]. Computed tomography (CT) and magnetic resonance imaging (MRI) are standard medical procedures for early detection, which can improve the survival rate of patients [6-7]. Previous intelligent methods used hand-drawn feature extraction methods, such as the sequential flood feature selection algorithm (SFFSA) or genetic algorithm (GA), which could help generate optimal features [8]. In recent years, CAD systems have used deep learning techniques for automatic extraction of image features [9]. Due to the use of deep learning techniques, many medical image processing programs have achieved success

[10] . Table 1 lists the accuracy rates of various algorithm models currently used for the early diagnosis of lung cancer.

[0006] Table 1 Comparative evaluation table of performance (accuracy rate %)

[11]

[0007]

[0008]

[0009] Meanwhile, Han et al.

[27] found that indicators such as ProGRP, NSE, CYFRA21-1, SCC, and HE4 have certain clinical value for the diagnosis of cancer and its pathological types, and the combined detection of each indicator is more valuable than single indicator detection. Cao et al.

[28] demonstrated that the levels of serum MMP-9, CYFRA21-1, SCC, NSE, and CEA in cancer patients were significantly elevated, which has important value for the diagnosis of peripheral cancer. The combined detection of the above indicators can improve the diagnostic accuracy of peripheral cancer in clinical practice. Birinci et al.

[29] proved that neuron-specific enolase has become the most commonly used tumor marker in pediatric hematology and oncology clinics. Compared with other tumor markers (cancer antigen 19-9, carcinoembryonic antigen, cancer antigen 15-3, etc.), the diagnostic rate of NSE detection is the highest. Neuron-specific enolase plays an important role as a tumor marker in the diagnosis of neuroendocrine tumors and small cell lung cancer. Chen et al.

[30] demonstrated through experiments that the combined application of tumor markers can improve the efficacy of differentiating small cell lung cancer, and established a prediction model for the differential diagnosis of small cell lung cancer, which has a good diagnostic effect. In addition, Li et al.

[31] found that the levels of serum TuM2 PK, CEA, and CYFRA21-1 have high clinical value in the diagnosis of non-small cell lung cancer and can effectively judge the prognosis of patients. In summary, according to biological theory and the aforementioned experiments, the current lung cancer-related test indicators do have a certain promoting effect on lung cancer identification.

[0010] Through literature and field research, the following deficiencies still exist in the current field of lung cancer recognition: First, the current lung cancer recognition models usually only include one modality, CT, and lack the support of multi-modal blood test indicators, and there are still certain deficiencies between the obtained results and the actual diagnosis of doctors; Second, traditional multi-modal fusion methods use simple splicing or weighted averaging and cannot dynamically capture cross-modal feature associations; Third, most of the current recognition models use a single model without ensemble learning, and when facing data imbalance or differentiation problems, the results may lack stability, while ensemble learning helps to better handle imbalanced classification problems. Summary of the Invention

[0011] The purpose of the present invention is to solve the problems pointed out in the background technology, and to propose a lung cancer diagnosis method and device.

[0012] To achieve the above purpose, the present invention adopts the following technical solutions:

[0013] A lung cancer diagnosis method includes the following steps:

[0014] Data collection and preprocessing: Collect CT image data and text data, and preprocess the data, including image normalization, text standardization and weighting;

[0015] Image feature extraction: Based on the pre-trained ResNet50 network, extract the deep features of the CT image, and enhance the key features through the attention mechanism;

[0016] Text feature processing: Extract features from the preprocessed text data to generate text feature vectors;

[0017] Multi-modal feature fusion: Fuse the image features and text features, and extract the fusion features through the fully connected layer and the regularization layer;

[0018] Classification: Classify lung cancer based on the fusion features, and output the probabilities of lung cancer and non-lung cancer;

[0019] Model integration: Train multiple models and average the prediction results for fusion.

[0020] A lung cancer diagnosis device includes a computer device, and the computer device at least includes a processor and a memory, and the computer device is used to execute the lung cancer diagnosis method.

[0021] The beneficial effects of the lung cancer diagnosis method proposed by the present invention are as follows:

[0022] This method adopts multi-modal information fusion to improve the robustness and performance of the model, making the contributions of various indicators to the classification results more explicit, so as to enhance the prediction stability and generalization ability, reduce the possible overfitting risk of a single model, ensure that the model maintains a good convergence speed and performance during training, prevent overfitting and training stagnation, comprehensively evaluate the accuracy, precision, recall, F1 score and discriminant ability of the model, and display the results in graphical and text forms for objective judgment of the model performance.

[0023] A lung cancer diagnosis device proposed by the present invention has the beneficial effect that the stability and accuracy of lung cancer diagnosis can be effectively improved through this device. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall system flowchart of the present invention;

[0025] Figure 2 It is the overall system architecture diagram of the present invention;

[0026] Figure 3 It is the attention mechanism structure diagram of the ResNet-Attention model of the present invention;

[0027] Figure 4 It is the neural network structure diagram of the ResNet-Attention model of the present invention for fusing image and text data;

[0028] Figure 5 It is the heat map of the correlation of multi-modal text information of the present invention;

[0029] Figure 6 It is the ROC curve diagram of the ResNet-Attention model of the present invention;

[0030] Figure 7 It is the ROC curve diagram of the ResNet-Attention ensemble learning model of the present invention;

[0031] Figure 8 It is the comparison diagram between the multi-modal ResNet-Attention model and the single-modal model of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0033] Referring to Figures 1-8 , a lung cancer diagnosis method includes the following steps:

[0034] Data collection and preprocessing: Collect CT image data and text data, and preprocess the data, including image normalization, text standardization, and weighting;

[0035] Image feature extraction: Based on the pre-trained ResNet50 network, extract the deep features of CT images, and enhance the key features through the attention mechanism;

[0036] Text feature processing: Extract features from the preprocessed text data to generate text feature vectors;

[0037] Multi-modal feature fusion: Fuse the image features and text features, and extract the fused features through the fully connected layer and the regularization layer;

[0038] Classification: Classify lung cancer based on the fused features, and output the probabilities of lung cancer and non-lung cancer;

[0039] Model integration: Train multiple models and average-fuse the prediction results.

[0040] Make the contributions of each index to the classification result more explicit to improve the prediction stability and generalization ability, and reduce the possible overfitting risk of a single model; Ensure that the model maintains a good convergence speed and performance during training, prevent overfitting and training stagnation, comprehensively evaluate the accuracy, precision, recall, F1 score, and discriminative ability of the model, and display the results in graphical and text forms for an objective judgment of the model effect.

[0041] As an implementation:

[0042] Data collection and preprocessing

[0043] Image data collection

[0044] The data directory structure is divided into three parts: training, validation, and testing; Each category (lung cancer and non-lung cancer) is stored in a separate folder.

[0045] Use preprocess_input in Keras to perform normalization preprocessing on the original image data to ensure that the images meet the input requirements of ResNet50 (size 224×224×3).

[0046] Text data collection

[0047] Each image folder is accompanied by a CSV file containing text features (such as image descriptions, pathology reports, etc.).

[0048] Read the data in the CSV file, perform standardization processing on the text features using StandardScaler, and then weight the text data using the preset importance weights to obtain the text data feature vectors.

[0049] 2. Model Architecture Design

[0050] Image Feature Extraction Module

[0051] Use the pre-trained ResNet50 as the basic network, remove the original fully connected layer, and extract deep image features.

[0052] Based on ResNet50, introduce the attention block.

[0053] The attention block includes channel attention: first use global average pooling to obtain channel statistical information, generate attention weights after two fully connected layers and ReLU activation and Sigmoid normalization, and then multiply with the original features pixel by pixel;

[0054] Spatial attention: use 7×7 convolution to obtain the spatial attention map, and then multiply with the original features;

[0055] Finally, add the two parts of attention features to get the final attention-enhanced feature.

[0056] Flatten, batch normalize the features after attention, and extract high-dimensional representations through several fully connected layers as the output of the image end.

[0057] Text Feature Processing Module

[0058] Take the preprocessed and weighted text vectors as text features as input, and extract hidden features through simple fully connected layers or other feature transformation layers.

[0059] Multi-modal Feature Fusion and Classification

[0060] Fuse the features at both the image and text ends through the Concatenate layer.

[0061] The fused features go through several fully connected layers (optional regularization measures such as Dropout and BatchNormalization), and finally connect to the Softmax layer or Sigmoid layer to output the class probability and achieve the prediction classification of lung cancer.

[0062] As an implementation:

[0063] 1. Data Loading and Preprocessing Process

[0064] Image Part

[0065] Traverse the training, validation, and test directories, load the pictures under each category, and call the pre-set image preprocessing function to normalize the pictures.

[0066] Use the preprocess_input preprocessing method in Keras to ensure that the image data meets the input requirements of ResNet50.

[0067] Text part

[0068] Read the text data from the CSV files corresponding to each category.

[0069] Normalize the text features using StandardScaler, and then multiply them element-wise with the preset importance weights to obtain the final input text features.

[0070] 2. Model construction and training

[0071] Construct the image feature extraction branch, call the ResNet50 model and connect the custom attention module.

[0072] Construct the text feature processing branch and project the text features through a fully connected layer.

[0073] Concatenate the features output by both networks to form a complete multi-modal feature vector.

[0074] Add a fully connected layer and a Dropout layer for feature fusion and dimensionality reduction, and finally use Softmax to output the class probabilities.

[0075] During the training process, use the cross-entropy loss function, and at the same time set callback functions such as EarlyStopping and ReduceLROnPlateau to improve the model convergence performance.

[0076] To improve the model robustness, a model ensemble strategy can be adopted, and after training multiple models, take the average of the prediction results.

[0077] 3. Model evaluation

[0078] After obtaining the prediction results on the test set, convert the output probabilities into prediction labels through a threshold.

[0079] Call sklearn.metrics.classification_report to generate index reports such as precision, recall, and F1-score.

[0080] Use the ROC curve and calculate the AUC value to evaluate and visually display the overall performance of the model.

[0081] As an implementation method:

[0082] Specific implementation environment

[0083] Hardware requirements (can be changed):

[0084] GPU: NVIDIA RTX 4060 (24GB video memory) or higher, supporting CUDA 11.6+.

[0085] CPU: Intel i7-12700K (12 cores / 20 threads) or higher.

[0086] Memory: 64GB DDR4 3600MHz.

[0087] Storage: 1TB NVMe SSD (for high-speed data reading).

[0088] Software dependencies (changeable):

[0089] Operating system: Ubuntu 22.04 LTS or Windows 11 Pro.

[0090] Python version: 3.9 (it is recommended to use Anaconda to manage the environment). Specific embodiments

[0092] 1. Overall program structure and description of dependent libraries

[0093] A large number of dependent libraries for deep learning, image processing, data analysis and evaluation are introduced at the beginning of the system. Their main uses are as follows:

[0094] os, time: For file path operations, file traversal, and measuring running time, etc.

[0095] cv2 (OpenCV): Mainly used for image reading, preprocessing, image cropping and resizing operations.

[0096] numpy: For array and matrix operations, saving image data, and numerical calculations.

[0097] pandas: For reading CSV files and loading text feature data into DataFrame.

[0098] keras and tensorflow: Implement deep learning neural network models, use the pre-trained ResNet50 to extract image features, and build attention modules and multi-modal fusion networks.

[0099] sklearn: Mainly used for calculating classification reports, ROC curves, AUC metrics, data standardization, etc.

[0100] matplotlib.pyplot: For plotting statistical result graphs such as ROC curves.

[0101] In addition, some Keras modules are introduced, such as model construction, layers (Dense, Flatten, BatchNormalization, Dropout, Input, GlobalAveragePooling2D, Concatenate, Multiply, Add, Reshape, Conv2D, etc.), optimizers (Adam), loss functions (categorical_crossentropy), and callback functions (EarlyStopping, ModelCheckpoint, and ReduceLROnPlateau) for monitoring and dynamically adjusting the learning rate during the training process.

[0102] 2. Implementation of the Attention Block

[0103] First, a function named attention_block is defined in the system, and its main function is to enhance the attention mechanism for the feature map of the convolutional layer. This part can be divided into two parts:

[0104] 2.1 Channel Attention

[0105] Use GlobalAveragePooling2D to perform global average pooling on the input feature map, which calculates the mean value of the pixel values on each channel to extract the statistical features of each channel.

[0106] Next, the number of channels is mapped through a dimensionality reduction process (i.e., inputs.shape[-1] / / 8) using a Dense layer, and the ReLU activation function is used for activation. This operation helps to capture the correlations between channels.

[0107] After passing through another Dense layer, the number of channels is mapped back to the original dimension, and the sigmoid activation function is used to normalize the output to the interval [0,1].

[0108] Use Reshape to reconstruct the one-dimensional vector into a tensor with the shape (1, 1, number of channels);

[0109] Finally, through the Multiply layer, the attention coefficient is multiplied element-wise with the original input to obtain the feature map with enhanced channel attention.

[0110] For the input feature map x (shape H×W×C), first perform global average pooling to obtain the channel statistical vector:

[0111] GAP(x)_c = (1 / (H×W)) ∑(i,j)x(i,j,c) (1)

[0112] Next, dimensionality reduction and restoration are achieved through two fully connected layers to generate the channel attention weight \(A_c\):

[0113] \(A_c = \text{sigmoid}(W 2 \cdot \text{ReLU}(W 1 \cdot \text{GAP}(x))) \quad (2)

[0114] where \(W 1 and \(W 2 are the weight matrices of the two Dense layers respectively, and sigmoid normalizes the result to \([0, 1]\). Finally, \(A_c\) (with shape changed to \(1\times1\times C\)) is multiplied element-wise with the original input:

[0115] \(x_{\text{channel}} = x\odot A_c \quad (3)

[0116] 2.2 Spatial Attention

[0117] The original input is convolved using a Conv2D layer with a \(7\times7\) kernel to obtain a single-channel spatial attention map, and the activation function also uses sigmoid, which can weight the features at each spatial position.

[0118] Similarly, this attention map is multiplied element-wise with the original input to obtain the feature result after spatial attention modulation.

[0119] For the input feature map \(x\), a spatial attention map is generated using a \(7\times7\) convolutional kernel:

[0120] \(A_{\text{spatial}} = \text{sigmoid}(\text{Conv} 7x7 (x)) \quad (4)

[0121] Here, \(\text{Conv} 7x7 represents convolution using a kernel of size \(7\times7\), and the output \(A_{\text{spatial}}\) has a shape of \(H\times W\times1\). It is multiplied element-wise with the input \(x\) according to the spatial position:

[0122] \(x_{\text{spatial}} = x\odot A_{\text{spatial}} \quad (5)

[0123] 2.3 Attention Output Integration

[0124] Finally, the outputs of channel attention and spatial attention are added using the Add layer to obtain the fused attention features. This process retains the feature enhancement information from two different perspectives.

[0125] Finally, the outputs of the two branches are added and fused:

[0126] x_attention = x_channel + x_spatial (6) This operation reflects the importance of simultaneously focusing on channels and spatial dimensions, enabling the model to adaptively emphasize key information.

[0127] 3. Multimodal Model Construction: Image and Text Branches and Fusion

[0128] The core of model construction is to extract image features based on the pre-trained ResNet50, simultaneously construct a text branch to extract clinical indicators, then fuse the two types of features, and finally output the prediction results for lung cancer classification.

[0129] 3.1 Image Branch Construction

[0130] Use ResNet50 as the backbone network, load the pre-trained model parameters (ImageNet weights) for its weights, and at the same time remove the top fully connected layer (include_top = False), with the input size limited to (224, 224, 3).

[0131] After the model output passes through ResNet50, the output feature map first undergoes attention mechanism enhancement through the attention_block defined in the system.

[0132] Next, the model flattens the processed feature map through Flatten, converting the multi-dimensional feature map into a one-dimensional vector for subsequent processing by the fully connected layer.

[0133] Immediately afterwards, use BatchNormalization for normalization processing, and then fuse and extract high-level features through a fully connected layer (Dense 512 units, ReLU activation), and pass through the Dropout layer to prevent overfitting.

[0134] Finally, define a new output feature layer Dense(256, activation='relu') for the image branch as the image-side feature during fusion.

[0135] In addition, there is also a setting for fine-tuning the pre-trained network. Freeze all layers in ResNet50 except the last 10 layers, that is, for layer in base_model.layers[:-10]: layer.trainable = False, to ensure that the model does not destroy the learned features due to randomly initialized weights, and at the same time only fine-tune the higher layers at the back.

[0136] Use the pre-trained ResNet50 to extract features. Denote the input image as I, then the output feature of a certain intermediate layer is F(I). After passing through the attention module, the enhanced feature is obtained:

[0137] F_att(I) = Flatten(BatchNorm(Dense(Attention(F(I))))) (7)

[0138] The specific steps are described as follows:

[0139] Feature extraction:

[0140] F(I) = ResNet50(I; weights = imagenet, include_top = False) (8)

[0141] Applying attention:

[0142] F_att 1 (I) = attention_block(F(I)) (9)

[0143] Flattening and normalization:

[0144] f = Flatten(F_att 1 (I)) -> BatchNorm(f) (10)

[0145] Fully connected and dimensionality reduction:

[0146] image_output = Dense(256, activation ='relu')(f) (11)

[0147] This part extracts the deep features of the image and further optimizes them through the attention module.

[0148] 3.2 Construction of the text branch

[0149] A text input layer is defined with an input shape of (5,), representing 5 clinical indicators (SCC, CEA, CK19, NSE, ProGRP respectively in subsequent data processing).

[0150] This input is mapped to a relatively low-dimensional feature space through a Dense(128, activation ='relu') layer, and then Dropout is used to prevent overfitting.

[0151] The output layer of the text branch is Dense(256, activation ='relu'), and its output features will be fused with the image-side features in the subsequent process.

[0152] Assume the input clinical text data is a vector (5 indicators), and it is mapped into high-dimensional features through two fully connected layers:

[0153] text_hidden = Dropout(ReLU(Dense(128)(x_text))) (12)

[0154] text_output = Dense(256, activation='relu')(text_hidden) (13)

[0155] Here, the goal is to extract a more representative 256-dimensional feature representation from the low-dimensional metrics.

[0156] 3.3 Multimodal Feature Fusion

[0157] The output of the image branch (256 dimensions) and the output of the text branch (256 dimensions) are concatenated through the Concatenate layer to form a fused feature vector of length 512.

[0158] The fused vector first undergoes a non-linear mapping through Dense(512, activation='relu'), followed by Dropout for dimensionality reduction to prevent overfitting; then it passes through another layer of Dense(256, activation='relu') and Dropout to further extract the fused features.

[0159] The final output layer uses Dense(2, activation='softmax'). Since the problem is a binary classification task (lung cancer vs. non-lung cancer), the probabilities of the two classes are output.

[0160] At the same time, the model uses the Adam optimizer (learning rate set to 0.0001) and categorical_crossentropy as the loss function, and monitors the accuracy metric.

[0161] The image output and the text output are combined through concatenation (Concatenate):

[0162]

[0163] The fused features are further extracted through the fully connected layer:

[0164] fuse = Dropout(ReLU(Dense(512)(combined))) (15)

[0165] fuse = Dropout(ReLU(Dense(256)(fuse))) (16)

[0166] Finally, output the classification result (for binary classification tasks, use softmax):

[0167] output = softmax(Dense(2)(fuse)) (17)

[0168] Therefore, the final model can be expressed as:

[0169] output = softmax(Dense 2 (Dropout(ReLU(Dense 1 ([image_output; text_output]))))) (18)

[0171] where Dense 1 and Dense 2 are the weight matrices of the fusion layer and the output layer respectively.

[0172] 4. Preprocessing of image data and chest region detection

[0173] To make the image data input into the model more focused on the regions related to diagnosis, the detect_and_crop_chest function is defined in the system, and the main process is as follows:

[0174] 4.1 Convert to grayscale image

[0175] First, use cv2.cvtColor to convert the RGB image to a grayscale image (single channel) for subsequent face / human detection operations.

[0176] 4.2 Detect using Haar feature cascade classifier

[0177] Call the built-in cascade classifier haarcascade_fullbody.xml in OpenCV (although generally this classifier is used for full-body detection, but in this example it is used to detect the chest region) to scan and detect the grayscale image.

[0178] The classifier returns one or more candidate regions (coordinates and width / height).

[0179] 4.3 Cropping operation

[0180] Traverse the returned candidate boxes (usually only take the first matching region), obtain the coordinates (x, y, w, h) of the chest region and perform cropping, and return the cropped image.

[0181] If the chest region is not detected, return the original image.

[0182] 5. Data loading: Reading and preprocessing of image and text data

[0183] The program is divided into two parts to load data:

[0184] 5.1 Image data loading

[0185] Define the function load_images_from_folder, which traverses all files in the specified folder and performs the following operations on each file:

[0186] Read the image using cv2.imread;

[0187] Call the aforementioned detect_and_crop_chest function to detect and crop the chest area of the image;

[0188] Resize the image to a fixed size of 224×224 using cv2.resize to meet the input requirements of ResNet50;

[0189] Save the processed image to a list and finally return the list composed of all images.

[0190] 5.2 Text data loading

[0191] Define the function load_text_data to read the text data in the CSV file.

[0192] This function uses pandas to read the CSV file, then selects the specified 5 feature columns ('SCC', 'CEA', 'CK19', 'NSE', 'ProGRP') as numerical features, and at the same time obtains the annotation column 'Lung_cancer' (used to indicate whether it is lung cancer) as the label.

[0193] Return the text data array and the label array.

[0194] 5.3 Dataset division

[0195] The system loads data from the training, validation, and test directories respectively. For each dataset, the image data is located in the "cancer" and "normal" folders respectively, and each category folder also contains the corresponding CSV file (file name: data.csv).

[0196] Call load_images_from_folder and load_text_data for the cancer group and the normal group respectively, and then integrate the two groups of data into an overall dataset.

[0197] The list of image data is finally converted into a NumPy array;

[0198] The text data is vertically stacked using vstack;

[0199] The label data is also uniformly converted into a NumPy array.

[0200] 6. Data Preprocessing and Standardization

[0201] After loading the original image and text data, the program performs the following preprocessing steps on the data:

[0202] 6.1 Image Data Preprocessing

[0203] Use the preprocess_input function built into Keras to preprocess the image data, normalizing the pixel values or converting them to the format expected by the model.

[0204] Call it separately for the training, validation, and test datasets.

[0205] 6.2 Text Data Standardization

[0206] For the text data, use StandardScaler in sklearn to standardize each feature so that each feature has a zero mean and unit variance.

[0207] Call fit_transform for the training data and transform for the validation and test data to maintain consistency.

[0208] In addition, for the text data, an importance weight (a weight vector of length 5) is also applied, multiplying the standardized text features element by element, aiming to highlight the importance of each indicator. This part of the weight is:

[0209] [0.09210638, 0.14757179, 0.22065559, 0.43121099, 0.10845525] (Before designing this system, a correlation and importance weight analysis was performed on each text data using the ExtraTreesClassifier model, selecting the 5 types of factors with the largest comprehensive correlation and sorting them by importance to obtain the above results)

[0210] 6.3 Label Conversion

[0211] The label data is converted into one-hot encoded form through to_categorical because the final output layer is softmax and the model output is the probability distribution of two classes.

[0212] 7. Model Training and Ensemble

[0213] 7.1 Multi-Model Training

[0214] The system uses an integrated method to train multiple models (here, num_models = 3 is set). Three models are constructed and trained separately through a loop, and each model uses the same structure.

[0215] During the training process of each model, three callback functions are set:

[0216] ModelCheckpoint: Monitor and save the best model weights according to the validation set loss value;

[0217] EarlyStopping: Stop training early when the validation performance does not improve for several consecutive epochs;

[0218] ReduceLROnPlateau: Reduce the learning rate when the validation set loss does not decrease, helping the model to jump out of local minima.

[0219] During training, model.fit is called, and the data input is in list form ([X_train_images, X_train_text], y_train), corresponding to the two input branches of images and text respectively, and the validation data is provided at the same time.

[0220] 7.2 Model Prediction and Integration

[0221] After training is completed, the test set data is predicted. Each model will predict the test images and text, obtaining a probability output.

[0222] The prediction results of multiple models are stored in the list y_preds, and finally the results are integrated by taking the average (np.mean(y_preds, axis = 0)) to improve the prediction stability and generalization ability.

[0223] The final prediction label is obtained from the integrated prediction results through np.argmax.

[0224] The model uses the Adam optimizer to minimize the cross-entropy loss. For a single sample, its loss function is:

[0225] L = -∑(k = 1, 2) y k log(output k ) (19)

[0226] where y k is the true one-hot label.

[0227] In actual training, callback functions such as EarlyStopping and ReduceLROnPlateau are also used to adjust the learning rate, so as to accelerate the convergence speed and prevent overfitting.

[0228] 8. Model Evaluation and Result Presentation

[0229] 8.1 Classification Evaluation Report

[0230] Use classification_report to generate a detailed classification report, including precision, recall, F1-score, and the support count for each category.

[0231] In the report, the performance metrics for each category (0: normal, 1: cancer) and the overall performance are calculated separately.

[0232] 8.2 ROC Curve and AUC Value Calculation

[0233] Call the roc_curve function to calculate the Receiver Operating Characteristic Curve (ROC Curve), with the true labels and predicted labels in the test set as the input.

[0234] Use the auc function to calculate the Area Under the Curve (AUC) of the ROC curve as a quantitative metric for the model's discriminative ability.

[0235] Finally, use matplotlib to plot the ROC curve graph to show the performance of the model at different thresholds.

[0236] Adopt this method:

[0237] 1. Integrate multi-modal features: Use pre-trained ResNet50 to extract image features, and enhance key features through custom attention blocks (channel and spatial attention); at the same time, combine clinical indicators such as pathology / blood (text data), construct a text feature branch through a fully connected layer, and complete the integration of multi-modal information through the Concatenate layer.

[0238] 2. Chest region detection: In the image preprocessing stage, use the Haar cascade detection method to crop the chest region in the image to ensure that the input image for the model focuses on the key diagnostic area, improving the robustness and effectiveness of the model.

[0239] 3. Data preprocessing and standardization: Normalize the images using Keras' preprocess_input; standardize the text data using StandardScaler and apply importance weights to make the contribution of each indicator to the classification result more explicit.

[0240] 4. Model ensemble strategy: Train multiple models with the same structure and average the prediction results for fusion to improve the prediction stability and generalization ability, and reduce the possible overfitting risk of a single model.

[0241] 5. Dynamic training process: Utilize multiple callback mechanisms (EarlyStopping, ReduceLROnPlateau, ModelCheckpoint) to ensure that the model maintains a good convergence speed and performance during training, preventing overfitting and training stagnation.

[0242] 6. Comprehensive evaluation metrics: Finally, through classification reports and ROC, AUC analysis, comprehensively evaluate the accuracy, precision, recall, F1-score, and discriminative ability of the model, and present the results in graphical and text forms for an objective judgment of the model's effect.

[0243] A lung cancer diagnosis device, including a computer device, the computer device at least includes a processor and a memory, and the computer device is used to execute the described lung cancer diagnosis method. Through this device, the stability and accuracy of lung cancer judgment can be effectively improved.

[0244] As described above, it is only the preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes to obtain technical solutions, concepts, and designs, which should all be covered by the protection scope of the present invention.

Claims

1. A method for diagnosing lung cancer, characterized in that: The following steps are involved: Data acquisition and preprocessing: Collect CT image data and text data, and preprocess the data, including image normalization, text standardization, and weighted processing; Image feature extraction: Based on the pre-trained ResNet50 network, deep features of CT images are extracted, and key features are enhanced through the attention mechanism; Text feature processing: extract features from preprocessed text data to generate text feature vectors; Multimodal feature fusion: fuse image features and text features, and extract fusion features through fully connected layers and regularization layers; Classification: Classify lung cancer based on fusion features and output the probability of lung cancer and non-lung cancer; Model ensemble: Train multiple models and average the prediction results.

2. A lung cancer diagnosis method according to claim 1, characterized in that: The attention mechanism includes: Channel attention: extract channel statistics through global average pooling, generate channel attention weights through two fully connected layers, and multiply them element-by-element with the original features; spatial attention: generate spatial attention maps through 7×7 convolutions and multiply them element-by-element with the original features; fusion attention: add the results of channel attention and spatial attention to obtain the final attention-enhanced features; And / or, in the text feature processing, the text data is standardized and weighted using a preset importance weight to obtain a text feature vector; And / or, in the multimodal feature fusion, the image features and the text features are concatenated through a Concatenate layer, and feature fusion and dimensionality reduction are performed through a fully connected layer and a Dropout layer; And / or, in the classification, a Softmax layer is used to output the probabilities of lung cancer and non-lung cancer, and a cross entropy loss function is used for training; And / or, in the model integration, multiple models with the same structure are trained, and the prediction results of each model are averaged and fused.

3. A lung cancer diagnosis method according to claim 2, characterized in that: The channel attention comprises the following steps: Use GlobalAveragePooling2D to perform global average pooling on the input feature map; The number of channels is mapped through a dimensionality reduction process through the Dense layer, and the activation function uses ReLU activation; After another Dense layer, the number of channels is mapped back to the original dimension, and the sigmoid activation function is used to normalize the output to the [0, 1] interval. Use Reshape to reshape the one-dimensional vector into a tensor with a shape of (1,1, number of channels); Finally, the attention coefficient is multiplied element-by-element with the original input through the Multiply layer to obtain the feature map after channel attention enhancement; For the input feature map x (shape H×W×C), first perform global average pooling to obtain the channel statistics vector: GAP(x)_c=(1 / (H×W))∑(i,j)x(i,j,c) Then, two fully connected layers are used to achieve dimensionality reduction and recovery, generating the channel attention weight A_c: A_c=sigmoid(W2·ReLU(W1·GAP(x))) Where W1 and W2 are the weight matrices of the two Dense layers respectively, and sigmoid normalizes the result to [0,1]. Finally, A_c (with a shape of 1×1×C) is multiplied element by element with the original input: x_channel = x⊙A_c; The spatial attention comprises the following steps: Use a Conv2D layer with a 7×7 kernel to convolve the original input to obtain a single-channel spatial attention map, and the activation function also uses sigmoid; Similarly, this attention map is multiplied element by element with the original input to obtain the feature result after spatial attention modulation; For the input feature map x, a 7×7 convolution kernel is used to generate a spatial attention map: A_spatial=sigmoid(Conv 7x7 (x)) Here Conv 7x7 Indicates that a convolution operation is performed using a convolution kernel of size 7×7, and the shape of the output A_spatial is H×W×1. It is multiplied element-by-element with the input x according to the spatial position: x_spatial = x⊙A_spatial; The fusion attention comprises the following steps: The outputs of channel attention and spatial attention are added together using the Add layer to obtain the fused attention features. This process retains the feature enhancement information from two different perspectives. Add and fuse the outputs of the two branches: x_attention=x_channel+x_spatial.

4. A lung cancer diagnosis method according to any one of claims 2-3, characterized in that: The image feature extraction comprises the following steps: Use ResNet50 as the backbone network, load the pre-trained model parameters (ImageNet weights) with its weights, remove the top fully connected layer (include_top=False), and limit the input size to (224,224,3); After the model output passes through ResNet50, the output feature map first passes through the attention_block defined in the system to enhance the attention mechanism; Next, the model flattens the processed feature map and converts the multi-dimensional feature map into a one-dimensional vector for subsequent fully connected layer processing; Next, BatchNormalization is used for normalization, followed by fusion extraction of high-level features through a fully connected layer (Dense 512 units, ReLU activation), and a Dropout layer is used to prevent overfitting. Finally, a new output feature layer Dense(256, activation='relu') is defined for the image branch as the image-side feature during fusion.

5. A lung cancer diagnosis method according to claim 4, characterized in that: The image feature extraction also includes the following steps: For fine-tuning of the pre-trained network, freeze all layers except the last 10 layers in ResNet50, i.e. forlayerin base_model.layers[:-10]:layer.trainable = False, to ensure that the model does not destroy the learned features due to random initialization of weights, and only fine-tune the higher layers later; Use the pre-trained ResNet50 to extract features. Let the input image be I, then the output feature of a certain middle layer is F(I); after the attention module, the enhanced feature is obtained: F_att(I)=Flatten(BatchNorm(Dense(Attention(F(I)))))).

6. A lung cancer diagnosis method according to claim 5, characterized in that: The image feature extraction comprises the following steps: Extract features: F(I)=ResNet50(I;weights=imagenet,include_top=False) Applied Attention: F_att1(I)=attention_block(F(I)) Flattening and normalization: f=Flatten(F_att1(I))→BatchNorm(f) Fully connected and dimension reduced: image_output=Dense(256, activation='relu')(f).

7. A lung cancer diagnosis method according to claim 5 or 6, characterized in that: The text feature processing comprises the following steps: A text input layer is defined with an input shape of (n,) representing n clinical indicators; The input is mapped to a relatively low-dimensional feature space through a Dense(128, activation = 'relu') layer, and then through Dropout to prevent overfitting; The output layer of the text branch is Dense(256, activation='relu'), and its output features will be fused with the image features later; Assume that the input clinical text data is a vector Mapped into high-dimensional features through two layers of full connection: text_hidden=Dropout(ReLU(Dense(128)(x_text))) text_output=Dense(256, activation='relu')(text_hidden).

8. A lung cancer diagnosis method according to claim 7, characterized in that: The multimodal feature fusion comprises the following steps: The output of the image branch (256 dimensions) and the output of the text branch (256 dimensions) are concatenated through the Concatenate layer to form a fused feature vector of length 512; The fused vector is first nonlinearly mapped through Dense(512, activation = 'relu'), and then reduced in dimension through Dropout to prevent overfitting; it is then passed through another layer of Dense(256, activation = 'relu') and Dropout to further extract fusion features; The final output layer uses Dense(2, activation = 'softmax'). Since the problem is a binary classification task (lung cancer and non-lung cancer), the probabilities of the two categories are output; At the same time, the model uses the Adam optimizer (learning rate is set to 0.0001) and categorical_crossentropy as the loss function, and monitors the accuracy indicator; Output the image With text output Merge by concatenation: Further extract fusion features through the fully connected layer: fuse=Dropout(ReLU(Dense(512)(combined))) fuse=Dropout(ReLU(Dense(256)(fuse))) Finally, the classification results are output: output=softmax(Dense(2)(fuse)) Therefore, the final model can be expressed as: output=softmax(Dense2(Dropout(ReLU(Dense1([image_output; text_output]))))) Among them, Dense2 and Dense2 are the weight matrices of the fusion layer and the output layer respectively.

9. A lung cancer diagnosis method according to claim 8, characterized in that: The data collection and preprocessing includes the following steps: Define the detect_and_crop_chest function, the process is as follows: Convert grayscale image First, use cv2.cvtColor to convert the RGB image to a grayscale image (single channel); Use Haar feature cascade classifier for detection; Call OpenCV's built-in cascade classifier haarcascade_fullbody.xml to scan and detect the grayscale image; The classifier returns one or more candidate regions; Crop Operation Traverse the returned candidate boxes, obtain the coordinates (x, y, w, h) of the chest area and crop it, and return the cropped image; If the chest region is not detected, the original image is returned.

10. A lung cancer diagnosis device, characterized in that: The method comprises a computer device, wherein the computer device comprises at least a processor and a memory, and the computer device is used to execute the lung cancer diagnosis method according to any one of claims 1 to 9.

Images