Method and device for predicting primary drug resistance of third-generation EGFR-TKI

Abstract

The invention discloses a third-generation EGFR-TKI primary drug resistance prediction method and device, and the method comprises the steps: carrying out the extraction of a chest plain-scanning CT image through a Swin Transformer encoder, and obtaining image features; performing feature extraction and processing on the clinical data through a Tab Transform encoder to obtain clinical features; inputting the image features and the clinical features into a third generation EGFR-TKI primary drug resistance multi-modal prediction model; the processing process of the prediction model of the third-generation EGFR-TKI primary drug resistance comprises the steps of splicing image features and clinical features based on a supervised multi-mode learning framework, performing feature fusion through a multi-layer perceptron to generate fusion features, and inputting the fusion features into a classifier to obtain a prediction result of the third-generation EGFR-TKI primary drug resistance.

Description

Technical Field

[0001] The present application relates to a method and device for predicting primary drug resistance of the third-generation EGFR-TKI, and specifically to a method and device for predicting primary drug resistance of the third-generation EGFR-TKI based on multimodal self-attention fusion, belonging to the field of medical artificial intelligence technology. Background Art

[0002] Lung cancer is the most common malignancy worldwide and the leading cause of cancer-related death. Activating mutations in the epidermal growth factor receptor (EGFR) are clinically significant in non-small cell lung cancer (NSCLC), particularly in Chinese patients, accounting for approximately 38%. The two most common EGFR mutations are exon 19 deletion (ex19Del) and exon 21 amino acid substitution (leucine to arginine codon 858 [L858R]). EGFR tyrosine kinase inhibitors (TKIs) have provided significant therapeutic benefits for NSCLC patients harboring EGFR mutations. First- and second-generation TKIs, such as gefitinib, have been shown to outperform standard chemotherapy in terms of overall response rate (ORR) and progression-free survival (PFS). Third-generation TKIs, such as osimertinib, have demonstrated promising efficacy in overcoming T790M mutations and intracranial progression that arise after resistance to first- and second-generation TKIs. However, approximately 20% of patients harboring EGFR mutations fail to benefit from third-generation TKIs, potentially due to the preexisting resistance. Studies have shown that mutations in genes such as TP53, EGFR exon 20 insertions (ex20ins), RB1, and KRAS may be associated with primary resistance to third-generation TKIs. Currently, there are no clear clinical or molecular predictors for identifying patients with primary resistance. Therefore, developing a method that can fully utilize patient information to predict the probability of primary resistance to third-generation EGFR-TKIs is of great significance.

[0003] Recent advances in artificial intelligence can help solve this problem. The development of image analysis technologies based on machine learning (ML), especially deep learning (DL) technologies, has shown that they can assist in diagnosis, identify new features, predict patient prognosis, and assist in personalized treatment, but they often have certain limitations. Traditional EGFR-TKI primary resistance prediction mainly relies on single-modality data, such as imaging or clinical indicators. However, these methods have certain limitations in accuracy and personalized prediction. By combining clinical information and medical imaging, researchers can build more accurate prediction models. However, most existing methods cannot fully capture the relationship between multimodal information; feature splicing or weighted averaging cannot dynamically adjust the weight prediction between modalities; the results are less interpretable, and other problems. Summary of the Invention

[0004] Objective: In view of at least one of the above technical problems, the present application provides a method and device for predicting primary drug resistance of the third-generation EGFR-TKI, which is suitable for solving the effective fusion of multimodal data, effectively utilizing clinical features and image features, improving prediction performance, and accurately predicting the efficacy of third-generation EGFR-TKI targeted therapy for lung cancer patients.

[0005] The technical solutions adopted in this application are:

[0006] In a first aspect, the present application provides a method for predicting primary drug resistance to third-generation EGFR-TKIs, comprising:

[0007] Obtain thin-slice chest plain CT images and clinical data of target patients;

[0008] The thin-slice chest plain scan CT image is downsampled into a series of patch images of the same size according to the spatial size, and the patch images are extracted through a multi-level hierarchical Swin Transformer encoder to obtain image features;

[0009] Tab Transformer encoder is used to extract and normalize clinical data to obtain clinical features;

[0010] The image features and clinical features were input into the third-generation EGFR-TKI primary resistance prediction model to obtain the third-generation EGFR-TKI primary resistance prediction results;

[0011] The processing process of the third-generation EGFR-TKI primary resistance prediction model includes: applying hierarchical normalization to image features and clinical features respectively, and splicing them in the feature dimension to form an initial multimodal fusion vector, and then inputting the initial multimodal fusion vector into a multi-layer perceptron for feature fusion to generate fusion features, and obtaining the third-generation EGFR-TKI primary resistance prediction result through a classifier; each layer of the multi-layer perceptron includes a fully connected layer, a batch normalization layer and a ReLU activation layer.

[0012] In a second aspect, the present application provides a third-generation EGFR-TKI primary resistance prediction device, comprising a processor and a storage medium;

[0013] The storage medium is used to store instructions;

[0014] The processor is configured to operate according to the instructions to perform the method according to the first aspect.

[0015] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the computer program is executed by a processor.

[0016] In a fourth aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0017] Beneficial Effects: The method and device for predicting primary resistance to third-generation EGFR-TKIs based on multimodal self-attention fusion provided in this application have the following advantages: To address the problems of low efficiency in multimodal medical data fusion, excessive memory usage of high-resolution chest CT images, and insufficient feature interaction in existing technologies, the SwinTransformer's PatchEmbed module divides the entire image into several patches in high-resolution chest CT images, which are then sequentially processed through multiple layers of Transformer Blocks, staged downsampling, and multi-scale feature representation, effectively reducing video memory usage. Clinical discrete variables are embedded and mapped, and continuous variables are linearly mapped after Min–Max normalization. These patches are then fed into a Tab Transformer encoder, where high-order features are captured through multi-head self-attention and a feedforward network. In the multimodal fusion stage, the image features and clinical features are concatenated along the feature dimension to complete feature fusion and regularization. Finally, a linear classification head is used to output the prediction results for primary resistance to third-generation EGFR-TKIs. This method achieved excellent performance with an AUC of 0.89 on an independent test set, providing clinicians with accurate and personalized treatment decision support. It has the following advantages:

[0018] (1) It fully combines image and clinical information and significantly improves the prediction performance of primary resistance to third-generation EGFR-TKI through a multi-layer perceptron fusion module;

[0019] (2) The Swin Transformer was used to construct multi-scale features, which greatly optimized the memory efficiency of high-resolution CT images. BatchNorm, Dropout and L2 normalization were used for clinical features, which effectively enhanced the feature interaction and the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of a process for predicting primary drug resistance of third-generation EGFR-TKI based on multimodal self-attention fusion according to an embodiment of the present application;

[0021] Figure 2 Schematic diagram of the model training and use method according to one embodiment of the present application;

[0022] Figure 3 A CT image visualization heat map generated according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The present application will be further described below in conjunction with the accompanying drawings and examples. The following examples are only used to more clearly illustrate the technical solutions of the present application and are not intended to limit the scope of protection of the present application.

[0024] In the description of this application, "several" means more than one, "plurality" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0025] Throughout the description of this application, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0026] The term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " generally indicates an "or" relationship between the related objects.

[0027] Example 1: This example provides a method for predicting primary drug resistance of the third-generation EGFR-TKI based on multimodal self-attention fusion, such as Figure 1 、 Figure 2 As shown, including:

[0028] Obtain thin-slice chest plain CT images and clinical data of target patients;

[0029] The thin-slice chest plain scan CT image is downsampled into a series of patch images of the same size according to the spatial size, and the patch images are extracted through a multi-level hierarchical Swin Transformer encoder to obtain image features;

[0030] Tab Transformer encoder is used to extract and normalize clinical data to obtain clinical features;

[0031] The image features and clinical features were input into the third-generation EGFR-TKI primary resistance prediction model to obtain the third-generation EGFR-TKI primary resistance prediction results;

[0032] The processing process of the third-generation EGFR-TKI primary resistance prediction model includes: applying hierarchical normalization to image features and clinical features respectively, and splicing them in the feature dimension to form an initial multimodal fusion vector, and then inputting the initial multimodal fusion vector into a multi-layer perceptron for feature fusion to generate fusion features, and obtaining the third-generation EGFR-TKI primary resistance prediction result through a classifier; each layer of the multi-layer perceptron includes a fully connected layer, a batch normalization layer and a ReLU activation layer.

[0033] In some embodiments, the encoders all use a multi-level hierarchical Swin Transformer. In some embodiments, the encoder training process includes:

[0034] Obtaining a thin-slice plain chest CT image of a lung cancer patient, wherein the thin-slice plain chest CT image is labeled, and the label indicates whether the corresponding lung cancer patient is primary resistant to a third-generation EGFR-TKI drug or not (a criterion for primary resistance is that the time to disease progression after taking the drug is less than 6 months);

[0035] The thin-slice chest plain scan CT image is downsampled into a series of patch images of the same size according to its spatial size for encoder feature extraction;

[0036] The multi-level hierarchical Swin Transformer encoder is trained using labeled images to obtain a trained multi-level hierarchical Swin Transformer encoder.

[0037] In this example, thin-layer chest plain scan CT images of 170 screened lung cancer patients were subjected to staged downsampling to compress image memory and improve processing efficiency. Each image was then mapped to a high-dimensional feature space through linear projection and position encoding. A multi-layer Swin Transformer and staged downsampling were used to achieve a step-by-step resolution reduction. Finally, multi-scale feature vectors were output at different depth stages to form a high-dimensional image representation.

[0038] In this embodiment, the clinical data include gender, age, tumor stage, T stage, N stage, M stage, drug manufacturer, whether surgery has been performed in the past, whether first or second generation EGFR-TKI has been taken in the past, whether chemotherapy has been performed in the past, whether radiotherapy has been performed in the past, whether immunotherapy has been used in the past, whether there is liver metastasis, whether there is bone metastasis, whether there is brain metastasis, gene mutation status obtained by NGS gene sequencing, blood routine, liver function, kidney function, four tumor indicators and coagulation function.

[0039] Clinical data are generally preprocessed first: columns that are not suitable for predicting recurrence and columns with a large number of missing values ​​are screened out, and columns with a small number of missing values ​​are filled with the median; then, feature extraction is performed on the preprocessed clinical data.

[0040] In some embodiments, extracting features and normalizing clinical data to obtain clinical features includes:

[0041] All categorical variables of multiple clinical data are mapped into fixed-dimensional embedding vectors. Continuous variables are first Min-Max normalized and then linearly mapped to the same dimension to obtain corresponding embedding vectors. All the above embedding vectors are concatenated according to the feature dimension and input into the Tab Transformer encoder. The Tab Transformer encoder includes multiple modules, each of which consists of a multi-head self-attention sublayer and a feedforward fully connected sublayer, which can adaptively capture high-order interactions between features. After each module, residual connections and layer normalization are added to stabilize training and enhance feature representation. All positions in the last layer of the Transformer are average pooled and L2 norm normalization is applied to obtain multiple clinical features.

[0042] The fused features are input into a classifier for final prediction. The classifier typically consists of one or more fully connected layers and ultimately outputs a prediction result for primary resistance to third-generation EGFR-TKIs. The prediction result can be a predicted probability of resistance or a classification label.

[0043] In some embodiments, the training process of the third-generation EGFR-TKI primary resistance prediction model includes:

[0044] The image features and corresponding clinical features of multiple labeled lung cancer patients are collected as a training dataset. The training dataset is combined with a loss function to train a third-generation EGFR-TKI primary resistance prediction model to obtain a trained third-generation EGFR-TKI primary resistance prediction model.

[0045] During the training process, the binary cross entropy loss function is used to optimize the classification performance of the model, and the image branch, clinical branch and their fusion module are jointly optimized end-to-end. The extracted image features and clinical features are sent to the third-generation EGFR-TKI primary resistance prediction model. All weights within the third-generation EGFR-TKI primary resistance prediction model are dynamically updated through back propagation; the entire training stage only uses the binary BCEWithLogitsLoss as the loss function, and the loss function Defined as:

[0046]

[0047] in Represents binary cross entropy with logits loss, is the true label (i.e. 0 / 1), is the score predicted by the model.

[0048] The model continuously updates network parameters by minimizing a loss function, gradually improving prediction accuracy. The model comprises a two-branch learning process. During training, each batch of samples is fed simultaneously into two parallel branches: one branch, a SwinTransformer, extracts multi-scale, patch-level features from 3D thin-slice chest CT images; the other branch, a TabTransformer, performs deep self-attention encoding on the standardized clinical table data. Image and clinical features are then concatenated along the feature dimension and fed into a fusion network defined by the model. Finally, a single-neuron output layer generates the predicted logit for primary drug resistance. Throughout training, the model uses only BCEWithLogitsLoss as a supervisory signal, updating all parameters of the image encoder, clinical encoder, fusion network, and classification head end-to-end using the AdamW optimizer. To improve training stability and control overfitting, Dropout (p=0.5) is added after each ReLU activation in the fusion network. LayerNorm and optional DropPath regularization are also applied to the intermediate feature outputs of the TabTransformer and SwinTransformer.

[0049] It should be noted that this training adopted a five-fold cross-validation strategy. In each fold, StratifiedKFold was used to divide the training set and the validation set, and after each epoch, indicators such as accuracy, precision, recall, F1 score, and ROC-AUC were calculated based on the validation set. Among them, ROC-AUC is the main evaluation criterion, which is used to dynamically monitor model performance and decide whether to save the current weights. In order to accelerate convergence and fine-tune the learning rate in the later stage, the StepLR scheduler was also introduced in the training, which decayed the current learning rate to 0.1 times the original every 10 epochs. The entire training process lasted 50 epochs, but it could be terminated in advance through the early stopping strategy if the validation set AUC did not improve, so as to obtain a final model that is both efficient and robust. This end-to-end multimodal fusion training framework not only fully utilizes the complementary advantages of image and clinical data, but also ensures that the model can maintain excellent generalization capabilities when dealing with imbalanced labels through strict regularization and validation strategies.

[0050] Furthermore, the method also includes generating a visualization heat map of the three-dimensional thin-layer chest CT image, overlaying the image's attention weight on the original image with a grayscale gradient, and viewing the part of the model that is of interest to improve the interpretability. Figure 3 As shown, brighter voxels represent higher attention, and darker ones represent lower attention.

[0051] Example 2: Based on Example 1, this example provides a prediction device for primary drug resistance of third-generation EGFR-TKI based on multimodal self-attention fusion, including a processor and a storage medium;

[0052] The storage medium is used to store instructions;

[0053] The processor is configured to operate according to the instructions to execute the method according to embodiment 1.

[0054] Example 3: Based on Example 1, this example provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described in Example 1 is implemented.

[0055] Example 4: Based on Example 1, this example provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in Example 1 when executing the computer program.

[0056] Example 5: Based on Example 1, this example provides a computer program product, including a computer program, which implements the method described in Example 1 when executed by a processor.

[0057] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0059] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0061] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for predicting primary resistance to third-generation EGFR-TKI, characterized in that: include: Obtain thin-slice chest plain CT images and clinical data of target patients; The thin-slice chest plain scan CT image is downsampled into a series of patch images of the same size according to the spatial size, and the patch images are extracted through a multi-level hierarchical Swin Transformer encoder to obtain image features; Tab Transformer encoder is used to extract and normalize clinical data to obtain clinical features; The image features and clinical features were input into the third-generation EGFR-TKI primary resistance prediction model to obtain the third-generation EGFR-TKI primary resistance prediction results; The processing process of the third-generation EGFR-TKI primary resistance prediction model includes: applying hierarchical normalization to image features and clinical features respectively, and splicing them in the feature dimension to form an initial multimodal fusion vector, and then inputting the initial multimodal fusion vector into a multi-layer perceptron for feature fusion to generate fusion features, and obtaining the third-generation EGFR-TKI primary resistance prediction result through a classifier; each layer of the multi-layer perceptron includes a fully connected layer, a batch normalization layer and a ReLU activation layer.

2. The method according to claim 1, characterized in that The processing of the multi-level hierarchical Swin Transformer encoder includes: The patch images are mapped to a high-dimensional feature space through linear projection and position encoding, and the resolution is gradually reduced through multi-layer Swin Transformer and staged downsampling. Finally, multi-scale feature vectors are output at different depth stages to form image features.

3. The method according to claim 1 or 2, characterized in that The training process of the multi-level hierarchical Swin Transformer encoder includes: Acquiring a thin-slice chest plain CT image of a patient, wherein the thin-slice chest plain CT image is labeled, and the label indicates whether the corresponding lung cancer patient is primary resistant to a third-generation EGFR-TKI drug or not; The thin-slice chest plain scan CT image is downsampled into a series of patch images of the same size according to the spatial size; The multi-level hierarchical Swin Transformer encoder is trained using labeled patch images to obtain a trained multi-level hierarchical Swin Transformer encoder.

4. The method according to claim 1, wherein The clinical data include gender, age, tumor stage, T stage, N stage, M stage, drug manufacturer, whether surgery has been performed before, whether the first or second generation EGFR-TKI has been taken before, whether chemotherapy has been performed before, whether radiotherapy has been performed before, whether immunotherapy has been used before, whether there is liver metastasis, whether there is bone metastasis, whether there is brain metastasis, gene mutation status obtained by NGS gene sequencing, blood routine, liver function, kidney function, four tumor indicators and coagulation function.

5. The method according to claim 1, wherein Extract and normalize clinical data to obtain clinical features, including: All categorical variables of multiple clinical data are mapped into fixed-dimensional embedding vectors. Continuous variables are first Min–Max normalized and then linearly mapped to the same dimension to obtain the corresponding embedding vectors. All embedding vectors are concatenated according to the feature dimension and input into the Tab Transformer encoder; the Tab Transformer encoder includes multiple modules, each of which includes a multi-head self-attention sublayer and a feed-forward fully connected sublayer, which can adaptively capture high-order interactions between features; after each module, residual connections and layer normalization are added to stabilize training and enhance feature representation; all positions in the last layer are average pooled and L2 norm normalization is applied to obtain multiple clinical features.

6. The method according to claim 1, characterized in that The training process of the third-generation EGFR-TKI primary resistance prediction model includes: The image features and corresponding clinical features of multiple labeled lung cancer patients are collected as a training dataset. The training dataset is combined with a loss function to train a third-generation EGFR-TKI primary resistance prediction model to obtain a trained third-generation EGFR-TKI primary resistance prediction model.

7. The method according to claim 6, characterized in that During the training process, the image features and clinical features were fed into the third-generation EGFR-TKI primary resistance prediction model. All weights within the third-generation EGFR-TKI primary resistance prediction model were dynamically updated through back propagation. During the entire training phase, only the binary BCEWithLogitsLoss was used as the loss function. Defined as: ， in Represents binary cross entropy with logits loss, is the true label, is the score predicted by the model.

8. A third-generation EGFR-TKI primary resistance prediction device based on cross-attention, characterized by: including processors and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images