Prediction method of EGFR gene mutation in lung cancer based on multi-instance learning and Transformer technology

Abstract

The present application relates to a method for predicting EGFR gene mutations in lung cancer based on multi-instance learning and Transformer technology. The method extracts instance-level features by segmenting lung CT images into image blocks, and uses soft pseudo-labeling technology in a self-generated manner to provide additional supervisory signals for instance-level features, guiding the multi-instance learning model to more effectively distinguish instances and enhance the identification ability of instance-level features. The unique relative spatial position information is integrated into the self-attention mechanism, which can flexibly and clearly mine the dependencies between instances and greatly improve the representation ability of heterogeneous tumors. The importance scores of different instance features are automatically learned through the adaptive gated instance aggregation module, which facilitates the dynamic weighting of feature aggregation instances. This method improves the generalization of the model on different data sets. The prediction method provided by the present application can delicately capture and analyze the heterogeneous features within the tumor, ensuring that the prediction of the EGFR mutation status is more accurate and targeted.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for predicting EGFR gene mutations in lung cancer based on multi-instance learning and Transformer technology. Background Art

[0002] Tumors play a key role in cancer-related deaths, and non-small cell lung cancer (NSCLC), as its main pathological classification, occupies a prominent position. In recent years, the leap forward in genomics and precision medicine has promoted the innovation of the NSCLC treatment paradigm. In particular, tyrosine kinase inhibitors (TKIs) targeting epidermal growth factor receptor (EGFR) have become one of the first choice treatment options for NSCLC patients. Clinical evidence clearly shows that compared with EGFR wild-type cases, mutant cases carrying EGFR mutations show a higher treatment response rate to TKIs, greatly improving the prognostic indicators and disease-free survival of these NSCLC patients, thus emphasizing the clinical significance of predicting EGFR mutation status before treatment and laying the foundation for the precise selection of EGFR-TKI treatment and personalized medical pathways.

[0003] Traditionally, histological samples obtained through tissue biopsy or tumor surgery are considered the gold standard for EGFR mutation prediction. However, this method has several limitations, including sampling bias, DNA degradation, high cost and long processing time, which together restrict its widespread application in the field of EGFR mutation screening. In addition, the invasiveness of the biopsy procedure itself and the possible complications such as pneumothorax and bleeding are particularly uncomfortable for patients in advanced disease states. In view of this, the development of efficient and non-invasive EGFR mutation prediction technology has become a top priority to overcome the shortcomings of existing methods and broaden the feasibility boundaries of pre-treatment molecular typing.

[0004] Recently, computed tomography (CT) has demonstrated its extraordinary value as a non-invasive imaging tool in the diagnosis and treatment of non-small cell lung cancer (NSCLC), especially in the non-invasive prediction of EGFR mutations.

[0005] However, traditional CT-based EGFR mutation prediction methods mostly use prediction models built on a fully supervised learning framework. These models generally assume that the EGFR mutation status within the tumor is evenly distributed and directly extract features from the overall CT image to predict the mutation status. This traditional method has a fatal flaw: it fails to fully consider the complexity and diversity of the tumor's internal structure, namely tumor heterogeneity.

[0006] Different subregions of the tumor may exhibit completely different mutation characteristics. Traditional CT-based EGFR mutation prediction methods fail to consider and utilize this spatial variation information, resulting in low prediction accuracy. Summary of the Invention

[0007] Based on this, it is necessary to provide a lung cancer EGFR gene mutation prediction method based on multi-instance learning and Transformer technology to address the problem that traditional lung cancer EGFR gene mutation prediction methods based on multi-instance learning and Transformer technology fail to fully consider the complexity and diversity of the internal structure of the tumor.

[0008] The present application provides a method for predicting EGFR gene mutations in lung cancer based on multi-instance learning and Transformer technology, the method comprising:

[0009] Obtain multiple lung CT images;

[0010] Each lung CT image is preprocessed to obtain an image of a region of interest in each lung CT image;

[0011] Divide each region of interest image into multiple image blocks of equal size;

[0012] Create an EGFR mutation prediction model;

[0013] Training the EGFR mutation prediction model k times to obtain a trained EGFR mutation prediction model, wherein k is a positive integer and k is greater than 1;

[0014] Obtaining a lung CT image to be tested, and inputting the lung CT image to be tested into the trained EGFR mutation prediction model;

[0015] Starting the trained EGFR mutation prediction model and obtaining a prediction result output by the trained EGFR mutation prediction model;

[0016] The training of the EGFR mutation prediction model k times to obtain the trained EGFR mutation prediction model comprises:

[0017] Using the feature encoder to encode all image blocks one by one, and obtaining the preliminary prediction result of each image block based on the encoding result of each image block;

[0018] The importance score of each image block is embedded in the preliminary prediction result of each image block to generate a soft pseudo label for each image block;

[0019] Create a spatial perception Transformer module with a multi-layer encoder structure. Based on the spatial perception Transformer module with a multi-layer encoder structure, it mines the spatial relative position relationship between different image blocks and obtains the comprehensive features of each image block;

[0020] Based on the adaptive gated instance aggregation module, the comprehensive features of each image block and the importance score of each image block are aggregated using a dynamic weighting method to obtain the packet-level CT image features of each image block;

[0021] Construct the prediction loss function of the package feature, the prediction loss function of the class label feature, and the supervision loss function respectively;

[0022] Construct an overall loss function using the prediction loss function of the bag feature, the prediction loss function of the class label feature, and the supervision loss function;

[0023] In the next training, the value of the overall loss function is minimized as the training goal.

[0024] The present application relates to a method for predicting EGFR gene mutations in lung cancer based on multi-instance learning and Transformer technology. The method extracts instance-level features by segmenting lung CT images into image blocks, and uses soft pseudo-labeling technology in a self-generated manner to provide additional supervisory signals for instance-level features, guiding the multi-instance learning model to more effectively distinguish instances and enhance the discrimination ability of instance-level features. The unique relative spatial position information is integrated into the self-attention mechanism, which can flexibly and clearly mine the dependencies between instances, greatly improve the representation ability of heterogeneous tumors, and take into account the importance of sub-region interactions in the prediction process. The importance scores of different instance features are automatically learned through the adaptive gated instance aggregation module, which facilitates the dynamic weighting of feature aggregation instances. This method ensures that more information-rich features are emphasized in heterogeneous tumors, thereby improving the generalization of the model on different data sets. The prediction method provided in the present application can delicately capture and analyze the heterogeneous features within the tumor, ensuring that the prediction of EGFR mutation status is more accurate and targeted. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a method for predicting EGFR gene mutations in lung cancer based on multi-instance learning and Transformer technology is provided in one embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0027] This application provides a method for predicting EGFR gene mutations in lung cancer based on multi-instance learning and Transformer technology. It should be noted that the method for predicting EGFR gene mutations in lung cancer based on multi-instance learning and Transformer technology provided in this application is applied to the prediction of gene mutations in EGFR cases of non-small cell lung cancer (NSCLC).

[0028] In addition, the lung cancer EGFR gene mutation prediction method based on multi-instance learning and Transformer technology provided in this application does not limit its execution subject. Optionally, the execution subject of the lung cancer EGFR gene mutation prediction method based on multi-instance learning and Transformer technology provided in this application can be a lung cancer EGFR gene mutation prediction system based on multi-instance learning and Transformer technology. Specifically, the execution subject of the lung cancer EGFR gene mutation prediction method based on multi-instance learning and Transformer technology provided in this application can be one or more processing terminals in the lung cancer EGFR gene mutation prediction based on multi-instance learning and Transformer technology.

[0029] It should be noted that all steps and processes described in this application are not intended for the purpose of treating or diagnosing diseases. The lung cancer EGFR gene mutation prediction method provided in this application based on multi-instance learning and Transformer technology only predicts lung cancer EGFR gene mutations by establishing a prediction model. This involves image processing and computer data analysis of lung CT images and has no direct effect on any qualitative diagnosis. It only serves as an auxiliary and reference role.

[0030] That is, the lung cancer EGFR gene mutation prediction result obtained in this application is only intermediate information for any subsequent application, or an intermediate data processing result for subsequent analysis.

[0031] like Figure 1 As shown, in one embodiment of the present application, the lung cancer EGFR gene mutation prediction method based on multi-instance learning and Transformer technology includes:

[0032] S100, acquiring multiple lung CT images.

[0033] S200 , preprocessing is performed on each lung CT image to obtain an image of a region of interest in each lung CT image.

[0034] S300: Segment each region of interest image into a plurality of image blocks of equal size.

[0035] S400, create an EGFR mutation prediction model.

[0036] S500: Training the EGFR mutation prediction model k times to obtain a trained EGFR mutation prediction model, where k is a positive integer and greater than 1.

[0037] S600: Acquire a lung CT image to be tested, and input the lung CT image to be tested into the trained EGFR mutation prediction model.

[0038] S700: Start the trained EGFR mutation prediction model and obtain the prediction result output by the trained EGFR mutation prediction model.

[0039] The S500 includes:

[0040] S510 , encoding all image blocks one by one using a feature encoder, and obtaining a preliminary prediction result of each image block according to the encoding result of each image block.

[0041] S520 , embedding the importance score of each image block into the preliminary prediction result of each image block to generate a soft pseudo label for each image block.

[0042] S530, creating a spatial perception Transformer module with a multi-layer encoder structure. Based on the spatial perception Transformer module with a multi-layer encoder structure, the spatial relative position relationship between different image blocks is mined to obtain the comprehensive features of each image block.

[0043] S540 , based on the adaptive gated instance aggregation module, the comprehensive features of each image block and the importance score of each image block are aggregated using a dynamic weighting method to obtain the package-level CT image features of each region of interest image.

[0044] S550 , constructing a prediction loss function of the package feature, a prediction loss function of the class label feature, and a supervision loss function respectively.

[0045] S560 , constructing an overall loss function using the prediction loss function of the bag feature, the prediction loss function of the class label feature, and the supervision loss function.

[0046] S570: Minimizing the value of the overall loss function in the next training is used as a training objective.

[0047] Specifically, S100 acquires multiple lung CT images, each of which is a sample. The lung CT images constitute a data set for predicting EGFR mutation status.

[0048] The datasets used to predict EGFR mutation status were derived from two NSCLC collections, one private and one public. Inclusion criteria included: (1) pathologically diagnosed primary NSCLC. (2) EGFR mutation detection. Exclusion criteria included: (1) large artifacts or poor quality in CT images. (2) lack of genetic testing or poor tissue quality, resulting in detection failure.

[0049] Among them, the private dataset was collected from the cases of eligible patients diagnosed or treated in a tertiary hospital in a certain province in China. Finally, 420 cases (228 EGFR mutant and 192 EGFR wild-type cases) were collected as an internal dataset, which has been approved by the Institutional Review Board of the domestic tertiary hospital (No.k2021-014). The collected CT image slices were 1.0 mm thick, and the tumor areas were manually annotated by two experienced radiologists. The public dataset is provided by the Cancer Imaging Archive and includes 211 original cases from the Stanford University School of Medicine and the Palo Alto Veterans Health Care System. After reviewing the data according to the same inclusion criteria, 140 cases were found with CT image slice thickness ranging from 0.625 to 3.0 mm.

[0050] S200: Preprocess each lung CT image to obtain an image of a region of interest in each lung CT image. The preprocessing process includes:

[0051] S210 collected CT images of 560 patients with EGFR mutations in non-small cell lung cancer as a sample set. The lung CT image data was resampled using a trilinear interpolation algorithm, adjusting the voxel size to 1mm×1mm×1mm. This was done to achieve uniform size.

[0052] S220: perform pixel intensity clipping on each 1mm×1mm×1mm lung CT image to a range of -1000 to 400 HU.

[0053] At step S230 , the region of interest (ROI) of the tumor is cropped based on the tumor center point marked by the physician and its pixel size is adjusted to a 224×224 pixel image. The tumor center point is the equivalent physical center of the tumor region, as the tumor region is an irregular shape. The ROI is the tumor region.

[0054] Next, S300 is executed.

[0055] S300 uses a sliding window of 16×16 pixels to segment the tumor region of interest into multiple sub-regions, and each sub-region serves as an instance of the model input in this method.

[0056] Therefore, each 224×224 pixel image can be cut into 196 16×16 pixel blocks. Each block can be denoted by x i,j , then each region of interest image has an image block dataset X i ,,X i ={x i,1 , x i,2 ,...,x i,j}. i is the serial number of the ROI image, and j is the serial number of the image block. i = 1, 2, 3, ..., N. j = 1, 2, 3, ..., M. N and M are both positive integers. N is the number of ROI images, and M is the number of image blocks contained in a single ROI image.

[0057] Assign each 224×224 pixel image a true label Y i , giving each 16×16 pixel image block a true label y i,j .

[0058] The relationship between the true label of the region of interest image and the true label of the region of interest image of the image patch can be expressed as follows based on the constraints of multi-instance learning:

[0059]

[0060] Among them, Y i is the true label of the image of the region of interest, y i,j is the true label of the image block, M is the number of image blocks contained in a single region of interest image, i is the sequence number of the region of interest image, and j is the sequence number of the image block.

[0061] The meaning of this constraint expression is that the mutation result of the region of interest image with a size of 224×224 pixels is 1, so if there is at least one small image block in the region of interest image with a size of 224×224 pixels, the mutation result is 1. 1 means there is a mutation, and 0 means there is no mutation. This is a binary expression method.

[0062] Because these lung CT images used for training are already mature cases, they already have mutation results, so the labels generated here are called true labels.

[0063] It should be noted that in the subsequent description of this application, ROI images = packages, and image blocks = instances. That is, the concept of packages refers to the category of ROI images, and the concept of instances refers to the category of image blocks. This will not be further explained.

[0064] In this embodiment, by dividing the lung CT image into image blocks for instance-level feature extraction, the soft pseudo-labeling technology is used in a self-generated manner to provide additional supervisory signals for instance-level features, guide the multi-instance learning model to distinguish instances more effectively, and enhance the identification ability of instance-level features. Integrating unique relative spatial position information into the self-attention mechanism can flexibly and clearly mine the dependencies between instances, which can greatly improve the representation ability of heterogeneous tumors and take into account the importance of sub-regional interactions in the prediction process. The importance scores of different instance features are automatically learned through the adaptive gated instance aggregation module, which facilitates the dynamic weighting of feature aggregation instances. This method ensures that more information-rich features are emphasized in heterogeneous tumors, thereby improving the generalization of the model on different data sets. The prediction method provided in this application can delicately capture and analyze the heterogeneous characteristics within the tumor, ensuring that the prediction of EGFR mutation status is more accurate and targeted.

[0065] In one embodiment of the present application, S510 includes encoding all image blocks one by one using the feature encoder, and obtaining a preliminary prediction result for each image block based on the encoding result of each image block, including:

[0066] S511 , encoding all image blocks one by one using a feature encoder based on Formula 1 to generate feature encoding information for each image block.

[0067] x′ i,j =F r (x i,j :θ r ) Formula 1.

[0068] Where x′ i,j is the image block x i,j The feature encoding information, x i,j is the jth image block in the i-th region of interest image, i is the serial number of the region of interest image, j is the serial number of the image block, F r (·:θ r ) is the expression symbol of coding behavior.

[0069] S521 : Based on Formula 2, the feature coding information of each image block is input into the instance prediction head for processing, and a preliminary prediction result of each image block output by the instance prediction head is obtained.

[0070]

[0071] in, is the image block x i,j The preliminary prediction results, x′ i,j is the image block x i,j The feature encoding information, ψ p(·:θ p ) is a symbolic representation of the processing behavior performed by the instance prediction head.

[0072] Specifically, the feature encoder used is based on the ResNet architecture. The prediction head is the last layer of the feature encoder, responsible for converting the previously extracted features into the final prediction results.

[0073] In this example, CT images of small cell lung cancer (NSCLC) often have low contrast and high variability, which may result in insufficient discriminative power of the extracted instance features for non-invasive EGFR mutation prediction. Especially within heterogeneous tumor regions, the intensity difference between EGFR mutant and wild-type tumor tissue can be extremely subtle, significantly increasing the difficulty of learning discriminative features from complex CT images. This step generates features by encoding image blocks. Further processing by the prediction head allows for feature refinement and classification.

[0074] In one embodiment of the present application, S520 includes embedding the importance score of each image block in the preliminary prediction result of each image block to generate a soft pseudo label for each image block, including:

[0075] S521: Generate an importance score for each image block.

[0076] S522: Create an image block importance score set, and add all image block importance scores into the image block importance score set one by one.

[0077] S523 , based on the image block importance score set, generate a soft pseudo label package for each image block using Formula 3.

[0078]

[0079] in, is the image block x i,j Soft fake labels, is the image block x i,j The importance score of is the set of image block importance scores for the i-th region of interest image, is the indicator function, and k is the symbol representing the current number of training times. i is the true label of the image of the region of interest.

[0080] Specifically, when Y i＝1 hour, The value of is 1. R is the range of the given set. A higher importance score indicates that the instance is embedded with richer mutation information, which reflects the contribution of the instance to EGFR prediction to a certain extent. For CT images of wild-type EGFR, we use their mutation status to initialize the pseudo labels of each instance, which conforms to the constraints in multi-instance learning. According to formula 0, it can be understood that the true labels of all image blocks in the lung CT images without EGFR mutations are non-mutated image blocks. For CT images with EGFR mutations, due to the heterogeneity within the tumor, all image blocks in the lung CT images may contain both mutated image blocks and non-mutated image blocks. Because we first calculate the importance score of each image block instance to indicate the contribution of each image block instance to the entire region of interest image, and then scale the image blocks in the region of interest image using formula 4 to obtain soft pseudo labels from the importance scores.

[0081] In this embodiment, soft false labels are generated based on the importance scores of instance features, which can further enhance the learning ability of instance features.

[0082] In one embodiment of the present application, S530 includes creating a spatially perceptual Transformer module having a multi-layer encoder structure, mining the spatial relative position relationship between different image blocks based on the spatially perceptual Transformer module having a multi-layer encoder structure, and obtaining comprehensive features of each image block, including:

[0083] S531: Create τ transformer encoder layers. τ is a positive integer and τ is greater than 2.

[0084] S532: Select a region of interest image.

[0085] S533: Input feature coding information of all image blocks of a region of interest image into a first transformer encoder layer, and calculate an output result of the first transformer encoder layer according to Formula 4.

[0086]

[0087] in, is the output of the first transformer encoder layer, x′ i,j is the image block x i,j The feature encoding information of , i is the serial number of the image of the region of interest, j is the serial number of the image block, is the class label feature of the i-th region of interest image, E c is the linear projection matrix, P i ab is the absolute position code of the i-th region of interest image.

[0088] S534, input the output result of the previous transformer encoder layer to the next transformer encoder layer, and calculate the encoding intermediate amount of the next transformer encoder layer according to Formula 5.

[0089]

[0090] in, is the intermediate quantity for encoding in the next transformer encoder layer, is the output of the previous transformer encoder layer, SA_SRPE[·] is the self-attention expression including spatial relative position encoding, for The output after the first normalization layer in the next transformer encoder layer.

[0091] S535, calculate the output result of the next transformer encoder layer according to Formula 6.

[0092]

[0093] in, is the output of the next transformer encoder layer, is the intermediate quantity for encoding in the next transformer encoder layer, for The output result after the second normalization layer in the next transformer encoder layer, for The output of the feed-forward network layer after the next transformer encoder layer.

[0094] S536 returns to S534, i.e., returns to the step of inputting the output result of the previous transformer encoder layer to the next transformer encoder layer, calculating the encoding intermediate quantity of the next transformer encoder layer according to Formula 5, until the output result of the last transformer encoder is obtained, and the output result of the last transformer encoder is used as the image block comprehensive feature set of the region of interest image i. Each element in the image block comprehensive feature set is a comprehensive feature of each image block of the region of interest image i.

[0095] S537, returning to S532, that is, returning to the step of selecting a region of interest image, until all region of interest images are selected.

[0096] Specifically, the comprehensive feature of each image block of the region of interest image i finally obtained in S536 is h i,j A comprehensive feature set H of the region of interest image i can be generated i . H i ={h i,1 ,h i,2 ,...,...h i,j},H i ∈R M×dx . H i ∈A matrix with M rows and dx columns. R M×dx It means a matrix with M rows and dx columns. If there is no special additional explanation, the symbol R at the end means the matrix.

[0097] Each transformer encoder layer of the encoder consists of a feedforward network (FFN), two layers of normalization (LN), and a self-attention layer with spatial relative position encoding. The specific connection relationship is: first normalization layer - SRP layer - second normalization layer - feedforward network layer.

[0098] Formula 5 is performed through the first normalization layer and the SRP layer, and Formula 6 is performed through the second normalization layer and the feedforward network layer. The SRP layer is also called the SA_SRPE layer.

[0099] To capture the overall characterization of heterogeneous tumors, radiologists often consider the relationships between different pathological subregions of CT images. For example, the spatial relationship between the tumor and surrounding areas, such as peritumoral pleural traction or vascular convergence, is crucial for inferring the elusive EGFR mutation status in complex CT images. Therefore, failure to capture the relationship between instances can limit the overall representation of the tumor. In this example, by designing a spatially aware Transformer to actively explore the dependencies between instances, the representation of heterogeneous tumors can be improved.

[0100] In one embodiment of the present application, S530 further includes creating a spatial perception Transformer module with a multi-layer encoder structure, mining the spatial relative position relationship between different image blocks based on the spatial perception Transformer module with a multi-layer encoder structure, and obtaining a comprehensive feature of each image block, further including:

[0101] S538, construct a self-attention expression including spatial relative position encoding to couple the spatial relative position encoding information and self-attention. The self-attention expression including spatial relative position encoding is shown in Formula 7.

[0102]

[0103] Where SA_SRPE[·] is the self-attention expression including spatial relative position encoding, X′ i is a set of feature coding information of each image block in the i-th region of interest image, x′ i,j is the image block x i,j The feature encoding information of , S(·) is the expression of the softmax function, W Q is the query parameter projection matrix, WK is the parameter projection matrix of the key, W V is the parameter projection matrix of the value, M is the number of image blocks contained in a single region of interest image, dx is the first preset matrix attribute parameter, H is the height dimension symbol, W is the width dimension symbol, x′ i,A and x′ i,B is the feature encoding information of the randomly selected image block A and image block B in the same region of interest image i, T is the transposed matrix symbol, is the height parameter matrix, is the width parameter matrix, is a learnable encoding of the relative height of image block A and image block B, is a learnable encoding of the relative widths of image blocks A and B.

[0104] Specifically, the extracted patches,image blocks are first arranged according to their spatial positions, and their relative heights and widths are,encoded according to the spatial information of the tumor CT image.,Next, the relative position encoding information between two random,instances is used to embed the attention feature.

[0105] S538 can be executed at the very beginning of S500. S538 can also be executed at the very end of S500 by substituting Formula 7 into Formula 5. The purpose of Formula 7 is to define SA_SRPE[·] in Formula 5.

[0106] One limitation of the ordinary Transformer used in existing methods is that it only uses absolute position encoding, which may not fully capture subtle position dependency information, which is indispensable for exploring the spatial relationship between highly complex tumors and their surrounding areas. To overcome this challenge, we integrate spatial relative position encoding into the self-attention mechanism to promote the exploration of relationships between instances. We first arrange the extracted patches and image blocks according to their spatial positions and encode their relative height and width according to the spatial information of the tumor CT image. Then, by randomly selecting two image blocks: image block A and image block B, their direct relative forged encoding information is embedded in the attention feature. W Q is the query parameter projection matrix, W K is the parameter projection matrix of the key, W V is the parameter projection matrix of the value, corresponding to the query, key and value in the attention mechanism.

[0107] In this embodiment, integrating spatial relative position encoding into the self-attention mechanism can effectively promote the exploration of relationships between instances. Unlike absolute position encoding that only captures static position information and implicitly implies spatial relationships, the spatial relative position encoding we proposed is good at flexibly and explicitly mining dependencies between instances, ensuring that key spatial relationships are not ignored, allowing the model to skillfully identify potential interactions between tumors and surrounding peritumoral areas, thereby improving the characterization ability of heterogeneous tumors and improving the prediction accuracy of EGFR mutations.

[0108] In one embodiment of the present application, S521 includes, that is, generating the importance score of each image block includes:

[0109] S521a: Generate a weight coefficient for each image block according to Formula 8.

[0110]

[0111] Among them, a i,j is the image block x i,j The weight coefficient is , W is the first learnable parameter matrix, U is the second learnable parameter matrix, μ is the third learnable parameter matrix, dx is the first preset matrix attribute parameter, dz is the second preset matrix attribute parameter, ⊙ is the element multiplication symbol, σ is the expression symbol of the S-type activation function, is the image block x i,j The comprehensive characteristics of h i,j The transpose of .

[0112] S521b: Generate an importance score for each image block according to Formula 8.

[0113]

[0114] in, is the image block x i,j The importance score, a i,j is the image block x i,j The weight coefficient is , j is the sequence number of the image block, i is the sequence number of the region of interest image, and M is the number of image blocks contained in a single region of interest image.

[0115] Specifically, due to the intratumor heterogeneity of NSCLC, the mutation status of EGFR varies significantly between different tumor subregions, resulting in uneven contributions to the overall EGFR mutation status of the patient. Therefore, a key aspect when generating bag-level features is to prioritize informative instances to ensure that the aggregated CT image features produce a high signal-to-noise ratio, thereby maintaining robustness to unseen tumors. Otherwise, the prediction model may rely on uninformative noisy instances, resulting in decreased generalization performance on unseen samples. In this method, this problem is addressed by an adaptive gating procedure that identifies valuable instances from different tumor subregions and highlights their contribution to feature aggregation.

[0116] In this embodiment, the comprehensive feature set H of the region of interest image i obtained based on the above content i , by introducing the adaptive gated instance aggregation module from H i Learning the feature importance scores of specific image blocks can improve the generalization ability of the model.

[0117] In one embodiment of the present application, S521 further includes, that is, generating the importance score of each image block, further including:

[0118] S521c, generating a weight coefficient vector set a based on the weight coefficients of each image block in each region of interest image i , a i ={a i,1 ,a i,2 ,a i,3 ,...,...a i,j}.

[0119] S521d, maps the weight coefficient vector set to a unit vector space with L2 normalization to generate a standardized scalar set v i , v i The expression of is shown in formula 10.

[0120]

[0121] Among them, v i is a standardized scalar set, a i is a weight coefficient vector set, j is the sequence number of the image block, i is the sequence number of the region of interest image, and M is the number of image blocks contained in a single region of interest image.

[0122] S521e, converting the normalized scalar set into a modified importance score set according to Formula 11.

[0123]

[0124] in, is the image block importance score set of the i-th region of interest image, v i is a normalized scalar set, and ⊙ is the element-wise multiplication symbol.

[0125] Specifically, one limitation of the adaptive gating module is that the softmax function may not sufficiently sharpen the importance scores. Due to the high heterogeneity of tumors, this may hinder the signal-to-noise ratio of the bag feature representation aggregated in the feature space, because the mutation status informed by the positive instances (positive instances: refers to image patches with mutations) and their relationships may be overwhelmed by the useless information in a large number of negative instances (negative instances: refers to image patches without mutations) from the background or tumor subregions with insufficient information (such as stromal tissue). Therefore, in order to promote the learning of instance scores by the adaptive gating module in this embodiment, we first map the weight coefficient vector set into a unit vector space with L2 normalization, v i Each element of represents a normalized scalar indicating the relative contribution of the corresponding instance.

[0126] Next, we convert these elements into importance scores in the range of [0,1] using Formula 11, calculate the modified importance score vector by applying the Hadamard product, and use the modified importance score vector as the image block importance score set.

[0127] In this embodiment, compared with a single softmax function, the processing method of this embodiment can better amplify the variance of the importance score, thereby encouraging the gating mechanism to focus more on information instances that are valuable for feature aggregation.

[0128] In one embodiment of the present application, S540 includes the adaptive gated instance aggregation module aggregating the comprehensive features of each image block and the importance score of each image block using a dynamic weighting method to obtain a packet-level CT image feature of each region of interest image, including:

[0129] S541, obtaining the packet-level CT image features of each region of interest image according to Formula 12.

[0130]

[0131] in, is the packet-level CT image feature of the image block of the i-th region of interest image, is x i,j The importance score of the image block, h i,j is the image block x i,j comprehensive characteristics.

[0132] Specifically, in this way, the packet-level CT image features of each region of interest image may include both the comprehensive features of the image block and the importance score of the image block.

[0133] The adaptive gated instance aggregation module proposed in this embodiment can effectively emphasize the contribution of valuable instance features and reduce the adverse effects of noise instances in heterogeneous tumors.

[0134] In one embodiment of the present application, S550 includes, that is, constructing the prediction loss function of the package feature, the prediction loss function of the class label feature, and the supervision loss function respectively, including:

[0135] S551: Obtain the true label of each region of interest image. The true label includes the actual gene mutation prediction result.

[0136] S552, define an expression of the loss function, the expression of the loss function is shown in Formula 13.

[0137]

[0138] Where N is the number of lung CT images, Y i is the true label of the image in the region of interest, λ is the weighting factor for balancing positive and negative samples, is a factor that balances the weight of difficult samples, ξ is a smoothing parameter, is the true label after smoothing, P i is the prediction result of the image of the region of interest.

[0139] S553: ​​Construct a prediction loss function of the class label feature. The expression of the prediction loss function of the class label feature is shown in Formula 14.

[0140]

[0141] in, is the prediction loss function of the class label feature, is the true label after smoothing, P i cla is the prediction result of the class label feature.

[0142] S554: Construct a prediction loss function of the packet feature. The expression of the prediction loss function of the packet feature is shown in Formula 15.

[0143]

[0144] in, is the prediction loss function of the package feature, is the true label after smoothing, is the prediction result of the package feature.

[0145] S555: Construct a supervision loss function. The expression of the supervision loss function is shown in Formula 16.

[0146]

[0147] in, is the supervised loss function, N is the number of region of interest images, M is the number of image patches contained in a single region of interest image, is the image block x i,j Soft fake labels, is the image block x i,j The importance score of .

[0148] Specifically, the loss function introduced in this embodiment is a focus loss function, which can eliminate the class imbalance problem that may exist in EGFR prediction and adjust the focus on samples of different categories through weighting factors, thereby improving the robustness of the model.

[0149] Formula 14 and Formula 15 use the real label. The lung CT images used for training are already mature cases, so they already have mutation results and therefore have real labels. i cla The prediction result of the class label feature is the prediction result of the class label feature. The prediction result of the class label feature is the prediction result of the model based on the class label feature. The prediction results obtained in formula 15 The prediction result of the package feature is the prediction result of the model based on the package feature. The prediction results are obtained. Package features are package-level CT image features

[0150] The prediction results of the class marker features are obtained through the MLP (MLP, Multilayer Perceptron) module in the model, and the prediction results of the package features are obtained through the package classifier head in the model. How the prediction results of the class marker features and the prediction results of the package features are specifically obtained is not the focus of protection of this application. The principle also adopts the more existing and common means in artificial intelligence machine learning, which will not be explained in detail here.

[0151] In addition, label smoothing technology is used to further improve the generalization ability of the model. to achieve.

[0152] The supervised loss function can enhance the model's ability to capture subtle features, explicitly supervise and enhance instance feature learning of tumor CT images, and cooperate with pseudo labels to provide additional supervisory signals, effectively improving the overall performance of EGFR mutation prediction.

[0153] In one embodiment of the present application, S560 includes constructing an overall loss function using the prediction loss function of the package feature, the prediction loss function of the class label feature, and the supervision loss function, including:

[0154] S561, construct an overall loss function according to Formula 17.

[0155]

[0156] in, is the overall loss function, is the prediction loss function of the package feature, is the prediction loss function of the class label feature, is the supervised loss function, α is the first trade-off factor, and β is the second trade-off factor.

[0157] Specifically, α is the first trade-off factor and β is the second trade-off factor, both of which can be pre-set constants. The goal of training the model k times is to minimize the value of the overall loss function after each training, and ultimately minimize the value of the overall loss function after the last training.

[0158] In the next training in S570 , minimizing the value of the overall loss function is used as a training goal, that is, the training goal is: in the process of k training times, making the value of the overall loss function smaller and smaller.

[0159] In this example, we used class marker features and package features, as well as supervisory factors, to comprehensively predict EGFR mutation status, and the model had high prediction accuracy.

[0160] The various technical features of the above-described embodiments can be combined arbitrarily, and the execution order of the method steps is not restricted. In order to make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0161] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for predicting lung cancer EGFR gene mutations based on multi-instance learning and Transformer technology, characterized by: The method comprises: Obtain multiple lung CT images; Each lung CT image is preprocessed to obtain an image of a region of interest in each lung CT image; Divide each region of interest image into multiple image blocks of equal size; Create an EGFR mutation prediction model; Training the EGFR mutation prediction model k times to obtain a trained EGFR mutation prediction model, wherein k is a positive integer and k is greater than 1; Obtaining a lung CT image to be tested, and inputting the lung CT image to be tested into the trained EGFR mutation prediction model; Starting the trained EGFR mutation prediction model and obtaining a prediction result output by the trained EGFR mutation prediction model; The training of the EGFR mutation prediction model k times to obtain the trained EGFR mutation prediction model comprises: Using the feature encoder to encode all image blocks one by one, and obtaining the preliminary prediction result of each image block based on the encoding result of each image block; The importance score of each image block is embedded in the preliminary prediction result of each image block to generate a soft pseudo label for each image block; Create a spatial perception Transformer module with a multi-layer encoder structure. Based on the spatial perception Transformer module with a multi-layer encoder structure, it mines the spatial relative position relationship between different image blocks and obtains the comprehensive features of each image block; Based on the adaptive gated instance aggregation module, the comprehensive features of each image block and the importance score of each image block are aggregated using a dynamic weighting method to obtain the packet-level CT image features of each region of interest image; Construct the prediction loss function of the package feature, the prediction loss function of the class label feature, and the supervision loss function respectively; Construct an overall loss function using the prediction loss function of the bag feature, the prediction loss function of the class label feature, and the supervision loss function; In the next training, the value of the overall loss function is minimized as the training goal.

2. The lung cancer EGFR gene mutation prediction method based on multi-instance learning and Transformer technology according to claim 1, characterized in that: The method of encoding all image blocks one by one using a feature encoder and obtaining a preliminary prediction result of each image block according to the encoding result of each image block includes: Based on formula 1, all image blocks are encoded one by one using a feature encoder to generate feature encoding information for each image block; x' i,j =F r (x i,j :θ r ) Formula 1; Among them, x' i,j is the image block x i,j The feature encoding information, x i,j is the jth image block in the i-th region of interest image, i is the serial number of the region of interest image, j is the serial number of the image block, F r (·:θ r ) is an expression symbol for coding behavior; Based on Formula 2, the feature encoding information of each image block is input into the instance prediction head for processing, and the preliminary prediction result of each image block output by the instance prediction head is obtained; in, is the image block x i,j The preliminary prediction results, x' i,j is the image block x i,j The feature encoding information, ψ p (·:θ p ) is a symbolic representation of the processing behavior performed by the instance prediction head.

3. The lung cancer EGFR gene mutation prediction method based on multi-instance learning and Transformer technology according to claim 2, characterized in that: The embedding of the importance score of each image block into the preliminary prediction result of each image block to generate a soft pseudo label for each image block includes: Generate an importance score for each image patch; Creating an image block importance score set, and adding all image block importance scores into the image block importance score set one by one; Based on the image block importance score set, use Formula 3 to generate a soft pseudo label package for each image block; in, is the soft pseudo label of the image block xi,j, is the importance score of image block xi,j, is the set of image block importance scores for the i-th region of interest image, is the indicator function, k is the symbol expressing the current number of training times, Y i is the true label of the image of the region of interest.

4. The lung cancer EGFR gene mutation prediction method based on multi-instance learning and Transformer technology according to claim 3, characterized in that: The spatial perception Transformer module with a multi-layer encoder structure is created. Based on the spatial perception Transformer module with a multi-layer encoder structure, the spatial relative position relationship between different image blocks is mined to obtain the comprehensive features of each image block, including: Create τ transformer encoder layers; τ is a positive integer and τ is greater than 2; Select an image of a region of interest; Input the feature encoding information of all image blocks of a region of interest image into the first transformer encoder layer, and calculate the output result of the first transformer encoder layer according to Formula 4; in, is the output of the first transformer encoder layer, x' i,j is the image block x i,j The feature encoding information of , i is the serial number of the image of the region of interest, j is the serial number of the image block, is the class label feature of the i-th region of interest image, E c is the linear projection matrix, P i ab Encode the absolute position of the i-th region of interest image; The output result of the previous transformer encoder layer is input to the next transformer encoder layer, and the encoding intermediate quantity of the next transformer encoder layer is calculated according to Formula 5; in, is the intermediate quantity for encoding in the next transformer encoder layer, is the output of the previous transformer encoder layer, SA_SRPE[·] is the self-attention expression including spatial relative position encoding, for The output after the first normalization layer in the next transformer encoder layer; Calculate the output of the next transformer encoder layer according to Formula 6; in, is the output of the next transformer encoder layer, is the intermediate quantity for encoding in the next transformer encoder layer, for The output result after the second normalization layer in the next transformer encoder layer, for The output after the feedforward network layer in the next transformer encoder layer; Return the output result of the previous transformer encoder layer to the next transformer encoder layer, calculate the encoding intermediate amount of the next transformer encoder layer according to Formula 5, until the output result of the last transformer encoder is obtained, and use the output result of the last transformer encoder as the image block comprehensive feature set of the region of interest image i; each element in the image block comprehensive feature set is the comprehensive feature of each image block of the region of interest image i; Return to the step of selecting a region of interest image until all region of interest images are selected.

5. The method for predicting lung cancer EGFR gene mutations based on multi-instance learning and Transformer technology according to claim 4, characterized in that: The step of creating a spatial perception Transformer module having a multi-layer encoder structure, mining the spatial relative position relationship between different image blocks based on the spatial perception Transformer module having a multi-layer encoder structure, and obtaining comprehensive features of each image block further includes: Construct a self-attention expression including spatial relative position encoding to couple the spatial relative position encoding information with self-attention. The self-attention expression including spatial relative position encoding is shown in Formula 7. Where SA_SRPE[·] is the self-attention expression including spatial relative position encoding, X' i is a set of feature coding information of each image block in the i-th region of interest image, x' i,j is the image block x i,j The feature encoding information of , S(·) is the expression of the softmax function, W Q is the query parameter projection matrix, W K is the parameter projection matrix of the key, W V is the parameter projection matrix of the value, M is the number of image blocks contained in a single region of interest image, dx is the first preset matrix attribute parameter, H is the height dimension symbol, W is the width dimension symbol, x' i,A and x' i,B is the feature encoding information of the randomly selected image block A and image block B in the same region of interest image i, T is the transposed matrix symbol, is the height parameter matrix, is the width parameter matrix, is a learnable encoding of the relative height of image block A and image block B, is a learnable encoding of the relative widths of image blocks A and B.

6. The method for predicting lung cancer EGFR gene mutations based on multi-instance learning and Transformer technology according to claim 5, characterized in that: Generating the importance score of each image block includes: Generate the weight coefficient of each image block according to formula 8; Among them, a i,j is the image block x i,j The weight coefficient is , W is the first learnable parameter matrix, U is the second learnable parameter matrix, μ is the third learnable parameter matrix, dx is the first preset matrix attribute parameter, dz is the second preset matrix attribute parameter, ⊙ is the element multiplication symbol, σ is the expression symbol of the S-type activation function, is the image block x i,j The comprehensive characteristics of h i,j The transpose of Generate the importance score of each image block according to formula 8; in, is the image block x i,j The importance score, a i,j is the image block x i,j The weight coefficient is , j is the sequence number of the image block, i is the sequence number of the region of interest image, and M is the number of image blocks contained in a single region of interest image.

7. The method for predicting lung cancer EGFR gene mutations based on multi-instance learning and Transformer technology according to claim 6, characterized in that: Generating the importance score of each image block further includes: A weight coefficient vector set a is generated based on the weight coefficients of each image block in each region of interest image i , a i ={a i,1 ,a i,2 ,a i,3 ,...,...a i,j }; Map the weight coefficient vector set into a unit vector space with L2 normalization to generate a standardized scalar set v i , v i The expression of is shown in formula 10; Among them, v i is a standardized scalar set, a i is a weight coefficient vector set, j is the sequence number of the image block, i is the sequence number of the region of interest image, and M is the number of image blocks contained in a single region of interest image; According to formula 11, the normalized scalar set is converted into a modified importance score set; in, is the image block importance score set of the i-th region of interest image, v i is a normalized scalar set, and ⊙ is the element-wise multiplication symbol.

8. The method for predicting lung cancer EGFR gene mutations based on multi-instance learning and Transformer technology according to claim 7, characterized in that: The adaptive gated instance aggregation module aggregates the comprehensive features of each image block and the importance score of each image block in a dynamic weighted manner to obtain the packet-level CT image features of each region of interest image, including: The packet-level CT image features of each region of interest image are obtained according to formula 12; in, is the packet-level CT image feature of the i-th region of interest image, is the image block x i,j The importance score, h i,j is the image block x i,j comprehensive characteristics.

9. The method for predicting lung cancer EGFR gene mutations based on multi-instance learning and Transformer technology according to claim 8, characterized in that: The respectively constructed prediction loss function of the package feature, the prediction loss function of the class label feature, and the supervision loss function include: Obtain the true label of each region of interest image; the true label includes the actual gene mutation prediction result; Define the expression of the loss function. The expression of the loss function is shown in Formula 13; Where N is the number of lung CT images, Y i is the true label of the image in the region of interest, λ is the weighting factor for balancing positive and negative samples, is a factor that balances the weight of difficult samples, ξ is a smoothing parameter, is the true label after smoothing, P i is the prediction result of the image of the region of interest; Construct a prediction loss function of the class label feature. The expression of the prediction loss function of the class label feature is shown in Formula 14. in, is the prediction loss function of the class label feature, is the true label after smoothing, P i cla is the prediction result of the class label feature; Construct a prediction loss function of the packet feature. The expression of the prediction loss function of the packet feature is shown in Formula 15. in, is the prediction loss function of the package feature, is the true label after smoothing, is the prediction result of the package feature; Construct a supervised loss function. The expression of the supervised loss function is shown in Formula 16. in, is the supervised loss function, N is the number of region of interest images, M is the number of image patches contained in a single region of interest image, is the image block x i,j Soft fake labels, is the image block x i,j The importance score of .

10. The lung cancer EGFR gene mutation prediction method based on multi-instance learning and Transformer technology according to claim 9, characterized in that: The overall loss function is constructed by using the prediction loss function of the package feature, the prediction loss function of the class label feature, and the supervision loss function, including: Construct the overall loss function according to formula 17; in, is the overall loss function, is the prediction loss function of the package feature, is the prediction loss function of the class label feature, is the supervised loss function, α is the first trade-off factor and β is the second trade-off factor.

Images