Gastric cancer postoperative recurrence prediction method and device based on cross attention
By predicting postoperative recurrence of gastric cancer based on cross attention, the problems of low fusion efficiency of multimodal data and insufficient feature interaction are solved, high accuracy and individualized prediction are achieved, and the interpretability of the model is improved.
Patent Information
- Application Number
- CN202510275588.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-10
- Publication Date
- 2025-06-20
AI Technical Summary
The existing methods for predicting postoperative recurrence of gastric cancer have problems in the low efficiency of multimodal data fusion, insufficient feature interaction, excessive memory usage and low interpretation of results.
A method based on cross attention is adopted to generate multiple patch maps through pyramid compression and segmentation, image features are extracted in combination with UNI encoder, and feature fusion is carried out through a weakly supervised multi-instance learning framework and cross attention mechanism to generate fusion features for prediction.
It improves the accuracy and individualized prediction ability of postoperative recurrence prediction of gastric cancer, reduces memory usage, enhances feature interaction and model interpretability, and achieves the prediction performance of AUC 0.866.
Smart Images
Figure CN120183612A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to a method and device for predicting gastric cancer recurrence after surgery based on cross-attention, belonging to the technical field of medical artificial intelligence. Background Art
[0002] Predicting a patient's postoperative recurrence is a key to targeted treatment for gastric cancer patients. Pathological images can directly observe the diseased tissue and play an important role in diagnosis. However, pathological images have characteristics such as excessive memory and usually far fewer lesion areas than normal areas. At the same time, direct observation also requires a certain threshold, the diagnosis process is long, and the consistency of the diagnosis results is relatively low. The expected results obtained by only observing and judging with pathological images are not good.
[0003] Therefore, it is very meaningful to develop a method for predicting gastric cancer recurrence after surgery that can make full use of patient information to predict the recurrence probability of patients.
[0004] The latest progress in digital pathology can help solve this problem. The development of image analysis technology based on machine learning (ML), especially deep learning (DL) technology, shows that they can assist in diagnosis, identify new features, predict the prognosis of patients, and assist in individualized treatment, but there are often certain limitations. Traditional gastric cancer recurrence prediction mainly relies on single-modal data, such as imaging or clinical indicators, etc. However, these methods have certain limitations in terms of accuracy and individualized prediction. By combining clinical information and medical images, researchers can build a more accurate prediction model. However, most of the existing methods cannot fully capture the relationships between multi-modal information; feature splicing or weighted averaging cannot dynamically adjust the weights between modalities for prediction; the interpretability of the results is relatively low, etc. Therefore, in the following work, the present application is committed to developing a multi-modal and interpretable model with good performance in predicting gastric cancer recurrence after surgery. Summary of the Invention
[0005] Objective: In view of at least one of the above technical problems, the present application provides a method and device for predicting gastric cancer recurrence after surgery based on cross-attention, which is suitable for solving the effective fusion of multi-modal data, effectively using clinical features and image features, improving prediction performance, and accurately predicting the postoperative recurrence of cancer patients.
[0006] The technical solution adopted by the present application is as follows: In the first aspect, the present application provides a method for predicting gastric cancer recurrence after surgery based on cross-attention, including: Obtaining the whole slide image of the gastric pathology and clinical data of the target patient; Performing pyramidal compression and segmentation on the whole slide image of the gastric pathology to generate multiple patch images, and extracting image features from the patch images through an encoder, where the image features include multiple patch image features; Extract features and normalize clinical data to obtain clinical features; Input the image features and clinical features into the postoperative gastric cancer recurrence prediction model to obtain the postoperative gastric cancer recurrence prediction result; the processing process of the postoperative gastric cancer recurrence prediction model includes: Based on the weakly supervised multi-instance learning framework, aggregate the multiple patch features of the image features through the introduction of clustering constraints in the attention mechanism to obtain the aggregated patch-level features; use the cross-attention mechanism to perform weighted fusion on the aggregated patch-level features and clinical features to generate the fused features; input the fused features into the classifier to obtain the postoperative gastric cancer recurrence prediction result.
[0007] In a second aspect, the present application provides a postoperative gastric cancer recurrence prediction device based on cross-attention, including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the method according to the first aspect.
[0008] In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method according to the first aspect is implemented.
[0009] In a fourth aspect, the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the method according to the first aspect is implemented.
[0010] Beneficial effects: The method and device for predicting gastric cancer recurrence after surgery based on cross-attention provided by this application have the following advantages: Aiming at the problems of low efficiency in multi-modal medical data fusion, excessive memory occupation of high-resolution pathological images, and insufficient feature interaction in the prior art, for high-quality whole-slide pathological images, the memory efficiency is optimized through pyramid-shaped image layering, and the cross-attention mechanism is adopted to achieve dynamic interactive weighting fusion of clinical information and pathological image features, and the multi-instance learning framework is combined to screen tissue region features with high diagnostic value. Image features are extracted through the UNI encoder, clinical data is processed by L2 normalization, and learnable weight parameters are constructed to achieve modal balance. Finally, recurrence prediction is completed by aggregating weighted instance features. The model achieves a prediction performance of AUC 0.866 on the test set, and is mainly applied to assist doctors in accurately evaluating the postoperative recurrence risk of gastric cancer patients based on H&E staining sections and clinical information, providing data support for personalized treatment decisions. It has the following advantages: (1) It more fully integrates feature information of different modalities, and significantly improves the probability of predicting gastric cancer recurrence by combining cross-attention on the basis of existing methods. (2) It provides a visual heat map, which can view the parts of interest to the model and improve the interpretability of the model. (3) Using pyramid-shaped processing of images greatly reduces the video memory occupation, and dropout and L2 normalization are performed on clinical features to enhance feature interaction and fusion. Description of the Drawings
[0011] Figure 1 It is a schematic flowchart of a method for predicting gastric cancer recurrence after surgery based on cross-attention according to an embodiment of the present application; Figure 2 It is a schematic flowchart of a model training and usage method according to an embodiment of the present application; Figure 3 It is a generated visual heat map according to an embodiment of the present application. Detailed Embodiments
[0012] The present application will be further described below with reference to the drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.
[0013] In the description of the present application, the meaning of several is more than one, the meaning of multiple is more than two, greater than, less than, exceeding, etc. are understood as not including the present number, above, below, within, etc. are understood as including the present number. If there is a description of first and second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features.
[0014] In the description of this application, the description with reference to terms such as "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.
[0015] The term "and / or" is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Additionally, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0016] Embodiment 1: This embodiment provides a method for predicting gastric cancer recurrence after surgery based on cross-attention, as Figure 1 shown, including: Obtain the whole slide image of gastric pathology and clinical data of the target patient; Perform pyramidal compression and segmentation on the whole slide image of gastric pathology to generate multiple patch images, and extract image features from the patch images through an encoder. The image features include multiple patch image features; Extract features from the clinical data and perform normalization to obtain clinical features; Input the image features and clinical features into the gastric cancer recurrence prediction model after surgery to obtain the gastric cancer recurrence prediction result after surgery; wherein the processing process of the gastric cancer recurrence prediction model after surgery includes: based on a weakly supervised multi-instance learning framework, aggregate the multiple patch image features of the image features through the introduction of clustering constraints in the attention mechanism to obtain aggregated patch-level features; use the cross-attention mechanism to perform weighted fusion of the aggregated patch-level features and clinical features to generate fused features; input the fused features into a classifier to obtain the gastric cancer recurrence prediction result after surgery.
[0017] In some embodiments, the encoder uses a UNI encoder. In this embodiment, the UNI encoder outputs a 1024-dimensional image feature vector. In some embodiments, the training process of the encoder includes: Obtain the whole slide image of gastric pathology of gastric cancer patients, where the whole slide image of gastric pathology has a label, and the label indicates recurrence or non-recurrence after surgery for the corresponding gastric cancer patient; Perform pyramidal multi-resolution segmentation on the whole slide image of gastric pathology to generate multiple patch images, and construct a labeled patch image dataset; Use the patch image dataset to train the encoder to obtain a trained encoder.
[0018] In this embodiment, the whole slide images of the stomach pathology of 380 gastric cancer patients after screening are processed in a pyramid shape to compress the image memory and improve the processing efficiency; then the tissues are segmented and any holes are excluded, and then the image is segmented into 256X256 patch images; then the pre-trained UNI encoder is used to extract image features.
[0019] We used a variety of models as encoders to extract image features respectively. While ensuring that the hyperparameters and other conditions remain unchanged, we used the same method to discover the encoder with the best performance to extract image features. During this process, we tried to use Resnet50, Conch, and UNI pre-trained on ImageNet as encoders to extract features. Finally, we found that the fusion training effect obtained by using the UNI encoder is the best, as shown in Table 1. By comparing the training effects of UNI, Resnet50, and CONCH in extracting features to select the encoder, it was found that the AUC value of the UNI encoder is the highest. Among them, the effect of using RESNET50 as the encoder is slightly inferior to that of the UNI encoder, and the effect of using CONCH as the encoder is worse than that of the RESNET50 encoder.
[0020] Table 1
[0021] In this embodiment, the clinical data includes gender, age, invasion humidity, lymph node metastasis, distant metastasis, carbohydrate antigen CA19-9, carcinoembryonic antigen CEA, alpha-fetoprotein AFP, neutrophils, lymphocytes, and inflammatory prognostic index IPI.
[0022] For clinical data, it is generally preprocessed first: columns that are not suitable for predicting recurrence and columns with a large number of missing values are screened and removed, and columns with a small number of missing values are filled with the median; then feature extraction is performed on the preprocessed clinical data.
[0023] In some embodiments, feature extraction and normalization are performed on clinical data to obtain clinical features, including: After multiple clinical data are respectively subjected to preliminary feature extraction and compression through the first fully connected layer, nonlinearity is introduced through the RELU activation function, then Dropout is used to reduce overfitting, and then mapping is performed through the second fully connected layer (mapping the 256-dimensional intermediate features to the final 128-dimensional embedding dimension), and finally normalization is performed through L2 norm normalization (scaling the norms of different vectors to the same size so that the clinical feature vectors are comparable), obtaining multiple clinical features.
[0024] In this embodiment, the cross-attention mechanism is used for feature fusion. By comparing different methods, the fusion effect obtained using the cross-attention mechanism is the best, as shown in Table 2. Cross-attention can weight according to the importance of each feature, so that the fused features can reflect the joint expression of images and clinical information. In this way, the model can capture the mutual relationship between two different modalities (images and clinical data).
[0025] Table 2
[0026] The fused features are input into the classifier for final prediction. The classifier usually consists of one or more fully connected layers and finally outputs the prediction result of gastric cancer recurrence after surgery. The prediction result can be the predicted recurrence probability or the classification label.
[0027] In some embodiments, the training process of the gastric cancer recurrence prediction model includes: Collect the image features and corresponding clinical features of multiple labeled gastric cancer patients as the training dataset. Combine the total loss function and use the training dataset to train the gastric cancer recurrence prediction model to obtain a trained gastric cancer recurrence prediction model.
[0028] During the training process, cross-entropy loss and instance-level loss are used to optimize the classification performance of the model.
[0029] The model continuously updates the network parameters by minimizing the total loss function and gradually improves the prediction accuracy. The model also includes an instance-level learning process, in which learning is carried out through an instance-level classifier. These classifiers classify each input instance (i.e., each small block in the image or clinical feature) and calculate the instance-level loss. During the training process, the instance-level loss is learned through an optimization objective function to improve the performance of the model on a single instance and help the model more finely adjust the importance of each feature. This multi-instance learning method can effectively improve the ability of the model to handle imbalanced data or sparse labels.
[0030] It should be noted that based on the weakly supervised multi-instance learning framework, the whole-slide pathology image of each patient is a bag, which contains several patch-level features (instances), and only one bag-level label (whether the slice of this disease is a high-risk recurrence patient / non-recurrence) is required, without the need to label each patch separately. This is the core of multi-instance learning: only given the bag-level label, the model needs to automatically learn which patches are the most critical. The model will learn the contribution degree of each patch to the final classification through the attention mechanism, introduce clustering constraints in the attention mechanism, restrict the patches within the same cluster to have similar attention or similar feature distributions, and separate different patches more widely, so as to better capture similar lesion regions in the pathology image, thereby improving the separability and interpretability. After the model aggregates the patch-level features and fuses the clinical features through cross-attention, it outputs the prediction result.
[0031] During training, through the way of joint optimization, the bag-level loss and the instance-level loss are combined. The bag-level loss is used to evaluate the classification effect of the entire input (such as a pathology image or the complete data of a patient), while the instance-level loss pays more attention to the prediction of each specific instance.
[0032] Furthermore, during the training process, initial learnable weights are added to the clinical features and image features, the contribution ratio of the image features and the clinical features is dynamically adjusted through backpropagation, and the feature distinguishability is enhanced through sparse regularization; the total loss function Total Loss is: The total loss function Total Loss is: ; Among them, = 0.7, = 0.3, Loss bag represents the bag-level loss, Loss instance represents the instance-level loss; The bag-level loss adopts the cross-entropy loss: ; Among them is the true label of the i-th sample, is the probability that the model predicts the i-th sample; The instance-level loss adopts the smooth support vector machine loss: ; Among them is the true label of the i-th sample, is the score of the model predicting the i-th sample.
[0033] By optimizing these two loss functions simultaneously, the model can not only improve the overall classification accuracy but also enhance its performance on each fine-grained feature, thus better handling the complex relationships of different modal features. We use a = 0.7 and β = 0.3. Meanwhile, to verify the effectiveness of multi-modal fusion, Table 3 shows the ablation experiments of the model.
[0034] Table 3
[0035] Furthermore, the method also includes generating a visualization heatmap, covering the attention weights of the image patches on the original image in a red gradient, and observing the parts that the model is interested in to improve interpretability. As Figure 3 shown, red or bright red represents higher attention, yellow or orange represents medium attention, blue or green represents lower attention, and light colors represent very low attention.
[0036] Example 2: Based on Example 1, this example provides a gastric cancer postoperative recurrence prediction device based on cross-attention, including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the method described in Example 1.
[0037] Example 3: Based on Example 1, this example provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in Example 1 is implemented.
[0038] Example 4: Based on Example 1, this example provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method described in Example 1 is implemented.
[0039] Example 5: Based on Example 1, this example provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method described in Example 1 is implemented.
[0040] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.
[0041] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for realizing the functions specified in multiple blocks.
[0042] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that realize the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for realizing the functions specified in multiple blocks.
[0043] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for realizing the functions specified in multiple blocks.
[0044] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.
Claims
1. A method for predicting gastric cancer recurrence after surgery based on cross-attention, characterized in that: include: Obtain gastric pathology full-slice images and clinical data of target patients; Performing pyramid compression and segmentation on the gastric pathology full-slice image to generate multiple patch images, extracting the patch images through an encoder to obtain image features, which include multiple patch image features; Extract features and normalize clinical data to obtain clinical features; The image features and clinical features are input into a gastric cancer postoperative recurrence prediction model to obtain a gastric cancer postoperative recurrence prediction result; wherein the processing process of the gastric cancer postoperative recurrence prediction model includes: Based on the weakly supervised multiple instance learning framework, clustering constraints are introduced in the attention mechanism to aggregate multiple patch map features of image features to obtain aggregated patch-level features; The cross-attention mechanism is used to perform weighted fusion of aggregated patch-level features and clinical features to generate fused features; the fused features are input into the classifier to obtain the prediction results of gastric cancer recurrence after surgery.
2. The method according to claim 1, characterized in that The encoder adopts UNI encoder.
3. The method according to claim 1 or 2, characterized in that: The training process of the encoder includes: Acquire a gastric pathology full-slice image of a gastric cancer patient, wherein the gastric pathology full-slice image is labeled, and the label indicates that the corresponding gastric cancer patient has recurrence after surgery or has not recurred after surgery; Perform pyramidal multi-resolution segmentation on gastric pathology full-slice images to generate multiple patch maps and construct a labeled patch map dataset; The encoder is trained using the patch image dataset to obtain a trained encoder.
4. The method according to claim 1, characterized in that The clinical data include gender, age, infiltration wetness, lymph node metastasis, distant metastasis, carbohydrate antigen CA19-9, carcinoembryonic antigen CEA, alpha-fetoprotein AFP, neutrophils, lymphocytes, and inflammatory prognostic index IPI.
5. The method according to claim 1, characterized in that Extract and normalize clinical data to obtain clinical features, including: After multiple clinical data are passed through the first fully connected layer for preliminary feature extraction and compression, nonlinearity is introduced through the RELU activation function, and then Dropout is used to reduce overfitting. They are then mapped through the second fully connected layer and finally normalized through L2 norm standardization to obtain multiple clinical features.
6. The method according to claim 1, characterized in that The training process of the gastric cancer postoperative recurrence prediction model includes: The image features and corresponding clinical features of multiple labeled gastric cancer patients are collected as a training data set, and the training data set is used in combination with the total loss function to train a gastric cancer postoperative recurrence prediction model to obtain a trained gastric cancer postoperative recurrence prediction model.
7. The method according to claim 6, characterized in that During the training process, initialized learnable weights were added to clinical features and image features, the contribution ratio of image features to clinical features was dynamically adjusted through back propagation, and feature distinguishability was enhanced through sparse regularization; The total loss function is: ； in, =0.7, =0.3, Loss bag Indicates packet-level loss, Loss instance represents instance-level loss; The packet-level loss uses cross entropy loss: ； in is the true label of the i-th sample, is the probability that the model predicts the i-th sample; The instance-level loss uses smoothed support vector machine loss: ； in is the true label of the i-th sample, is the score predicted by the model for the i-th sample.
8. A device for predicting gastric cancer recurrence after surgery based on cross-attention, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the method according to any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.