Method and device for predicting lymph node metastasis and total lifetime after neoadjuvant chemotherapy of local development stage gastric cancer

Abstract

The invention discloses a method and device for predicting lymph node metastasis and total lifetime after neoadjuvant chemotherapy of gastric cancer in a local development stage, and the method comprises the steps: collecting and preprocessing time sequence CT image data of an LAGC patient and a corresponding label, and obtaining a needed data set; constructing a TSMamba segmentation model, and performing coding task training on the TSMamba segmentation model by using the data set to obtain a TSMamba encoder capable of extracting time sequence features; freezing and extracting TSMamba features from the CT image of the pre-NAC before neoadjuvant chemotherapy and the CT image of the post-NAC after chemotherapy by using a trained TSMamba encoder, so as to obtain a first TSMamba feature vector and a second TSMamba feature vector; performing lymph node metastasis LNM and total lifetime OS prediction on the first TSMamba feature vector and the second TSMamba feature vector based on a preset common attention mechanism network; the invention provides a method for simultaneously and accurately predicting lymph node metastasis and total lifetime by using a preoperative time sequence CT image, and personalized treatment suggestions and prognosis decision support are provided for clinic.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and specifically relates to a method and device for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer. Background Art

[0002] The main treatment for locally advanced gastric cancer (LAGC) is radical gastrectomy, but even so, the prognosis of patients is still poor, with a 5-year survival rate of less than 40%, mainly due to tumor recurrence and metastasis, especially lymph node metastasis (LNM). Neoadjuvant chemotherapy (NAC) has become the standard treatment for LAGC, which can reduce the stage of the primary tumor, eliminate micrometastases, and improve the resectability rate, thereby improving the survival outcomes of patients. However, due to the high heterogeneity of gastric cancer, only about 30% of patients can achieve lymph node regression and prolong overall survival after receiving NAC. Therefore, accurate preoperative identification of LNM after NAC is crucial to improve treatment decisions and optimize prognosis.

[0003] Computed tomography (CT) is the routine method of choice for diagnosing LNM, monitoring NAC response, and assessing prognosis. Pre-treatment CT provides direct information on the original tumor and lymph nodes, while time-series CT reflects the dynamic changes of the tumor. However, the subjective assessment of LNM after NAC is challenging, with a diagnostic sensitivity of less than 57%. Previous studies have confirmed that certain clinical predictors are associated with LNM and overall survival (OS), but these factors are controversial and have limited clinical application, often ignoring the dynamic changes of the tumor caused by NAC treatment. To date, there is no non-invasive and accurate method to predict LNM and OS preoperatively.

[0004] Deep learning (DL) technology can automatically quantify complex tumor features beyond human visual perception and has attracted widespread attention in the field of medical image research. This technology has been widely used to predict LNM, treatment efficacy, and prognosis of LAGC. However, existing DL models are mostly based on single-task predictions of two-dimensional images, ignoring the spatiotemporal heterogeneity of tumors and the interactions between multiple tasks. Multi-task deep learning (MDL) exploits the regularity and shared features between interrelated tasks and can simultaneously optimize multiple learning tasks in the same model. MDL is widely recognized for its high data efficiency and ability to mitigate overfitting, thereby improving the generalization and prediction performance of the model. Given the complex spatiotemporal heterogeneity of LAGC during NAC, MDL provides a new opportunity to fully mine dynamic information from time-series CT images.

[0005] Therefore, how to simultaneously predict LNM and OS and improve the prediction accuracy is one of the technical problems that technical personnel in this field need to solve urgently. Summary of the invention

[0006] The main objective of the present invention is to overcome the drawbacks and deficiencies of the prior art, and to provide a method and device for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, which can combine longitudinal CT image data, accurately extract the image features of gastric cancer patients at different treatment stages, and perform multi-task learning based on the TSMamba framework to automatically identify and predict lymph node metastasis and overall survival, so as to provide personalized treatment recommendations and prognostic decision-making support for clinical practice.

[0007] To achieve the above objective, the present invention adopts the following technical solutions:

[0008] In the first aspect, the present invention provides a method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, including the following steps:

[0009] Collect and preprocess the time-series CT image data and their corresponding labels of locally advanced gastric cancer (LAGC) patients to obtain the required data set;

[0010] Construct a TSMamba segmentation model, and use the data set to train the TSMamba segmentation model for the encoding task to obtain a TSMamba encoder capable of extracting time-series features; the TSMamba segmentation model includes an encoder and a decoder, the encoder includes multiple Mamba feature downsampling modules composed of TSMamba modules and downsampling modules, and the decoder includes multiple upsampling modules composed of transposed convolution modules and residual modules;

[0011] Freeze and use the trained TSMamba encoder to extract TSMamba features from the CT images before neoadjuvant chemotherapy (pre-NAC) and after neoadjuvant chemotherapy (post-NAC) of gastric cancer patients to obtain a first TSMamba feature vector and a second TSMamba feature vector; the first TSMamba feature vector and the second TSMamba feature vector are feature vectors containing tumor information;

[0012] Based on a preset co-attention mechanism network, perform lymph node metastasis (LNM) and overall survival (OS) prediction on the first TSMamba feature vector and the second TSMamba feature vector; the co-attention mechanism network includes co-attention and self-gating mechanisms, and the self-gating mechanism is used to assign co-attention confidence to each attention summary to improve the prediction accuracy.

[0013] As a preferred technical solution, the step of collecting and preprocessing the time-series CT image data and their corresponding labels of locally advanced gastric cancer (LAGC) patients to obtain the required data set is specifically:

[0014] All LAGC patients underwent enhanced CT examinations within two weeks before NAC and before surgery;

[0015] Use the pre-segmentation model AILEN to perform initial segmentation on the tumor region of interest in the CT images of LAGC patients;

[0016] Manually calibrate the pre-segmented tumor regions of interest in the pre-treatment and preoperative portal venous phase enhanced CT of all LAGC patients.

[0017] As a preferred technical solution, use the dataset to train the TSMamba segmentation model for the encoding task, specifically:

[0018] Input the CT images of different time series of the same patient in the obtained dataset into the TSMamba encoder respectively to obtain feature vectors containing patients of different time series; the CT images of different time series include CT images of two time series before and after neoadjuvant chemotherapy;

[0019] Input the obtained feature vectors of different time series into the decoder to restore them to the original shape, and obtain segmentation images of different time series with two channels;

[0020] Calculate the loss between the obtained segmentation images of different time series and the annotated segmentation results in the obtained dataset, and backpropagate the loss value in the model to obtain the TSMamba segmentation model that can segment the CT images of the same patient at different time series including the encoder and the decoder;

[0021] Extract the encoder of the obtained TSMamba segmentation model and freeze it to obtain the TSMamba encoder that can obtain the time series features of the same patient;

[0022] During the encoding task training process:

[0023] The input data for training is a four-dimensional matrix composed of CT images containing pre-NAC and CT images containing post-NAC, and the matrix shape is C*D*H*W; where C is the channel dimension of the CT image data, and D, H, and W refer to the length, width, and height of the four-dimensional matrix;

[0024] The output target label for training the TSMamba segmentation model is a label matrix composed of a set of position points of the tumor in the above input data;

[0025] After training, the TSMamba segmentation model can automatically extract the positions where tumors exist in pre-NAC and post-NAC.

[0026] As a preferred technical solution, calculate the loss between the obtained segmentation images of different time series and the segmentation results annotated by doctors in the obtained dataset, and backpropagate the loss value in the model, specifically:

[0027] When predicting LNM, the binary cross-entropy (BCE) loss is used to train the prediction performance, and the BCE loss L BCE is defined as follows:

[0028]

[0029] where y i is the true label of the i-th sample, taking values of 0 or 1, where 0 indicates no lymph node metastasis and 1 indicates lymph node metastasis; is the predicted probability of the i-th sample, representing the probability that the model predicts lymph node metastasis, with a value range of [0,1]; N is the number of samples;

[0030] When predicting OS, the negative log-likelihood function is used to train the prediction performance, and the loss L surv is a negative log-likelihood function:

[0031]

[0032] where is the survival prediction result of the i-th sample, representing the survival probability predicted by the model, with a value range of [0,1]; is the time label vector of the i-th sample, representing the time information of the event occurrence: all time intervals before the event occurrence are set to 1, and all time intervals after the event occurrence are set to 0; represents the censoring label vector of the i-th sample, indicating whether censoring occurs: set to 1 at the time interval of the event occurrence, and set to 0 for the remaining time intervals; ε is a very small positive number used to avoid the logarithm being negative infinity or numerical instability;

[0033] The losses of the LNM prediction task and the OS prediction task are calculated using the total loss L, and the calculation results of the losses are backpropagated and the network parameters are updated to complete one iteration;

[0034] L = α * L BCE + (1 - α)L surv ,

[0035] After multiple iterative calculations until the parameters converge, the training is completed.

[0036] As a preferred technical solution, the TSMamba module is composed of a GSC module, a layer normalization module, a ToM module, and a multi-layer perceptron module;

[0037] Among them, the GSC module processes the input 3D features through two convolutional blocks, one of which uses a 3x3x3 convolutional kernel and the other uses a 1x1x1 convolutional kernel; the feature maps processed by the two convolutional blocks are multiplied point by point at the pixel level, then convolved and concatenated with the original features to form a gate structure;

[0038] Among them, the ToM module is crucial for modeling the global information of high-dimensional features and is achieved by calculating feature dependencies in three directions;

[0039] The input 3D features are flattened into three sequences, allowing effective interaction and integration between the corresponding feature information;

[0040] ToM(V) = Mamba(V f ) + Mamba(Vr) + Mamba(Vs), where the Mamba model extracts the global information in each direction within the sequence, and f, r, and s represent the forward, reverse, and inter-layer directions respectively.

[0041] As a preferred technical solution, based on a preset co-attention mechanism network, the first TSMamba feature and the second TSMamba feature are used for predicting lymph node metastasis (LNM) and overall survival (OS), specifically:

[0042] The Pre-NAC and Post-NAC images are respectively input into two frozen and trained TSMamba modules to obtain the first TSMamba feature vector Va and the second TSMamba feature vector Vb; V a and V b are used as the input of the co-attention mechanism network, and finally two prediction results are output: one is the prediction of whether the sample belongs to the positive class, and the other is the prediction of the final survival time of the sample.

[0043] As a preferred technical solution, the prediction of lymph node metastasis and overall survival based on the preset co-attention mechanism network for the first TSMamba feature and the second TSMamba feature is specifically:

[0044] The co-attention mechanism mines the correlation between different modalities by calculating the similarity or affinity between different modalities;

[0045] The co-attention mechanism first calculates an affinity matrix, which represents the matching degree between two modalities, and the affinity matrix is obtained by matrix multiplication of the features of one modality and the features of the other modality;

[0046] The affinity matrix is normalized in the row and column directions by the softmax function to obtain the attention distribution within each modality, that is, the attention degree of each element to the elements of other modalities;

[0047] The gated confidence score calculated through the self-gating mechanism updates the attention summary, which is achieved by multiplying the gated score with the original features and then concatenating the result with the original features, thereby obtaining an updated feature representation;

[0048] The features enhanced by the co-attention mechanism are input into the prediction network for predicting lymph node metastasis (LNM) and operative risk (OS).

[0049] In a second aspect, the present invention provides a system for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, which is applied to the method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, and includes a data acquisition module, a segmentation model construction module, a feature extraction module, and a prediction module;

[0050] The data acquisition module is used to collect and preprocess the temporal CT image data of patients with locally advanced gastric cancer (LAGC) and their corresponding labels to obtain the required data set;

[0051] The segmentation model construction module is used to construct a TSMamba segmentation model, and train the TSMamba segmentation model for the encoding task using the data set to obtain a TSMamba encoder capable of extracting temporal features; the TSMamba segmentation model includes an encoder and a decoder, the encoder includes multiple Mamba feature downsampling modules composed of TSMamba modules and downsampling modules, and the decoder includes multiple upsampling modules composed of transposed convolution modules and residual modules;

[0052] The feature extraction module is used to freeze and use the trained TSMamba encoder to extract TSMamba features from the CT images before neoadjuvant chemotherapy (pre-NAC) and after neoadjuvant chemotherapy (post-NAC) to obtain a first TSMamba feature vector and a second TSMamba feature vector; the first TSMamba feature vector and the second TSMamba feature vector are feature vectors containing tumor information;

[0053] The prediction module is used to predict lymph node metastasis (LNM) and overall survival (OS) based on the first TSMamba feature vector and the second TSMamba feature vector using a preset co-attention mechanism network; the co-attention mechanism network includes co-attention and self-gating mechanisms, and the self-gating mechanism is used to assign co-attention confidence to each attention summary to improve the prediction accuracy.

[0054] In a third aspect, the present invention provides an electronic device, characterized in that the electronic device includes:

[0055] At least one processor; and,

[0056] A memory communicatively connected to the at least one processor; wherein,

[0057] The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer.

[0058] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which when executed by a processor, implements the method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] (1) Innovative application of multi-task deep learning model: The CTSMamba model developed in the present invention is the first to apply multi-task deep learning technology to the study of dynamic changes in gastric tumors induced by NAC, and can simultaneously predict postoperative LNM and OS.

[0061] (2) Improved prediction accuracy: The CTSMamba model shows superior performance in predicting LNM and OS compared to models based on single-time-point images, which benefits from the model's ability to capture more comprehensive tumor change information from temporal CT images.

[0062] (3) Prognostic value independent of clinicopathological factors: The prognostic value of the CTSMamba model is confirmed to be independent of clinicopathological factors, which means it can be used as an independent tool to assist clinical decision-making without being restricted by traditional clinicopathological indicators.

[0063] (4) Large-scale multi-center study: Compared with previous studies with small sample sizes and lack of external validation, the present invention utilizes multi-center data of 1,021 patients to enhance the robustness and universality of the model's prediction performance.

[0064] (5) Reduction of computational complexity: The CTSMamba model adopts an innovative TSMamba architecture, effectively reducing the computational complexity related to long feature sequences and spatial positions, and significantly reducing the computational cost without sacrificing prediction accuracy compared to traditional deep learning networks.

[0065] (6) Visualization analysis: Through Grad-CAM maps, the present invention provides an in-depth understanding of the model's decision-making process, revealing the impact of tumor images before and after NAC on model prediction, as well as the relationship between tumor heterogeneity and NAC response.

[0066] (7) Comprehensive prognostic assessment: The CTSMamba model can integrate the prediction tasks of LNM and OS into one model, providing a more comprehensive and integrated prognostic assessment tool considering their unique and interrelated natures.

[0067] (8) Non-invasive risk assessment: The CTSMamba model provides a non-invasive and simple method to reflect biomarkers significantly associated with the prognosis after NAC, which is a safer and more comfortable option for patients.

[0068] (9) Improving prognostic effect: Combining the CTSMamba survival score with clinical prognostic predictors can further improve the prognostic effect of LAGC patients, indicating the practical value of the model in clinical applications.

[0069] (10) Individualized treatment decision-making: By accurately predicting LNM and OS, the CTSMamba segmentation model helps doctors make more individualized treatment decisions for LAGC patients, optimize treatment plans, and improve treatment effects. Description of the Drawings

[0070] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0071] Figure 1 It is a flowchart of a method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer in an embodiment of the present invention;

[0072] Figure 2 It is a schematic structural diagram of a system for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer in an embodiment of the present invention;

[0073] Figure 3 It is a structural diagram of an electronic device in an embodiment of the present invention. Detailed Embodiments

[0074] To enable those skilled in the art of the present technology to better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0075] References to "embodiments" in this application mean that the specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0076] Please refer to Figure 1 , this embodiment provides a method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, including the following steps:

[0077] S1. Collect and preprocess the time-series CT image data and their corresponding labels of patients with locally advanced gastric cancer (LAGC) to obtain the required dataset.

[0078] Further, before obtaining the dataset, the following steps are also included:

[0079] Recruit patients to participate in the research work and collect and analyze clinicopathological factors, specifically:

[0080] The patients were from three medical centers and underwent radical gastrectomy after receiving NAC;

[0081] The treatment time of the participating patients was from October 2013 to October 2022;

[0082] All recruited patients obtained the approval of the ethics committees and institutional review boards of all participating hospitals, and all patients were anonymized and complied with the Declaration of Helsinki and its subsequent amendments;

[0083] All sample data came from medical electronic medical records, including age, gender, body mass index (BMI), carcinoembryonic antigen (CEA) level, carbohydrate antigen (CA) 199 level, tumor location, degree of differentiation, Borrmann type, Lauren type, NAC regimen, NAC cycle, and tumor pathological tissue stage;

[0084] All patients were divided into the LNM group and the non-LNM group, and their pathological histology was defined as N+ or N0;

[0085] The follow-up data of the patients (including the results of blood tests or CT scans) were reviewed every 3-6 months in the first two years and then once a year thereafter.

[0086] Further, step S1 includes the following steps:

[0087] S11. All LAGC patients underwent contrast-enhanced CT examinations within two weeks before NAC and before surgery;

[0088] S12. Use the pre-segmentation model AILEN to perform initial segmentation on the tumor region of interest in the CT images of LAGC patients;

[0089] S13. For the pre-segmented tumor regions of interest in the pre-treatment and preoperative portal venous phase enhanced CT of all patients, they are manually calibrated by experienced radiologists, and reviewed by senior radiologists, and the differences are resolved through consultation and consensus.

[0090] S2. Construct a TSMamba segmentation model, and use the dataset to train the TSMamba segmentation model for the encoding task to obtain a TSMamba encoder capable of extracting temporal features.

[0091] Furthermore, the TSMamba segmentation model includes an encoder and a decoder;

[0092] Among them, the encoder consists of multiple mamba feature downsampling modules, which are composed of a TSMamba module and a downsampling module. The TSMamba module in the encoder of the TSMamba feature extraction model is specifically:

[0093] The TSMamba module consists of a gated spatial convolution module (GSC), a layer normalization module, a three-way spatial mamba (ToM), and a multi-layer perceptron module;

[0094] Among them, the GSC module processes the input 3D features through two convolution blocks. One convolution block uses a 3x3x3 convolution kernel, and the other uses a 1x1x1 convolution kernel. These two feature maps are multiplied point by point at the pixel level, then convolved and concatenated with the original features to form a gated structure;

[0095] Among them, the ToM module is crucial for modeling the global information of high-dimensional features and is achieved by calculating feature dependencies in three directions;

[0096] The 3D input features are flattened into three sequences, allowing effective interaction and integration between the corresponding feature information;

[0097] ToM(V) = Mamba(V f ) + Mamba(Vr) + Mamba(Vs), where the Mamba model extracts the global information in each direction within the sequence, and f, r, and s represent the forward, reverse, and inter-layer directions respectively.

[0098] Among them, the decoder includes multiple upsampling modules, which are composed of a transposed convolution module and a residual module.

[0099] Furthermore, using the dataset to train the TSMamba segmentation model for the encoding task is specifically:

[0100] S21. Input the CT images of the same patient at different time series in the obtained dataset into the TSMamba encoder respectively to obtain the feature vectors of the patient at different time series; the CT images of different time series include the CT images at two time series before and after neoadjuvant chemotherapy.

[0101] S22. Input the obtained feature vectors of different time series into the decoder to restore them to the original shape, and obtain the segmentation images of different time series with two channels.

[0102] S23. Calculate the loss between the obtained segmentation images of different time series and the labeled segmentation results in the obtained dataset, and backpropagate the loss value in the model to obtain the TSMamba segmentation model that can segment the CT images of the same patient at different time series, including the encoder and the decoder.

[0103] S24. Extract the encoder from the obtained TSMamba segmentation model and freeze it to obtain the TSMamba encoder that can obtain the time series features of the same patient.

[0104] It can be understood that during the training process of the encoding task:

[0105] The input data for training is a four-dimensional matrix composed of the CT images of pre-NAC and post-NAC, and the matrix shape is C*D*H*W; where C is the channel dimension of the CT image data, and D, H, and W refer to the length, width, and height of the four-dimensional matrix.

[0106] The output target label for training the TSMamba segmentation model is the label matrix composed of the set of the tumor location points in the above input data.

[0107] After being trained, the TSMamba segmentation model can automatically extract the locations where tumors exist in pre-NAC and post-NAC.

[0108] Furthermore, during the training of the encoding task, calculate the loss between the obtained segmentation images of different time series and the segmentation results marked by doctors in the obtained dataset, and backpropagate the loss value in the model. Specifically:

[0109] In predicting LNM, binary cross-entropy (BCE) is used to train the prediction performance. While in predicting OS, the negative log-likelihood function is used to train the prediction performance. The BCE loss (L BCE ) is defined as follows:

[0110]

[0111] In predicting OS, the loss Lsurv is a negative log-likelihood function:

[0112]

[0113] where S pred is the output result of survival prediction, and S time and α*L censored are two label vectors generated according to the true survival results (event occurrence time and censored or uncensored status). For S time , all time intervals before the event occurrence are set to 1, and the remaining time intervals are 0. For S censored , only the time interval of the event occurrence is set to 1, and the remaining time intervals are set to 0.

[0114] L = α*L BCE +(1 - α)L surv ,

[0115] Calculate the losses of the LNM prediction task and the OS prediction task using the loss L, and backpropagate the loss calculation results to update the network parameters to complete one iteration;

[0116] Perform multiple iterative calculations until the parameters converge, and the training is completed.

[0117] S3. Freeze and use the trained TSMamba encoder to extract TSMamba features from the pre - NAC CT images before neoadjuvant chemotherapy and the post - NAC CT images after neoadjuvant chemotherapy, obtaining a first TSMamba feature vector and a second TSMamba feature vector; the first TSMamba feature vector and the second TSMamba feature vector are feature vectors containing tumor information.

[0118] Specifically, this frozen encoder can automatically extract key tumor features from CT images at different time points; by using this encoder, the high - dimensional data of CT images at different time series is converted into a set of more compact and representative feature vectors.

[0119] S4. Based on a preset co - attention mechanism network, perform lymph node metastasis LNM and overall survival OS predictions on the first TSMamba feature vector and the second TSMamba feature vector, specifically:

[0120] S41. The co - attention co - attention mechanism mines the correlation between different modalities by calculating the similarity or affinity between different modalities;

[0121] S42. The co-attention mechanism first calculates an affinity matrix, which represents the matching degree between two modalities. The affinity matrix is obtained by performing matrix multiplication on the features of one modality and the features of the other modality;

[0122] S43. The affinity matrix is normalized in the row and column directions through the softmax function to obtain the attention distribution within each modality, that is, the degree of attention of each element to the elements of other modalities;

[0123] S44. The attention summary is updated through the gating confidence score calculated by the self-gating mechanism, which is achieved by multiplying the gating score with the original features and then concatenating the results with the original features, thereby obtaining the updated feature representation;

[0124] S45. The features enhanced by the co-attention mechanism are input into the prediction network for predicting lymph node metastasis LNM and operative risk OS.

[0125] In a specific embodiment, the co-attention mechanism network is specifically:

[0126] This network passes through two input vectors (V a ) and (V b ), representing two different features or entities respectively. These two input vectors pass through their respective linear transformations and to calculate (Z a ) and (Z b ), where (S) is the attention weight matrix calculated through . This attention weight matrix (S) reflects the mutual relationship between (V a ) and (V b ) and is used to enhance or suppress certain features.

[0127] Next, the (Z a ) and (Z b ) pass through the self-gating mechanism,

[0128] f g (Z a ) = sigmoid(w f Z a +b f ) ∈ [0,1] DHW ,

[0129] f g (Z b ) = sigmoid(w f Z b +b f ) ∈ [0,1] DHW ,

[0130] where w f and b f are the convolution kernel and bias respectively. The gating f g dynamically determines the amount of information retained from the reference frame and automatically adapts during the training process,

[0131] is further processed through the self-gating mechanism and then the information is integrated through a merging layer. The integrated information is subjected to feature extraction and transformation through a series of fully connected layers, and finally two prediction results are output: one is the prediction on whether the sample belongs to the positive class (N+), and the other is the prediction on the final survival time of the sample.

[0132] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously.

[0133] Based on the same idea as the method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer in the above embodiments, the present invention also provides a system for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, and this system can be used to execute the method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer. For the sake of convenience of description, in the structural schematic diagram of the embodiment of the system for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, only the parts related to the embodiments of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than those illustrated, or combine certain components, or have different component arrangements.

[0134] Please refer to Figure 2 , in another embodiment of the present application, there is provided a system 100 for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, and this system includes a data acquisition module 101, a segmentation model construction module 102, a feature extraction module 103, and a prediction module 104;

[0135] The data acquisition module 101 is used to collect and preprocess the temporal CT image data of locally advanced gastric cancer (LAGC) patients and their corresponding labels to obtain the required data set;

[0136] The segmentation model construction module 102 is used to construct a TSMamba segmentation model, and use the data set to perform encoding task training on the TSMamba segmentation model to obtain a TSMamba encoder capable of extracting temporal features; the TSMamba segmentation model includes an encoder and a decoder, the encoder includes a plurality of Mamba feature downsampling modules composed of TSMamba modules and downsampling modules, and the decoder includes a plurality of upsampling modules composed of transposed convolution modules and residual modules;

[0137] The feature extraction module 103 is used to freeze and use the trained TSMamba encoder to extract TSMamba features from the CT image of pre-NAC before neoadjuvant chemotherapy and the CT image of post-NAC after neoadjuvant chemotherapy to obtain a first TSMamba feature vector and a second TSMamba feature vector; the first TSMamba feature vector and the second TSMamba feature vector are feature vectors containing tumor information;

[0138] The prediction module 104 is used to predict lymph node metastasis LNM and overall survival OS for the first TSMamba feature vector and the second TSMamba feature vector based on a preset common attention mechanism network; the common attention mechanism network includes common attention and self-gating mechanisms, and the self-gating mechanism is used to assign common attention confidence to each attention summary to improve prediction accuracy.

[0139] It should be noted that the lymph node metastasis and overall survival prediction system after neoadjuvant chemotherapy for locally advanced gastric cancer of the present invention corresponds one to one with the lymph node metastasis and overall survival prediction method after neoadjuvant chemotherapy for locally advanced gastric cancer of the present invention. The technical features and beneficial effects described in the embodiment of the lymph node metastasis and overall survival prediction method after neoadjuvant chemotherapy for locally advanced gastric cancer are applicable to the embodiment of lymph node metastasis and overall survival prediction after neoadjuvant chemotherapy for locally advanced gastric cancer. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0140] In addition, in the implementation of the system for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the system for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer is divided into different program modules to complete all or part of the functions described above.

[0141] See also Figure 3, in one embodiment, an electronic device for implementing a method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer is provided. The electronic device 200 may include a first processor 201, a first memory 202, and a bus, and may further include a computer program stored in the first memory 202 and executable on the first processor 201, such as a program 203 for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer.

[0142] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as the mobile hard disk of the electronic device 200. In some other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 200. Further, the first memory 202 may also include both the internal storage unit and the external storage device of the electronic device 200. The first memory 202 can be used not only to store application software installed on the electronic device 200 and various types of data, such as the code of the program 203 for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, but also to temporarily store data that has been output or is to be output.

[0143] In some embodiments, the first processor 201 may be composed of integrated circuits. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same or different functions, including a combination of one or more Central Processing Units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core of the electronic device. It connects various components of the entire electronic device through various interfaces and lines, and by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202, it executes various functions of the electronic device 200 and processes data.

[0144] Figure 3 Only the electronic device with components is shown. Those skilled in the art can understand that Figure 3The structure shown does not limit the electronic device 200, and it may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0145] The lymph node metastasis and overall survival prediction program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0146] Collect and preprocess the temporal CT image data of locally advanced gastric cancer (LAGC) patients and their corresponding labels to obtain the required dataset;

[0147] Construct a TSMamba segmentation model, and use the dataset to train the TSMamba segmentation model for the encoding task to obtain a TSMamba encoder capable of extracting temporal features; the TSMamba segmentation model includes an encoder and a decoder. The encoder includes multiple Mamba feature downsampling modules composed of TSMamba modules and downsampling modules, and the decoder includes multiple upsampling modules composed of transposed convolution modules and residual modules;

[0148] Freeze and use the trained TSMamba encoder to extract TSMamba features from the CT images before neoadjuvant chemotherapy (pre-NAC) and after neoadjuvant chemotherapy (post-NAC) to obtain a first TSMamba feature vector and a second TSMamba feature vector; the first TSMamba feature vector and the second TSMamba feature vector are feature vectors containing tumor information;

[0149] Based on a preset co-attention mechanism network, perform lymph node metastasis (LNM) and overall survival (OS) prediction on the first TSMamba feature vector and the second TSMamba feature vector; the co-attention mechanism network includes co-attention and self-gating mechanisms, and the self-gating mechanism is used to assign co-attention confidence to each attention summary to improve prediction accuracy.

[0150] Furthermore, if the modules / units integrated in the electronic device 200 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory).

[0151] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0152] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0153] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention should be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer, characterized in that, Including the following steps: Collect and preprocess the temporal CT image data and their corresponding labels of locally advanced gastric cancer (LAGC) patients to obtain the required dataset; Construct a TSMamba segmentation model, and use the dataset to train the TSMamba segmentation model for an encoding task to obtain a TSMamba encoder capable of extracting temporal features; the TSMamba segmentation model includes an encoder and a decoder, the encoder includes multiple Mamba feature downsampling modules composed of TSMamba modules and downsampling modules, and the decoder includes multiple upsampling modules composed of transposed convolution modules and residual modules; Freeze and use the trained TSMamba encoder to extract TSMamba features from the CT images before neoadjuvant chemotherapy (pre-NAC) and after neoadjuvant chemotherapy (post-NAC) to obtain a first TSMamba feature vector and a second TSMamba feature vector; the first TSMamba feature vector and the second TSMamba feature vector are feature vectors containing tumor information; Based on a preset co-attention mechanism network, perform lymph node metastasis (LNM) and overall survival (OS) predictions on the first TSMamba feature vector and the second TSMamba feature vector; the co-attention mechanism network includes co-attention and a self-gating mechanism, and the self-gating mechanism is used to assign co-attention confidence to each attention summary to improve prediction accuracy.

2. The method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer according to claim 1, wherein The collecting and preprocessing of the temporal CT image data and their corresponding labels of locally advanced gastric cancer (LAGC) patients to obtain the required dataset is specifically as follows: All LAGC patients underwent contrast-enhanced CT examinations within two weeks before NAC and before surgery; Use the pre-segmentation model AILEN to perform initial segmentation on the tumor regions of interest in the CT images of LAGC patients; Manually calibrate the pre-segmented tumor regions of interest in the pre-treatment and pre-operative portal venous phase contrast-enhanced CT of all LAGC patients.

3. The method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer according to claim 1, wherein The training of the TSMamba segmentation model for the encoding task using the dataset is specifically as follows: Input the CT images of different time series of the same patient in the obtained dataset into the TSMamba encoder respectively to obtain feature vectors of patients with different time series; the CT images of different time series include CT images of two time series before and after neoadjuvant chemotherapy; Input the obtained feature vectors of different time series into the decoder to restore them to the original shape to obtain segmentation images of different time series with two channels; Calculate the loss between the obtained segmentation images of different time series and the labeled segmentation results in the obtained dataset, and backpropagate the loss value in the model to obtain a TSMamba segmentation model that includes an encoder and a decoder and can segment the CT segmentation images of the same patient at different time series; Extract the encoder from the obtained TSMamba segmentation model and freeze it to obtain a TSMamba encoder capable of obtaining the temporal features of the same patient; During the encoding task training process: The input data for training is a four-dimensional matrix composed of pre-NAC CT images and post-NAC CT images, with the matrix shape being C*D*H*W; where C is the channel dimension of the CT image data, and D, H, and W refer to the length, width, and height of the four-dimensional matrix. The output target label for training the TSMamba segmentation model is a label matrix composed of a set of tumor location points in the above input data. After being trained, the TSMamba segmentation model can automatically extract the locations where tumors exist in pre-NAC and post-NAC.

4. The method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer according to claim 3, wherein Calculate the loss between the obtained segmentation images at different time series and the segmentation results annotated by doctors in the obtained dataset, and backpropagate the loss value in the model. Specifically: When predicting LNM, the binary cross-entropy (BCE) loss is used to train the prediction performance, and the BCE loss \(L\) BCE is defined as follows: where y i is the true label of the i-th sample, taking values of 0 or 1, where 0 indicates no lymph node metastasis and 1 indicates lymph node metastasis; is the predicted probability of the i-th sample, representing the probability that the model predicts lymph node metastasis, with a value range of [0, 1]; N is the number of samples; When predicting OS, the negative log-likelihood function is used to train the prediction performance, and the loss L surv is a negative log-likelihood function: Among them, is the survival prediction result of the i-th sample, representing the survival probability predicted by the model, and the value range is [0, 1]; is the time label vector of the i-th sample, representing the time information of the event occurrence: all time intervals before the event occurrence are set to 1, and all time intervals after the event occurrence are set to 0; represents the censoring label vector of the i-th sample, indicating whether censoring occurs: set to 1 at the time interval of the event occurrence, and set to 0 at the remaining time intervals; ε is a very small positive number used to avoid the logarithm being negative infinity or numerical instability; Use the total loss L to calculate the losses of the LNM prediction task and the OS prediction task, backpropagate the loss calculation results, and update the network parameters to complete one iteration. L = α * L BCE + (1 - α)L surv , Perform multiple iterative calculations until the parameters converge, and the training is completed.

5. The method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer according to claim 1, wherein The TSMamba module consists of a GSC module, a layer normalization module, a ToM module, and a multi-layer perceptron module. Among them, the GSC module processes the input 3D features through two convolutional blocks. One convolutional block uses a 3x3x3 convolutional kernel, and the other uses a 1x1x1 convolutional kernel; the feature maps processed by the two convolutional blocks are multiplied point by point at the pixel level, then convolved and concatenated with the original features to form a gate structure. Among them, the ToM module is crucial for modeling the global information of high-dimensional features and is achieved by calculating feature dependencies in three directions. The input 3D features are flattened into three sequences, allowing effective interaction and integration between the corresponding feature information. ToM(V) = Mamba(V f ) + Mamba(Vr) + Mamba(Vs), where the Mamba model extracts global information in each direction within the sequence, and f, r, and s represent the forward, reverse, and inter-layer directions respectively.

6. The method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer according to claim 1, wherein Based on a preset co-attention mechanism network, perform lymph node metastasis LNM and overall survival OS predictions on the first TSMamba feature and the second TSMamba feature. Specifically: Input the Pre-NAC and Post-NAC images into two frozen and trained TSMamba modules respectively to obtain the first TSMamba feature vector Va and the second TSMamba feature vector Vb; Use Va and Vb as the input of the co-attention mechanism network, and finally output two prediction results: one is the prediction on whether the sample belongs to the positive class, and the other is the prediction on the final survival time of the sample. a a b b as the input of the co-attention mechanism network, and finally output two prediction results: one is the prediction on whether the sample belongs to the positive class, and the other is the prediction on the final survival time of the sample.

7. The method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer according to claim 1, wherein The prediction of lymph node metastasis and overall survival on the first TSMamba feature and the second TSMamba feature based on a preset co-attention mechanism network is as follows: The co-attention mechanism mines the correlation between different modalities by calculating the similarity or affinity between different modalities. The co-attention mechanism first calculates an affinity matrix, which represents the matching degree between two modalities. The affinity matrix is obtained by multiplying the features of one modality with the features of another modality through matrix multiplication. The affinity matrix is normalized in the row and column directions through the softmax function to obtain the attention distribution within each modality, that is, the attention degree of each element to other modality elements. The gated confidence score calculated through the self-gating mechanism is used to update the attention summary, which is achieved by multiplying the gated score with the original features and then concatenating the results with the original features, thereby obtaining an updated feature representation. The features enhanced by the co-attention mechanism are input into the prediction network for the prediction of lymph node metastasis LNM and surgical risk OS.

8. A lymph node metastasis and overall survival prediction system after neoadjuvant chemotherapy for locally advanced gastric cancer, characterized in that, Applied to the method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer described in any one of claims 1-7, including a data acquisition module, a segmentation model construction module, a feature extraction module, and a prediction module; The data acquisition module is used to collect and preprocess the time-series CT image data of locally advanced gastric cancer (LAGC) patients and their corresponding labels to obtain the required data set; The segmentation model construction module is used to construct a TSMamba segmentation model, and use the data set to train the TSMamba segmentation model for an encoding task to obtain a TSMamba encoder capable of extracting time-series features; the TSMamba segmentation model includes an encoder and a decoder, and the encoder includes multiple Mamba feature downsampling modules composed of TSMamba modules and downsampling modules, and the decoder includes multiple upsampling modules composed of transposed convolution modules and residual modules; The feature extraction module is used to freeze and use the trained TSMamba encoder to extract TSMamba features from the CT images before neoadjuvant chemotherapy (pre-NAC) and after neoadjuvant chemotherapy (post-NAC) to obtain a first TSMamba feature vector and a second TSMamba feature vector; the first TSMamba feature vector and the second TSMamba feature vector are feature vectors containing tumor information; The prediction module is used to predict lymph node metastasis (LNM) and overall survival (OS) based on the first TSMamba feature vector and the second TSMamba feature vector by a preset co-attention mechanism network; the co-attention mechanism network includes co-attention and a self-gating mechanism, and the self-gating mechanism is used to assign co-attention confidence to each attention summary to improve prediction accuracy.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the method for predicting lymph node metastasis and overall survival after neoadjuvant chemotherapy for locally advanced gastric cancer described in any one of claims 1-7.

Images