Breast cancer neoadjuvant chemotherapy curative effect prediction method based on graph neural network

Abstract

The invention relates to the field of biomedical engineering, and discloses a breast cancer neoadjuvant chemotherapy curative effect prediction method based on a graph neural network, and the method comprises the steps: obtaining multi-modal data of a breast cancer patient, and extracting a time sequence feature vector; constructing a plurality of meta-tasks containing a support set and a query set, and constructing the time sequence feature vector of the patient sample in each meta-task into graph structure data; inputting the graph structure data into a graph neural network model for curative effect prediction and loss calculation; back propagation prediction loss update model parameters; and constructing the time sequence feature vector of a new patient to be predicted and the support set into new graph structure data, inputting the new graph structure data into the trained model, and outputting a prediction result. The system comprises a data processing module, a graph construction module, a model training module and a curative effect prediction module. According to the method, through the meta-learning strategy and the graph neural network, multi-modal time series data are effectively integrated, accurate prediction of the breast cancer neoadjuvant chemotherapy curative effect in a small sample scene is realized, and support is provided for clinical diagnosis and treatment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biomedical engineering, in particular to a breast cancer neoadjuvant chemotherapy efficacy prediction method based on a graph neural network. BACKGROUND

[0002] As one of the most common malignant tumors in women worldwide, breast cancer plays a key role in reducing tumor size, reducing surgical difficulty and evaluating chemotherapy sensitivity. Therefore, accurately predicting the efficacy of neoadjuvant chemotherapy is crucial for achieving individualized treatment, improving efficacy and reducing unnecessary side effects.

[0003] Currently, medical imaging technology has been widely used in breast cancer diagnosis and efficacy monitoring, and its data contains rich tumor information. However, traditional image feature extraction methods mainly rely on manual measurement, which not only consumes time and effort, but also is difficult to fully and objectively capture the complex biological characteristics of tumors.

[0004] Although in recent years, image-based and artificial intelligence-based methods have been introduced into this field and have made some progress, there are still significant shortcomings. On the one hand, traditional machine learning methods (such as support vector machines, random forests) have limitations in dealing with complex nonlinear data relationships and are highly dependent on manual feature engineering. On the other hand, deep learning methods represented by convolutional neural networks (CNN) can automatically extract image features, but they mainly focus on local information and ignore the complex relationships and interactions between tumors and their surroundings, different tissues, and the whole patient. In addition, existing models mostly fail to effectively integrate multi-modal time-series data and are difficult to ensure the accuracy of the prediction under small sample conditions, which limits the universality and effectiveness of existing technologies in clinical applications.

[0005] Therefore, the present application proposes a breast cancer neoadjuvant chemotherapy efficacy prediction method based on a graph neural network to solve the shortcomings of the prior art. SUMMARY

[0006] In view of the shortcomings of the prior art, the present application provides a breast cancer neoadjuvant chemotherapy efficacy prediction method based on a graph neural network, which solves the problem that existing neoadjuvant chemotherapy efficacy prediction methods fail to effectively utilize the dynamic evolution information of patients at multiple time points and have limited model robustness and prediction accuracy when dealing with small sample data.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] The first aspect of the present application provides a breast cancer neoadjuvant chemotherapy efficacy prediction method based on a graph neural network. The method converts multi-time point multi-modal data of patients into a dynamic heterogeneous graph structure through a meta-learning framework, and processes it using a specially designed graph neural network model to achieve accurate prediction of neoadjuvant chemotherapy efficacy. The method comprises the following steps:

[0009] First, multi-time point multi-modal data of breast cancer patients during treatment is obtained, and a time series feature vector is extracted for each time point. The process is as follows: imageomics features, deep learning features and clinical features are extracted from multi-modal data. To process high-dimensional imageomics features and deep learning features, a sequential dimension reduction and selection process is applied:

[0010] First, Mann-Whitney U test is used to screen features with statistically significant differences between different efficacy categories;

[0011] Second, the selected features are sorted according to the correlation between the features and the efficacy label and the redundancy between the features using the least redundancy and maximum correlation method, and a part of the features with high ranking are selected;

[0012] Third, LASSO algorithm is used to perform final screening on the selected features to obtain the most representative feature subset.

[0013] Finally, the imageomics features, deep learning features and original clinical features that have undergone final screening are combined to form a time series feature vector corresponding to each time point.

[0014] Second, a plurality of meta-tasks containing support sets and query sets are constructed, and the time series feature vectors of patient samples in each meta-task are constructed into graph structure data. The graph structure data is a dynamic heterogeneous graph sequence composed of heterogeneous graph snapshots covering multiple time points, which is used to represent the dynamic evolution process of the patient population. In each heterogeneous graph snapshot, the nodes include patient nodes representing each patient individual, and clinical feature nodes and time series image feature nodes instantiated according to the clinical features and image features contained in the time series feature vectors.

[0015] To determine the association between patient nodes in the graph, a final adjacency matrix needs to be generated for each heterogeneous graph snapshot. The generation process includes:

[0016] First, an edge weight matrix is calculated. A convolutional neural network is used to calculate the similarity between two patient nodes at a specific time point by taking their time series feature vectors as input, to quantify their association. The edge weight formula is as follows:

[0017]

[0018] wherein, is the edge weight between patient node i and patient node j in the mth layer graph; s i and s j are the feature vectors of patient node i and patient node j, respectively; CNN θ is a convolutional neural network with learnable parameters θ; abs(·) is an element-wise absolute value operation.

[0019] Secondly, an edge probability matrix is constructed. This matrix is constructed based on the known efficacy labels of the support set samples, and is used to introduce prior knowledge. The edge probability formula is as follows:

[0020]

[0021] wherein, e ij is the element value between node i and node j in the edge probability matrix; l i and l j are the known efficacy labels of node i and node j, respectively.

[0022] Thirdly, the edge weight matrix and the edge probability matrix are multiplied element by element to obtain the final adjacency matrix. The adjacency matrix generation formula is as follows:

[0023]

[0024] wherein, is the final adjacency matrix of the mth layer graph; E (m) is the edge probability matrix of the mth layer graph; W (m) is the edge weight matrix of the mth layer graph; ° represents element by element multiplication operation.

[0025] Next, the constructed graph structure data is input into a graph neural network model for processing. The model first performs intra-graph information propagation through a heterogeneous graph attention network for each heterogeneous graph snapshot to update the node representation;

[0026] Then, the time series node representation sequence obtained after updating each patient node in all heterogeneous graph snapshots is input into a gated recurrent unit network for inter-graph time series information aggregation to generate a final patient comprehensive representation encapsulating dynamic evolution information, and based on this representation, the efficacy prediction results of the samples in the query set are output. At the same time, the cross-entropy loss function is used to calculate the prediction loss between the prediction results and the true efficacy labels of the query set. The loss function formula is as follows:

[0027]

[0028] wherein, to predict the loss value; L is a set of all efficacy categories; l is a specific efficacy category in the set L; y is a one-hot encoding of the true efficacy label of the query set sample; y l is an element corresponding to the category l in the one-hot encoding y; is the efficacy prediction result of the model; is the probability of the efficacy prediction result of the model being the category l for the query set sample under the condition of a given meta-task sample set S.

[0029] Then, the parameters of the graph neural network model are updated by backpropagating the prediction loss. This process uses a gradient descent optimizer to jointly update all learnable parameters in the heterogeneous graph attention network and the gated recurrent unit network end-to-end according to the gradient calculated from the prediction loss, so as to obtain a trained graph neural network model.

[0030] Finally, in the prediction stage, the time series feature vector of a new patient to be predicted is combined with a support set, and a new graph structure data containing the new patient is constructed by completely following the aforementioned graph structure data construction method. The new graph structure data is input into the trained graph neural network model to output the neoadjuvant chemotherapy efficacy prediction result of the new patient.

[0031] The second aspect of the present application provides a breast cancer neoadjuvant chemotherapy efficacy prediction system based on a graph neural network. The system implements the aforementioned method, and is characterized in that the system comprises:

[0032] a data processing module for acquiring multi-modal data of breast cancer patients covering multiple time points, and extracting a corresponding time series feature vector for each time point from the multi-modal data;

[0033] a graph construction module for constructing multiple meta-tasks containing a support set and a query set, and constructing the time series feature vector of the patient sample in each meta-task into graph structure data;

[0034] a model training module for inputting the graph structure data into a graph neural network model, performing efficacy prediction on the samples in the query set, calculating a prediction loss according to the efficacy prediction result and the known efficacy label of the query set, and updating the parameters of the graph neural network model by backpropagating the prediction loss, so as to obtain a trained graph neural network model;

[0035] an efficacy prediction module for constructing the time series feature vector of a new patient to be predicted and a support set into new graph structure data, and inputting the new graph structure data into the trained graph neural network model to output the neoadjuvant chemotherapy efficacy prediction result of the new patient.

[0036] The application provides a breast cancer neoadjuvant chemotherapy efficacy prediction method based on a graph neural network.

[0037] 1、 The application can comprehensively capture the dynamic pathophysiological changes of patients during neoadjuvant chemotherapy by extracting imageomics, deep learning and clinical multi-modal features from multiple time points in the treatment process of patients and integrating them into a time sequence feature vector. Not only static information at a single time point is utilized, but more importantly, the evolution trajectory of the patient state is modeled, providing a richer and more time-dimension information basis for subsequent efficacy prediction, thereby improving the accuracy of the prediction results.

[0038] 2、 The application adopts a meta-learning training framework, and trains the model by constructing multiple meta-tasks containing a support set and a query set. The framework enables the model to learn a general measurement or comparison ability rather than a fixed classification rule for a specific data set. Therefore, when facing a medical scene with limited sample size or needing to predict new patients, the model can utilize the generalization ability learned from the meta-task for effective prediction, reducing the dependence on large-scale labeled data sets and enhancing the applicability and robustness of the model under small sample conditions.

[0039] 3、 The application constructs the time sequence feature vector of the patient population into a dynamic heterogeneous graph sequence, where each time point corresponds to a graph snapshot. On this basis, a heterogeneous graph attention network is used to process the node information within a single graph snapshot, and a gated recurrent unit network is used to aggregate the time sequence information between different graph snapshots. This combination of intra-graph information propagation and inter-graph time sequence aggregation structure can effectively capture the evolution trajectory of individual patient characteristics during treatment, model the dynamic response process of patients, and improve the utilization efficiency of time-dependent features by the model.

[0040] 4、 In constructing the graph structure, the application generates an adjacency matrix by a combination method to define the relevance between patient nodes. On the one hand, a convolutional neural network is used to learn the deep similarity of high-dimensional features between patients to generate an edge weight matrix; on the other hand, an edge probability matrix is constructed using the known efficacy labels in the support set to introduce prior knowledge. The combination of the two generates the final adjacency matrix, so that the graph structure not only reflects the intrinsic feature distribution relationship of the data, but also integrates task-related supervised information, thereby constructing a patient correlation graph that is richer in information and more targeted for efficacy prediction tasks.

[0041] 5、The application can clearly screen out imageomics and deep learning features that have a significant impact on efficacy prediction by performing Mann-Whitney-U test, minimum redundancy maximum relevance and LASSO algorithm. At the same time, in the graph construction, by distinguishing different types of nodes such as patient nodes, clinical feature nodes and image feature nodes, the model can learn the specific relationship between heterogeneous information. This explicit screening of key features and structured modeling of heterogeneous information sources provides a technical approach to analyzing model decision-making basis, to some extent, enhancing the technical interpretability of the prediction model. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A flowchart of the breast cancer neoadjuvant chemotherapy efficacy prediction method based on the graph neural network of the application is shown in the figure.

[0043] Figure 2 A schematic diagram of the graph neural network model training and application process of the application is shown in the figure.

[0044] Figure 3 A schematic diagram of the graph neural network layer internal information updating process of the application is shown in the figure.

[0045] Figure 4 A block diagram of the breast cancer neoadjuvant chemotherapy efficacy prediction system based on the graph neural network of the application is shown in the figure.

[0046] Among them, 10, data processing module; 20, graph construction module; 30, model training module; 40, efficacy prediction module. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the application will be described below in conjunction with the drawings of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0048] Reference Figure 1 The embodiment of the application provides a breast cancer neoadjuvant chemotherapy efficacy prediction method based on a graph neural network. The method can include the following steps in one embodiment:

[0049] Step S1: Obtain multi-modal data of breast cancer patients covering multiple time points, and extract corresponding time series feature vectors for each time point from the multi-modal data. Specifically, the multi-modal data includes magnetic resonance imaging (MRI) image data and clinical pathology data. Imageomics features, deep learning features and clinical features are extracted from the multi-modal data. To process high-dimensional imageomics features and deep learning features, a sequential dimension reduction and selection process is applied:

[0050] In the first step, Mann-Whitney-U test is used to screen features with statistically significant differences between different efficacy categories;

[0051] In the second step, the minimum redundancy maximum correlation method is used to sort the screened features according to the correlation between the features and the efficacy label and the redundancy between the features, and a predetermined number of top-ranked features are selected.

[0052] In the third step, LASSO algorithm is used to perform final screening on the selected features. The imageomics features, deep learning features and clinical features after final screening are combined to form a time series feature vector corresponding to each time point.

[0053] Step S2: Construct multiple meta-tasks containing support sets and query sets, and construct the time series feature vectors of patient samples in each meta-task into graph structure data. The graph structure data is a dynamic heterogeneous graph sequence composed of heterogeneous graph snapshots covering multiple time points. In each heterogeneous graph snapshot, the nodes include patient nodes representing each patient individual, and clinical feature nodes and time series image feature nodes instantiated according to the clinical features and image features contained in the time series feature vectors. To determine the relevance between patient nodes in the graph, a final adjacency matrix is generated for each heterogeneous graph snapshot. The generation process includes calculating an edge weight matrix, constructing an edge probability matrix, and combining the two.

[0054] Step S3: Input the graph structure data into a graph neural network model to perform efficacy prediction on the samples in the query set, and calculate the prediction loss based on the efficacy prediction result and the known efficacy label of the query set. For each heterogeneous graph snapshot, the graph neural network model updates the node representation through a heterogeneous graph attention network for intra-graph information propagation; then, the time series node representation sequence obtained after updating each patient node in all heterogeneous graph snapshots is input into a gated recurrent unit network for inter-graph time series information aggregation to generate a final patient comprehensive representation, and the efficacy prediction result is output based on the representation. In this step, the cross-entropy loss function is used to calculate the prediction loss between the prediction result and the true efficacy label.

[0055] Step S4: Update the parameters of the graph neural network model by backpropagating the prediction loss to obtain a trained graph neural network model. This process uses a gradient descent optimizer to update all learnable parameters in the heterogeneous graph attention network and the gated recurrent unit network end-to-end based on the gradients calculated from the prediction loss.

[0056] Step S5: constructing the time series feature vector of a new patient to be predicted and a support set into new graph structure data, inputting the new graph structure data into the trained graph neural network model to output a neoadjuvant chemotherapy efficacy prediction result of the new patient.

[0057] With reference to Figure 4 The embodiment of the present application also provides a breast cancer neoadjuvant chemotherapy efficacy prediction system based on a graph neural network, which is used for implementing the foregoing method and can comprise, in an embodiment:

[0058] a data processing module 10 configured to acquire multi-modal data of a breast cancer patient covering multiple time points and extract a corresponding time series feature vector for each time point from the multi-modal data;

[0059] a graph construction module 20 configured to construct multiple meta-tasks containing a support set and a query set and construct the time series feature vector of a patient sample in each meta-task into graph structure data;

[0060] a model training module 30 configured to input the graph structure data into a graph neural network model, perform efficacy prediction on samples in the query set, calculate a prediction loss according to the efficacy prediction result and a known efficacy label of the query set, and update parameters of the graph neural network model by back-propagating the prediction loss to obtain a trained graph neural network model;

[0061] an efficacy prediction module 40 configured to construct the time series feature vector of a new patient to be predicted and a support set into new graph structure data and input the new graph structure data into the trained graph neural network model to output a neoadjuvant chemotherapy efficacy prediction result of the new patient.

[0062] With reference to Figure 1 The specific implementation process of the technical solutions in the embodiment of the present application is described in detail.

[0063] I. Data acquisition and time series feature vector extraction

[0064] In a specific embodiment of the present application, this step aims to construct a comprehensive and quantitative feature representation for each breast cancer patient at multiple treatment time points. This process first acquires multi-modal data of the patient during neoadjuvant chemotherapy. These data mainly include two types:

[0065] One type is magnetic resonance imaging (MRI) image data acquired from a picture archiving and communication system (PACS);

[0066] Another type is the clinical pathology data obtained from the electronic medical record system (EMRS), such as the status of estrogen receptor (ER), progesterone receptor (PR) and human epidermal growth factor receptor 2 (HER2), which are usually obtained by analyzing the biopsy tissue of the tumor core region. To capture the dynamic changes of the patient's state, the above data are collected at multiple preset time points during the treatment.

[0067] After obtaining the original multi-modal data, feature extraction is performed on the data at each time point to generate a sequence of feature vectors with temporal relationships. The feature extraction process is specifically divided into three parallel branches:

[0068] First, radiomic features are extracted. In the MRI image at each time point, the tumor region is first accurately segmented. Then, based on the segmented three-dimensional tumor region, a large number of radiomic features are calculated and extracted using automated software tools. In one embodiment, a total of 1672 features are extracted, which can be classified as: 23 features based on gray level co-occurrence matrix (GLCM), 16 features based on gray level run length matrix (GLRLM), 16 features based on gray level size zone matrix (GLSZM), 14 features based on gray level dependence matrix (GLDM), 5 features based on neighborhood gray tone difference matrix (NGTDM), 16 shape features, and 18 first-order intensity statistical features. In addition, to obtain more abundant information, 17 different image filters (such as wavelet filters, Laplacian of Gaussian filters, etc.) are applied to the original MRI images, and the feature extraction process is repeated on the filtered images.

[0069] Second, deep learning features are extracted. This process uses a transfer learning method, using a pre-trained convolutional neural network model (such as DenseNet) on a large image dataset. When extracting features, the MRI image at each time point is first normalized in size and standardized in intensity. Then, the processed image is input into the pre-trained DenseNet network model. The last layer (i.e. the classification layer) of the model is removed, and the output of the second last layer (i.e. the last fully connected layer or global pooling layer) is extracted as the deep learning feature. By this method, a high-dimensional feature vector can be extracted for each patient at each time point, for example, a feature vector containing 1664 elements.

[0070] The clinical features directly use the structured data obtained from the electronic medical record system, such as the status values of ER, PR and HER2.

[0071] Since the directly extracted radiomic features and deep learning features have high dimensions and information redundancy, the invention independently performs a three-stage sequential dimension reduction and selection process on these two types of features to screen out the most relevant feature subsets for chemotherapy efficacy.

[0072] In the first stage, Mann-Whitney-U test is used. The patient samples are divided into two groups according to their final efficacy results (e.g., pathologic complete remission or non-pathologic complete remission), and a non-parametric test is performed on each feature to screen features with statistically significant differences in distribution between the two groups, for example, features with P value less than 0.05 are selected.

[0073] In the second stage, the minimum redundancy maximum relevance (mRMR) method is used. The features screened by the first stage are sorted using the mRMR algorithm. This algorithm is based on mutual information theory and aims to maximize the correlation between the selected features and the efficacy label while minimizing the redundancy between the selected features. After sorting, a preset number of features with high ranking are selected, for example, the top 50 features are retained.

[0074] In the third stage, the least absolute shrinkage and selection operator (LASSO) algorithm is used. The features retained in the last stage are input into the LASSO regression model. The LASSO algorithm can compress the coefficients of some unimportant features to zero by adding an L1 regularization term in the loss function, thereby realizing automatic selection of features. Finally, only the features with non-zero coefficients are retained as the final screening results.

[0075] Finally, the imageomics feature subset, deep learning feature subset obtained by the above three-stage screening, and the original clinical features (ER, PR, HER2, etc.) are combined. The combined vector constitutes the time series feature vector of the patient at a specific time point. Repeat this process for each data collection time point to ultimately obtain a time series feature vector sequence covering multiple time points, which will be used as input for subsequent graph construction and model training.

[0076] Reference Figure 2 The specific implementation process of the technical solutions in the embodiments of the present application is described in detail.

[0077] II. Graph structure data construction based on meta tasks

[0078] This step, in one specific embodiment of the present application, aims to organize the time series feature vectors extracted in the previous step into graph structure data suitable for processing by graph neural network models. This construction process is based on the training framework of meta learning, and a series of meta tasks are constructed to organize the data.

[0079] A meta task randomly extracts a small sample subset S from the training data set. The subset is divided into a support set containing known efficacy labels and a query set to be predicted for efficacy labels. The specific definition of the sample set S is as follows:

[0080]

[0081] where x i represents a sample, i is the known efficacy label of the sample, L is the total number of efficacy categories, is a single unknown label sample in the query set. The support set contains L x k samples, where each category contains k samples, so the total number of samples N = L x k + 1. The goal of the meta-task is to predict the efficacy label of the query sample according to the information of the support set.

[0082] For each constructed meta-task, the present application constructs the time series feature vector of each patient sample at each time point into a dynamic graph sequence. Since the present application deals with data with time series relationship, a corresponding graph snapshot will be constructed for each time point. A graph snapshot is an isomorphic graph, where the nodes are all patient nodes, representing each patient in the meta-task sample set S.

[0083] In the graph snapshot, the initial feature representation v i of each patient node i is concatenated from its multi-modal feature vector at the time point and label information. Specifically, the construction method of the node initial feature representation is as follows:

[0084] v i = concat(l i ,c i ,r i ,d i );

[0085] where v i is the initial feature vector of node i; concat(·) represents the vector concatenation operation; l i is the one-hot encoding vector of the efficacy label of node i. For nodes in the support set, the vector is generated according to the known label, and for the query node, the vector is a zero vector; c i is the clinical feature vector of the node; r i is the imageomics feature vector of the node after screening; d i is the deep learning feature vector of the node after screening.

[0086] The core of the graph structure is to define the connection relationship between the nodes, which is quantitatively represented by the adjacency matrix A. In the present application, the adjacency matrix A is generated by combining an edge weight matrix W and an edge probability matrix E, so that the edges of the graph can reflect both the intrinsic feature similarity of the data and the prior supervision information related to the task.

[0087] The calculation of the edge weight matrix W aims to automatically evaluate the feature similarity between nodes through a learnable metric function. Specifically, the initial feature vectors si and s j The absolute difference value is input into a convolutional neural network (CNN), which outputs a scalar value as the edge weight between two nodes. The calculation formula is as follows:

[0088]

[0089] In the formula, is the edge weight between node i and node j in the mth layer graph; CNN θ is a convolutional neural network with learnable parameters θ; s i and s j are the initial feature vectors of nodes i and j; abs(·) is an element-wise absolute value operation.

[0090] The construction of the edge probability matrix E utilizes the known efficacy labels of the samples in the support set to introduce explicit supervision information into the graph structure. The element e ij is set according to the label relationship of nodes i and j, and the calculation formula is as follows:

[0091]

[0092] In the formula, e ij is the element value between nodes i and j in the edge probability matrix; l i and l j are the known efficacy labels of nodes i and j. When the labels of the two support set nodes are the same, the edge probability is 1; when they are different, the edge probability is 0. When any node is a query node (the label is unknown), the edge probability is set to 0.5.

[0093] The final adjacency matrix A is obtained by performing element-level multiplication on the edge weight matrix W and the edge probability matrix E, and the generation formula is as follows:

[0094]

[0095] In the formula, is the final adjacency matrix of the mth layer graph; E (m) is the edge probability matrix of the mth layer graph; W (m) is the edge weight matrix of the mth layer graph; represents element-level multiplication operation; before inputting into the graph neural network layer, each row of the adjacency matrix will be normalized by a softmax function.

[0096] Referring to Figure 2 and Figure 3 , the specific implementation process of the technical solutions in the embodiments of the present application is described in detail.

[0097] III. Training of the graph neural network model

[0098] This step aims to train the graph neural network model with the multiple meta-tasks constructed in the previous step, so that it learns a generalizable node similarity measurement method. The overall process of model training is a meta-learning loop, in which the model iteratively optimizes its internal parameters by processing a series of graph structure data (i.e., meta-tasks).

[0099] The core of the graph neural network model (labeled as MFFGCN in the accompanying drawings) employed in the present application is stacked by one or more graph neural network layers with probability constraints. Referring to Figure 3 , the graph neural network layer is responsible for information propagation and node feature update within a single graph snapshot. Specifically, for the m-th layer network, it receives the node feature matrix V (m) and the normalized adjacency matrix calculated in the previous step as inputs.

[0100] The update process of information in the graph neural network layer is as follows: first, the neighborhood information is aggregated and new node features are generated through a graph convolution operation Gn(·). Its calculation formula is as follows:

[0101]

[0102] In the formula, Gn(V (m) ) is the updated feature calculated by this layer; is a set of adjacency operators, which mainly contains the normalized adjacency matrix B is an operator in the operator set; is the degree matrix; V (m) is the input node feature matrix of the m-th layer; is the learnable weight matrix corresponding to the operator B, which is used for linear transformation of node features; Leaky-ReLU(·) is a leaky rectified linear unit activation function.

[0103] Subsequently, in order to preserve the original information and alleviate the problem of gradient vanishing, the output feature V (m+1) of this layer is composed by concatenating the input feature V (m) and the updated feature Gn(V (m) ) obtained by graph convolution operation. Its node feature update formula is as follows:

[0104] V (m+1) = concat(V (m) , Gn(V (m) ));

[0105] In the formula, concat(·) represents the vector concatenation operation. After processing through multiple stacked graph neural network layers, in the last layer, the model outputs the final feature representation of the query set sample (i.e., the unknown label node), and normalizes it through a softmax function to obtain the predicted probability of the sample belonging to each therapeutic category.

[0106] In each meta-task, after the model makes a prediction on the query set samples, the cross-entropy loss function is used to quantify the prediction result. The difference between the predicted and actual efficacy label y. The formula for calculating this predicted loss is as follows:

[0107]

[0108] In the formula, The predicted loss value; L is the set of all efficacy categories; l is a specific efficacy category in set L; y is the one-hot encoding of the true efficacy label of the query set sample; y l For the element in y corresponding to category l in the one-hot encoding; The model's efficacy prediction results; Given a meta-task sample set S, this represents the probability that the model's prediction of the therapeutic effect of a query set sample belongs to category l.

[0109] The model's parameter update mechanism is based on the backpropagation algorithm and gradient descent optimizer. After processing a meta-task and calculating the prediction loss, this loss value is used to calculate all learnable parameters in the model (e.g., CNN). θ and in each layer of the graph neural network The gradient of the parameter is then calculated. The optimizer subsequently updates the parameters based on this gradient. (See reference...) Figure 2 The training process shown cyclically occurs across a large number of meta-tasks (from task0 to taskT-1). The abstract process of parameter updates can be represented as:

[0110] P t+1 =f(l t ,P t );

[0111] In the formula, P t Let l be the set of model parameters before training the t-th meta-task; t Let f(·) be the prediction loss calculated on the t-th meta-task; f(·) represents the gradient descent optimization algorithm; P t+1 This is the updated set of parameters, which will serve as the initial parameters for processing the next meta-task. By training on numerous meta-tasks with diverse structures, the model learns a general similarity metric that is independent of any specific graph structure, thereby gaining the ability to inductively learn and independently test new samples.

[0112] Reference Figure 2 The specific implementation process of the technical solutions in the embodiments of the present application will be described in detail.

[0113] Four, efficacy prediction for new patients

[0114] This step describes, in one specific embodiment of the present application, how to apply the graph neural network model after it has been trained through the aforementioned meta-learning process to a completely new patient whose efficacy is unknown. At this stage, all the learnable parameters of the model (such as CNN θ and the graph neural network in each layer have been fixed and no longer updated.

[0115] When a new patient needs to be predicted for efficacy, first follow the procedure exactly the same as step S1 to obtain the multi-modal data of the new patient and extract the corresponding time-series feature vector. The new patient at this time serves as the query sample in the prediction task.

[0116] Subsequently, a support set is selected from the existing patient data set with known efficacy labels. The support set contains Lxk samples, where L is the total number of efficacy categories and k is the number of samples included in each category. The new patient (query sample) is combined with the selected support set to form a new sample set for prediction.

[0117] Next, follow the method exactly the same as step S2 to construct a completely new graph structure data for this new sample set. Specifically, each patient in the set (including the new patient and all patients in the support set) is instantiated as a node in the graph. The initial feature representation of the node, the calculation of the edge weight matrix W, the construction of the edge probability matrix E, and the generation of the final adjacency matrix A all use the same mechanism and formula as in the training phase. Among them, the calculation of the edge weight matrix W will use the trained CNN θ model.

[0118] Finally, this graph structure data constructed for the new patient is sent as input into the graph neural network model that has been trained and has fixed parameters. The model performs a forward propagation process:

[0119] Information is transmitted and updated in the graph neural network layer according to the learned parameters, and the feature representation of the query node (i.e. the new patient node) is updated after aggregating the information of the support set nodes. The last layer of the model will output the probability distribution of the new patient node belonging to each efficacy category. Finally, the category with the highest probability is selected as the final prediction result of the neoadjuvant chemotherapy efficacy of the new patient, such as pathological complete remission (pCR) or non-pathological complete remission (Non-pCR).

[0120] Reference Figure 4 , Figure 4 is a structural block diagram of a breast cancer neoadjuvant chemotherapy efficacy prediction system based on a graph neural network according to an embodiment of the present application. The system is used to implement the aforementioned prediction method, which in an embodiment can include:

[0121] A data processing module 10 is configured to obtain multi-modal data of breast cancer patients covering multiple time points, and extract corresponding time series feature vectors for each time point from the multi-modal data. Specifically, the module is configured to obtain MRI image data and clinical pathology data from a picture archiving and communication system and an electronic medical record system, and perform imageomics feature extraction, deep learning feature extraction based on a pre-trained DenseNet model, and a three-stage sequential dimension reduction and selection process including Mann-Whitney-U test, minimum redundancy maximum relevance (mRMR) method and least absolute shrinkage and selection operator (LASSO) algorithm on the extracted imageomics features and deep learning features respectively. Finally, the filtered features and clinical features are combined into time series feature vectors.

[0122] A graph construction module 20 is configured to construct multiple meta-tasks containing support sets and query sets, and construct the time series feature vectors of patient samples in each meta-task into graph structure data. Specifically, the module is configured to construct sample data of each meta-task into a dynamic graph sequence, where the nodes in the graph snapshot at each time point are initialized and a final adjacency matrix is calculated for them. The calculation of the adjacency matrix includes: calculating the feature similarity between nodes through a convolutional neural network to generate an edge weight matrix, constructing an edge probability matrix according to the known efficacy labels of the support set samples, and combining the two matrices through element-level multiplication.

[0123] A model training module 30 is configured to input the graph structure data into a graph neural network model, perform efficacy prediction on the samples in the query set, calculate a prediction loss according to the efficacy prediction result and the known efficacy label of the query set, and update the parameters of the graph neural network model by backpropagating the prediction loss to obtain a trained graph neural network model. Specifically, the module contains the graph neural network model and is configured to perform forward propagation to obtain a prediction result, calculate a loss using a cross-entropy loss function, and update all learnable parameters in the model according to the loss using a gradient descent optimizer.

[0124] The efficacy prediction module 40 is configured to construct a support set of the time-series feature vector of a new patient to be predicted into a new graph structure data, and input the new graph structure data into the trained graph neural network model to output a new adjuvant chemotherapy efficacy prediction result of the new patient. Specifically, this module receives a trained and parameter-fixed graph neural network model, and is configured to repeat the function of the graph construction module 20 on new patient data to generate a new graph structure data for prediction, and then obtain the final efficacy prediction category through a forward propagation process of the model.

[0125] The implementation of the present application requires the following equipment or devices:

[0126] Computing device: including but not limited to one or more high-performance computers or servers, which are configured with a graphics processing unit (GPU) to meet the computing resources required for deep learning and graph neural network model training and inference.

[0127] Software tools: a software environment running on the computing device, including but not limited to a deep learning framework (such as TensorFlow or PyTorch), and one or more image processing libraries (such as ITK or OpenCV), for implementing data processing, feature extraction, and model construction, etc.

[0128] Although embodiments of the present application have been shown and described, it will be understood by those having ordinary skill in the art that various changes, modifications, alternatives, and variations can be made thereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for predicting the efficacy of neoadjuvant chemotherapy in breast cancer based on graph neural networks, characterized in that, Includes the following steps: S1. Obtain multimodal data of breast cancer patients covering multiple time points, and extract the corresponding temporal feature vector for each time point from the multimodal data; S2. Construct multiple meta-tasks containing support sets and query sets, and construct the temporal feature vectors of patient samples in each meta-task as graph structure data; S3. Input the graph structure data into the graph neural network model, perform efficacy prediction on the samples in the query set, and calculate the prediction loss based on the efficacy prediction results and the known efficacy labels of the query set. S4. Update the parameters of the graph neural network model by backpropagating the prediction loss to obtain the trained graph neural network model. S5. Construct a new graph structure data from the temporal feature vector and support set of the new patient to be predicted, and input the new graph structure data into the trained graph neural network model to output the neoadjuvant chemotherapy efficacy prediction result of the new patient.

2. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on graph neural networks according to claim 1, characterized in that, In step S1, the step of acquiring multimodal data covering multiple time points of breast cancer patients and extracting the corresponding temporal feature vector for each time point from the multimodal data includes: Extract radiomics features, deep learning features, and clinical features from the multimodal data; A sequential dimensionality reduction and selection process is performed on the aforementioned image omics features and deep learning features. The process is as follows: The Mann-Whitney-U test was used to screen for characteristics with statistically significant differences; For the features that pass the screening, the minimum redundancy maximum relevance method is used to sort them and select a portion of the top-ranked features; The LASSO algorithm is used to perform the final screening of the selected features; The radiomics features, deep learning features, and clinical features that have undergone final screening are combined to form the temporal feature vector.

3. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on graph neural networks according to claim 1, characterized in that, Step S2, which involves constructing multiple meta-tasks containing support sets and query sets, and then constructing the temporal feature vectors of patient samples in each meta-task into graph-structured data, includes the following steps: A dynamic heterogeneous graph sequence is constructed as the graph structure data, and the dynamic heterogeneous graph sequence consists of heterogeneous graph snapshots covering multiple time points; The nodes in the heterogeneous graph snapshot include patient nodes representing each individual patient, as well as clinical feature nodes and temporal image feature nodes instantiated based on the clinical and image features contained in the temporal feature vector.

4. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on graph neural networks according to claim 3, characterized in that, Step S2, which involves constructing multiple meta-tasks containing support sets and query sets, and then constructing the temporal feature vectors of patient samples in each meta-task into graph-structured data, further includes: Determine the relationships between the patient nodes; The correlation is quantified by calculating an edge weight matrix for each heterogeneous graph snapshot, and the calculation method is as follows: Using a convolutional neural network, the similarity between two patient nodes is calculated by taking their temporal feature vectors at a specific time point as input. The edge weight formula is as follows: In the formula, Let s be the edge weights between patient node i and patient node j in the m-th layer graph; i and s j These are the feature vectors of patient node i and patient node j, respectively; CNN θ This is a convolutional neural network with learnable parameters θ; abs(·) is the element-wise absolute value operation.

5. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on graph neural networks according to claim 4, characterized in that, Step S2, which involves constructing multiple meta-tasks containing support sets and query sets, and then constructing the temporal feature vectors of patient samples in each meta-task into graph-structured data, further includes: A side probability matrix is ​​constructed based on the known therapeutic labels in the support set, and the side probability matrix is ​​combined with the side weight matrix to generate the final adjacency matrix. The elements of the edge probability matrix are calculated using the edge probability formula: In the formula, e ij Let l be the element value between node i and node j in the edge probability matrix; i and l j These are the known therapeutic labels for nodes i and j, respectively; The final adjacency matrix is ​​obtained by element-wise multiplication of the edge probability matrix and the edge weight matrix, calculated using the adjacency matrix generation formula: In the formula, E represents the final adjacency matrix of the m-th layer graph. (m) W is the edge probability matrix of the m-th layer graph; (m) Let be the edge weight matrix of the m-th layer graph; ° denotes element-wise multiplication.

6. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on graph neural networks according to claim 1, characterized in that, Step S3, which involves inputting the graph structure data into a graph neural network model to predict the therapeutic effect of samples in the query set, includes: For each heterogeneous graph snapshot in the graph structure data, intra-graph information is propagated through a heterogeneous graph attention network to update the node representation; The temporal node representation sequence obtained by updating each patient node in all heterogeneous graph snapshots is input into a gated recurrent unit network to aggregate inter-graph temporal information, so as to generate a final comprehensive patient representation that encapsulates dynamic evolution information, and output the efficacy prediction result based on the final comprehensive patient representation.

7. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on graph neural networks according to claim 1, characterized in that, Step S3, the step of calculating the prediction loss based on the efficacy prediction result and the known efficacy labels of the query set, includes: The cross-entropy loss function is used, with the true efficacy labels and efficacy prediction results of the query set samples as input. The calculation is performed using the loss function formula: In the formula, The predicted loss value; L is the set of all efficacy categories; l is a specific efficacy category in set L; y is the one-hot encoding of the true efficacy label of the query set sample; y l For the element in y corresponding to category l in the one-hot encoding; The model's efficacy prediction results; Given a meta-task sample set S, this represents the probability that the model's prediction of the therapeutic effect of a query set sample belongs to category l.

8. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on graph neural networks according to claim 6, characterized in that, Step S4, which involves updating the parameters of the graph neural network model through backpropagation of the prediction loss to obtain the trained graph neural network model, includes the following steps: A gradient descent optimizer is used, and all learnable parameters in the heterogeneous graph attention network and the gated recurrent unit network are jointly updated end-to-end based on the gradient calculated from the predicted loss.

9. The method for predicting the efficacy of neoadjuvant chemotherapy for breast cancer based on graph neural networks according to claim 1, characterized in that, Step S5, which involves constructing a new graph structure data from the temporal feature vector and support set of the new patient to be predicted, and inputting the new graph structure data into the trained graph neural network model to output the neoadjuvant chemotherapy efficacy prediction result for the new patient, includes the following steps: Combine the temporal feature vector of the new patient with the temporal feature vectors of each patient sample in the support set; Based on the combined temporal feature vector, a dynamic heterogeneous graph sequence consisting of heterogeneous graph snapshots covering multiple time points is constructed as the new graph structure data. The nodes of the heterogeneous graph snapshots include patient nodes representing new patients and patient nodes supporting each patient in the set. A final adjacency matrix is ​​generated for the dynamic heterogeneous graph sequence to determine the associations between patient nodes. The generation process includes: A convolutional neural network is used to calculate the feature similarity between patient nodes to obtain an edge weight matrix; A side probability matrix is ​​constructed based on the known efficacy labels of each patient in the support set; The edge weight matrix and the edge probability matrix are multiplied element-wise to obtain the final adjacency matrix.

10. A graph neural network-based system for predicting the efficacy of neoadjuvant chemotherapy for breast cancer, applied to the method described in any one of claims 1-9, characterized in that, The system includes: The data processing module is used to acquire multimodal data of breast cancer patients covering multiple time points, and extract the corresponding temporal feature vector for each time point from the multimodal data; The graph construction module is used to construct multiple meta-tasks containing support sets and query sets, and to construct the temporal feature vectors of patient samples in each meta-task into graph structure data; The model training module is used to input the graph structure data into a graph neural network model, perform efficacy prediction on the samples in the query set, calculate the prediction loss based on the efficacy prediction result and the known efficacy labels of the query set, and update the parameters of the graph neural network model by backpropagating the prediction loss to obtain the trained graph neural network model. The efficacy prediction module is used to construct a new graph structure data by combining the temporal feature vector of the new patient to be predicted with a support set, and input the new graph structure data into the trained graph neural network model to output the neoadjuvant chemotherapy efficacy prediction result of the new patient.

Images