NAC curative effect prediction method and system based on multi-modal image and dual-channel GCN-Transformer

Abstract

The invention relates to an NAC curative effect prediction method and system based on a multi-modal image and a dual-channel GCN-Transform, and the method comprises the steps: extracting the features of multi-modal images, such as an ultrasonic gray-scale image and a color Doppler blood flow image, through ViT, constructing a knowledge graph, and carrying out the aggregation of a graph attention network; meanwhile, an SVS pathological section is segmented into pixel patches, patch features are extracted, and global representation is obtained through a four-layer graph convolutional network; after deep learning of the two paths of Transformers, weighted soft voting fusion is adopted to predict the complete remission probability of neoadjuvant chemotherapy pathology, complementarity of image and histological information can be fully mined, and the prediction accuracy is remarkably improved. Compared with a plurality of existing optimization algorithms, the advantages of the algorithm in the aspect of improving the breast cancer NAC curative effect prediction accuracy are verified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a NAC efficacy prediction method and system based on multi-modal images and a dual-path GCN-Transformer. BACKGROUND

[0002] Neoadjuvant chemotherapy (NAC) has become the main treatment regimen for patients with locally advanced breast cancer, but different patients have significant differences in response to NAC, and only about 19% to 30% of patients can achieve pathological complete remission (pCR), and about 5% to 20% of patients may continue to develop. At present, medical imaging technologies such as MRI, PET / CT and ultrasound are commonly used in clinical practice to dynamically evaluate the efficacy of NAC. MRI has excellent soft tissue resolution, but the examination cost is high and time-consuming; PET / CT can provide tumor metabolic information, but is limited by ionizing radiation and equipment accessibility. However, the interpretation of the above image results and efficacy prediction still mainly rely on the subjective visual experience of imaging physicians, resulting in unstable diagnosis results affected by observer differences.

[0003] In summary, how to improve the accuracy and stability of pCR prediction after NAC for breast cancer is a technical problem to be solved. SUMMARY

[0004] The purpose of the present application is to overcome the defects of the prior art and provide a NAC efficacy prediction method and system based on multi-modal images and a dual-path GCN-Transformer, aiming to enhance the stability and generalization ability of prediction.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] In one aspect of the present application, a NAC efficacy prediction method based on multi-modal images and a dual-path GCN-Transformer is provided, comprising the following steps:

[0007] Step S1, obtaining pathological images of lesion sites and multi-modal ultrasound images of a plurality of cases before NAC;

[0008] Step S2, performing region of interest annotation and image preprocessing on the ultrasound images and pathological images;

[0009] Step S3, extracting high-dimensional features of multi-modal ultrasound images, establishing a multi-modal knowledge graph for each case and its corresponding ultrasound image node, performing feature extraction on pathological sections obtained by cutting the pathological images, and constructing a position coding pathological knowledge graph;

[0010] Step S4, in a first path, node feature graphs are obtained by applying a graph attention network to realize feature aggregation between nodes for the multi-modal knowledge graph, and in a second path, global feature graphs are obtained by using a multi-layer graph convolution network to extract global topological information for the location coding pathology knowledge graph, and the aggregated features and global topological information output by the two paths are input into the corresponding Transformers of the respective paths in parallel for deep representation learning;

[0011] Step S5, an ultrasound image and a pathology image to be predicted are obtained, and the Transformers of the first path and the second path are used to predict the pathological complete remission rates of the respective paths;

[0012] Step S6, based on the pathological complete remission rates of the two paths, a therapeutic effect prediction result is obtained by weighted soft voting.

[0013] As a preferred technical solution, the step S3 comprises:

[0014] Step S301, for the ultrasound images of each ultrasound modality of each case, pre-trained visual Transformers are used to extract high-dimensional features of imageomics;

[0015] Step S302, taking an individual as a center node of a heterogeneous knowledge graph, pre-trained visual Transformers are used to extract deep features of images under each ultrasound modality, each set of features is taken as a modality feature node, all modality feature nodes are connected to corresponding intermediate nodes through undirected edges, the intermediate nodes are connected to the center node, the clinical features (such as ER state, PR state, HER2 state, etc.) of each case are one-hot encoded, and the “clinically_similar” edges are established according to whether the Euclidean distance in the clinical feature space is lower than a set threshold, connections are established between the same modalities of different cases, and a multi-modal knowledge graph carrying multi-modal information complementation is formed;

[0016]

[0017] Wherein, x and y are clinical features of different cases, which are composed of 7-dimensional vectors, and d represents the Euclidean distance between the two cases.

[0018] Step S303, for each pathology slice obtained by cutting the pathology image, the deep features of each partition are extracted by dividing the partition into several sub-regions without overlap according to a preset size, each sub-region is taken as a graph node, and the deep features are taken as node attributes, undirected edges are constructed between adjacent graph nodes according to the spatial adjacency relationship of the sub-regions in the slice, and a case-level location coding pathology knowledge graph representing the spatial topology and phenotype features of the organization is formed.

[0019] As a preferred technical solution, the step S4 comprises:

[0020] In step S401, based on the multi-modal knowledge graph, the node feature matrix is formed by splicing the multi-modal deep features, and the edge index matrix is constructed according to the node relationship in the multi-modal knowledge graph;

[0021] In step S402, for the multi-modal knowledge graph, the three-layer multi-head graph attention network structure is formed by calculating the attention coefficient between nodes, attention weight normalization and node feature updating, so as to realize adaptive aggregation of multi-modal features and obtain the node feature map;

[0022] In step S403, based on the position coding pathological knowledge graph, the local information of the nodes in the knowledge graph is aggregated by using the four-layer graph convolution network.

[0023] In step S404, in the process of aggregation by using the four-layer graph convolution network in step S403, for each layer of the graph convolution network, the node feature representation of the current graph convolution network layer is updated based on the neighbor node set of the current node, and for the node feature representation of the last layer of the graph convolution network, the features of all nodes in the position coding pathological knowledge graph are aggregated by global average pooling to obtain the global topological information representing the overall structure and histological features of the pathological image, as the global feature map.

[0024] In step S405, the aggregated features output by the two paths and the global topological information are input into the corresponding Transformer of each path in parallel for deep feature learning.

[0025] As a preferred technical solution, in the multi-layer graph convolution network, the node feature representation updating of each layer of the graph convolution network and the global average pooling are realized by the following formula:

[0026]

[0027] Wherein, h i (l+1) is the updated node feature representation of the (l+1) th layer of the graph convolution network, σ is the ReLU activation function, N(i) is the neighbor node set of node i in the position coding pathological knowledge graph, d i and d i are the degrees of node i and node j respectively, W l is the learnable weight matrix of the l th layer of the graph convolution network, h global is the global topological information after global average pooling, N is the number of nodes, and h i (L) is the feature of the i th node in the L th layer of the graph convolution network.

[0028] As a preferred technical solution, the step S5 comprises:

[0029] Step S501: Obtain the clinical features of the case to be predicted. The clinical features may include ER status, PR status, HER2 status, ki67 status and molecular subtype three classification. Map the clinical features and the node feature map to the same dimension, and form a feature sequence together with the trainable classification label. Combine the feature sequence with the trainable position encoding as the input of the Transformer of the first path.

[0030] In step S502, in the Transformer of the first pathway, feature interaction between clinical features, node feature maps and classification labels is achieved through global attention. The global representation of classification labels is extracted through the feedforward network and classification head to generate the pathological complete remission rate predicted by the first pathway.

[0031] Step S503: Perform SMOTE oversampling and standardization on the clinical features and the global feature map, and map them to the hidden space. Combine the new feature sequence with the trainable classification label to form a new feature sequence. Combine the new feature sequence with the new trainable position encoding as the input of the Transformer of the second pathway.

[0032] In step S504, in the Transformer of the second pathway, the global dependency between clinical features, global feature map and new location encoding is captured by sub-attention encoding, and the pathological complete remission rate predicted by the second pathway is generated by layer normalization and classification head.

[0033] As a preferred technical solution, step S6 includes:

[0034] Based on the predicted complete remission rates of the first and second pathways, the final efficacy prediction results were obtained by normalization and formaldehyde combination.

[0035] As a preferred technical solution, the image preprocessing includes grayscale normalization and pathological image segmentation.

[0036] As a preferred technical solution, after acquiring pathological images and multimodal ultrasound images, the method further includes:

[0037] The acquired pathological images and multimodal, multi-sectional ultrasound images were randomly divided into five equal parts. Four parts were used as the training set and one part as the validation set. The process was repeated five times to test the stability and generalization ability of the model.

[0038] As a preferred technical solution, the multimodal ultrasound images include grayscale multi-section ultrasound images, elastic multi-section ultrasound images, and blood flow multi-section ultrasound images.

[0039] Another aspect of the present invention provides a NAC efficacy prediction system based on multimodal imaging and dual-pathway GCN-Transformer, comprising:

[0040] The clinical feature acquisition module is used to acquire pathological images and multimodal ultrasound images of the lesion site to be predicted.

[0041] The dual-pathway GCN-Transformer prediction module is used to obtain efficacy prediction results based on pathological images and multimodal ultrasound images of the lesion site to be predicted, through the aforementioned NAC efficacy prediction method based on multimodal imaging and dual-pathway GCN-Transformer.

[0042] An interactive module is used to display the efficacy prediction results.

[0043] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0044] (1) Enhance the stability and generalization ability of prediction: This invention constructs a dual Transformer network architecture. The two-path models are trained independently and then fused by weighted soft voting, which effectively balances the prediction results of each path and reduces the impact of single path error on the overall performance.

[0045] (2) Enhance the model’s ability to perceive tumor heterogeneity: This invention integrates ultrasound grayscale images, color Doppler blood flow images and pathological slide images, and processes them in parallel through multiple pathways to effectively explore the complementarity of imaging and histological features, thereby enhancing the model’s ability to perceive tumor heterogeneity.

[0046] (3) Ensure the richness and accuracy of global feature representation: The graph attention network GAT and graph neural network GCN of this invention are responsible for the aggregation of local information in the neighborhood of nodes in the graph structure, and the Transformer is responsible for modeling global dependencies. The two complement each other and can improve the richness and accuracy of local-global feature representation. Attached Figure Description

[0047] Figure 1 The flowchart below shows the NAC efficacy prediction method based on multimodal imaging and dual-pathway GCN-Transformer in the embodiments.

[0048] Figure 2 This is a schematic diagram of the NAC efficacy prediction method in the embodiments;

[0049] Figure 3 This is a schematic diagram of the dual-path GCN-Transformer architecture in the embodiment;

[0050] Figure 4 This is a schematic diagram of the multimodal knowledge graph in the embodiment;

[0051] Figure 5 This is a schematic diagram of receiver operating characteristic curves for early prediction of NAC efficacy in breast cancer using different algorithms in the examples. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0053] Example 1

[0054] To address the problems of the aforementioned existing technologies, this embodiment provides a method for predicting the efficacy of NAC based on multimodal imaging and a dual-pathway GCN-Transformer. The method uses pre-chemotherapy (NAC0) multimodal ultrasound images (grayscale, color Doppler flow mapping, elastography) and pathological slide images to predict efficacy. Specifically, it deeply fuses and models multimodal information, extracts image features using a visual Transformer (ViT), aggregates graph structure information using a graph attention network (GAT) / graph convolutional network (GCN), performs parallel learning with dual Transformers, and employs weighted soft voting to fuse the output. This allows for efficient and accurate prediction of the probability of complete pathological remission in the early stages of chemotherapy, providing decision support.

[0055] See Figure 1 and Figure 2 The method includes the following steps:

[0056] Step S1: Retrospectively collect high-quality multimodal (GUSI, CDFI, and EI) multi-section ultrasound images and pathological images of breast cancer cases undergoing NAC before chemotherapy.

[0057] Specifically, this step may include steps S101-S102:

[0058] Step S101: A retrospective collection of multimodal ultrasound images and pathological images before NAC was collected from 148 breast cancer cases diagnosed with breast cancer and who underwent NAC. All cases completed a full cycle of NAC and underwent surgery after NAC.

[0059] Step S102: Under the premise of ensuring that each modality is preserved completely and of good quality, a five-fold cross-validation strategy is used to evaluate all case samples.

[0060] The data is randomly divided into five equal parts, with four parts used as the training set and one part as the validation set, and this process is repeated five times to fully test the stability and generalization ability of the model.

[0061] Step S2 involves preprocessing ultrasound and pathological images. This process includes manually annotating regions of interest (ROIs) and image preprocessing to improve image quality and stability, and to ensure the accuracy of subsequent feature extraction.

[0062] Specifically, this step may include steps S201-S205:

[0063] Step S201, Region of Interest (ROI) annotation: The ROI is manually annotated by 3 ultrasound physicians to eliminate irrelevant information such as text and icons.

[0064] Step S202, Multimodal Consistency: For different modal ultrasound images (GUSI, CDFI, EI) of the same case, ensure that the ROI annotation is consistent across modal images, that is, try to keep the annotation area covering the same lesion site.

[0065] Step S203, grayscale normalization: For ultrasound images, the grayscale values ​​of the images are adjusted to a uniform range (e.g., 0-255) to eliminate the influence of equipment differences.

[0066] Step S204, Image size adjustment: For pathological images, crop the images to remove irrelevant parts, and normalize all images to the same size 512x512 small patch so that the graph structure can be constructed later based on its patch position.

[0067] Step S205, Denoising Processing: Considering that ultrasound images often contain noise, such as shot noise, Gaussian filtering is used for denoising to remove or reduce noise interference in the image. Gaussian filtering is a commonly used linear smoothing filtering method. Its basic principle is to remove noise by weighting each pixel in the image. The weights are determined by a Gaussian function, with the center point having the highest weight, and the weight gradually decreasing as the distance increases.

[0068] Step S3: A pre-trained Visual Transformer (ViT) is used to extract high-dimensional features from multimodal ultrasound images. A knowledge graph is established for each case and its corresponding ultrasound image nodes. ResNet-50 is used to extract features from pathological slice patches that are segmented into 512×516 pixels. At the same time, a corresponding node-edge structure graph is constructed based on graph topology.

[0069] Specifically, this step may include steps S301-S303:

[0070] Step S301: For the three ultrasound modalities (grayscale, elasticity, and blood flow) images of each case, retain all slices obtained under that modality and perform feature extraction on each.

[0071] For each case, two-dimensional grayscale images, elastography images, and blood flow images are used as inputs to extract radiomics features, including but not limited to: texture features, morphological features, first-order statistical features, wavelet features, and color distribution features. These features collectively provide solid support for quantitative analysis. Specific steps include steps S3011-S3013:

[0072] Step S3011, Model and Preprocessing.

[0073] A pre-trained Vision Transformer (ViT-B / 16) model was used. The input image was uniformly resized to 256×256, and the central region of the image was cropped to 224×224. Normalization was then performed (mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225]) to match the ViT pre-training settings and ensure consistent feature distribution.

[0074] Step S3012, image traversal and feature extraction.

[0075] For each case, the extract_vit_features function is called sequentially for the GUSI, CDFI, and EI modal and multi-section images. A 768-dimensional output vector is obtained through ViT forward inference and then reduced to one-dimensional features. For each case, all frame features are concatenated in the format of "modality_frame number_feature dimension" to form a high-dimensional omics feature matrix.

[0076] Step S3013, Feature storage and preparation.

[0077] The extracted ViT feature list is encapsulated into a DataFrame and grouped by case ID and saved as a CSV file for easy fusion with clinical phenotypes and pathological features later.

[0078] Step S302: Construction of the image knowledge graph.

[0079] See Figure 4A heterogeneous knowledge graph centered on individual case nodes was constructed. First, ViT was used to extract depth features from multimodal images such as grayscale ultrasound, color Doppler flow imaging, and elastography. Each set of features was represented as a modal feature node. Then, all modal feature nodes were connected to their corresponding intermediate nodes through undirected edges, thereby reducing the impact of differences in the number of images on the subsequent graph neural network. The intermediate nodes were then connected to individual case nodes, and individual cases were connected based on the similarity of clinical features. This formed a knowledge graph structure that could carry complementary multimodal information and facilitate subsequent GAT aggregation. Connections were directly established between the same modalities of different individual cases, allowing the model to learn the global sharing of modal features, thus obtaining a multimodal knowledge graph.

[0080] Step S303: Construction of a pathological image knowledge graph.

[0081] In this embodiment, for each SVS pathological slide, it is first divided into several patches of 516×516 pixels with no overlap, and the depth features of each patch are extracted using ResNet-50. Then, each patch is regarded as a graph node, and its ResNet-50 features are used as node attributes. Based on the spatial adjacency relationship of the patch in the slide, undirected edges are constructed between adjacent patch nodes, finally forming a pathological patch knowledge graph (i.e., a location-encoded pathological knowledge graph) that can reflect the spatial topology and phenotypic features of the tissue. The specific steps include steps S3031-S3033:

[0082] Step S3031: ResNet-50 feature extraction of pathological slide patch.

[0083] (1) Model loading and preprocessing.

[0084] We used the official PyTorch ResNet-50 pre-trained model (ImageNet1K_V1 weights), cropped each 512×512 patch, resized it to 224×224, and normalized it (mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225]) before feeding it into the network to ensure input consistency.

[0085] (2) Patch batch processing.

[0086] Traverse all .tiff format patches in the pathology slide directory of each case, use the extract_resnet50_features function in no_grad mode for forward inference to obtain a 2048-dimensional feature vector; record the patch_id and match it with the feature vector to form a patch-level feature table.

[0087] (3) Feature storage.

[0088] All patch features under each ROI are aggregated into a DataFrame, with the first column being patch_id and the following 2048 columns being the feature dimensions. This DataFrame is then saved to a ".csv" file for subsequent graph construction and GCN input.

[0089] Step S3032, Multimodal feature splicing

[0090] (1) Feature vector concatenation: Feature vectors extracted from the three modalities of the same case are concatenated at the vector level to form a high-dimensional vector with a total of 2598 features. This feature vector contains feature information from different modalities and can more comprehensively reflect the various features of the lesion.

[0091] (2) Feature matrix construction: Combine the concatenated feature vectors of all patients into a feature matrix, where rows represent patients and columns represent features. This matrix will serve as input data for subsequent dimensionality reduction and modeling.

[0092] Step S3033, Construction of a multimodal ultrasound knowledge graph. Specifically, as follows... Figure 3 As shown.

[0093] (1) Loading empty graphs and clinical information.

[0094] First, read the clinical information of 148 cases extracted from the Excel file, convert the "serial number" field to a string and rename it to patient_id, and create a NetworkX undirected graph G and a node ID mapping table node_id_map to prepare for adding nodes later.

[0095] (2) Individual case nodes and clinical attributes.

[0096] Traverse the two groups of cases, pCR(1) and non-pCR(0), add each patient_id with complete ViT features as a node in the graph, label it as "patient", and store all its clinical fields (such as age, ER status, lesion size, etc.) as node attributes in G.nodes[patient_id] to ensure that the clinical phenotype information and imaging features are homologous.

[0097] (3) Modal intermediate nodes and image nodes.

[0098] For each case, three modalities (ela, gray, doppler) are each represented by an intermediate node (marked with type="middle") and connected to the patient node with an edge of "has_modality". Then, a corresponding image node (type="modality") is added to each intermediate node and connected with an edge of "has_image", forming a three-level topology structure of case-modality-image, realizing the hierarchical organization of multimodal information.

[0099] (4) Graph size statistics.

[0100] After construction, output G.number_of_nodes() and G.number_of_edges() to record the size of the graph and check whether each case has 3 modal links and corresponding image nodes to ensure the integrity and connectivity of the knowledge graph.

[0101] Step S3034: Construction of the pathological patch atlas.

[0102] (1) ResNet features and location loading.

[0103] Read the patch_id and 2048-dimensional ResNet-50 feature vector from the CSV file corresponding to each ROI, and load the JSON metadata to obtain the spatial coordinates (left, top, right, bottom) of the patch, providing a spatial benchmark for building neighborhood relationships.

[0104] (2) k-NN neighborhood calculation.

[0105] Using sklearn.NearestNeighbors(k=8) (including itself), a neighborhood search is performed on all patch features. The seven nearest neighbor indices after removing itself are extracted to generate an adjacency list between patches, providing a connection basis based on feature similarity for edge relationships in the graph structure.

[0106] (3) Edge index matrix and node feature tensor.

[0107] The adjacency list is converted into a PyTorch tensor edge_index(2×E), and all patch features are merged into a node feature matrix x(N×2048) to form a Graph Data object input format.

[0108] (4) Graph Data encapsulation.

[0109] The single ROI graph object is created using PyG's Data(x=x, edge_index=edge_index), and data.num_nodes and data.num_edges are output to record the number of patches and edges, providing standardized input for the subsequent GCN aggregation module.

[0110] Step S4: Apply Graph Attention Network (GAT) to aggregate features between nodes on the multimodal knowledge graph, and use a four-layer Graph Convolutional Network (GCN) to extract global topological information on the pathological patch graph. Then, input the two outputs in parallel into their respective Transformers for deep representation learning.

[0111] Specifically, this step may include steps S401-S404:

[0112] Step S401: Knowledge graph data preparation and initialization.

[0113] To organically combine multimodal features with graph structure, this step first concatenates the deep features of each modality extracted by ViT into a node feature matrix X = [h1, h2, ..., h N ] T ∈R N×F Where N is the total number of nodes (including patient, middle, and modality nodes), and F is the feature dimension (a 2048-dimensional feature vector). The clinical features of each case (such as ER status, PR status, HER2 status, etc.) are one-hot encoded, and the differences in clinical features between cases are measured using Euclidean distance. Connection edges are established based on whether the Euclidean distance in the clinical feature space is below a set threshold. This step maps heterogeneous modality information to a unified tensor space, ensuring the input normalization and data consistency of GATConv, thereby improving the stability and efficiency of subsequent attention aggregation.

[0114] The purpose of this step is to uniformly map heterogeneous modal information to tensor space, providing normalized input for subsequent GATConv operations and ensuring consistency between features and topological information.

[0115] Step S402: Multi-head graph attention aggregation calculation to obtain node feature maps.

[0116] To adaptively fuse features from various modalities, this step employs a three-layer multi-head GAT structure, and the node update process is shown in steps S4021-S4023:

[0117] Step S4021: Unnormalized attention coefficient:

[0118]

[0119] in, This represents the unnormalized attention coefficients. LeakyReLU() is a non-linear activation function. (l,k)T W is the attention coefficient of the current layer, used to weight the relationships between node features. (l,k) Represents the weight matrix. and These represent the feature representations of node i and node j in the current layer, respectively.

[0120] This step aims to model the initial association strength between nodes. By introducing learnable linear transformations and nonlinear activations, it can highlight the characteristic differences of important modal nodes, laying the foundation for attention allocation.

[0121] Step S4022, attention weight normalization:

[0122]

[0123] Where, α i,k (l) represents the attention coefficient of node i in layer l in layer k, and N(i) is the set of neighboring nodes of node i.

[0124] This step maps attention energy to a probability distribution using Softmax, ensuring that the weights of different neighboring nodes sum to 1, thereby achieving stable and interpretable feature weighted aggregation.

[0125] Step S4023, Node Feature Update:

[0126] Multi-head attention is aggregated in parallel and then concatenated to capture various semantic relationships and enrich node representations; ELU activation increases non-linear expressive power, and the final fused features have stronger discriminativeness and robustness, significantly improving the downstream pCR prediction performance.

[0127] The following is a detailed explanation of the processing procedure for each of the three layers of GAT:

[0128] (1) First layer GATConv: Input X and edge_index, calculate the attention coefficients between nodes through 8 parallel attention heads and aggregate neighbor features, output Nonlinearity is introduced based on the ELU activation function.

[0129] (2) Second layer GATConv: H (1) Input a second multi-head attention layer, continue using 8-head aggregation and compress to ELU activation is also applied to enhance feature robustness.

[0130] (3) Third layer GATConv: Use single-head attention to focus on H (2) Aggregation to final output No activation function is used to obtain the final node representation.

[0131] Through the multi-head attention mechanism, each layer can learn the relative importance of nodes in parallel in different subspaces and adaptively assign higher weights to key modal nodes, thereby improving the accuracy of multimodal information complementarity and fusion; hierarchical aggregation gradually expands the receptive field, taking into account both local and cross-modal global dependencies, significantly enhancing the discriminative ability of node features and the robustness of the model.

[0132] Step S403, GCN aggregation on the pathological patch image.

[0133] In this step, a four-layer Graph Convolutional Network (GCN) is used to extract features from the pathological patch images constructed for each case. Each case corresponds to one image, where each patch is a node, and the node features are derived from the deep features extracted by ResNet-50. The GCN aggregates the local information of these nodes and combines the node features of each image into a 128-dimensional global feature set, which is ultimately used for the NAC efficacy prediction task. This process effectively captures the spatial dependencies and topological structure between nodes in the pathological images, thus providing more accurate global features for subsequent prediction models.

[0134] Specifically, firstly, the CSV file corresponding to each ROI is read, loading the patch_id and its 2048-dimensional ResNet-50 feature vector, and the spatial coordinates (left, top, right, bottom) of each patch are obtained by reading the JSON metadata. Using these features and location information, the adjacency relationships between patches are calculated through a k-NN (k=7), generating an edge index matrix. and node feature tensor Where N is the number of patches and E is the number of adjacent edges. This step ensures that the features input to the GCN accurately correspond to the graph topology, providing a standardized input format for subsequent graph convolution operations.

[0135] Step S404: The four-layer structure of GCN and global feature extraction are used to obtain the global feature map.

[0136] GCN aggregates node features through four convolutional layers. Each convolutional layer combines the features of the current node with information from its neighboring nodes to generate richer node representations. Specific steps include S4041-S4046:

[0137] Step S4041, Initialize node features: The features of each pathological patch node are high-dimensional features extracted by ResNet-50, which serve as the initial features for the GCN input. Assume the features of node i are h... i(0), where h i (0) is the 2048-dimensional feature vector extracted by ResNet-50, representing the initial representation of the node.

[0138] Step S4042, GCN Convolution Operation: Each layer of the GCN aggregates information from neighboring nodes through convolution operations. The node feature update formula for the l-th layer of the GCN is:

[0139]

[0140] Among them, h i (l+1) represents the node features of the (l+1)th layer, W l This is the learnable weight matrix of this layer, where N(i) represents the set of neighboring nodes of node i, and d i and d i σ represents the degree of node i and node j, respectively, and σ is the ReLU activation function. Through this operation, GCN weighted aggregates the features of each node with the features of its neighbors.

[0141] Step S4043, Multi-layer Stacking. In this step, GCN contains four convolutional layers, each updating node features by aggregating information from neighboring nodes. Each layer extracts information at different levels, gradually strengthening the interdependencies between nodes. Finally, a deep representation of each node is obtained through four convolutional layers.

[0142] Step S4044, Global Feature Extraction: After the last GCN layer, the features of all nodes are aggregated using a pooling method to generate a global feature vector representing the overall information of the pathological image. We use Global Average Pooling to extract the 128-dimensional global features of the pathological image, as shown in the following formula:

[0143]

[0144] Among them, h global It is a global feature of the pathological image, where N is the number of nodes and h is the number of nodes. i (L) represents the feature of the i-th node in the L-th layer GCN. Through average pooling, the features of all nodes are pooled into a single global feature representation, preserving the information of the entire pathological map.

[0145] Step S4045, through GCN aggregation and global pooling of node features, extracts global features from pathological slide images, reflecting the overall structure and histological characteristics of the pathological images. These global features provide strong information support for subsequent pCR prediction models, helping to more accurately assess the efficacy of neoadjuvant chemotherapy.

[0146] Step S4046: Advantages of Node and Global Features. Node features extracted using GCN can retain fine-grained information from the pathological image, providing details about histological structure; while global features can summarize this local information to form a concise representation, facilitating learning for subsequent tasks. Global features extracted through GCN are more representative and can effectively improve the accuracy of pCR prediction.

[0147] The following is a detailed explanation of the processing procedure for each of the four layers of GCN:

[0148] (1) First convolutional layer: Input The edge_index is processed by GCNConv(2048→128) and ReLU activation, and the output is... Low-level structural information used for initial fusion of neighbor patches.

[0149] (2) Second convolution: H (1) Inputting GCNConv(128→128) will activate ReLU and yield the following result. Further enhance the feature transfer and representation capabilities between nodes to capture a wider range of local topological relationships.

[0150] (3) Third convolutional layer: H (2) Activated by GCNConv (128→64) and ReLU, the output is... While reducing dimensionality, key features are extracted, balancing expressive power and computational efficiency.

[0151] (4) Fourth layer convolution and global convergence: H (3) Input GCNConv(64→128) and activate ReLU to get Then, average pooling is performed on all nodes to generate a 128-dimensional global topological feature vector for the ROI.

[0152] The four-layer GCN structure can extract spatial dependencies between patches from local to global levels through layer-by-layer neighborhood aggregation; the final average pooling aggregation operation integrates all node information into global features, effectively preserving the overall organizational structure information and providing high-quality, compact global representations for subsequent Transformer learning.

[0153] Step S5: Independent prediction by dual-path Transformer.

[0154] See Figure 3This invention constructs two independent Transformer prediction pathways: the first pathway takes a multimodal imaging knowledge graph and clinical information as input, and generates a pCR probability distribution P1(y) through feature projection, position encoding, self-attention encoding, and a classification head. The second pathway takes global features of pathological slices extracted by GCN and the same clinical information as input, and independently outputs a pCR probability distribution P2(y) through sequence construction, two-layer self-attention, and a feedforward classification head.

[0155] Specifically, this step may include steps S501-S504:

[0156] Step S501, Feature Projection and Sequence Construction. Specifically, this includes steps S5011-S5012:

[0157] Step S5011, linear projection of clinical and GAT features: clinical features c with different dimensions (such as ER status, PR status, HER2 status, ki67 status and molecular subtype three-class multi-dimensional clinical features) and knowledge graph GAT node features g are mapped to the same hidden dimension D, which facilitates subsequent parallel processing by Transformer.

[0158] c′=W c ·c+b c

[0159] g′=W g ·g+b g

[0160] in Linear transformation ensures that features from different sources obtain the same representation space, improving fusion efficiency and consistency.

[0161] Step S5012, CLS labeling and positional encoding are introduced: a classification label is reserved at the beginning of the sequence to provide a global focus node for the Transformer; positional encoding is used to distinguish the relative order of sequence elements. Let CLS be a trainable vector, and the sequence is concatenated. make For trainable location encoding;

[0162] The final input is X = S + P. CLS provides a global context aggregation entry point, and positional encoding enables the model to have sequence awareness, laying the foundation for the self-attention mechanism.

[0163] Step S502, Transformer encoding and probability output. Specifically, this includes steps S5021-S5022:

[0164] Step S5021, Multi-head self-attention calculation: Capture the global dependencies between elements within the sequence, enabling the model to dynamically focus on the interactions between CLS, GAT and clinical features.

[0165] Calculate query Q = X·W for each X. Q Key K = X·W K Value V = X·W V And perform multi-head attention:

[0166]

[0167] After concatenation of multiple heads, a single layer of encoding is completed through linear transformation, residual connection, and LayerNorm. The multi-head mechanism can learn association patterns in different subspaces in parallel, enhancing the diversity and robustness of feature representation.

[0168] In step S5022, the feedforward network and classification head undergo further nonlinear transformation based on self-attention to refine the global representation of the CLS position and generate the pCR prediction probability.

[0169] Take the output h of the first position (CLS) after encoding cls h′ is obtained by LayerNorm normalization;

[0170] S1=W1·h′+b1

[0171] a = ReLU(S1)

[0172] S2=W2·a+b2

[0173] P (y＝i) =exp(S i ) / (exp(S0)+exp(S1)),i∈{0,1}

[0174] The feedforward network increases nonlinear expressive power, and the Softmax output ensures that interpretable binary classification probabilities are obtained for fusion.

[0175] Step S503, first pathway feature projection and sequence construction. Specifically includes steps S5031-S5033:

[0176] Step S5031, SMOTE balancing and feature preparation: To address the class imbalance problem in pathological features, SMOTE oversampling is introduced to generate minority class samples, ensuring that Transformer can fully learn from both pCR and non-pCR cases during training.

[0177] In each cross-validation tradeoff, the clinical features c and global features g of the training set are oversampled using SMOTE, and then standardized to separate the two sub-vectors. Balancing the data distribution can prevent the model from being biased towards the majority class and improve the ability to identify the minority class (pCR or non-pCR).

[0178] Step S5032, Clinical and GCN feature mapping, maps the standardized multi-source features to the Transformer hidden space and introduces appropriate Dropout to suppress overfitting.

[0179] C′=Dropout(DropoutRate)·ReLU(W c ·c+b c )

[0180] g′=Dropout(DropoutRate)·ReLU(W g ·g+b g )

[0181] in Combining Dropout with ReLU can enhance feature sparsity during the mapping stage and improve the robustness of the model on small sample pathological data.

[0182] Step S5033, sequence concatenation and position encoding, provides the Transformer with a unified format of three-element sequences and injects spatial position information to distinguish CLS, pathological and clinical features. CLS is a trainable vector;

[0183]

[0184] X = S + PosEmbedding, where PosEmbedding is trainable and has a shape of 1×3×D. In scenarios where small-scale patch diagrams are fused with clinical data, positional encoding ensures that the model perceives the relative order and semantic hierarchy of elements within the sequence.

[0185] Step S504, second-path Transformer encoding and probability output. Specifically, this includes steps S5041-S5042:

[0186] Step S5041, the two-layer self-attention encoding quickly captures the global dependencies between CLS, pathological and clinical features in a shallower two-layer structure, and reduces the risk of overfitting by using fewer layers.

[0187] After calculating Q, K, and V for X, two EncoderLayers are executed. This two-layer design maintains sufficient expressive power while better adapting to small pathological samples, improving convergence speed and generalization.

[0188] Step S5042, LayerNorm and classification head: The CLS output is stabilized by LayerNorm, and the final score is generated using a feedforward classification head with Dropout. The class weights are combined to suppress the impact of data imbalance.

[0189] h cls=LayerNorm(x[:,0,:])

[0190] S = W2·ReLU(W1·h cls +b1)+b2

[0191]

[0192] Loss=CrossEntropy(s,y;weight)

[0193] LayerNorm ensures stable feature distribution, and the weighted loss function helps improve minority class prediction performance. The final output P2 is the pCR probability distribution of the second path.

[0194] Step S6: Weighted soft voting fusion and performance evaluation.

[0195] To fully utilize the complementary advantages of the two pathways, fusion weights ω1 and ω2 (ω1 + ω2 = 1) are set, and a weighted soft vote is performed on the two outputs. The final category prediction is the fused prediction probability P. f The larger of the two probabilities is used as the confidence score.

[0196] On the test set, the performance of the fusion model was evaluated by accuracy (ACC) and area under the receiver operating characteristic curve (AUC) to measure the overall prediction accuracy and classification ability, respectively. The results were compared with those of the single-modal Transformer and traditional clinical assessment methods to verify the advantages of the dual-pathway weighted soft voting strategy in this embodiment in improving prediction accuracy and stability.

[0197] Specifically, to fully utilize the two independent Transformer prediction pathways, this invention employs a weighted soft voting strategy to fuse the outputs of the two models. The pCR prediction probabilities output by each model are weighted and merged according to preset weights, ultimately generating a more stable and more predictive composite probability.

[0198] The SoftVotingEnsemble class is used to combine the outputs of two independent models after softmax normalization:

[0199] P f =α×P1+(1-α)×P2

[0200] Where P1 and P2 are the predicted probabilities of the two models, and α is a hyperparameter controlling the fusion weights, α∈[0,1]. In practical applications, α can be adjusted using the validation set to obtain the best fusion effect. The preferred α = 0.8 indicates a slightly higher weight for the first pathway (multimodal imaging + clinical), reflecting its slightly better performance on the validation set; (1-α) = 0.2 retains the supplementary information of the second pathway (pathological GCN + clinical). After fusion, the fusion weights are adjusted for P1 and P2. f The category corresponding to the maximum value is taken as the final prediction, and this maximum probability value is output as the confidence score. This strategy can smooth the prediction noise of individual pathways and improve the overall stability and accuracy.

[0201] By combining the advantages of two pathways through a weighted soft voting strategy, the robustness and prediction accuracy of the model under different data perspectives can be improved, avoiding the bias that may be caused by a single model.

[0202] The fusion model was evaluated using a test set, and its accuracy (ACC) and area under the ROC curve (AUC) were calculated. Comparative analysis was conducted with the single-modal Transformer and other common evaluation methods to verify the advantages of the soft voting strategy. All models were compared on the same test set data and using the same feature subset to ensure fairness in the comparison.

[0203] See Figure 5 The receiver operating characteristic curves of different algorithms for early prediction of NAC efficacy in breast cancer are shown in Table 1, which shows the efficacy of the test set in predicting pCR.

[0204] Table 1. Performance of the test set in predicting pCR

[0205]

[0206]

[0207] Therefore, the fusion model proposed in this application outperforms any single-path model in both accuracy (0.818) and AUC (0.846), while maintaining a balanced level of sensitivity and specificity (0.714 / 0.880), demonstrating its stronger discriminative ability in distinguishing between pCR and non-pCR patients. Compared with traditional MLP and random forest, the fusion model significantly outperforms in all metrics, showcasing the superiority of deep multimodal fusion strategies for complex and heterogeneous data. Compared to single-branch Transformers that rely solely on imaging or pathological information, the fusion model can fully integrate the complementary features of both, significantly improving the stability and accuracy of early prediction. In summary, the dual-pathway Transformer weighted soft-voting fusion method of this embodiment effectively combines multimodal knowledge graphs with global pathological topological information, significantly improving the performance of pCR prediction after NAC in breast cancer.

[0208] To evaluate the impact of clinical feature similarity thresholds on graph structure construction and subsequent graph neural network representation, the distance threshold was set from 0.1 to 0.9, with a step size of 0.1, and heterogeneous graphs were constructed cyclically.

[0209] Based on the above indicators, as shown in Table 2, it can be seen that when the threshold is set to 0.7, the model achieves an optimal balance in various performance aspects: accuracy reaches 80.4%, sensitivity is 87.1%, specificity is 87.4%, AUC is 0.841, and the positive predictive value (PPV) is as high as 86.7%, while the negative predictive value (NPV) remains at 92.1%. In contrast, although the sensitivity is higher (90.3%) and the PPV is the highest (92.5%) when the threshold is 0.5, its NPV (83.6%) and AUC (0.835) are slightly inferior; while other thresholds either sacrifice sensitivity or specificity. Considering the need to minimize missed diagnoses (high sensitivity, high NPV) and avoid false positives (high specificity, high PPV) in the early prediction of neoadjuvant chemotherapy response, a connection threshold of 0.7 was selected.

[0210] Table 2. The effect of different Euclidean thresholds on the results.

[0211]

[0212] Next, the impact of the contribution value α of the two models in soft voting on the final results was tested, and the comparison results are shown in Table 3. α was traversed in steps from 0.1 to 0.9, and the accuracy, sensitivity, specificity, and AUC of the models were recorded. The results show that when α is small (e.g., α = 0.1), although the specificity reaches 0.885, the sensitivity is only 0.483, and the overall accuracy and AUC are also low (0.750 and 0.806, respectively). As α gradually increases to 0.4, the sensitivity increases to a peak of 0.738, while maintaining high specificity (0.912) and AUC (0.846). When α is further increased to 0.6, the model's accuracy and AUC simultaneously reach their highest points, at 0.812 and 0.919, respectively, at which point the sensitivity remains at 0.703, and the specificity is 0.924. When α continues to increase to 0.9, the sensitivity drops sharply to 0.408, causing the overall performance to decline. Therefore, when α = 0.6, the model achieves the best balance between accuracy, sensitivity and specificity, resulting in optimal performance in the early pCR prediction task.

[0213] Table 3. Impact of different soft voting contribution values ​​on the results

[0214]

[0215] Example 2

[0216] Based on Example 1, this example provides a NAC efficacy prediction system based on multimodal imaging and dual-pathway GCN-Transformer, including:

[0217] (1) Clinical feature acquisition module, used to acquire pathological images and multimodal ultrasound images of the lesion site to be predicted.

[0218] (2) The dual-pathway GCN-Transformer prediction module is used to obtain the efficacy prediction results based on the pathological images and multimodal ultrasound images of the lesion site to be predicted, through the NAC efficacy prediction method based on multimodal imaging and dual-pathway GCN-Transformer in Example 1.

[0219] (3) Interactive module, used to display the efficacy prediction results.

[0220] Existing models for predicting the efficacy of neoadjuvant chemotherapy (NAC) in breast cancer are mostly based on single-modality imaging data or clinical features, often extracting information from only limited dimensions, resulting in low prediction accuracy. This invention combines multimodal ultrasound images, pathological images, and clinical features, fully utilizing information from different modalities for accurate prediction. First, by introducing a graph neural network (GNN) and a Transformer model, the model can process multimodal data and learn deep-level correlations between modalities, significantly improving prediction accuracy and stability. Second, by combining a weighted soft-voting strategy to fuse the outputs of two independent Transformers, the comprehensive ability of the prediction results is optimized, giving the model stronger generalization ability on diverse datasets. This fusion strategy can fully combine the advantages of imaging and clinical information, effectively improving the accuracy of pCR prediction. Furthermore, the model enhances its ability to perceive data order and spatial relationships through self-attention mechanisms and positional encoding, further strengthening the expressive power of multimodal data. Finally, experimental verification shows that the model of the present invention has achieved significant improvements in accuracy and area under the ROC curve (AUC) compared with traditional single-modal methods and other existing methods in the task of predicting the efficacy of NAC in breast cancer.

[0221] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for predicting the efficacy of NAC (Non-Acute Circulatory Collapse) based on multimodal imaging and dual-pathway GCN-Transformer, characterized in that, Includes the following steps: Step S1: Obtain pathological images and multimodal ultrasound images of the lesion sites in multiple cases before NAC; Step S2: Perform region of interest annotation and image preprocessing on the ultrasound image and pathological image; Step S3: Extract high-dimensional features from multimodal ultrasound images, establish a multimodal knowledge graph for each case and its corresponding ultrasound image nodes, extract features from pathological slices obtained by segmenting pathological images, and construct a location-encoded pathological knowledge graph. Step S4: In the first path, for the multimodal knowledge graph, feature aggregation between nodes is achieved by applying a graph attention network to obtain a node feature map. In the second path, for the location-encoded pathological knowledge graph, global topological information is extracted using a multi-layer graph convolutional network to obtain a global feature map. The aggregated features output from the two paths and the global topological information are input in parallel into the Transformer corresponding to each path for deep representation learning. Step S5: Obtain the ultrasound image and pathological image to be predicted, and use the Transformer of the first and second pathways to predict the complete pathological remission rate of each pathway. Step S6: Based on the pathological complete remission rates of the two pathways, efficacy prediction results are obtained through weighted soft voting.

2. The method for predicting NAC efficacy based on multimodal imaging and dual-pathway GCN-Transformer according to claim 1, characterized in that, Step S3 includes: Step S301: For each case, extract high-dimensional features of radiomics using a pre-trained visual Transformer for each ultrasound modality of ultrasound images. Step S302: Using an individual as the central node of the heterogeneous knowledge graph, the depth features of the images under each ultrasound modality are extracted using a pre-trained visual Transformer. Each set of features is taken as a modal feature node. All modal feature nodes are connected to the corresponding intermediate nodes through undirected edges. The intermediate nodes are connected to the central node. Connections are established between the same modalities of different individuals to form a multimodal knowledge graph that carries complementary multimodal information. Step S303: For each pathological slice obtained by segmenting the pathological image, divide it into several non-overlapping partitions according to a preset size, extract the depth features of each partition, treat each partition as a graph node, and use the depth features as node attributes. Based on the spatial adjacency relationship of the partitions in the slice, construct undirected edges between adjacent graph nodes to form a case-level location-coded pathological knowledge graph that represents the spatial topology and phenotypic features of the tissue.

3. The method for predicting NAC efficacy based on multimodal imaging and dual-pathway GCN-Transformer according to claim 1, characterized in that, Step S4 includes: Step S401: Based on the multimodal knowledge graph, a node feature matrix is ​​formed by concatenating the deep features of each modality, and an edge index matrix is ​​constructed according to the node relationships in the multimodal knowledge graph. Step S402: For the multimodal knowledge graph, a three-layer multi-head graph attention network structure is formed by calculating the attention coefficients between nodes, normalizing the attention weights, and updating the node features, so as to achieve adaptive aggregation of features of each modality and obtain the node feature map. Step S403: Based on the location-encoded pathological knowledge graph, a four-layer graph convolutional network is used to aggregate the local information of the nodes in the knowledge graph; In step S404, during the aggregation process using the four-layer graph convolutional network in step S403, for each layer of the graph convolutional network, the node feature representation of the current graph convolutional network layer is updated based on the set of neighboring nodes of the current node. For the node feature representation of the last layer of the graph convolutional network, the features of all nodes in the location-encoded pathological knowledge graph are aggregated by global average pooling to obtain global topological information that characterizes the overall structure and histological features of the pathological image, which serves as the global feature map. Step S405: Input the aggregated features of the two pathway outputs and the global topological information into the Transformer corresponding to each pathway in parallel for deep representation learning.

4. The method for predicting NAC efficacy based on multimodal imaging and dual-pathway GCN-Transformer according to claim 1, characterized in that, In the aforementioned multi-layer graph convolutional network, the node feature representation update of each layer and the global average pooling are implemented using the following formula: Among them, h i (l+1) represents the node feature representation after the (l+1)th layer graph convolutional network update, σ is the ReLU activation function, N(i) is the set of neighboring nodes of node i in the position-encoded pathological knowledge graph, and d i and d i W represents the degree of node i and node j, respectively. l h is the learnable weight matrix of the l-th layer graph convolutional network. global This represents the global topology information after global average pooling, where N is the number of nodes and h is the number of nodes. i (L) is the feature of the i-th node in the L-th layer graph convolutional network.

5. The method for predicting NAC efficacy based on multimodal imaging and dual-pathway GCN-Transformer according to claim 1, characterized in that, Step S5 includes: Step S501: Obtain the clinical features of the case to be predicted, map the clinical features and the node feature map to the same dimension, and form a feature sequence together with the trainable classification label. Combine the feature sequence with the trainable position encoding as the input of the Transformer of the first path. In step S502, in the Transformer of the first pathway, feature interaction between clinical features, node feature maps and classification labels is achieved through global attention. The global representation of classification labels is extracted through the feedforward network and classification head to generate the pathological complete remission rate predicted by the first pathway. Step S503: Perform SMOTE oversampling and standardization on the clinical features and the global feature map, and map them to the hidden space. Combine the new feature sequence with the trainable classification label to form a new feature sequence. Combine the new feature sequence with the new trainable position encoding as the input of the Transformer of the second pathway. In step S504, in the Transformer of the second pathway, the global dependency between clinical features, global feature map and new location encoding is captured by sub-attention encoding, and the pathological complete remission rate predicted by the second pathway is generated by layer normalization and classification head.

6. The method for predicting NAC efficacy based on multimodal imaging and dual-pathway GCN-Transformer according to claim 1, characterized in that, Step S6 includes: Based on the predicted complete remission rates of the first and second pathways, the final efficacy prediction results were obtained by normalization and formaldehyde combination.

7. The method for predicting NAC efficacy based on multimodal imaging and dual-pathway GCN-Transformer according to claim 1, characterized in that, The image preprocessing includes grayscale normalization and pathological image segmentation.

8. The method for predicting NAC efficacy based on multimodal imaging and dual-pathway GCN-Transformer according to claim 1, characterized in that, After acquiring pathological images and multimodal ultrasound images, the process also includes: The acquired pathological images and multimodal, multi-sectional ultrasound images were randomly divided into five equal parts. Four parts were used as the training set and one part as the validation set. The process was repeated five times to test the stability and generalization ability of the model.

9. The method for predicting NAC efficacy based on multimodal imaging and dual-pathway GCN-Transformer according to claim 1, characterized in that, The multimodal ultrasound images include grayscale multi-section ultrasound images, elastic multi-section ultrasound images, and blood flow multi-section ultrasound images.

10. A NAC efficacy prediction system based on multimodal imaging and dual-pathway GCN-Transformer, characterized in that, include: The clinical feature acquisition module is used to acquire pathological images and multimodal ultrasound images of the lesion site to be predicted. The dual-pathway GCN-Transformer prediction module is used to obtain efficacy prediction results based on pathological images and multimodal ultrasound images of the lesion site to be predicted, using the NAC efficacy prediction method based on multimodal imaging and dual-pathway GCN-Transformer as described in any one of claims 1-9. An interactive module is used to display the efficacy prediction results.

Images