Postoperative complication prediction method and device based on two-way graph neural network

Abstract

The invention relates to a postoperative complication prediction method and device based on a two-way map neural network, and the method comprises the steps: collecting ICG video images before and after cerebrovascular bypass surgery, and obtaining the clinical feature data of a patient; preprocessing the collected ICG video image, extracting a blood vessel network by adopting an image segmentation algorithm, and generating corresponding blood vessel graph structures before and after bridging; extracting blood vessel features, blood flow dynamic features and clinical features from the collected ICG video images and clinical feature data, and constructing a multi-modal high-dimensional feature set; and taking the multi-modal high-dimensional feature set and the corresponding vascular graph structures before and after bridging as input, and generating a complication prediction score through a pre-trained prediction model based on a two-way graph neural network. Compared with the prior art, the method has the advantages that the multi-modal feature information is fused, the graph neural network structure is optimized, the model performance is stable, the accuracy is high, and the method has important clinical application value.

Description

Technical Field

[0001] The present invention relates to the field of medical image analysis technology, and in particular to a method and device for predicting postoperative complications based on a dual-path graph neural network. Background Art

[0002] Moyamoya disease (MMD) is a chronic cerebrovascular disease of unknown etiology, characterized by stenosis or occlusion of the middle cerebral artery, distal internal carotid artery, and proximal anterior cerebral artery. MMD is also characterized by an irregular network of perforated vessels, with areas of occlusion or stenosis corresponding to serrated arteries and thalamic perforations. Currently, there is no evidence that drug therapy can slow or even reverse the progression of MMD. Current treatments for MMD target clinical symptoms of ischemia and hemorrhage solely through anticoagulation or hemostasis. Surgery is an effective treatment option aimed at improving brain blood supply and alleviating neurological deficits. Some studies have shown that surgical interventions are superior to medical treatment or the natural course of the disease, particularly in reducing the incidence of cerebral ischemia. Surgical revascularization is an effective method for improving cerebral blood perfusion in patients with MMD. Direct bypass surgery (such as superficial temporal artery-middle cerebral artery bypass grafting) is particularly effective in rapidly improving cerebral blood flow after surgery.

[0003] Cerebrovascular bypass surgery is an important surgical procedure commonly used to treat complex cerebrovascular diseases (such as moyamoya disease, intracranial aneurysms, and ischemic stroke). It improves cerebral blood supply by providing a new blood flow pathway to the ischemic area. However, patients may develop a series of complications after surgery, including cerebral ischemia, cerebral hemorrhage, and cerebral hyperperfusion syndrome. These complications not only seriously affect the patient's postoperative recovery but may also lead to permanent neurological damage or even death. Therefore, early prediction of postoperative complications is of great significance for formulating postoperative management strategies and improving patient prognosis.

[0004] Indocyanine green (ICG) is a near-infrared fluorescent tricarbonate dye that is injected intravenously. It binds tightly to plasma proteins and produces fluorescence when excited by near-infrared light. Studies have demonstrated that ICG's dynamic flow through anastomotic vessels can aid in determining intraoperative blood flow patency and direction.

[0005] The prediction of postoperative complications mainly relies on the patient's clinical characteristics and intraoperative monitoring data. For example, a multivariate regression model is used to analyze the correlation between the patient's preoperative health status (such as age, gender, hypertension, diabetes, etc.) and the occurrence of complications. However, these methods often rely on only a small number of clinical features and fail to fully utilize the deep information of pre- and post-surgery imaging data. The accuracy and robustness of the prediction results are limited. In addition, traditional imaging analysis methods mostly focus on the extraction of single-modality image features and fail to achieve effective fusion of multimodal data, which further limits the comprehensive assessment of the risk of complication occurrence.

[0006] In recent years, radiomics and machine learning technologies have been increasingly applied to the analysis and diagnosis of medical imaging data. Radiomics extracts features such as intensity, texture, and shape from medical images at high throughput, providing a rich set of quantitative metrics for disease detection and prediction. Machine learning, on the other hand, significantly improves disease classification and prediction by learning nonlinear patterns from data. However, traditional machine learning methods are susceptible to feature redundancy and noise when processing high-dimensional imaging features, leading to increased model complexity and overfitting. Summary of the Invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies, which are mostly limited to clinical feature analysis or single modality image feature extraction, making it difficult to fully capture the dynamic blood perfusion changes before and after surgery, resulting in limited accuracy and robustness of the prediction results, and to provide a postoperative complication prediction method and device based on a dual-path graph neural network.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A method for predicting postoperative complications based on a dual-path graph neural network comprises the following steps:

[0010] Collect ICG video images before and after cerebral vascular bypass surgery and obtain the patient's clinical characteristics data;

[0011] The collected ICG video images are pre-processed and the vascular network is extracted using an image segmentation algorithm to generate the corresponding vascular map structure before and after bypass surgery;

[0012] Extract vascular features, blood flow dynamic features, and clinical features from the collected ICG video images and clinical feature data to construct a multimodal high-dimensional feature set;

[0013] Using a multimodal high-dimensional feature set and the corresponding vascular graph structures before and after bypass surgery as input, a pre-trained prediction model based on a dual-path graph neural network is used to generate a complication prediction score.

[0014] Among them, the prediction model based on the dual-path graph neural network includes a node embedding layer, a global embedding layer, a cross-graph interaction layer and an aggregate prediction layer. The node embedding layer is used to calculate the node representation matrix based on the multimodal high-dimensional feature set and the vascular graph structure. The global embedding layer is used to learn the association relationship between nodes based on the node representation matrix based on the attention mechanism to obtain the global embedding vector before and after the bypass; the cross-graph interaction layer is used to calculate the relationship between the global embedding vectors before and after the bypass, and output the complication prediction score through the aggregate prediction layer.

[0015] Furthermore, the input of the node embedding layer includes node features based on a multimodal high-dimensional feature set. and an adjacency matrix based on the vascular graph structure, where N is the total number of nodes, D is the characteristic dimension corresponding to the node, and the adjacency matrix is ​​used to indicate whether two nodes are connected;

[0016] Then a weight matrix with shared parameters is trained for all nodes to perform a shared self-attention mechanism for each node and calculate the attention correlation coefficient;

[0017] The expression of the attention correlation coefficient is:

[0018]

[0019] Where, e ij is the attention correlation coefficient between node i and node j, W node is the weight matrix, is the feature of node i, is the feature of node j;

[0020] The attention correlation coefficient is normalized, and the corresponding calculation expression is:

[0021]

[0022] Where, α ij is the normalized attention correlation coefficient, T Indicates transposition, || is a connection symbol, is the parameter of the feedforward neural network a, LeakyReLU is a nonlinear activation function, N i For node v i An adjacent node of

[0023] Based on the normalized attention correlation coefficient, the final representation vector of each node is calculated, and the corresponding calculation expression is:

[0024]

[0025] Where, is the final representation vector of node i, and σ is the nonlinear activation function.

[0026] Furthermore, the global embedding layer first averages the representation matrix of each node and transforms it through a nonlinear activation function to obtain an initial global embedding vector. The calculation expression of the initial global embedding vector is:

[0027]

[0028] Where C is the initial global embedding vector, ReLU is the nonlinear activation function, and W graph is the weight matrix, N is the total number of nodes, h i is the feature of node i;

[0029] Take the inner product of the embedded feature vector of each node and the initial global embedding vector as the attention weight of each node to obtain the global embedding vector;

[0030] The calculation expression of the attention weight is:

[0031]

[0032] Where a i is the attention weight of node i, σ is the nonlinear activation function;

[0033] The calculation expression of the global embedding vector is:

[0034]

[0035] Where G_all is the global embedding vector.

[0036] Furthermore, the cross-graph interaction layer calculates the relationship between the two global embedding vectors before and after bridging through a neural tensor network. The corresponding expression is:

[0037]

[0038] Where, is the relationship between the two global embedding vectors G1 and G2, σ is a nonlinear activation function, is a weight tensor, there are K, K is a hyperparameter, is the number of interaction scores generated by the model for each pair of graphs, is the weight matrix of the standard neural network, [·] is the inner product of the two images before and after surgery, is the bias.

[0039] Furthermore, the cross-graph interaction layer obtains a correlation score matrix based on the relationship between the global embedding vectors before and after bridging;

[0040] The aggregate prediction layer concatenates the correlation score matrix and the global embedding vectors before and after bridging, and uses multiple fully connected neural networks to gradually reduce the dimension of the variability score vector to obtain a complication prediction score.

[0041] Furthermore, the ICG video image acquisition process is specifically as follows:

[0042] During cerebral vascular bypass surgery, ICG fluorescence angiography videos of patients before and after bypass surgery are collected through a microscope that integrates a near-infrared light emission device and a fluorescence acquisition system.

[0043] Furthermore, the generation process of the vascular map structure is specifically as follows:

[0044] For the last frame of the ICG video before and after bypass surgery, blood vessel segmentation is performed to extract the blood vessel area and generate a binary image containing blood vessel information.

[0045] Perform thinning processing on the binary image containing vascular information to generate a vascular skeleton map;

[0046] Detecting the intersection positions of blood vessels on the blood vessel skeleton graph and defining them as nodes of the blood vessel network;

[0047] Connecting vascular segments according to the positions of nodes on the vascular skeleton graph, and defining vascular segments as paths between nodes, i.e., edges of the vascular network;

[0048] Based on the detected node and edge information, the vascular graph structure before and after bypass is constructed. The vascular graph structure is represented in the form of an adjacency matrix. Each node contains position information and brightness value, and each edge contains length, diameter and corresponding blood flow parameters.

[0049] Furthermore, the blood vessel characteristics include total blood vessel length, total blood vessel area, number of connected blood vessels, blood vessel diameter characteristics and blood vessel width characteristics;

[0050] The blood flow dynamic characteristics include optical flow characteristics and fluorescence brightness characteristics;

[0051] In the process of extracting the vascular features and blood flow dynamic features, the collected ICG video images are divided into multiple segments according to time, and the vascular features and blood flow dynamic features of each ICG video image are counted respectively, and the sum of them is a multi-dimensional feature;

[0052] The clinical characteristics include gender, age, surgical side, lesion type, smoking history, drinking history and NIHSS score.

[0053] Furthermore, during the training process of the prediction model based on the dual-path graph neural network, the patient's postoperative complications are classified and labeled based on the collected ICG video images, and corresponding classification labels are constructed; a standardized data set is constructed for the selected ICG video images and the corresponding classification labeling results, which is used for the training of the prediction model of the graph neural network.

[0054] The present invention also provides a postoperative complication prediction device based on a dual-path graph neural network, comprising a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of the above-mentioned method.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] (1) This invention combines the static imaging features of bypass ICG fluorescence videos with the dynamic blood perfusion change features, as well as the patient's clinical characteristics, to form a comprehensive multimodal feature set. Compared with traditional methods that rely on only a single type of feature, this invention significantly improves the ability to predict postoperative complications, with richer and more comprehensive feature information.

[0057] This paper proposes a dual-pathway graph neural network model. By constructing a pre- and post-surgery vascular network graph, this model uses a graph attention network to embed nodes. This model then captures the dynamic changes in the vascular network before and after vascular bypass surgery through global embedding and cross-graph interaction analysis. Compared to traditional machine learning classifiers (such as SVM or logistic regression), this model achieves end-to-end modeling of the structural characteristics and dynamic changes of vascular networks, significantly improving the model's classification accuracy and robustness.

[0058] By designing a cross-graph interaction module based on a neural tensor network (NTN), the present invention can quantify the degree of change in the vascular network before and after bypass surgery and perform deep learning modeling on these changes. The present invention introduces a global embedding weighting method based on an attention mechanism, which can focus on the dynamic changes in the recipient vessels and their surrounding key areas. This method significantly improves the classification ability of positive samples (cases with complications), especially when the number of samples is limited, achieving even better classification performance, meeting the clinical demand for high-sensitivity prediction.

[0059] (2) Through systematic optimization of the node update mechanism, number of network layers, and global embedding method, the present invention determined the optimal graph neural network structure, enabling the model to achieve high levels of accuracy (Acc) and area under the curve (AUC), significantly outperforming existing technical methods. Based on the differential analysis of the vascular network before and after bypass surgery, the present model can quickly and accurately predict postoperative complications, helping clinicians to adjust treatment plans in a timely manner during the perioperative period, reducing the incidence of complications and improving patient prognosis.

[0060] (3) The present invention is based on the node embedding layer in the prediction model of the dual-path graph neural network. In order to obtain sufficient expression ability, the input features are converted into higher-level features. By training a weight matrix with shared parameters for all nodes, the attention correlation coefficient is constructed to realize the correlation calculation with adjacent nodes. Thus, applying a node embedding layer once on a node can be regarded as aggregating the features of its first-order neighbors and itself. After the second layer, each node also contains the information of the adjacent nodes of the adjacent nodes. By stacking L layers continuously, more and more information is involved in the operation, and the learned receptive field becomes larger. Information from distant neighboring nodes is gradually collected, and the final features of a node will be obtained.

[0061] (4) The present invention is based on the global embedding layer in the prediction model of the dual-path graph neural network. First, based on the attention mechanism, the node embeddings are weighted averaged to obtain the initial global embedding vector. In order to give more weight to more important nodes, the embedding feature vector of each node is taken as the inner product with the initial global embedding vector as the attention weight of each node, so that the node embedding feature that is most similar to the initial global embedding vector will obtain a higher weight, which can ensure that nodes with more structural importance have a greater impact on the embedding of the entire graph than other ordinary nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic flow chart of a method for predicting postoperative complications based on a dual-path graph neural network provided by an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of constructing a vascular network according to an embodiment of the present invention. In the figure, (a) shows the circle test result, (b) shows the intensity curve result, (c) shows the vascular network construction result before bypassing, and (d) shows the vascular network construction result after bypassing.

[0064] Figure 3 An overall schematic diagram of a prediction model based on a dual-path graph neural network provided by an embodiment of the present invention;

[0065] Figure 4 This is a ROC curve diagram for hyperparameter optimization provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0067] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0068] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0069] Example 1

[0070] like Figure 1 As shown, this embodiment provides a method for predicting postoperative complications based on a dual-path graph neural network, comprising the following steps:

[0071] S1: Collect ICG video images before and after cerebral vascular bypass surgery and obtain the patient's clinical characteristics data;

[0072] S2: Preprocess the collected ICG video images and use image segmentation algorithms to extract the vascular network and generate the corresponding vascular map structure before and after bypass surgery;

[0073] S3: Extract vascular features, blood flow dynamic features, and clinical features from the collected ICG video images and clinical feature data to construct a multimodal high-dimensional feature set;

[0074] S4: Using the multimodal high-dimensional feature set and the corresponding vascular graph structure before and after bypass surgery as input, a pre-trained prediction model based on a dual-path graph neural network is used to generate a complication prediction score;

[0075] The prediction model based on the dual-path graph neural network includes a node embedding layer, a global embedding layer, a cross-graph interaction layer, and an aggregate prediction layer. The node embedding layer is used to calculate the node representation matrix based on the multimodal high-dimensional feature set and the vascular graph structure. The global embedding layer is used to learn the association relationship between nodes based on the node representation matrix based on the attention mechanism to obtain the global embedding vector before and after the bypass. The cross-graph interaction layer is used to calculate the relationship between the global embedding vectors before and after the bypass and output the complication prediction score through the aggregate prediction layer:

[0076] S5: Optimizing the structure of a dual-path graph neural network (i.e., a prediction model based on a dual-path graph neural network)

[0077] To build a dual-path graph neural network model suitable for complication prediction, this example confirms the model's hyperparameters one by one through comparative experiments to find the optimal network architecture. Experiments are conducted in three aspects: node update mechanism, number of graph convolution layers, and whether the global embedding method is weighted;

[0078] S6: Model training and validation

[0079] In this embodiment, the model was trained and tested using a five-fold cross-validation method, and the model performance was evaluated using indicators such as classification accuracy (Acc), area under the curve (AUC), sensitivity (Sens), and specificity (Spec). The stability and effectiveness of the model were verified through experiments.

[0080] In step S1, the data collection process is specifically as follows:

[0081] A1. Patient screening. According to the "Chinese Expert Consensus on the Diagnosis and Treatment of Moyamoya Disease and Moyamoya Syndrome (2017)", patients who met the diagnostic criteria for moyamoya disease and were confirmed by DSA were screened. Patients who underwent superficial temporal artery-middle cerebral artery bypass grafting and had an ICG video recorder using the built-in FLOW800 software were also selected.

[0082] A2. Video data acquisition before and after bypass surgery. During surgery, ICG fluorescence angiography videos were collected using a microscope equipped with a near-infrared light emitter and fluorescence acquisition system. The ICG contrast agent was diluted with 20 ml of saline and injected into the patient's peripheral vein. The microscope was switched to fluorescence mode, capturing vascular fluorescence signals in real time, and recording approximately one minute of video. Video acquisition parameters were set to a frame rate of 25 frames per second and an image resolution of 720 × 576 pixels.

[0083] A3. Clinical characteristics collection. Basic clinical information of each patient was recorded, including gender, age, lesion type (hemorrhagic or ischemic), surgical side, smoking history, alcohol consumption history, and National Institutes of Health Stroke Scale (NIHSS) score, which ranges from 0 to 22.

[0084] A4. Video data quality screening. To ensure the quality and reliability of ICG fluorescence video data, the study conducted strict screening of angiography videos before and after the bypass. First, check whether each video fully records the entire process from the injection of the fluorescent agent to the end of the reaction, including the initial stage, peak stage and end stage of blood perfusion. Secondly, strict requirements are placed on the quality of the video image to exclude the presence of obstructions by surgical instruments, hands or gauze, as well as blurred images due to improper focus of the microscope or improper exposure settings. Each frame in the video must be able to clearly reflect the dynamic changes in the blood perfusion path, especially the change process from dark to bright in the recipient blood vessels. Angiography video data that does not meet the above quality standards will be eliminated to ensure that the data entering subsequent analysis is highly consistent and scientific.

[0085] A5. Data labeling. After completing the video screening, the patient data that meets the quality requirements are classified and labeled for postoperative complications. The specific method is: the doctor observes the patient in the perioperative period after the operation, and follows up the patient within three months after discharge, recording the patient's recovery and feedback on the physical condition in detail. By combining the patient's postoperative records and follow-up data, the doctor strictly classifies whether the patient has complications after the operation. Among them, "symptomatic" patients show obvious complications such as cerebral ischemia, cerebral hemorrhage or cerebral hyperperfusion syndrome after surgery; while "asymptomatic" patients did not report any related symptoms or discomfort after the operation. During the classification process, special attention is paid to the combination of the patient's subjective feedback and the doctor's objective diagnosis to ensure the accuracy and reliability of the labeling results. All classification labels are recorded in a standardized format and associated with the corresponding ICG video and clinical feature data, providing clear labels for subsequent model training.

[0086] A6. Dataset construction. After completing the video screening and classification annotation, all data that meet the standards will be systematically organized to construct a standardized data set. The data set includes ICG fluorescence videos before and after the bypass surgery, basic clinical feature data of the patient, and classification labels for postoperative complications. Among them, the ICG video dynamically records the blood perfusion process of the recipient blood vessels, providing a high-quality data basis for extracting imaging features; the clinical feature data covers the patient's gender, age, surgical side, lesion type, smoking history, drinking history, and National Institutes of Health Stroke Scale (NIHSS) score and other multidimensional information, providing additional semantic information support for the model. All data are organized and saved in a unified format to ensure the correlation and consistency between the data, laying a solid foundation for subsequent multimodal feature extraction, model training and experimental verification.

[0087] In step S2, Figure 2 As shown in Figure 2, the generation process of the vascular graph structure is as follows:

[0088] B1. Vascular Segmentation. The MedSAM network is used to segment the last frame of the ICG fluorescence video before and after bypass surgery to extract the vascular region. The vascular region is separated from the background, generating a binary image containing vascular information. The foreground region represents the vascular structure, while the background region represents other irrelevant information. Automated methods are used during the segmentation process to ensure the integrity of the vascular morphology and eliminate noise interference. The segmented image is manually verified to ensure the accuracy of the segmentation results.

[0089] B2. Vessel Skeleton Extraction. The segmented binary vascular image is thinned to generate a vascular skeleton. This thinning process uses the Guo thinning algorithm to ensure that all vessels have a uniform diameter of 1 pixel and maintain 8-connectivity of foreground pixels, preserving the topological structure of the vascular network. This thinning process extracts the centerline of each vessel, laying the foundation for subsequent node and edge detection.

[0090] B3. Vascular network node detection. The intersection locations of blood vessels are detected on the skeleton diagram and defined as nodes of the network diagram. Node detection uses a ridge-based multi-circle test method. Multiple circles of different radii are placed in each candidate intersection area. The pixel characteristics (including intensity value and number of blood vessels) on the circle are calculated to determine whether the point is an intersection. The criteria for intersection detection include the following conditions: the pixel intensity on the circle is similar to the center of the circle; the pixel distribution on the circle meets certain geometric rules; and there are three or more blood vessel branches on the circle. Through multiple tests, the locations of all nodes in the network diagram are finally determined.

[0091] B4. Constructing vascular network edges. In the skeleton graph, vascular segments are connected according to node positions. These segments are defined as paths between nodes, i.e., edges in the network graph. Using a breadth-first search algorithm, a path of white pixels in the skeleton graph is searched from one node to another, while recording the path length. For each edge, the vessel diameter and brightness information are also calculated to supplement the edge attributes.

[0092] B5. Vascular network graph structure generation. Based on the detected node and edge information, a vascular network graph before and after bypass surgery is constructed. The network graph is represented as an adjacency matrix, where each node contains location information, brightness value, and other attributes, and edges contain length, diameter, and corresponding blood flow parameters. The network graph structures before and after bypass surgery are stored separately to provide input data for subsequent graph neural network analysis.

[0093] In step S3, the feature extraction process is specifically as follows:

[0094] C1. Vascular feature extraction. Vascular features are mainly used to describe the geometric morphology, connectivity characteristics, and structural scale of the vascular network, specifically including the following:

[0095] Total vascular length: The total length of all vascular segments in the vascular network is calculated to reflect the scale of the entire vascular network.

[0096] Total vascular area: The total area of ​​the vascular territory was calculated to quantify the spatial extent occupied by the vessels during preoperative and postoperative perfusion.

[0097] Number of connected vessels: Count the number of independent connected regions in the vascular network to evaluate the connectivity of the vessels.

[0098] Blood vessel diameter characteristics: Records the maximum diameter, minimum diameter, and average diameter of blood vessels to analyze the distribution characteristics of blood vessel thickness and flow potential.

[0099] Vascular width features: The maximum width, minimum width, and average width of the blood vessels are extracted and combined with the diameter features to further reflect the morphological characteristics and perfusion capacity of the blood vessels.

[0100] Segmented statistics: The video is divided into five segments evenly according to time. All the above vascular features are counted for each segment to capture the dynamic changes in blood perfusion at different time periods. The nine features counted for each video segment total 45 dimensions.

[0101] C2. Blood flow dynamic feature extraction

[0102] This type of feature is used to capture the dynamic changes in fluorescence brightness and blood flow velocity during blood perfusion, specifically including the following:

[0103] Optical flow characteristics: Calculate the maximum optical flow and its direction (x and y) of blood vessels in each frame of the video to describe the direction and rate of blood flow. Optical flow characteristics can effectively reflect the dynamic characteristics of blood flow in blood vessels before and after surgery.

[0104] Fluorescence Brightness Characteristics: Records the time and brightness of the vessel's maximum fluorescence brightness, and calculates the ratio of the time required for the fluorescence brightness to reach 50% of its maximum value to the total time. These characteristics are used to analyze the speed and efficiency of blood flow to the target location, as well as the temporal dynamics of perfusion.

[0105] Segment statistics: The video is divided into 5 segments, and 7 types of optical flow features are counted in each segment, totaling 35 dimensions.

[0106] C3. Integration of clinical features

[0107] The patients' clinical characteristic data were included in the feature set, including gender, age, surgical side, lesion type (ischemic or hemorrhagic), smoking history, drinking history and NIHSS score, a total of 7 features, to supplement the semantic information of image features.

[0108] C4. Feature Summary

[0109] By integrating all extracted features, including vascular features (45 dimensions), blood flow dynamic features (35 dimensions), and clinical features (7 dimensions), the data for each patient ultimately contains 87 dimensions of multimodal features, providing input for subsequent model training and experimental analysis.

[0110] The prediction process in step S4 is specifically as follows:

[0111] Because it is necessary to infer complications by comparing the blood perfusion changes of ICG videos before and after bypass surgery, the present invention proposes a new complication prediction network based on graph change comparison (Compare Graph Changing Network, ComGCN), such as Figure 3 As shown in Figure 2, ComGCN effectively learns variations in node features. ComGCN consists of four layers: 1) node embedding layer; 2) global embedding layer; 3) cross-graph interaction layer; and 4) aggregation and prediction layer. The convolutional neural network layer transforms the correlation calculation problem into a pattern recognition problem, providing multi-scale features to the fully connected network to obtain the final predicted complication score.

[0112] The following is a detailed description of each part.

[0113] D1. Node Embedding Layer

[0114] The input of the node embedding layer includes node features based on a multimodal high-dimensional feature set and an adjacency matrix based on the vascular graph structure;

[0115] Then a weight matrix with shared parameters is trained for all nodes to perform a shared self-attention mechanism for each node and calculate the attention correlation coefficient;

[0116] Based on the normalized attention correlation coefficient, the final representation vector of each node is calculated.

[0117] The specific process includes:

[0118] Each graph G = (V, E) is represented as a set of N nodes V and edges E, where each edge e∈W is a set of two vertices of V. For each node v in the list i ∈V, all have D-dimensional features, and these original features are represented as an N*D-dimensional matrix X, where each edge (v i ,v j )∈E is associated with a feature vector, i.e., each edge is a set of two vertices of V. If a node or an edge lacks relevant features, we set the corresponding vector value to 1. The input of GAT is a set of node features, and an adjacency matrix A representing the graph structure, if the node v i and v j are connected, then A ij =1, otherwise 0. Then, the graph convolution layer calculates a new set of node features based on the input features and graph structure.

[0119] In order to obtain sufficient expressive power, it is necessary to transform the input features into higher-level features, that is, to learn at least one linear transformation based on the input features to obtain the output features. Therefore, we train a weight matrix with shared parameters for all nodes. This weight matrix describes the relationship between D input features and D′ output features. The weight matrix is ​​denoted as Then a shared self-attention mechanism a is performed on each node: And calculate the attention correlation coefficient

[0120]

[0121] e ij Represents the node v j For node v i The importance of. Let N i is node v i In order to make the correlation coefficient easier to calculate and easier to compare between different neighboring nodes, the softmax function is used to normalize the coefficient.

[0122]

[0123] In the first convolutional layer, these neighboring nodes are nodes v i The first-order neighbors (including v i ). The attention mechanism a is a single-layer feedforward neural network.

[0124]

[0125] in· T Indicates transposition, || is a connection symbol, is the parameter of the feedforward neural network a, and LeakyReLU is a nonlinear activation function. At this time, the final representation vector of the node can be obtained

[0126]

[0127] H (l) is the feature matrix of all nodes of the l-th layer GAT. For the input layer, H (0) =X. Intuitively, applying a GAT layer on a node can be regarded as aggregating the features of its first-order neighbors and itself. After the second layer, each node also contains the information of the neighboring nodes of its neighboring nodes. By stacking L layers continuously, more and more information is involved in the operation, and the learned receptive field becomes larger. Information from distant neighboring nodes is gradually collected, and the final feature of a node will be obtained.

[0128] D2. Graph-level global embedding layer

[0129] The global embedding layer first averages the representation matrix of each node and transforms it through a nonlinear activation function to obtain the initial global embedding vector;

[0130] Take the inner product of the embedded feature vector of each node and the initial global embedding vector as the attention weight of each node to obtain the global embedding vector;

[0131] The specific process is:

[0132] After D1 obtains the node representation matrix H, the next step is to effectively combine them to generate a global embedding C for the entire graph. This uses an attention-based mechanism to perform a weighted average of node embeddings. However, the weighting of more important nodes is learned based on a relevance metric. Giving more attention to certain nodes based on relevance ensures that nodes with greater structural importance have a greater impact on the overall graph embedding than other nodes.

[0133] To achieve this, we first average the embedding matrices h of each node i , and transform it through the nonlinear activation function ReLU to obtain the initial global embedding vector

[0134]

[0135] in is the weight matrix. By learning the weight matrix, C collects the structural attribute information of the entire graph. Nodes similar to the global background should receive higher attention weights. Therefore, the present invention takes each node v i The embedded feature vector h i Do the inner product with C as the attention weight a of each node i , the node embedding features that are most similar to C will get a higher weight. Once the attention weight a of each node is calculated i , we can get the final attention-based whole graph embedding That is, the weighted sum of node embeddings, where σ(·) is the sigmoid function, ensuring that the attention weight value is in the range (0,1),

[0136]

[0137] D3. Cross-graph interaction layer

[0138] The cross-graph interaction layer calculates the relationship between the global embedding vectors before and after bridging through a neural tensor network.

[0139] The specific process includes:

[0140] After D2 obtains the global embedding G_all of the entire image, it is necessary to effectively compare the embeddings of the two images to estimate the changes in the postoperative images. A simple way to model their relationship is to find the inner product of the two, but this method often leads to insufficient or weak interaction between the two. For this reason, the present invention uses Neural Tensor Networks (NTN) to explore the relationship between a pair of images to solve the deep relationship between the blood perfusion images before and after the bypass. The advantage of NTN is that it can effectively compare two embedding vectors in multiple dimensions. Given the global embedding vectors of the two images before and after the operation are G1_all and G2_all, NTN calculates the relationship between the two embeddings:

[0141]

[0142] Where σ is a nonlinear activation function, is a weight tensor, with K as the weight tensor. K is a hyperparameter and is the number of interaction (correlation) scores produced by the model for each pair of graphs. is the weight matrix of the standard neural network, [·] is the inner product of the two images before and after surgery, It is a bias. Assuming that the number of nodes before and after surgery is N1 and N2 respectively, to prevent the different number of nodes from affecting matrix operations, the present invention fills the node embedding matrix of the graph with fewer nodes with all zero rows, so that both graphs contain max(N1, N2) nodes.

[0143] D4. Aggregation and prediction layer

[0144] The cross-graph interaction layer obtains a correlation score matrix based on the relationship between the global embedding vectors before and after bridging;

[0145] The aggregate prediction layer concatenates the correlation score matrix and the global embedding vectors before and after bridging, and uses multiple fully connected neural networks to gradually reduce the dimension of the variability score vector to obtain the complication prediction score.

[0146] The specific process is:

[0147] The goal of network learning is to generate a prediction score y=s(G1,G2) for whether or not complications occur. When performing complication classification tasks, y∈{0,1} is a binary label, 0 means no complications, and 1 means complications. After obtaining a list of correlation scores through the NTN network, the correlation score matrix and the global graph before and after surgery are embedded into the matrix. A total of three matrices are concatenated, and three fully connected neural networks are used to gradually reduce the dimension of the variability score vector. The fully connected layer is a nonlinear neural network that uses multiple layers of weighted neurons to map the input matrix of dimension m to the desired output dimension n. From the last layer of the fully connected neural network, a probability score is calculated and compared with the true label using the mean square error loss function.

[0148]

[0149] Among them, M is the set of training graph pairs, is the true label of whether or not complications occur.

[0150] The process of performing dual-path graph neural network structure optimization in step S5 specifically includes:

[0151] In order to build a dual-path graph neural network model suitable for complication prediction, the present invention confirmed the model's hyperparameters one by one through comparative experiments to find the optimal network architecture. Experiments were conducted from three aspects: node update mechanism, number of graph convolution layers, and whether the global embedding method is weighted.

[0152] E1. Optimization of node information update mechanism

[0153] To ensure that graph neural networks can effectively capture the relational characteristics between nodes, different node information update mechanisms have been explored, including graph convolutional networks (GCNs), graph sampling and aggregation networks (GraphSAGEs), and graph attention networks (Graph Attention Networks (GATs). GCNs aggregate neighboring node information to a target node through fixed-weight convolutions, but are unable to distinguish the importance of different neighboring nodes. GraphSAGEs improve computational efficiency by sampling a fixed number of neighboring nodes for aggregation, but this can lead to redundant or insufficient information. GATs utilize a self-attention mechanism to assign different weights to neighboring nodes, allowing for more flexible capture of relational characteristics between key nodes.

[0154] Model A: GCN+2 layers+unweighted embedding

[0155] Model B: GraphSAGE+2 layers+unweighted embedding

[0156] Model C: GAT+2 layers+unweighted embedding

[0157] Table 1 shows that model C's GAT network performs best, achieving high AUC and Acc, while also significantly improving the SVM model's low Sens value caused by class imbalance. Model A's GCN performs the worst for two main reasons: First, GCN fuses the features of each node's neighbors by summing fixed weights, failing to distinguish the contributions of different nodes; second, it uses full-graph convolutional fusion for gradient updates across the entire graph. This node update method is unsuitable for data with numerous vascular nodes and large network graphs, such as ICG images. GAT addresses the first issue by assigning a learnable weight to each edge, allowing the model to independently learn the importance of different neighboring nodes. This allows the model to adaptively adjust parameters based on the task when fusing features. GraphSAGE addresses the second issue, but GraphSAGE can only update a fixed number of neighboring nodes. However, the number of edges connecting each node in cerebral vascular networks varies greatly, making it possible for the number of neighboring nodes to be less than the fixed number. Consequently, the sampling with replacement approach employed may result in redundant update information. Furthermore, there's a chance that the number of adjacent nodes exceeds the fixed number, preventing the network from accessing all neighboring nodes and resulting in inadequate information updates. In summary, GraphSAGE's fixed node count update method lacks flexibility, making it difficult to select a suitable fixed value for all nodes. Because GAT outperformed the other two models on all four metrics, it was selected as the base model for subsequent experiments.

[0158] E2. Optimizing the number of graph convolution layers

[0159] After confirming GAT as the node update mechanism, we further studied the impact of the number of graph convolutional layers on model performance. We compared the performance of 1-layer, 2-layer, and 3-layer GATs and used evaluation metrics to find the optimal number of layers.

[0160] Model D: GAT+1 layer+unweighted embedding

[0161] Model E: GAT+3 layers+unweighted embedding

[0162] The classification results of model D and model E are shown in the fourth and fifth rows of Table 1 respectively. Combined with model C, it can be seen that GAT with more layers (i.e. 2 layers, 3 layers) provides better and more stable performance, and the AUC is significantly higher. However, model D with the least layers has a small number of network layers and parameters, and does not learn the corresponding representation of each node very well, so the model is prone to underfitting. The fact that this model obtains the lowest Sens among all models shows that underfitting causes the model to have a high classification error rate for a small number of positive samples. Further comparison of C and E shows that the performance of C is still better than E. The main reason may be that model E is a three-layer update mechanism, and too many parameters can easily lead to overfitting of the model. Therefore, for the GAT network, the two-layer parameter structure used by model C is the best. The present invention uses two-layer GAT as the node information update mechanism of the model to discuss the next parameter.

[0163] E3. Global embedding optimization

[0164] After deciding to use a two-layer GAT network as the node update mechanism, experiments were conducted on global feature embedding for the entire graph. The experiments compared the performance of a weighted approach (Model F: GAT + two layers + weighted embedding) with an unweighted approach (Model C). The performance statistics for Model F are shown in the last row of Table 1. Comparing the performance of Models C and F, we see that Model F's Sens (Sens) score improved by 13 points compared to Model C. This significant improvement in the accuracy of detecting patients with disease is clinically significant, as the high accuracy of this metric can provide doctors with timely and effective feedback on patients' risk of disease. Models F differ from Model C in whether they use a weighted global embedding approach. A weighted approach uses an attention mechanism to aggregate nodes, selecting important nodes from the entire graph. Since pre- and post-ICG perfusion differences are primarily concentrated near the recipient vessel, the weighted mechanism helps the network focus more closely on the recipient vessel and its surrounding vascular nodes. Consequently, Model F significantly improved performance, achieving a classification accuracy of 0.85, significantly enhancing the effectiveness of clinical judgment. Therefore, the present invention uses model F, a weighted global embedding model with 2-layer GAT, as the final prediction model for complication classification.

[0165] Table 1 Hyperparameter optimization evaluation index values

[0166] Acc AUC Sens Spec Model A 0.7819 0.7798 0.6571 0.8462 Model B 0.8083 0.7730 0.7143 0.8621 Model C 0.8172 0.7954 0.7357 0.8889 Model D 0.7567 0.6838 0.6250 0.8333 Model E 0.7423 0.7572 0.6600 0.8200 Model F 0.8513 0.8396 0.8600 0.8679

[0167] In step S6, the model training and verification process specifically includes:

[0168] The model was trained and tested using the five-fold cross-validation method. The model performance was evaluated using indicators such as classification accuracy (Acc), area under the curve (AUC), sensitivity (Sens), and specificity (Spec). The stability and effectiveness of the model were verified through experiments.

[0169] F1. Model training

[0170] The preprocessed vascular network data and its extracted multimodal features were partitioned according to the principle of five-fold cross-validation. Pre- and post-surgery data from 101 patients were randomly divided into five groups: four for training and one for testing, ensuring that there was no overlap between the test and training samples. An optimized two-way graph neural network model (GAT + weighted global embedding) was trained. The pre- and post-surgery vascular network graphs were used as dual-way inputs, and model parameters were optimized layer by layer through the node embedding layer, the global embedding layer, and the cross-graph interaction layer. The model output was converted into a binary probability score (complication / no complication) using a softmax activation function. A cross-entropy loss function was used to minimize the error between the predicted value and the true label. The Adam optimizer was used with a learning rate of 0.5e-3 for 300 epochs to ensure model convergence on the training set. The performance of the trained model was verified using the cross-validation test set.

[0171] F2. Model testing and performance evaluation

[0172] Select classification accuracy (Accuracy, Acc), area under the curve (AUC), sensitivity (Sensitivity, Sens) and specificity (Spec) as evaluation indicators. And draw the Receiver Operating Characteristic Curve (ROC) curve, such as Figure 4 As shown in the figure, AUC measures the model's ability to distinguish between positive and negative samples, Sens evaluates the recognition ability of positive samples (cases with complications), and Spec evaluates the detection ability of negative samples (cases without complications). The evaluation results for each fold are recorded and the average of all folds is calculated to ensure the stability and reliability of the performance evaluation.

[0173] This embodiment also provides a postoperative complication prediction device based on a dual-path graph neural network, including a memory and a processor, the memory storing a computer program, and the processor calling the computer program to execute the steps of the above postoperative complication prediction method based on a dual-path graph neural network.

[0174] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A postoperative complication prediction method based on a dual-path graph neural network, characterized in that: The following steps are involved: Collect ICG video images before and after cerebral vascular bypass surgery and obtain the patient's clinical characteristics data; The collected ICG video images are pre-processed and the vascular network is extracted using an image segmentation algorithm to generate the corresponding vascular map structure before and after bypass surgery; Extract vascular features, blood flow dynamic features, and clinical features from the collected ICG video images and clinical feature data to construct a multimodal high-dimensional feature set; Using a multimodal high-dimensional feature set and the corresponding vascular graph structures before and after bypass surgery as input, a pre-trained prediction model based on a dual-path graph neural network is used to generate a complication prediction score. Among them, the prediction model based on the dual-path graph neural network includes a node embedding layer, a global embedding layer, a cross-graph interaction layer and an aggregate prediction layer. The node embedding layer is used to calculate the node representation matrix based on the multimodal high-dimensional feature set and the vascular graph structure. The global embedding layer is used to learn the association relationship between nodes based on the node representation matrix based on the attention mechanism to obtain the global embedding vector before and after the bypass; the cross-graph interaction layer is used to calculate the relationship between the global embedding vectors before and after the bypass, and output the complication prediction score through the aggregate prediction layer.

2. A postoperative complication prediction method based on a dual-path graph neural network according to claim 1, characterized in that: The input of the node embedding layer includes node features based on a multimodal high-dimensional feature set and an adjacency matrix based on the vascular graph structure, where N is the total number of nodes, D is the characteristic dimension corresponding to the node, and the adjacency matrix is ​​used to indicate whether two nodes are connected; Then a weight matrix with shared parameters is trained for all nodes to perform a shared self-attention mechanism for each node and calculate the attention correlation coefficient; The expression of the attention correlation coefficient is: Where, e ij is the attention correlation coefficient between node i and node j, W node is the weight matrix, is the feature of node i, is the feature of node j; The attention correlation coefficient is normalized, and the corresponding calculation expression is: Where, α ij is the normalized attention correlation coefficient, T Indicates transposition, || is a connection symbol, is the parameter of the feedforward neural network a, LeakyReLU is a nonlinear activation function, N i For node v i An adjacent node of Based on the normalized attention correlation coefficient, the final representation vector of each node is calculated, and the corresponding calculation expression is: Where, is the final representation vector of node i, and σ is the nonlinear activation function.

3. The method for predicting postoperative complications based on a dual-path graph neural network according to claim 1, characterized in that: The global embedding layer first averages the representation matrix of each node and transforms it through a nonlinear activation function to obtain an initial global embedding vector. The calculation expression of the initial global embedding vector is: Where C is the initial global embedding vector, ReLU is the nonlinear activation function, and W graph is the weight matrix, N is the total number of nodes, h i is the feature of node i; Take the inner product of the embedded feature vector of each node and the initial global embedding vector as the attention weight of each node to obtain the global embedding vector; The calculation expression of the attention weight is: Where a i is the attention weight of node i, σ is the nonlinear activation function; The calculation expression of the global embedding vector is: Where G_all is the global embedding vector.

4. The method for predicting postoperative complications based on a dual-path graph neural network according to claim 1, characterized in that: The cross-graph interaction layer calculates the relationship between the global embedding vectors before and after bridging through a neural tensor network. The corresponding expression is: Where, is the relationship between the two global embedding vectors G1 and G2, σ is a nonlinear activation function, is a weight tensor, there are K, K is a hyperparameter, is the number of interaction scores generated by the model for each pair of graphs, is the weight matrix of the standard neural network, [·] is the inner product of the two images before and after surgery, is the bias.

5. The method for predicting postoperative complications based on a dual-path graph neural network according to claim 1, characterized in that: The cross-graph interaction layer obtains a correlation score matrix based on the relationship between the global embedding vectors before and after bridging; The aggregate prediction layer concatenates the correlation score matrix and the global embedding vectors before and after bridging, and uses multiple fully connected neural networks to gradually reduce the dimension of the variability score vector to obtain a complication prediction score.

6. The method for predicting postoperative complications based on a dual-path graph neural network according to claim 1, characterized in that: The ICG video image acquisition process is specifically as follows: During cerebral vascular bypass surgery, ICG fluorescence angiography videos of patients before and after bypass surgery are collected through a microscope that integrates a near-infrared light emission device and a fluorescence acquisition system.

7. The method for predicting postoperative complications based on a dual-path graph neural network according to claim 1, characterized in that: The generation process of the vascular map structure is specifically as follows: For the last frame of the ICG video before and after bypass surgery, blood vessel segmentation is performed to extract the blood vessel area and generate a binary image containing blood vessel information. Perform thinning processing on the binary image containing vascular information to generate a vascular skeleton map; Detecting the intersection positions of blood vessels on the blood vessel skeleton graph and defining them as nodes of the blood vessel network; Connecting vascular segments according to the positions of nodes on the vascular skeleton graph, and defining vascular segments as paths between nodes, i.e., edges of the vascular network; Based on the detected node and edge information, the vascular graph structure before and after bypass is constructed. The vascular graph structure is represented in the form of an adjacency matrix. Each node contains position information and brightness value, and each edge contains length, diameter and corresponding blood flow parameters.

8. The method for predicting postoperative complications based on a dual-path graph neural network according to claim 1, characterized in that: The blood vessel characteristics include total blood vessel length, total blood vessel area, number of connected blood vessels, blood vessel diameter characteristics and blood vessel width characteristics; The blood flow dynamic characteristics include optical flow characteristics and fluorescence brightness characteristics; In the process of extracting the vascular features and blood flow dynamic features, the collected ICG video images are divided into multiple segments according to time, and the vascular features and blood flow dynamic features of each ICG video image are counted respectively, and the sum of them is a multi-dimensional feature; The clinical characteristics include gender, age, surgical side, lesion type, smoking history, drinking history and NIHSS score.

9. The method for predicting postoperative complications based on a dual-path graph neural network according to claim 1, characterized in that: During the training process of the prediction model based on the dual-path graph neural network, the patient's postoperative complications are classified and labeled based on the collected ICG video images, and corresponding classification labels are constructed; a standardized data set is constructed for the selected ICG video images and the corresponding classification labeling results for the training of the prediction model of the graph neural network.

10. A postoperative complication prediction device based on a dual-path graph neural network, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of any one of the methods according to claims 1 to 9.

Images