A multi-modal neural image brain disease diagnosis system based on a pulse neural network

Abstract

A multi-modal neural image brain disease diagnosis system based on pulse neural network belongs to the technical field of medical information processing and specifically relates to the cross technical field of artificial intelligence and neural image analysis. It solves the core defects that the traditional brain disease diagnosis method cannot analyze the brain nonlinear dynamics mechanism, the multi-modal data fusion is insufficient, the cross-center batch effect is obvious, the feature reliability is not quantitatively evaluated, and the explainability is poor. The system comprises: a unit for multi-modal neural image data preprocessing, a unit for extracting brain dynamics features based on Koopman EvoNet, a unit for constructing a multi-modal graph neural network to fuse features, a unit for eliminating site batch effects through neural heuristic domain adaptation, a unit for jointly training the model through multiple loss functions, and a unit for realizing brain disease diagnosis through attention graph convolution classification.

Description

Technical Field

[0001] This invention belongs to the field of medical information processing technology, specifically involving the interdisciplinary field of artificial intelligence and neuroimaging analysis. Background Technology

[0002] Brain activity exhibits highly nonlinear and complex dynamic characteristics. Traditional methods for diagnosing brain diseases are mostly based on static neuroimaging feature analysis or by fitting brain region time series using traditional time-series modeling methods, which have the following core limitations: 1. It cannot reveal the intrinsic laws from the perspective of brain dynamics mechanisms; it can only achieve data-level fitting and is difficult to extract biomarkers of dynamics with biological significance. 2. The fusion effect of multimodal neuroimaging data (brain connectivity matrix, ROI time series, phenotypic data, etc.) is poor, and the guiding role of phenotypic information in image feature analysis is not fully utilized; 3. In cross-center neuroimaging analysis, there is a batch effect caused by scanning sites. Traditional methods have difficulty learning site-invariant feature representations, resulting in weak model generalization ability. 4. There is a lack of quantitative assessment methods for the reliability of brain dynamics characteristics, and low-quality samples can affect the accuracy of diagnostic models; 5. The feature representations of traditional neural networks lack sparsity and interpretability, making it difficult to match the physiological characteristics of brain neural activity.

[0003] The aforementioned shortcomings make it difficult for traditional methods to meet clinical needs in terms of accuracy and interpretability in the diagnosis of brain diseases, and they are unable to effectively capture abnormal dynamic interactions and abnormal local coordination patterns between brain regions. Summary of the Invention

[0004] To address the core shortcomings of traditional brain disease diagnostic methods, such as the inability to analyze nonlinear brain dynamics mechanisms, insufficient multimodal data fusion, significant cross-center batch effects, lack of quantitative assessment of feature reliability, and poor interpretability, this invention aims to provide a multimodal neuroimaging brain disease diagnostic system based on spiking neural networks. This system achieves intelligent diagnosis of brain diseases with high accuracy, high interpretability, and strong generalization. Simultaneously, it extracts biologically significant brain dynamics biomarkers, providing technical support for research on the pathological mechanisms of brain diseases and meeting the practical needs of clinical diagnosis and research.

[0005] The system includes: Units for multimodal neuroimaging data preprocessing: used to construct data containing different sites Brain connectivity matrix of 10 subjects , phenotypic data ROI time series and clinical labels dataset , Indicates the first 10 subjects; The unit used for extracting brain dynamics features based on KoopmanEvoNet: taking ROI time series as input, it extracts dynamic features through sliding time window, Koopman observation network, Koopman operator layer, pulse encoder, and spatiotemporal integration and decoding, and completes dynamics quality assessment and association graph construction; The unit used to construct multimodal graph neural network fusion features: Based on the association graph obtained by the unit used to extract brain dynamics features based on KoopmanEvoNet, a phenotypic guidance graph is constructed. Adaptive fusion of image and phenotypic features is achieved through dual-channel graph neural network and cross-modal attention fusion to obtain the fused graph structure and enhanced features. Units for eliminating site batch effects through neurally inspired domain adaptation: Introducing a gradient inversion domain adversarial mechanism to eliminate batch effects generated by different scan sites; Unit for jointly training a model using multiple loss functions: setting an overall training loss function and training each neural network used by the system; A unit for diagnosing brain diseases through attention graph convolution: information is propagated on the graph through attention graph convolution, and a fully connected classifier is used for the final diagnostic prediction.

[0006] Furthermore, the sliding time window is used to extract statistical features, given the ROI time series. ,in and The meaning is brain regions and At each time point, a sliding window strategy is used to extract dynamic statistical features: Mean:

[0007] Standard deviation:

[0008] Skewness:

[0009] in, Indicates window size. Indicates the step size. The index represents the sliding time window, indicating the nth time window. A sliding time window, Indicates the first The first subject The brain region, in the _ ... Neuroimaging signal values ​​at each time point; Calculate the mean of each window for the time series data of each brain region. Standard deviation skewness The window feature is obtained by concatenating the three statistics. Stack all window features to form the input tensor , This represents the number of windows.

[0010] Furthermore, the Koopman observation network is used to map the statistical features of the high-dimensional ROI to the latent Koopman space to obtain the initial latent state. ; The Koopman operator layer is used to perform Koopman evolution along the time window dimension, capturing the dynamic interactions between brain regions over time, and obtaining the first... A sliding window Koopman latent state at any given moment .

[0011] Furthermore, the pulse encoder is used to realize sparse feature representation, specifically by defining the membrane potential. , It is a learnable firing threshold vector, and the pulse firing probability is calculated using the sigmoid function. : During training, a pass-through estimator is used for gradient backpropagation, and Bernoulli sampling is employed. Generate pulse Deterministic thresholds were used during the test. Furthermore, a refractory period mechanism was introduced to prevent excessive neuronal firing.

[0012] Furthermore, the spatiotemporal integration and decoding specifically involves performing spatiotemporal integration on the pulses of all time windows and time steps to obtain compact neural features. : ,in, For the number of evolution steps of the Koopman algorithm, a decoding network is used to... Recovering to interpretable dynamic characteristics .

[0013] Furthermore, the dynamic quality assessment specifically involves introducing a quantitative quality score that integrates information richness and system stability. : ,in, Let be the impulse value of the k-th window, the t-th step, and the j-th latent dimension. Learnable linear operators after applying spectral radius constraints ; The construction of the association graph specifically involves calculating the similarity between samples based on Koopman features and then constructing the association graph. : ,in It is the first The dynamic characteristics of each sample This is the sigmoid function.

[0014] Furthermore, the specific operations for the units used to construct the fusion features of the multimodal graph neural network are as follows: S1. Constructing a phenotypic guidance graph: A phenotypic guidance graph includes edges of three semantic types: Strengthening edges: established when two samples have similar phenotypic features and belong to the same diagnostic category; weakening edges: established when samples have similar phenotypic features but different diagnoses; stimulating edges: connecting samples in the training set and the validation / test set to promote information transfer; the final hybrid association graph is the weighted sum of the edge weights of each phenotypic dimension. S2. Graph Neural Network Feature Extraction: A dual-channel parallel structure is used, employing TransformerConv to construct an image network and a phenotypic network. The image network processes the functional connectivity features of the brain connectivity matrix, while the phenotypic network processes the phenotypic embeddings encoded by VAE. The importance of neighboring nodes is adaptively learned through an attention mechanism. S3. Cross-modal attention fusion: A cross-modal attention fusion mechanism is introduced. First, the shared representation is calculated. Then, the attention scores of each modality and the shared representation are calculated separately. Finally, the fusion weight is determined by the scores after softmax normalization.

[0015] Furthermore, the gradient inversion domain adversarial mechanism is specifically as follows: Construct a site discriminator The input is the learned feature representation, and the output is the site label. An identity mapping is applied during forward propagation, and the gradient is multiplied during backpropagation. Through this adversarial training, the feature learner is forced to generate representations that cannot be distinguished by the site discriminator, thereby eliminating domain bias.

[0016] Furthermore, the overall training loss function includes classification loss. Graph structure loss kinetic loss Domain Adaptation Loss .

[0017] Furthermore, the attention graph convolution performs spatial information aggregation on node features, employing a dual-channel parallel structure to process image features and phenotypic features separately. This includes graph convolution branches, gating quality filtering, and cross-modal attention fusion. The outputs of the image branch and the phenotypic branch are adaptively fused through cross-modal attention fusion to obtain a joint representation. Fully connected classifier pairs A fully connected classifier is used for classification and final diagnostic prediction.

[0018] The beneficial effects of the system described in this invention are as follows: 1. High diagnostic accuracy: By capturing dynamic interaction abnormalities and local coordination pattern abnormalities in brain regions and combining adaptive fusion of multimodal data, the accuracy of brain disease diagnosis is significantly improved compared with traditional static feature analysis methods. 2. Strong interpretability of features: The dynamic features extracted based on the Koopman operator theory directly correspond to the dynamic interaction patterns of brain regions. The feature representation of the spiking neural network conforms to the physiological characteristics of brain neural activity, providing interpretable biomarkers for the study of the pathological mechanisms of brain diseases. 3. Excellent generalization ability: By eliminating cross-center batch effects through site adversarial training, the model can maintain stable diagnostic performance on neuroimaging data from different scanning sites, making it suitable for clinical cross-center studies; 4. Good robustness: The introduction of quantification of dynamic quality score and gating quality screening effectively filters low-quality samples, reduces the impact of noisy data on diagnostic results, and improves the model's anti-interference ability; 5. High data utilization: It fully integrates information from multimodal neuroimaging data such as brain connectivity matrices, ROI time series, and phenotypic data, and designs phenotypic guidance maps to provide semantic guidance for image analysis, thus solving the problem of insufficient multimodal data fusion in traditional methods; 6. Excellent physiological fit: The design of the spiking neural network simulates the spiking and refractory period characteristics of brain neurons, making the feature learning process of the model more consistent with the actual neural activity of the brain and having stronger biological rationality. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the overall system workflow in an embodiment of the present invention. Figure 2 This is a flowchart of the KoopmanEvoNet network workflow in an embodiment of the present invention; Figure 3 This is a comparison chart of mass fraction analysis in an embodiment of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0021] Example 1 This embodiment provides a multimodal neuroimaging brain disease diagnostic system based on a spiking neural network, including: Units for multimodal neuroimaging data preprocessing: used to construct data containing different sites Brain connectivity matrix of 10 subjects , phenotypic data ROI time series and clinical labels dataset , Indicates the first 10 subjects; The unit used for extracting brain dynamics features based on KoopmanEvoNet (Koopman Evolutionary Network): Taking ROI time series as input, it extracts dynamic features through sliding time window, Koopman observation network, Koopman operator layer, pulse encoder, and spatiotemporal integration and decoding, and completes dynamics quality assessment and association graph construction; The unit used to construct multimodal graph neural network fusion features: Based on the association graph obtained by the unit used to extract brain dynamics features based on KoopmanEvoNet, a phenotypic guidance graph is constructed. Adaptive fusion of image and phenotypic features is achieved through dual-channel graph neural network and cross-modal attention fusion to obtain the fused graph structure and enhanced features. Units for eliminating site batch effects through neurally inspired domain adaptation: Introducing a gradient inversion domain adversarial mechanism to eliminate batch effects generated by different scan sites; Unit for jointly training a model using multiple loss functions: setting an overall training loss function and training each neural network used by the system; A unit for diagnosing brain diseases through attention graph convolution: information is propagated on the graph through attention graph convolution, and a fully connected classifier is used for the final diagnostic prediction.

[0022] The system in this embodiment is based on multimodal neuroimaging data. It extracts brain dynamics features through a Koopman-constrained spiking neural network, fuses image and phenotypic information through a multimodal graphical neural network, eliminates site batch effects through neural heuristic domain adaptation, and achieves final diagnostic prediction through attention map convolution.

[0023] Example 2 This embodiment further defines embodiment 1, by combining... Figure 1 , Figure 2 And written descriptions to provide a detailed explanation of the entire system's workflow.

[0024] Unit for preprocessing multimodal neuroimaging data: Given Multimodal data from individual subjects, including brain connectivity matrices , phenotypic data (Demographic information such as age, gender, IQ, and scanning sites), ROI time series ( brain regions in Blood oxygen level dependent (BOLD) signal at each time point, and clinical labels. (Health control / Clinical disease group), construct the training set The ROI time series was normalized, and the phenotypic data was filled with missing values ​​and standardized.

[0025] Units used for extracting brain dynamics features based on KoopmanEvoNet: KoopmanEvoNet architecture diagram as follows Figure 2 As shown.

[0026] Using the ROI time series as input, dynamic features are extracted through a sliding time window, a Koopman observation network, a Koopman operator layer, a pulse encoder, spatiotemporal integration, and decoding. Dynamic quality assessment and correlation graph construction are then performed. The KoopmanEvoNet network consists of four core components: a Koopman observation network, a Koopman operator layer, a pulse encoder, and a feature decoder. Specific sub-steps are as follows: (1) Extracting statistical features using a sliding time window Given ROI time series ( Each brain region, (at several time points), dynamic statistical features were extracted using a sliding window strategy: Mean: ; Standard deviation: ; Skewness: ; Set window size Step length , Indicates the index of the sliding time window (the k-th sliding time window). Indicates the first The first subject The brain region, in the _ ... The neural imaging signal values ​​at each time point (i.e., the element values ​​at the corresponding positions in the ROI time series); Calculate the mean of each window for the time series data of each brain region. Standard deviation skewness The window feature is obtained by concatenating the three statistics. Stack all window features to form the input tensor ( (This refers to the number of windows). This sliding window design ensures temporal overlap between adjacent windows, thereby capturing the continuous dynamic changes of a time series.

[0027] (2) Koopman observation network maps the latent space For each time window, through a nonlinear observation network Mapping high-dimensional ROI statistical features to the latent Koopman space: ; Where input For the first The statistical features of each window are spliced ​​together. , , , , , These are the weight coefficients and bias coefficients of a three-layer fully connected network, respectively. The network employs a three-layer fully connected structure, with each layer followed by LayerNorm and Tanh activation functions. The output dimension is set to... .

[0028] (3) Koopman operator layer characterizes brain dynamics evolution In the latent space, through learnable linear operators This invention achieves brain dynamics evolution. Unlike traditional methods that iterate on static features, it performs Koopman evolution over a time window to capture the dynamic interactions between brain regions over time. ; in The time constant controls the smoothness of the evolution. Indicates the first The first sliding time window The step-by-step Koopman latent hidden state, this window-level evolution makes What is learned is the coordinated change pattern of brain regions across different time windows, rather than simple static feature transformations.

[0029] To ensure that the learned dynamics are physically sound, for Apply spectral radius constraints This ensures the system remains stable and does not diverge. Specifically, it calculates during each forward propagation. Maximum Singularity ,like Then it is normalized. This constraint makes the network tend to learn "edge-stable" dynamic patterns, that is, subtle dynamic changes associated with brain disease states.

[0030] (4) Pulse encoder realizes sparse feature representation To achieve sparse yet information-dense feature representation, spiking neurons are used to convert continuous Koopman patterns into discrete pulse sequences. Membrane potentials are defined. ( (It is a learnable firing threshold vector), and the pulse firing probability is calculated using the sigmoid function. : ; in For the sigmoid function, during training, a pass-through estimator is used for gradient backpropagation, and impulses are generated through Bernoulli sampling. Deterministic thresholds were used during the test. , This represents an indicator function (unit step function); furthermore, a refractory period mechanism is introduced to prevent neuronal overfiring. ,in Controlling the decay of the refractory period, Indicates the first The first sliding time window The refractory period state variable of the neuron in the step.

[0031] (5) Spatiotemporal integration and decoding generation dynamics characteristics Spatiotemporal integration of impulses across all time windows and time steps yields compact neural features. : ; in For the number of windows, This represents the number of Koopman evolution steps. This spatiotemporal two-dimensional integral simultaneously captures the time dimension (pulse changes at different evolutionary stages) and the window dimension (dynamic evolution across time windows), and finally recovers interpretable dynamic features through a decoding network. This is used as an augmentation input for subsequent graph neural networks.

[0032] ; Indicates the decoding network. The operation specifically means: to convert the dimension to... neural characteristics Through hidden layer Perform linear mapping and feature transformation to obtain dimension . The neural characteristics are processed by the ReLU activation function for nonlinear activation, and then passed through a second fully connected layer. A quadratic linear mapping and feature transformation are performed, followed by a non-linear activation using the ReLU activation function, ultimately yielding a dimension of... Dynamic characteristics ; in As a potential spatial dimension, This is the output dimension for dynamic features. Each hidden layer includes LayerNorm normalization and Dropout regularization (with a rate of 0.2).

[0033] (5) Dynamic mass assessment To evaluate the reliability of the extracted dynamic features, a quantitative quality score that integrates information richness and system stability is introduced. : ; in , For the first The first window, the Step, First Impulse values ​​for each latent dimension For the number of windows, For the number of evolutionary steps in the Koopman algorithm, As a potential spatial dimension, this indicator is calculated across all windows, time steps, and potential dimensions, reflecting the richness of information about dynamic features; The stability of the brain dynamics system characterized by the Koopman operator K, which measures model learning.

[0034] A high-quality score signifies two things: 1) active pulse firing and sufficient information in the kinetic characteristics; 2) a stable and reliable kinetic pattern. This score serves as a quantitative indicator of the reliability of the kinetic characteristics, providing a basis for filtering low-quality samples in subsequent analyses. We determined this score using receiver operating characteristic (ROC) curves combined with grid search. Valid threshold of scores When the sample If a sample is deemed low-quality, its feature contribution weight will be reduced or it will be removed from the subsequent model.

[0035] To verify the proposed The reasonableness of the score definition, the reliability of real data, and the ability to distinguish disease states were assessed on the ABIDE dataset. Fraction, spectral radius An empirical analysis was conducted using three global Koopman dynamics indices: average impulse rate, mean impulse rate, and mean impulse rate. After validating normality using the Shapiro-Wilk test, independent samples were selected. The difference between groups was analyzed using the test or the Mann-Whitney U test, and the results were as follows: Figure 3 As shown. Figure 3 This study presents a comparison of the distributions and mean ± 95% confidence intervals of Q scores, spectral radius ρ(K), and Spike Rate between 403 patients with autism spectrum disorder (ASD) and 468 healthy controls (HC). Each indicator is labeled with its mean ± standard deviation, effect size (Cohen's d), and statistical p-value. Statistical results show no significant differences in the three global indicators between the disease and healthy groups (Q: p=0.36, ρ(K): p=0.66, Spike Rate: p=0.10, all statistically insignificant), indicating that at the global scale of population average, the overall statistical indicators of the model-learned Koopman operator K cannot effectively distinguish between ASD and HC. Global Q score, The lack of significant differences in Spike Rate essentially indicates that the overall stability and information richness of brain dynamics in ASD patients are not significantly different from those in healthy controls. The disease-related abnormalities are not disturbances in global dynamics, but rather abnormalities in the coordination patterns of local brain regions.

[0036] (6) Construction of the association graph Similarity between samples is calculated based on Koopman features, and a correlation graph is constructed. : ; in It is the first The dynamic characteristics of each sample The function is sigmoid. This association graph will be merged with the phenotypic guide graph to form the final graph structure input.

[0037] Units used to construct multimodal graph neural network fusion features: Phenotypic guidance maps are constructed by combining the association graphs obtained from units used for extracting brain dynamics features based on KoopmanEvoNet with phenotypic data. Adaptive fusion of image and phenotypic features is achieved through a dual-channel graph neural network and cross-modal attention fusion, such as... Figure 1 As shown, , These represent the inter-sample dynamic correlation graph constructed based on Koopman dynamics features and the inter-sample phenotypic guidance graph constructed based on phenotypic data, respectively.

[0038] The specific sub-steps are as follows: (1) Constructing a phenotypic guide map To fully utilize phenotypic information to guide the construction of the graph structure, a phenotypic guiding graph is used, which contains edges of three semantic types: Strengthen the edges: This indicates that a diagnosis is established when two samples have similar phenotypic characteristics and belong to the same diagnostic category. Indicates the first Clinical diagnostic labels of the subjects Indicates the first The first subject The phenotypic features include demographic information such as age, gender, IQ, and scanning site. Dimension index for phenotypic features.

[0039] Weakening edges: This indicates that a system is established when samples have similar phenotypes but different diagnoses; Incentive edge: Connecting samples from the training set with those from the validation / test set facilitates information transfer. It belongs to the training set, and the samples The validation / test set is created in real time. Indicates sample , Indicates sample , Represents the training set, Represents the validation / test set; Formalistically, for the first Phenotypic characteristics ,definition:

[0040]

[0041]

[0042] in This is the phenotypic similarity threshold. The final hybrid association graph is a weighted sum of the edge weights for each phenotypic dimension.

[0043] (2) Feature extraction from graph neural networks Using a dual-channel parallel architecture, an Imaging Network (Img-Net) and a Phenotypic Network (Ph-Net) are constructed using TransformerConv (Graph Transformer convolutional layers). The Imaging Network processes the functional connectivity features of the brain connectivity matrix (obtained from the output of the Imaging Data branch: first, the time series of each brain region (ROI) defined by the brain template is divided into time sliding windows, and then the dynamic features are extracted through the KoopmanEvoNet module to obtain the brain region functional connectivity / dynamic feature matrix, i.e., the functional connectivity features of the brain connectivity matrix mentioned in this step). The Phenotypic Network processes the phenotypic embeddings encoded by VAE (obtained from the output of the Phenotypic Data branch: after constructing the phenotypic association between samples based on phenotypic information (Age, Sex, IQ), the original phenotypic data is encoded and dimensionality reduced through a Variational Autoencoder (VAE) to obtain a low-dimensional dense phenotypic embedding representation, i.e., the phenotypic embeddings encoded by VAE mentioned in this step). The importance of neighboring nodes is adaptively learned through an attention mechanism. ; in, Indicates the first In layer graph Transformer convolution, nodes The output feature vector (i.e., node) After the first (Updated feature representation after convolutional layers) Represents the nodes in the graph The set of neighboring nodes (containing all nodes related to node) A node that is connected by an edge can also contain other nodes. (It is defined according to the graph convolution implementation). Represents a node With nodes Attention coefficient between, The values ​​of the attention mechanism are projected onto the weight matrix. Indicates the first Nodes in a layered graph Transformer convolution Input feature vector, attention coefficient ,in, The query projection weight matrix represents the attention mechanism. The key-projection weight matrix represents the attention mechanism. Represents the node The input feature vector of the current layer, Represents a node The input feature vector of the current layer.

[0044] (3) Cross-modal attention fusion To adaptively fuse information from both image and phenotypic modalities, a cross-modal attention fusion mechanism is introduced.

[0045] First, calculate the shared representation. ,in, This represents the image modal feature vector output by the image network (Img-Net). This represents the phenotypic modality feature vector output by the Ph-Net network; Then, the attention scores for each modality and the shared representation are calculated separately. : ; in, This represents the trace operation of a matrix (i.e., the sum of the elements on the main diagonal of the matrix). This indicates that the feature matrix is ​​shared in batches. The final fusion weights are determined by the softmax-normalized scores, and are jointly represented as: ; , , This represents a learnable linear projective weight matrix. This represents the multimodal joint feature vector obtained after cross-modal attention fusion.

[0046] exist Figure 1 middle, / / The `` tag represents the enhancement / activation / suppression branches of the phenotypic association graph, used to construct phenotypic associations between samples of different strengths; `img_features` represents the image dynamics feature matrix output by KoopmanEvoNet, which is subsequently used to construct enhanced image features; `img_enhanced` represents the enhanced image feature matrix, obtained by adding `img_features` to the linear layer output, and input to the image network; `h_ph_emb` represents the phenotypic embedding after VAE encoding, input to the phenotypic network; `f_dyn` represents the dynamics features input to the gating network, corresponding to the core dynamics information extracted from the image branch; `g` represents the gating vector output by the gating network, used for... Perform element-wise weighting; h_img_gated represents the gated image features, by It is obtained by element-wise multiplication with the gate vector g.

[0047] Units for eliminating site batch effects through neurally inspired domain adaptation: Batch effects from different scanning sites are a major challenge in cross-center neural image analysis. To learn site-invariant representations, a gradient inversion domain adversarial mechanism is introduced.

[0048] Specifically, construct a site discriminator. The input is the learned feature representation, and the output is the site label. The training objective is: This means minimizing the classification loss. And maximize domain adaptation loss This is to enable the feature learner to align the data distribution across different sites while maintaining its classification capabilities; Representation domain adaptation loss; , It is a gradient inversion layer. This represents the cross-entropy loss function, which is an identity mapping during forward propagation and multiplies the gradient by a factor during backpropagation. , This represents the gradient scaling factor of the gradient inversion layer. Through this adversarial training, the feature learner is forced to generate representations that cannot be distinguished by the site discriminator, thereby eliminating domain bias.

[0049] Units used for jointly training models using multiple loss functions: Set the overall training loss function , , and For coefficients; Each loss term is defined as follows: Classification loss Using standard cross-entropy ; Graph structure loss Includes smoothing regularization loss Degree Regularization Loss , ,in Penalize feature differences between adjacent nodes. The feature matrix of all nodes in the graph structure. For the graph Laplace matrix, The degree matrix of the graph. Represents the adjacency matrix of the merged graph; Prevent the formation of supernodes with excessively high degrees; Dynamic loss : This includes consistency and stability regularization; express, This represents the L2 norm (Euclidean norm). and is the regularization weight coefficient (hyperparameter), used to balance the loss weight coefficient between the L2 regularization term and the dynamic stability constraint term; Domain adaptation loss Cross-entropy of site categories.

[0050] Units for diagnosing brain diseases using attention graph convolutional classes: (1) Information fusion of Attention Graph Convolution After obtaining the fused graph structure and enhanced features, an attention graph convolution mechanism is used to aggregate spatial information of the node features. This module adopts a dual-channel parallel structure, processing image features and phenotypic features separately. The image feature processing branch includes two sub-modules: a graph convolution branch and a gating quality screening branch. First, spatial dependencies are extracted using graph convolution, then quality gating is performed based on dynamic quality scores. The phenotypic feature processing branch uses only the graph convolution branch to directly aggregate spatial information of the phenotypic features. After the outputs of the two branches are combined through cross-modal attention fusion, the final joint features are obtained.

[0051] Graph Convolution Branch: A multi-layered TransformerConv is used to perform message passing on the input graph structure, learning the spatial dependencies between nodes. Let... These are the initial node features. The initial node feature matrix of the graph convolution (composed of the fused enhanced features) The structure consists of the input node features of the attention map convolutional module, after which... After layer graph convolution, we get ,in This is the fused relationship diagram.

[0052] Gated mass screening: using the kinetic mass fraction output by KoopmanEvoNet Dynamically adjust the contribution weights of image features: , ; in This is a gating vector used to filter out low-quality samples. For the Sigmoid function, The learnable linear projective weight matrix for the gated quality screening module. The brain dynamics feature vector output by KoopmanEvoNet (strongly correlated with the dynamics quality score Q, used to characterize the feature quality).

[0053] Cross-modal attention fusion: Adaptively fuses the outputs of the image branch and the phenotypic branch to obtain a joint representation. (The cross-modal fusion in this module and the fusion in step 3 are fusions at different levels: step 3 is the pre-fusion in the feature extraction stage, which aligns the basic information of the multimodal based on the original single-modal features and the fusion graph structure; this module is the final fusion before classification, which first filters low-quality features through gating quality screening, then completes spatial information aggregation through attention map convolution, and finally fuses the high-quality enhanced features to output the joint features finally used for diagnosis and classification.) (2) Fully connected classifier prediction Joint statement Classification is performed, and a fully connected classifier is used for the final diagnostic prediction: , , in, This is the weight matrix. Indicates batch normalization. This is the Sigmoid function. Output. Predict the probability of health conditions / illnesses, and determine the diagnosis based on the probability value.

[0054] Example 3 This embodiment further defines Embodiment 2, and provides a detailed explanation of the working principle and innovations of the system in Embodiment 2.

[0055] I. Working Principle: The core working principle of this system is to combine Koopman operator theory with spiking neural networks to achieve linear analysis of brain nonlinear dynamics. Simultaneously, it integrates complementary information from multimodal neuroimaging data to eliminate cross-site batch effects, ultimately achieving high accuracy and high interpretability in the diagnosis of brain diseases. 1. Linearization analysis of Koopman operator: By mapping the high-dimensional nonlinear brain dynamics state space to a low-dimensional linear space through Koopman eigenfunctions, the dynamic interaction between brain regions evolves linearly in the Koopman space, breaking through the nonlinear fitting limitations of traditional time series modeling. 2. Physiological matching of spiking neural networks: spiking neurons are used to convert continuous Koopman patterns into discrete spiking sequences, achieving sparse feature representations that conform to the characteristics of brain neural activity, and introducing a refractory period mechanism to simulate the physiological characteristics of brain neurons; 3. Adaptive fusion of multimodal data: Constructing a phenotypic guidance map to provide semantic guidance for image feature analysis, and adaptively learning the fusion weights of images and phenotypic features through a cross-modal attention mechanism to make full use of the information in multimodal data; 4. Improved generalization of domain adversarial training: Through gradient reversal site adversarial training, the model is forced to learn feature representations that are independent of the scanning sites, eliminating batch effects and improving the model's generalization ability on cross-center data. 5. Quantitative evaluation of kinetic characteristics: through mass fraction By integrating feature richness and system stability, low-quality samples can be screened, thereby improving the robustness of model diagnosis. 6. Spatial information aggregation of graph convolution: Based on the dynamic similarity and phenotypic correlation between samples, an association graph is constructed. Spatial dependencies between samples are aggregated through attention graph convolution to capture abnormal local cooperative patterns in brain diseases.

[0056] II. Innovation Points 1. Propose the KoopmanEvoNet network architecture: For the first time, Koopman operator theory is combined with spiking neural networks to design learnable Koopman eigenfunctions and linear dynamic constraints, realize linear analysis of brain nonlinear dynamics, and extract biomarkers of dynamics with biological interpretability; 2. Design of a quantitative evaluation system for dynamic characteristics: A mass fraction integrating average impulse rate and Koopman operator spectral radius is proposed. This enables a quantitative assessment of the reliability of brain dynamics features, providing a quantitative basis for screening low-quality samples and solving the problem of traditional methods lacking feature-based quality assessment. 3. Construct a phenotypic-guided multimodal fusion graph neural network: Design a phenotypic-guided graph containing reinforcement edges, weakening edges, and activation edges, and combine it with a dynamic feature association graph to form a hybrid graph structure. Adaptive fusion of image and phenotypic features is achieved through a dual-channel TransformerConv and a cross-modal attention mechanism, making full use of the complementary information of multimodal data; 4. Introducing a neurally inspired domain adaptation mechanism: Based on site adversarial training with gradient inversion layers, the model learns site-invariant feature representations, effectively eliminating batch effects in cross-center neural image analysis and improving the model's generalization ability. 5. A multi-loss joint training strategy is proposed: classification loss, graph structure loss, dynamics loss, and domain adaptation loss are integrated to achieve end-to-end joint training of the model, while constraining the rationality of graph structure, the stability of dynamic features, and domain invariance, thereby improving the diagnostic accuracy and robustness of the model; 6. Design an attention graph convolutional integral module for gating quality screening: Integrate dynamic quality scores into the feature aggregation process of graph convolution, dynamically adjust the contribution weights of sample features through gating vectors, filter low-quality samples, and further improve the accuracy of the diagnostic model.

Claims

1. A multimodal neuroimaging diagnostic system for brain diseases based on spiking neural networks, characterized in that, The system includes: Units for multimodal neuroimaging data preprocessing: used to construct data containing different sites Brain connectivity matrix of 10 subjects , phenotypic data ROI time series and clinical labels dataset , Indicates the first 10 subjects; The unit used for extracting brain dynamics features based on KoopmanEvoNet: taking ROI time series as input, it extracts dynamic features through sliding time window, Koopman observation network, Koopman operator layer, pulse encoder, and spatiotemporal integration and decoding, and completes dynamics quality assessment and association graph construction; The unit used to construct multimodal graph neural network fusion features: Based on the association graph obtained by the unit used to extract brain dynamics features based on KoopmanEvoNet, a phenotypic guidance graph is constructed. Adaptive fusion of image and phenotypic features is achieved through dual-channel graph neural network and cross-modal attention fusion to obtain the fused graph structure and enhanced features. Units for eliminating site batch effects through neurally inspired domain adaptation: Introducing a gradient inversion domain adversarial mechanism to eliminate batch effects generated by different scan sites; Unit for jointly training a model using multiple loss functions: setting an overall training loss function and training each neural network used by the system; A unit for diagnosing brain diseases through attention graph convolution: information is propagated on the graph through attention graph convolution, and a fully connected classifier is used for the final diagnostic prediction.

2. The multimodal neuroimaging brain disease diagnostic system based on a spiking neural network according to claim 1, characterized in that, The sliding time window is used to extract statistical features, given a ROI time series. ,in and The meaning is brain regions and At each time point, a sliding window strategy is used to extract dynamic statistical features: Mean: Standard deviation: Skewness: in, Indicates window size. Indicates step size, The index represents the sliding time window, indicating the nth time window. A sliding time window, Indicates the first The first subject The brain region, in the _ ... Neuroimaging signal values ​​at each time point; Calculate the mean of each window for the time series data of each brain region. Standard deviation skewness The window feature is obtained by concatenating the three statistics. Stack all window features to form the input tensor , This represents the number of windows.

3. The multimodal neuroimaging brain disease diagnostic system based on a spiking neural network according to claim 2, characterized in that, The Koopman observation network is used to map the statistical features of high-dimensional ROIs to the latent Koopman space to obtain the initial latent state. ; The Koopman operator layer is used to perform Koopman evolution along the time window dimension, capturing the dynamic interactions between brain regions over time, and obtaining the first... A sliding window Koopman latent state at any given moment .

4. The multimodal neuroimaging brain disease diagnostic system based on a spiking neural network according to claim 3, characterized in that, The pulse encoder is used to realize sparse feature representation, specifically by defining the membrane potential. , It is a learnable firing threshold vector, and the pulse firing probability is calculated using the sigmoid function. : During training, a pass-through estimator is used for gradient backpropagation, and Bernoulli sampling is employed. Generate pulse ; Deterministic thresholds were used during the test. Furthermore, a refractory period mechanism was introduced to prevent excessive neuronal firing.

5. The multimodal neuroimaging brain disease diagnostic system based on a spiking neural network according to claim 4, characterized in that, The spatiotemporal integration and decoding specifically involve performing spatiotemporal integration on the pulses of all time windows and time steps to obtain compact neural features. : ,in, For the number of evolution steps of the Koopman algorithm, a decoding network is used to... Recovering to interpretable dynamic characteristics .

6. The multimodal neuroimaging brain disease diagnostic system based on a spiking neural network according to claim 5, characterized in that, The dynamic quality assessment specifically involves introducing a quantitative quality score that integrates information richness and system stability. : ,in, Let be the impulse value of the k-th window, the t-th step, and the j-th latent dimension. Learnable linear operators after applying spectral radius constraints ; The construction of the association graph specifically involves calculating the similarity between samples based on Koopman features and then constructing the association graph. : ,in It is the first The dynamic characteristics of each sample This is the sigmoid function.

7. The multimodal neuroimaging brain disease diagnostic system based on a spiking neural network according to claim 6, characterized in that, The specific operations for the unit used to construct the fusion features of a multimodal graph neural network are as follows: S1. Constructing a phenotypic guidance graph: A phenotypic guidance graph includes edges of three semantic types: Strengthening edges: established when two samples have similar phenotypic features and belong to the same diagnostic category; weakening edges: established when samples have similar phenotypic features but different diagnoses; stimulating edges: connecting samples in the training set and the validation / test set to promote information transfer; the final hybrid association graph is the weighted sum of the edge weights of each phenotypic dimension. S2. Graph Neural Network Feature Extraction: A dual-channel parallel structure is used, employing TransformerConv to construct an image network and a phenotypic network. The image network processes the functional connectivity features of the brain connectivity matrix, while the phenotypic network processes the phenotypic embeddings encoded by VAE. The importance of neighboring nodes is adaptively learned through an attention mechanism. S3. Cross-modal attention fusion: A cross-modal attention fusion mechanism is introduced. First, the shared representation is calculated. Then, the attention scores of each modality and the shared representation are calculated separately. Finally, the fusion weight is determined by the scores after softmax normalization.

8. The multimodal neuroimaging brain disease diagnostic system based on a spiking neural network according to claim 7, characterized in that, The gradient inversion domain adversarial mechanism is as follows: Construct a site discriminator The input is the learned feature representation, and the output is the site label. An identity mapping is applied during forward propagation, and the gradient is multiplied during backpropagation. Through this adversarial training, the feature learner is forced to generate representations that cannot be distinguished by the site discriminator, thereby eliminating domain bias.

9. A multimodal neuroimaging brain disease diagnostic system based on a spiking neural network according to claim 8, characterized in that, The overall training loss function includes classification loss. Graph structure loss kinetic loss Domain Adaptation Loss .

10. A multimodal neuroimaging brain disease diagnostic system based on a spiking neural network according to claim 9, characterized in that, The attention graph convolution aggregates spatial information of node features and employs a dual-channel parallel structure to process image features and phenotypic features separately. This includes graph convolution branches, gating quality filtering, and cross-modal attention fusion. The outputs of the image branch and the phenotypic branch are adaptively fused through cross-modal attention fusion to obtain a joint representation. Fully connected classifier pairs A fully connected classifier is used for classification and final diagnostic prediction.

Images