Multi-modal brain network analysis system and method, electronic equipment and storage medium

Abstract

The invention provides a multi-modal brain network analysis system and method, electronic equipment and a storage medium, and relates to the field of brain networks, the multi-modal brain network analysis system comprises an acquisition module, a preprocessing module, a first processing module, a second processing module, a fusion convolution module and an output module; the acquisition module is used for acquiring functional magnetic resonance imaging data and diffusion tensor imaging data; the preprocessing module is used for processing to obtain an edge time sequence and an initial structure brain network; the first processing module is used for constructing a dynamic effect brain network; the second processing module is used for constructing an edge center structure brain network; and the fusion convolution module is used for carrying out structure fusion and directed space-time diagram convolution. According to the method, structure fusion and directed space-time diagram convolution are carried out on the dynamic effect brain network and the edge center structure brain network based on a self-attention mechanism, target information representing the brain nerve state is obtained, accurate cognitive load evaluation is carried out, and the limitation of a traditional node center method is solved.

Description

Technical Field

[0001] This application relates to the field of brain network technology, and specifically relates to a multimodal brain network analysis system, method, electronic device, and storage medium. Background Art

[0002] A brain network refers to the network structure formed by the complex connections between neurons in the brain. These connections include nerve fiber bundles, synaptic connections, etc., enabling different regions of the brain to communicate and cooperate with each other to achieve various cognitive, emotional, and behavioral functions. Brain network research is of great significance for cognitive load research. Brain networks can support complex cognitive functions. Various cognitive functions of the brain, such as perception, attention, memory, language, thinking, etc., all rely on the cooperation of multiple brain regions. These brain regions are interconnected and communicate through the brain network, forming a highly complex information processing system.

[0003] Traditional cognitive load research mainly relies on behavioral indicators, such as reaction time, error rate, etc., to infer the magnitude of cognitive load. Brain network research, on the other hand, directly starts from the level of neural activities in the brain and reveals the neural mechanisms behind cognitive load. By observing the network activation patterns and connection changes in the brain under different cognitive loads, we can understand more deeply how cognitive load is generated and regulated in the brain, and how different types of cognitive tasks rely on specific brain networks to be completed.

[0004] In related technologies, the functional brain network obtained by functional magnetic resonance imaging (fMRI) and the structural brain network obtained by diffusion tensor imaging (DTI) can characterize the structural and functional states of the brain. Existing technologies mainly use node-centered methods to extract features of brain regions. However, this method has obvious deficiencies in capturing the complex interaction dynamics between brain regions. The simple node-centered method cannot fully reveal the true transmission mechanism of neural information and complex interaction patterns. In addition, existing multimodal edge-centered methods usually ignore the high-order edge causal effects and structural asymmetries during the analysis process; at the same time, the traditional assumption that the structural brain network is symmetric and non-directional may ignore structural lateralization abnormalities, thereby affecting the accurate modeling of the overall organizational characteristics of the brain. Summary of the Invention

[0005] Aiming at the deficiencies of the existing technology, this application provides a multimodal brain network analysis system, method, electronic device, and storage medium, which solves the problem that the traditional node-centered method only focuses on the features of a single brain region and has limitations in capturing dynamic information and structural asymmetries.

[0006] To achieve the above objectives, this application is implemented through the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a multimodal brain network analysis system, which includes an acquisition module, a preprocessing module, a first processing module, a second processing module, a fusion convolution module, and an output module.

[0008] Specifically, the acquisition module is used to acquire functional magnetic resonance imaging data and diffusion tensor imaging data; the preprocessing module is used to preprocess the functional magnetic resonance imaging data to obtain edge time series; and preprocess the diffusion tensor imaging data to obtain an initial structural brain network based on nodes; the first processing module is used to perform fine-grained temporal segmentation on the edge time series by using a sliding window partitioning method, and convert the time-domain signals in each segment into high-order time-varying causal relationships through conditional Granger causality analysis to construct a dynamic effect brain network; the second processing module is used to convert the initial structural brain network into an edge-centered structural brain network that captures brain structure lateralization features through an adaptive construction and graph diffusion method; the fusion convolution module is used to perform structural fusion and directed spatio-temporal graph convolution on the dynamic effect brain network and the edge-centered structural brain network based on the self-attention mechanism to obtain target information characterizing the brain nerve state; the output module is used to classify the brain nerve state of the target information by using a multi-layer perceptron and output the classification result for visualization processing.

[0009] According to the first aspect of the embodiment of the present application, the foregoing preprocessing of the functional magnetic resonance imaging data to obtain edge time series may specifically include the following steps: processing the original functional image corresponding to the functional magnetic resonance imaging data, and extracting the BOLD time series of each brain region ROI; for the BOLD time series, perform slice time series correction by removing the first 10 time points, and perform motion correction to exclude subjects with head movement exceeding 2.5 mm or 2.5 degrees; perform co-registration with the T1-weighted structural image by using statistical parametric mapping SPM, and perform segmentation of gray matter, white matter, and cerebrospinal fluid, and then apply DARTEL normalization to a preset standard space; perform 4-mm full-width smoothing processing on the functional image through a Gaussian of full width at half maximum FWHM to reduce registration variability; perform band-pass filtering on the functional time series between 0.01 Hz and 0.1 Hz, and for each subject, obtain the target BOLD signal; normalize the target BOLD signal, calculate the element product of each region pair to characterize the instantaneous co-fluctuation between region pairs, and repeat the process for all edges to obtain edge time series.

[0010] According to the first aspect of the embodiments of the present application, the foregoing preprocessing of diffusion tensor imaging data to obtain an initial structural brain network based on nodes may specifically include the following steps: for the diffusion tensor imaging data, use the PANDA toolbox to apply the deterministic tractography method; after removing the skull cortex parameters, use the FACT fiber tracking algorithm for fiber tracking; determine the angle threshold to be 45 degrees, and if the fractional anisotropy FA is less than 0.2 or greater than 1, terminate the tracking process; after the fiber tracking is completed, use a filter to smooth the tracking result; use the drawn white matter fibers as the edge weights of the brain network to obtain the initial structural brain network.

[0011] According to the first aspect of the embodiments of the present application, the dynamic effect brain network is a dynamic edge-centered effect brain network and is used to characterize dynamic information transmission and enhance the capture of edge dynamics; the initial structural brain network is a node-centered brain network.

[0012] According to the first aspect of the embodiments of the present application, the foregoing conversion of the initial structural brain network into an edge-centered structural brain network that captures the brain structure lateralization characteristics through adaptive construction and graph diffusion method may specifically include the following steps: perform adaptive construction on the initial structural brain network to obtain a target structural brain network centered on edges Apply heat diffusion to the target structural brain network to simulate the propagation of neurophysiological signals to analyze the lateralization abnormalities in the brain structure network and obtain the edge-centered structural brain network.

[0013] According to the first aspect of the embodiments of the present application, the process of adaptive construction satisfies the expression:

[0014]

[0015] where α represents a learnable parameter, FA_ij represents the anisotropy fraction of the white matter fiber bundle between brain regions i and j, and FA_uv represents the anisotropy fraction of the white matter fiber bundle between brain regions u and v; corresponding to the local network between brain regions i, j, u, and v, and being a subset of the target structural brain network of.

[0016] According to the first aspect of the embodiments of the present application, the foregoing process corresponding to heat diffusion satisfies the expression:

[0017] In the formula, e is the natural constant, t represents a learnable parameter that can adaptively adjust the diffusion scale, L is the Laplacian matrix of, represents the edge-centered structural brain network.

[0018] According to the first aspect of the embodiments of the present application, the foregoing fusion convolution module is specifically configured to: use a self-attention mechanism to assign different weights to the effect information and structure information corresponding to the dynamic effect brain network and the edge-center structure brain network for fusion to obtain a fusion network; adopt graph convolution and a gated recurrent unit to aggregate the spatio-temporal features of each edge in the fusion network, and mine the dynamic association information of the fusion network to obtain target information representing the brain nerve state.

[0019] According to the first aspect of the embodiments of the present application, the fusion process of the foregoing dynamic effect brain network and the edge-center structure brain network satisfies the expression:

[0020]

[0021]

[0022] In the formula, W is a trainable weight matrix, b is a bias vector, q is a shared attention vector, and tanh represents an activation function; α e , α s respectively represent the weights corresponding to the dynamic effect brain network and the edge-center structure brain network; represents the dynamic effect brain network, represents the edge-center structure brain network, represents the fused network, and l represents the segmented time series index.

[0023] According to the first aspect of the embodiments of the present application, the foregoing adoption of graph convolution and a gated recurrent unit to aggregate the spatio-temporal features of each edge in the fusion network, mine the dynamic association information of the fusion network, and obtain target information representing the brain nerve state may specifically include the following steps: through a bidirectional graph convolutional network Bi-GCN, use forward graph convolution and backward graph convolution to process the directed graph corresponding to the fusion network to capture bidirectional node dependencies and perform directed spatial feature extraction; based on the bidirectional node dependencies, obtain a spatial representation through fusion direction embedding; based on the spatial representation, use a gated recurrent unit to model the temporal dependencies, and perform average pooling aggregation at the 0 time step to obtain the target information.

[0024] Second aspect, an embodiment of the present application provides a multimodal brain network analysis method, and the multimodal brain network analysis method includes: obtaining functional magnetic resonance imaging data and diffusion tensor imaging data through an acquisition module; preprocessing the functional magnetic resonance imaging data through a preprocessing module to obtain edge time series; and preprocessing the diffusion tensor imaging data to obtain an initial structural brain network based on nodes; using a sliding window partitioning method by a first processing module to perform fine-grained temporal segmentation on the edge time series, and converting the time-domain signals in each segment into high-order time-varying causal relationships through conditional Granger causality analysis to construct a dynamic effect brain network; converting the initial structural brain network into an edge-centered structural brain network that captures brain structure lateralization characteristics through an adaptive construction and graph diffusion method by a second processing module; performing structural fusion and directed spatio-temporal graph convolution on the dynamic effect brain network and the edge-centered structural brain network based on a self-attention mechanism through a fusion convolution module to obtain target information representing the brain nerve state; using a multi-layer perceptron by an output module to classify the target information of the brain nerve state, and outputting a classification result for visualization processing.

[0025] Third aspect, an embodiment of the present application provides an electronic device, and the electronic device includes: a processor, a memory, and a program stored on the memory and executable on the processor, and when the program is executed by the processor, the multimodal brain network analysis method in the foregoing second aspect is implemented.

[0026] Fourth aspect, an embodiment of the present application provides a computer-readable storage medium, and a program or instruction is stored on the computer-readable storage medium, and when the program or instruction is executed by the processor, the multimodal brain network analysis method in the foregoing second aspect is implemented.

[0027] The present application provides a multimodal brain network analysis system, method, electronic device, and storage medium. Compared with the prior art, the following beneficial effects are achieved:

[0028] This application processes functional magnetic resonance imaging data into edge time series and processes diffusion tensor imaging data to obtain an initial structural brain network based on nodes; during the construction of a dynamic effect brain network, multiple time series segments are divided based on the edge time series, and the time domain signals of each segment are converted into high-order time-varying causal relationships; during the construction of an edge-center structural brain network, brain structure lateralization features are captured; by fusing edge time-varying causal information and structural asymmetry features, accurate analysis of the brain network structure can be achieved; the edge information between different brain regions has multi-dimensional and dynamic characteristics, which can fully reveal the true transmission mechanism of neural information and complex interaction patterns; this application is based on the self-attention mechanism, performs structural fusion and directed spatio-temporal graph convolution on the dynamic effect brain network and the edge-center structural brain network to obtain target information representing the brain's neural state for accurate cognitive load assessment and obtain an accurate cognitive load assessment classification result; it solves the limitations of traditional node-centered methods in capturing dynamic information and structural asymmetry. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0030] Figure 1 is a schematic structural diagram of a multi-modal brain network analysis system provided by an embodiment of this application;

[0031] Figure 2 is a partial detailed schematic diagram of a multi-modal brain network analysis method provided by an embodiment of this application;

[0032] Figure 3 is an exemplary flowchart of a multi-modal brain network analysis method provided by an embodiment of this application;

[0033] Figure 4 is a schematic structural diagram of an electronic device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0035] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0036] By providing a multimodal brain network analysis system, method, electronic device and storage medium, embodiments of the present application solve the problem that traditional node-centered methods only focus on the characteristics of individual brain regions and have limitations in capturing dynamic information and structural asymmetry.

[0037] The overall idea of the technical solution in the embodiments of the present application to solve the above technical problems is as follows:

[0038] A brain network refers to the network structure formed by the complex connections between neurons in the brain. These connections include nerve fiber bundles, synaptic connections, etc., enabling different regions of the brain to communicate and cooperate with each other to achieve various cognitive, emotional and behavioral functions. Brain networks are divided into structural networks, functional networks and effective networks. Structural networks are mainly based on the anatomical structure of the brain and depict the physical connections between various regions of the brain by methods such as tracing nerve fiber bundles. The structural network provides the infrastructure for information transmission in the brain, just like the road system of a city, determining the path and manner of information propagation in the brain. Functional networks are constructed by observing the activity correlations between different brain regions when the brain is performing various tasks or in different states. Even in the resting state, there are some spontaneous and continuous neural activities in the brain, and these activities show specific cooperative patterns between different brain regions, forming functional networks. For example, the default mode network is a functional network that is highly active in the resting state and is closely related to functions such as self-referential thinking and episodic memory. Effective networks not only consider the functional correlations between brain regions, but also further focus on the causal relationships and information flow directions between brain regions. It attempts to reveal how different brain regions interact and regulate each other when the brain processes information, and focuses more on describing the dynamic functional characteristics of the brain network.

[0039] Brain network research is of great significance to cognitive load research. Brain networks can support complex cognitive functions. Various cognitive functions of the brain, such as perception, attention, memory, language, and thinking, all rely on the synergy between multiple brain regions. These brain regions are interconnected and communicated through brain networks, forming a highly complex information processing system. For example, in the process of memory formation, there is a close functional connection between the hippocampus and multiple brain regions such as the prefrontal cortex and temporal lobe, and they jointly participate in the encoding, storage, and retrieval of memory.

[0040] Brain network research can meet the cognitive load assessment of some occupations, such as pilots and air traffic controllers; it is understandable that pilots need to process a large amount of complex information during flight, including flight instrument data, weather conditions, air traffic control instructions, etc. Cognitive load assessment of pilots can help optimize flight training programs and cockpit design, ensure that pilots can complete tasks accurately and efficiently in a high-load working environment, and ensure flight safety. The work of air traffic controllers requires a high degree of concentration, real-time processing of flight information of many aircraft, and commanding aircraft takeoffs and landings and flight paths. Cognitive load assessment can help optimize control processes and work environments, avoid controllers making mistakes due to excessive cognitive load, and ensure the safety and smoothness of air traffic.

[0041] Traditional cognitive load research is mainly based on behavioral indicators, such as reaction time, error rate, etc., to infer the size of cognitive load. Brain network research starts directly from the level of neural activity in the brain, revealing the neural mechanism behind cognitive load. By observing the network activation patterns and connection changes of the brain under different cognitive loads, we can have a deeper understanding of how cognitive load is generated and regulated in the brain, and how different types of cognitive tasks rely on specific brain networks to complete. When different individuals face the same cognitive tasks, there are differences in their cognitive load performance and brain network activity patterns; brain network research can reveal the neural basis of these individual differences.

[0042] Currently, researchers can comprehensively characterize the structural and functional state of the brain by using functional brain networks obtained by functional magnetic resonance imaging (fMRI) and structural brain networks obtained by diffusion tensor imaging (DTI). Existing technologies mainly use the node-centric method to extract features from brain regions, but this method is obviously insufficient in capturing the complex interaction dynamics between brain regions. Brain function is not only reflected in the activation of a single brain region, but more importantly, the edge information between different brain regions, that is, the connection information; the interaction between these edges has multi-dimensional and dynamic characteristics. Therefore, the simple node-centric method cannot fully reveal the true transmission mechanism and complex interaction pattern of neural information.

[0043] In addition, existing multi-modal edge center methods usually ignore high-order edge causal effects and structural asymmetries during the analysis process. The interactions between brain regions are not only cooperative relationships, but also contain explicit causal information, and simple functional analysis methods are difficult to reveal the true mechanism of information flow. Although conditional Granger causality analysis provides an effective tool for capturing causal relationships between brain regions, in practical applications, how to accurately characterize such causal effects in a dynamic environment remains a major challenge. At the same time, the traditional assumption that the structural brain network is symmetric and undirected may ignore abnormal structural lateralization, thereby affecting the accurate modeling of the overall organizational characteristics of the brain.

[0044] To better understand the above technical solution, the above technical solution will be described in detail below in combination with the accompanying drawings of the specification and specific implementation manners.

[0045] First, a multi-modal brain network analysis system 100 provided by an embodiment of the present application will be introduced below.

[0046] A structural schematic diagram of a multi-modal brain network analysis system 100 provided by an embodiment of the present application is shown in Figure 1 As shown, the multi-modal brain network analysis system 100 may specifically include the following modules:

[0047] An acquisition module 110, configured to acquire functional magnetic resonance imaging data and diffusion tensor imaging data;

[0048] A preprocessing module 120, configured to preprocess the functional magnetic resonance imaging data to obtain edge time series; and preprocess the diffusion tensor imaging data to obtain an initial structural brain network based on nodes;

[0049] A first processing module 130, configured to perform fine-grained time series segmentation on the edge time series by using a sliding window partitioning method, and convert the time domain signals in each segment into high-order time-varying causal relationships through conditional Granger causality analysis to construct a dynamic effect brain network;

[0050] A second processing module 140, configured to convert the initial structural brain network into an edge center structural brain network that captures brain structure lateralization characteristics through an adaptive construction and graph diffusion method;

[0051] A fusion convolution module 150, configured to perform structural fusion and directed spatio-temporal graph convolution on the dynamic effect brain network and the edge center structural brain network based on a self-attention mechanism to obtain target information representing the brain nerve state;

[0052] An output module 160, configured to classify the brain nerve state of the target information by using a multi-layer perceptron, and output a classification result for visualization processing.

[0053] The above is the specific implementation of the multimodal brain network analysis system 100 provided by the embodiments of the present application. Please refer to Figure 2 , Figure 2 wherein fMRI is functional magnetic resonance imaging data and DTI is diffusion tensor imaging data in Figure 2 ; it can be understood that the present application processes the functional magnetic resonance imaging data into edge time series, and processes the diffusion tensor imaging data to obtain an initial structural brain network based on nodes; in the process of constructing a dynamic effect brain network, a plurality of time series segments are divided based on the edge time series, and the time domain signals of each segment are converted into high-order time-varying causal relationships; in the process of constructing an edge-center structural brain network, the brain structure lateralization characteristics are captured; by fusing the edge time-varying causal information and the structural asymmetry characteristics, accurate analysis of the brain network structure can be achieved.

[0054] It should be noted that the edge information between different brain regions has the characteristics of multi-dimension and dynamics, which can fully reveal the real transmission mechanism of nerve information and complex interaction patterns; based on the self-attention mechanism, the present application performs structural fusion and directed spatio-temporal graph convolution on the dynamic effect brain network and the edge-center structural brain network to obtain target information representing the brain nerve state for accurate cognitive load assessment; it solves the limitations of traditional node-centered methods in capturing dynamic information and structural asymmetry.

[0055] According to the embodiments of the present application, any multiple of the acquisition module 110, the preprocessing module 120, the first processing module 130, the second processing module 140, the fusion convolution module 150, and the output module 160 can be combined into one module for implementation, or any one of them can be split into multiple modules. Or, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0056] In some embodiments, the foregoing preprocessing of the functional magnetic resonance imaging data to obtain edge time series may specifically include the following steps:

[0057] S210. Process the original functional image corresponding to the functional magnetic resonance imaging data, and extract the BOLD time series of each region of interest (ROI) of the brain.

[0058] S220. For the BOLD time series, perform slice time series correction by removing the first 10 time points, and perform motion correction to exclude subjects with head movement exceeding 2.5 mm or 2.5 degrees.

[0059] S230. Co-register the statistical parametric mapping (SPM) with the T1-weighted structural image, and perform segmentation of gray matter, white matter, and cerebrospinal fluid, and then apply DARTEL normalization to a preset standard space.

[0060] S240. Smooth the functional image with a 4-mm full-width Gaussian at half maximum (FWHM) to reduce registration variability;

[0061] S250. Band-pass filter the functional time series between 0.01 Hz and 0.1 Hz to obtain the target BOLD signal for each subject;

[0062] S260. Normalize the target BOLD signal, calculate the element-wise product for each region pair to characterize the instantaneous co-fluctuations between region pairs, and repeat the process for all edges to obtain the edge time series.

[0063] Exemplarily, when extracting the BOLD time series of each brain region ROI, the Data Processing Assistant for Resting-State fMRI (DPARSF) software package can be used to process the original functional image. During the use of DARTEL, the aforementioned preset standard space can adopt the Montreal Neurological Institute space.

[0064] In some embodiments, the aforementioned preprocessing of the diffusion tensor imaging data to obtain the initial structural brain network based on nodes may specifically include the following steps:

[0065] S310. For the diffusion tensor imaging data, apply the deterministic tractography method using the PANDA toolbox; after removing the skull cortex parameters, use the FACT fiber tracking algorithm for fiber tracking;

[0066] S320. Set the angular threshold to 45 degrees. If the fractional anisotropy (FA) is less than 0.2 or greater than 1, terminate the tracking process; after fiber tracking is completed, smooth the tracking results using a filter;

[0067] S330. Use the drawn white matter fibers as the edge weights of the brain network to obtain the initial structural brain network.

[0068] In the embodiments of the present application, it can be understood that the FACT fiber tracking algorithm is based on diffusion tensor imaging data and estimates the direction of nerve fibers by calculating the diffusion tensor in each voxel; the diffusion tensor describes the diffusion characteristics of water molecules in tissues. In nerve fibers, the diffusion of water molecules is easier along the fiber direction. Therefore, the fiber orientation can be inferred by analyzing the principal direction of the diffusion tensor.

[0069] In some embodiments, the aforementioned conversion of the initial structural brain network into an edge-centered structural brain network that captures brain structure lateralization features through adaptive construction and graph diffusion methods may specifically include the following steps:

[0070] S410. Adaptively construct the initial structural brain network to obtain a target structural brain network centered on edges.

[0071] S420. Apply heat diffusion to the target structural brain network to simulate the propagation of neurophysiological signals to analyze the lateralization abnormality in the brain structural network and obtain the edge-centered structural brain network.

[0072] In the embodiments of the present application, please refer to Figure 2 , after obtaining the initial structural brain network, the present application performs adaptive construction processing and asymmetric diffusion processing successively; it can be understood that the initial structural brain network is a node-centered brain network, and the present application converts the node-centered brain network into an edge-centered brain network through adaptive construction; and further captures and analyzes the lateralization abnormality in the brain structural network, which can improve the accuracy of brain network analysis.

[0073] In one example, the process of the foregoing adaptive construction satisfies the expression:

[0074]

[0075] where α represents a learnable parameter, FA_ij represents the fractional anisotropy of the white matter fiber bundle between brain region i and brain region j, and FA_uv represents the fractional anisotropy of the white matter fiber bundle between brain region u and brain region v;

[0076] corresponding to the local network between brain region i, brain region j, brain region u, and brain region v, and is a subset of the target structural brain network of.

[0077] In another example, the process corresponding to the foregoing heat diffusion satisfies the expression:

[0078]

[0079] In the formula, e is the natural constant, t represents a learnable parameter that can adaptively adjust the diffusion scale, L is the Laplacian matrix of, represents the edge-centered structural brain network.

[0080] In some embodiments, the foregoing fusion convolution module 150 is specifically configured to: use the self-attention mechanism to assign different weights to the effect information and structural information corresponding to the dynamic effect brain network and the edge-centered structural brain network for fusion to obtain a fusion network; adopt graph convolution and gated recurrent units to aggregate the spatio-temporal features of each edge in the fusion network to mine the dynamic correlation information of the fusion network and obtain the target information representing the brain nerve state.

[0081] In the embodiments of the present application, it can be understood that the dynamic effect brain network is a dynamic edge-centered effect brain network, which is used to represent dynamic information transmission and enhance the capture of edge dynamics; please refer to Figure 2 , in the brain network research corresponding to cognitive load assessment, the present application performs information fusion from two perspectives of effect and structure, and successively performs directed spatial feature extraction and temporal feature extraction to mine and analyze the dynamic correlation information of the fusion network.

[0082] In one example, the fusion process of the dynamic effect brain network and the edge-centered structure brain network satisfies the expression:

[0083]

[0084]

[0085] In the formula, W is a trainable weight matrix, b is a bias vector, q is a shared attention vector, and tanh represents an activation function; α e , α s respectively represent the weights corresponding to the dynamic effect brain network and the edge-centered structure brain network; represents the dynamic effect brain network, represents the edge-centered structure brain network, represents the fused network, and l represents the segmented time series index.

[0086] In some embodiments, the foregoing uses graph convolution and gated recurrent units to aggregate the spatio-temporal features of each edge in the fusion network, mine the dynamic correlation information of the fusion network, and obtain the target information representing the brain nerve state, which may specifically include the following steps:

[0087] S510. Through the bidirectional graph convolutional network Bi-GCN, use forward graph convolution and backward graph convolution to process the directed graph corresponding to the fusion network to capture bidirectional node dependencies and perform directed spatial feature extraction;

[0088] S520. Based on the bidirectional node dependencies, obtain the spatial representation through the fusion direction embedding;

[0089] S530. Based on the spatial representation, use the gated recurrent unit to model the temporal dependencies, and perform average pooling aggregation at the 0 time step to obtain the target information.

[0090] In some embodiments, the present application provides a multi-modal brain network analysis method, as Figure 3 shown, the multi-modal brain network analysis method may include the following steps:

[0091] S610. Obtain functional magnetic resonance imaging data and diffusion tensor imaging data through the acquisition module;

[0092] S620, preprocessing the functional magnetic resonance imaging data through a preprocessing module to obtain an edge time series; and preprocessing the diffusion tensor imaging data to obtain an initial structural brain network based on nodes;

[0093] S630, using the first processing module to perform fine-grained time series segmentation on the edge time series using a sliding window partitioning method, and converting the time domain signal in each segment into a high-order time-varying causal relationship through conditional Granger causality analysis to construct a dynamic effector brain network;

[0094] S640, converting the initial structural brain network into an edge-center structural brain network that captures the lateralization characteristics of brain structure through the second processing module through adaptive construction and graph diffusion method;

[0095] S650, by using the fusion convolution module based on the self-attention mechanism, the dynamic effect brain network and the edge center structure brain network are structurally fused and directed spatiotemporal graph convolution is performed to obtain target information representing the brain neural state;

[0096] S660, using a multi-layer perceptron to classify the target information into brain neural states through an output module, and outputting the classification results for visualization.

[0097] Figure 3 The shown S610-S660 has the functions and effects of implementing the aforementioned acquisition module 110, preprocessing module 120, first processing module 130, second processing module 140, fused convolution module 150 and output module 160, which will not be repeated here for the sake of brevity.

[0098] In some embodiments, the present application provides an electronic device, the structure diagram of the electronic device is as follows Figure 4 shown.

[0099] The electronic device may include a processor 710 and a memory 720 storing computer program instructions.

[0100] Specifically, the processor 710 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0101] Memory 720 may include a mass storage for data or instructions. By way of example and not limitation, memory 720 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In a suitable case, memory 720 may include removable or non-removable (or fixed) media. In a suitable case, memory 720 may be inside or outside the integrated gateway disaster recovery device. In a particular embodiment, memory 720 is a non-volatile solid-state memory.

[0102] Memory 720 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, generally, memory 720 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in a multi-modal brain network analysis method in the above embodiments.

[0103] Processor 710 reads and executes computer program instructions stored in memory 720 to implement a multi-modal brain network analysis method in the above embodiments.

[0104] In one example, the electronic device may further include a communication interface 730 and a bus 700. Among them, as Figure 4 shown, the processor 710, the memory 720, and the communication interface 730 are connected through the bus 700 and complete communication with each other.

[0105] The communication interface 730 is mainly used to implement communication between the modules, devices, units, and / or devices in the embodiments of the present application.

[0106] The bus 700 includes hardware, software, or both, and couples the components of the online data flow metering device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, the bus 700 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0107] In addition, embodiments of the present application may be implemented by providing a computer storage medium. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, a multi-modal brain network analysis method in the above embodiments is implemented.

[0108] It should be clear that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present application.

[0109] The functional blocks shown in the above structural block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present application are programs or code segments for performing the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or communication link. A "machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0110] It should also be noted that in the exemplary embodiments mentioned in this application, some methods or systems are described based on a series of steps or devices. However, this application is not limited to the order of the above steps. That is, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0111] The above has described various aspects of the present disclosure with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block in the flowchart and / or block diagram, and the combination of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more blocks of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It can also be understood that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can also be implemented by dedicated hardware that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0112] In summary, compared with the prior art, this application has the following beneficial effects:

[0113] 1. This application processes functional magnetic resonance imaging data into edge time series and processes diffusion tensor imaging data to obtain an initial structural brain network based on nodes. In the process of constructing a dynamic effect brain network, multiple time series segments are divided based on the edge time series, and the time-domain signals of each segment are converted into high-order time-varying causal relationships. In the process of constructing an edge-center structural brain network, the brain structure lateralization characteristics are captured. By fusing edge time-varying causal information and structural asymmetry characteristics, accurate analysis of the brain network structure can be achieved, solving the limitations of traditional node-centered methods in capturing dynamic information and structural asymmetry.

[0114] 2. Based on the self-attention mechanism, this application performs structural fusion and directed spatio-temporal graph convolution on the dynamic effect brain network and the edge-center structural brain network, can adaptively fuse the structural and causal characteristics from different edges, fully excavate the potential directed associations between edges, and further fully learn spatio-temporal characteristics through a graph neural network and a gated recurrent unit to realize brain network modeling to obtain target information representing the brain nerve state and perform accurate cognitive load assessment.

[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal brain network analysis system, characterized in that Including: An acquisition module, configured to acquire functional magnetic resonance imaging data and diffusion tensor imaging data; A preprocessing module, configured to preprocess the functional magnetic resonance imaging data to obtain edge time series; And preprocess the diffusion tensor imaging data to obtain an initial structural brain network based on nodes; A first processing module, configured to perform fine-grained temporal segmentation on the edge time series by using a sliding window partitioning method, and convert the time-domain signals in each segment into high-order time-varying causal relationships through conditional Granger causality analysis to construct a dynamic effect brain network; A second processing module, configured to convert the initial structural brain network into an edge-centered structural brain network capturing brain structure lateralization features through adaptive construction and graph diffusion methods; A fusion convolution module, configured to perform structural fusion and directed spatio-temporal graph convolution on the dynamic effect brain network and the edge-centered structural brain network based on a self-attention mechanism to obtain target information characterizing the brain nerve state; An output module, configured to classify the brain nerve state of the target information by using a multi-layer perceptron and output a classification result for visualization processing.

2. The multimodal brain network analysis system according to claim 1, wherein The preprocessing of the functional magnetic resonance imaging data to obtain edge time series includes: Processing the original functional image corresponding to the functional magnetic resonance imaging data, and extracting the BOLD time series of each brain region ROI; For the BOLD time series, perform slice time series correction by removing the first 10 time points, and perform motion correction, excluding subjects with head movement exceeding 2.5 mm or 2.5 degrees; Perform co-registration with the T1-weighted structural image by using statistical parametric mapping SPM, and perform segmentation of gray matter, white matter, and cerebrospinal fluid, and then apply DARTEL normalization to a preset standard space; Perform 4-mm full-width smoothing processing on the functional image through a half-maximum FWHM Gaussian to reduce registration variability; Perform band-pass filtering on the functional time series between 0.01 Hz and 0.1 Hz, and obtain the target BOLD signal for each subject; Normalize the target BOLD signal, calculate the element product of each region pair to characterize the instantaneous co-fluctuation between region pairs, and repeat the process for all edges to obtain edge time series.

3. The multimodal brain network analysis system according to claim 1, wherein, The preprocessing of the diffusion tensor imaging data to obtain an initial structural brain network based on nodes includes: For the diffusion tensor imaging data, apply a deterministic tractography method by using the PANDA toolbox; after removing the skull cortex parameters, perform fiber tracking by using the FACT fiber tracking algorithm; Determine the angle threshold to be 45 degrees, and if the fractional anisotropy FA is less than 0.2 or greater than 1, terminate the tracking process; after the fiber tracking is completed, perform smoothing processing on the tracking result by using a filter; Use the drawn white matter fibers as the edge weights of the brain network to obtain an initial structural brain network.

4. The multimodal brain network analysis system according to claim 1, characterized in that, The dynamic effect brain network is a dynamic edge-centered effect brain network and is used to characterize dynamic information transmission and enhance the capture of edge dynamics; the initial structural brain network is a node-centered brain network; Converting the initial structural brain network into an edge-centered structural brain network that captures brain structure lateralization features through an adaptive construction and graph diffusion method, including: An adaptive construction is performed on the initial structural brain network to obtain a target structural brain network centered on edges. Apply heat diffusion to the target structural brain network to simulate the propagation of neurophysiological signals, analyze the lateralization abnormalities in the brain structural network, and obtain the edge-centered structural brain network.

5. The multimodal brain network analysis system according to claim 4, wherein The process of the adaptive construction satisfies the expression: where α represents a learnable parameter, FA_ij represents the fractional anisotropy of the white matter fiber bundle between brain region i and brain region j, and FA_uv represents the fractional anisotropy of the white matter fiber bundle between brain region u and brain region v; The local network among corresponding brain regions i, j, u, and v, and it is a subset of the target structural brain network ; The process corresponding to the heat diffusion satisfies the expression: where e is the natural constant, t represents a learnable parameter that can adaptively adjust the diffusion scale, and L is the Laplacian matrix of representing the edge-centered structural brain network.

6. The multimodal brain network analysis system according to claim 1, wherein The fusion convolution module is specifically used for: Using a self-attention mechanism to assign different weights to the effect information and structural information corresponding to the dynamic effect brain network and the edge-centered structural brain network for fusion to obtain a fusion network; Adopting graph convolution and gated recurrent units to aggregate the spatio-temporal features of each edge in the fusion network, and mining the dynamic correlation information of the fusion network to obtain target information representing the brain nerve state; where the fusion process of the dynamic effect brain network and the edge-centered structural brain network satisfies the expression: Wherein, W is a trainable weight matrix, b is a bias vector, q is a shared attention vector, and tanh represents an activation function; α e , α s respectively represent the weights corresponding to the dynamic effect brain network and the limbic central structure brain network; represents the dynamic effect brain network, represents the limbic central structure brain network, represents the fused network, and l represents the segmented time series index.

7. The multimodal brain network analysis system according to claim 6, wherein The adopting graph convolution and gated recurrent units to aggregate the spatio-temporal features of each edge in the fusion network, and mining the dynamic correlation information of the fusion network to obtain target information representing the brain nerve state, including: Through a bidirectional graph convolutional network Bi-GCN, using forward graph convolution and backward graph convolution to process the directed graph corresponding to the fusion network to capture bidirectional node dependencies and perform directed spatial feature extraction; Based on the bidirectional node dependencies, obtaining a spatial representation through fusing direction embeddings; Based on the spatial representation, using a gated recurrent unit to model the temporal dependencies, and performing average pooling aggregation at time step 0 to obtain target information.

8. A multi-modal brain network analysis method, characterized in that, Including: Obtaining functional magnetic resonance imaging data and diffusion tensor imaging data through an acquisition module; Preprocessing the functional magnetic resonance imaging data through a preprocessing module to obtain edge time series; And preprocessing the diffusion tensor imaging data to obtain an initial structural brain network based on nodes; Using a sliding window partitioning method by a first processing module to perform fine-grained temporal segmentation on the edge time series, and converting the time-domain signals in each segment into high-order time-varying causal relationships through conditional Granger causality analysis to construct a dynamic effect brain network; Converting the initial structural brain network into an edge-centered structural brain network that captures brain structure lateralization features through an adaptive construction and graph diffusion method by a second processing module; Based on a self-attention mechanism through a fusion convolution module, performing structural fusion and directed spatio-temporal graph convolution on the dynamic effect brain network and the edge-centered structural brain network to obtain target information representing the brain nerve state; Using a multi-layer perceptron by an output module to classify the brain nerve state of the target information, and outputting a classification result for visualization processing.

9. An electronic device, characterized in that, Including: A processor, a memory, and a program stored on the memory and executable on the processor, where when the program is executed by the processor, it implements the multi-modal brain network analysis method as claimed in claim 8.

10. A computer-readable storage medium, characterized in that, A program or instructions are stored on the computer-readable storage medium, and when the program or instructions are executed by a processor, the multi-modal brain network analysis method described in claim 8 is implemented.

Images