Image processing method and system based on deep learning of hierarchical brain network graph convolution

Abstract

The present invention discloses an image processing method and system based on hierarchical brain network graph convolution deep learning, which relates to the technical fields of artificial intelligence and medical image processing, and includes: receiving multimodal image data and performing preprocessing; dividing the cerebral cortex into brain regions with high spatial resolution according to the results of nerve fiber bundle tracing; extracting deep image features, deep connection features, imaging genomics features and connectomics features of the partitions; constructing a hierarchical tree diagram of the brain network; establishing graph convolution kernels for aggregating spatial adjacent node information and graph convolution kernels for aggregating brain network connection node information respectively; establishing graph pooling and de-pooling based on the hierarchical brain network tree diagram; constructing a residual graph convolution neural network for graph classification; and constructing a UNet++ graph convolution neural network for node classification. The method realizes modeling of the topological structure of the hierarchical modularity of the brain network, and provides an effective technology for brain network analysis.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and medical image processing technology, and specifically to an image processing method and system based on hierarchical brain network graph convolution deep learning. Background Art

[0002] The brain network is a complex network system formed by interconnected neurons in the brain. Its function is to process and transmit information. It is the neurophysiological basis of functions such as cognition, movement and perception. The brain network has the characteristics of modularity and hierarchy. Modularity refers to the existence of relatively independent functional modules in the brain, which play an important role in specific functions (such as movement, vision and language, etc.) and are interconnected with other modules to form a complex information processing network. Hierarchy refers to the hierarchical structure of the brain network in terms of function and structure. This hierarchical structure enables the brain to efficiently realize information transmission and coordinate various cognitive functions.

[0003] Technologies such as magnetic resonance imaging (MRI), positron emission tomography (PET), electroencephalogram (EEG), and magnetoencephalography (MEG) can visualize brain networks in vivo and non-invasively. Depending on the construction method, brain networks can be divided into functional brain networks, structural brain networks, and causal brain networks. Functional brain networks are constructed by evaluating the temporal correlation of functional MRI (fMRI) blood oxygen level dependent (BOLD) signals, EEG or MEG signals between brain regions, reflecting the functional organization and activity patterns of the brain. Structural brain networks reflect the connections between brain regions by tracing fiber bundles using diffusion magnetic resonance imaging (dMRI), including diffusion tensor imaging (DTI), diffusion spectrum imaging (DSI), and high angular resolution diffusion imaging (HARDI). Causal brain networks establish causal connections between brain regions by analyzing the correlation between T1-weighted MRI (T1WI) cortical thickness and PET metabolism at the group level. Currently, brain networks are mostly constructed based on existing brain maps, using the regions of interest (ROI) of the brain maps as brain network nodes. However, the ROI of brain maps is usually based on anatomical structures, and the same ROI may contain multiple functional areas, making it difficult to deeply reflect the modular hierarchy of brain networks. Summary of the invention

[0004] In order to address the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide an image processing method and system based on hierarchical brain network graph convolution deep learning, to finely partition the brain based on brain network connections, and to establish a graph convolutional neural network to achieve a deep representation of the hierarchy of brain network modules.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: an image processing method based on hierarchical brain network graph convolution deep learning, the method comprising the following steps:

[0006] Receiving three-dimensional brain multimodal image data and performing preprocessing to obtain preprocessed three-dimensional brain multimodal image data, and segmenting and obtaining high-resolution cerebral cortex partitions according to the voxel fiber bundle tracking results of the gray-white matter interface of the preprocessed three-dimensional brain multimodal image data;

[0007] Perform representation learning on high-resolution cerebral cortex partitions to extract deep local imaging features and brain network connection features, and extract imaging omics and connectomics features. Generate a spatial adjacency graph of high-resolution partition ROI based on spatial adjacency relationships, and merge brain network nodes based on a greedy algorithm to generate a hierarchical brain network dendrogram.

[0008] A graph convolution kernel is established based on circular harmonic functions on the sphere to aggregate local node information of the spatial adjacency graph. A graph convolution kernel based on spherical harmonic functions is established to aggregate remote node information connected through the brain network. The aggregated nodes are pooled and unpooled according to the hierarchical brain network dendrogram to obtain a residual graph convolution neural network model for graph classification and a UNet++ graph convolution neural network model for node classification.

[0009] By processing images using the residual graph convolutional neural network model for graph classification and the UNet++ graph convolutional neural network model for node classification, a deep hierarchical representation of brain network modules is achieved.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of segmenting and obtaining high-resolution cerebral cortex partitions based on the gray-white matter interface voxel fiber bundle tracing results of the preprocessed three-dimensional brain multimodal imaging data:

[0011] The voxels on the gray-white matter interface surface are used as seed points for probabilistic fiber bundle tracing to establish a fiber bundle density map. A weighted undirected graph is established based on the similarity of the fiber bundle density map. The edge weight is:

[0012]

[0013] η The value is η 0 multiplied by the SSD of all adjacent pairs / d Euc Mean;

[0014] Use Dijkstra algorithm to calculate the geodesic distance between all point pairs in ROI and establish the geodesic distance matrix W;

[0015] Spectral clustering is performed based on the geodesic distance matrix W to obtain K high-resolution cerebral cortical partitions.

[0016] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of extracting the high-resolution partition ROI:

[0017] Taking the ROI on the gray-white matter interface surface as the seed point, the seed point region growing algorithm is used to project the partitioning result to the cortex and subcortical superficial white matter to obtain the Voronoi partition;

[0018] The Voronoi partitions of multimodal images are input into the convolutional autoencoder to extract local deep image features and image omics features;

[0019] Functional and structural brain networks were established based on gray matter Voronoi partitioning, and the deep connection features were obtained by inputting the autoencoder. Connectomics features were extracted and all the extracted features were combined into high-resolution partition ROI features.

[0020] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of generating a dendrogram of the hierarchical brain network:

[0021] The high-resolution cerebral cortex partition ROI is used as the leaf node of the hierarchical brain network dendrogram;

[0022] A weighted undirected graph is established based on the spatial adjacency relationship of node ROIs, and adjacent similar ROIs are continuously aggregated based on a greedy algorithm to construct a higher-level ROI;

[0023] Each layer of nodes is partitioned to establish functional brain networks and structural brain networks respectively, and a tree diagram of the hierarchical brain network is obtained.

[0024] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of aggregating the graph convolution kernel based on the circular harmonic function on the sphere:

[0025] The gray-white matter interface surface is topologically deformed into a spherical surface, and the distance between the vertices of the surface is kept unchanged during the deformation process;

[0026] Partition S the nodes on the sphere, and extract the spherical great circle C0 that passes through the center point of S and the north pole of the sphere;

[0027] Extract the great circle that passes through the center point S and the center point of the adjacent node N, and calculate the angle θ between it and C0 in the clockwise direction;

[0028] The graph convolution kernel on the ring is represented by a circular harmonic function:

[0029]

[0030] in and is a circular harmonic function, k=0,1,2,⋯,∞

[0031] set up h i For thei The local feature vector of a node, For the i The spatial neighboring nodes of a node, w 0 ’ is the weight coefficient of the node to be learned. The graph convolution of the circular harmonic function can aggregate the feature vectors of spatially adjacent nodes to the current node:

[0032] .

[0033] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of aggregating remote node information connected through the brain network by the graph convolution kernel based on spherical harmonics:

[0034] The gray-white matter interface surface is further deformed into a unit sphere;

[0035] For the connections within the ipsilateral cerebral hemisphere and between the cerebral hemispheres, two spherical functions are established respectively. and ;

[0036] set up g i For the i The brain network connection feature vector of nodes is represented by the graph convolution of the aggregated brain network connection node information:

[0037]

[0038] in, For the j The coordinates of the nodes on the unit sphere, θ and φ are the elevation and azimuth angles, respectively, e ij is the brain network connection between the current node and the jth node in the ipsilateral hemisphere, Indicates the contralateral hemisphere j The brain network of nodes is connected. and Represents the ipsilateral and contralateral cerebral hemispheres j The feature vector of a node, For Node i Its own weight coefficient;

[0039] The graph convolution kernel is further expressed using spherical harmonic functions as:

[0040]

[0041] in is the linear combination coefficient, Y l,m are spherical harmonics:

[0042]

[0043] .

[0044] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of pooling and de-pooling the aggregated nodes according to the hierarchical brain network dendrogram:

[0045] The pooling process merges network nodes into nodes at a higher level according to the hierarchical brain network dendrogram, while changing the brain network structure;

[0046] During the node merging process, the feature vectors in the nodes are pooled, and maximum pooling or average pooling can be selected;

[0047] De-pooling splits the network nodes into nodes at a lower level according to the dendrogram structure, and de-pools the feature vectors in the nodes, selecting de-maximum pooling or de-average pooling.

[0048] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the residual graph convolutional neural network model for graph classification is based on the graph convolution kernel C conv and S conv , graph pooling and fully connected layers to establish a residual graph convolutional neural network for graph classification;

[0049] The network uses ResNet as the basic structure, including 6 graph convolution layers, 3 graph pooling layers, 1 global average pooling layer and 1 fully connected layer. The Leaky ReLU function is used as the intermediate layer activation function, and the Sigmoid function or Softmax function is used as the output layer activation function.

[0050] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the UNet++ graph convolutional neural network model for node classification is based on the graph convolution kernel C conv and S conv , as well as graph pooling and unpooling operations to establish the UNet++ graph convolutional neural network;

[0051] The network uses ResNet as the backbone network, including 8 layers of graph convolution layers, 3 layers of graph pooling layers and 3 layers of upsampling layers, using dense skip connections and deep supervision mechanism.

[0052] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses an image processing system based on hierarchical brain network graph convolution deep learning, comprising:

[0053] A partitioning and segmentation module is used to receive and preprocess the three-dimensional brain multimodal image data to obtain the preprocessed three-dimensional brain multimodal image data, and segment and obtain high-resolution cerebral cortex partitions according to the voxel fiber bundle tracking results of the gray-white matter interface of the preprocessed three-dimensional brain multimodal image data;

[0054] The feature extraction module is used to perform representation learning on high-resolution cerebral cortex partitions to extract deep local imaging features and brain network connection features, and to extract imaging omics and connectomics features. It generates a spatial adjacency graph of high-resolution partition ROI based on spatial adjacency relationships, and merges brain network nodes based on a greedy algorithm to generate a tree diagram of a hierarchical brain network.

[0055] The model generation module is used to establish a graph convolution kernel based on circular harmonic functions on a sphere, aggregate local node information of a spatial adjacency graph, and establish a graph convolution kernel based on spherical harmonic functions to aggregate remote node information connected through a brain network. The aggregated nodes are pooled and unpooled according to a hierarchical brain network dendrogram to obtain a residual graph convolution neural network model for graph classification and a UNet++ graph convolution neural network model for node classification.

[0056] The image processing module is used to process images according to the residual graph convolutional neural network model for graph classification and the UNet++ graph convolutional neural network model for node classification, so as to achieve a deep characterization of the hierarchy of brain network modules. Beneficial effects of the present invention:

[0057] The present invention establishes a hierarchical tree diagram structure of a brain network, and establishes a graph convolutional neural network based on the tree diagram, providing an effective deep learning method for in-depth analysis of the module hierarchy of the brain network, and aggregates the spatially adjacent node information in the brain network. This method solves the problem that traditional convolution methods are not suitable for highly complex non-convex cortexes, and has the advantages of small parameters and easy training. It aggregates the node information of remote connections in the brain network, and generalizes the convolution operation to the brain network structure in non-Euclidean space. A residual graph convolutional neural network for graph classification and a UNet++ graph convolutional neural network model for node classification are established, providing a new tool for brain network analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 It is a schematic flow chart of the method of the present invention;

[0060] Figure 2 It is a schematic diagram of high-resolution cerebral cortex partitioning, feature extraction and hierarchical brain network dendrogram construction in an embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of a graph convolution kernel for aggregating spatial adjacent node information in an embodiment of the present invention;

[0062] Figure 4 is a schematic diagram of the expansion of the spherical harmonic function of the graph convolution kernel for aggregating brain network connection node information in an embodiment of the present invention;

[0063] Figure 5 is a schematic diagram of graph pooling based on a hierarchical brain network dendrogram in an embodiment of the present invention;

[0064] Figure 6 It is a structural diagram of a residual graph convolutional neural network for graph classification in an embodiment of the present invention;

[0065] Figure 7 It is a structural diagram of a UNet++ graph convolutional neural network for node classification in an embodiment of the present invention;

[0066] Figure 8 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Embodiment 1:

[0069] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0070] Graph Convolution Neural Networks (GCNNs) methods are divided into two categories, spectral domain-based methods and spatial domain-based methods. Spectral domain-based methods define graph convolution by introducing filters from the perspective of graph signal processing, where the graph convolution operation is interpreted as removing noise from the graph signal. Spatial domain-based methods represent graph convolution as aggregating feature information from neighbors.

[0071] like Figure 1 As shown, the image processing method based on hierarchical brain network graph convolution deep learning includes the following steps:

[0072] S101: receiving three-dimensional brain multimodal image data and performing preprocessing to obtain preprocessed three-dimensional brain multimodal image data, and segmenting and obtaining high-resolution cerebral cortex partitions according to the voxel fiber bundle tracking results of the gray-white matter interface of the preprocessed three-dimensional brain multimodal image data;

[0073] The preprocessing process includes removing non-brain tissue, gray-white matter segmentation, registration, fiber bundle tracking, extracting BOLD signals, etc. Specifically, the present invention is further described below through examples:

[0074] The subjects' T1WI, fluid attenuated inversion recovery MRI (FLAIR), dMRI and PET images were acquired in the same body position, and the acquired DICOM format images were converted into easy-to-process NIfTI format or NPZ format;

[0075] T1WI and FLAIR preprocessing steps include head motion correction, removal of non-brain tissue, pixel value normalization, spatial standardization, gray-white matter segmentation, gray-white matter interface surface mesh reconstruction, and topology correction;

[0076] fMRI preprocessing steps, removal of the first 10 time points, temporal layer correction, head motion correction, registration, smoothing, removal of linear drift, covariate regression, and bandpass filtering;

[0077] dMRI preprocessing steps include head motion correction, eddy current correction, registration, fitting water molecule diffusion model, and probabilistic fiber bundle tracking;

[0078] PET preprocessing steps include head motion correction, registration, partial volume effect correction, and pixel value normalization.

[0079] The process of segmenting and obtaining high-resolution cerebral cortical partitions based on the gray-white matter interface voxel fiber tract tracing results of the preprocessed three-dimensional brain multimodal imaging data:

[0080] The voxels on the gray-white matter interface surface are used as seed points for probabilistic fiber bundle tracing, and a fiber bundle density map is established. A weighted undirected graph is established based on the similarity of the fiber bundle density map, and the connection vertices are set v 1 and v The side of 2 is e ,by v 1 and v 2 is the seed point for fiber bundle tracing and the fiber density maps obtained are TDI 1 and TDI 2, TDI 1 and TDI The sum of squares of 2 is SSD ( TDI 1 , TDI 2),v 1 and v The Euclidean distance between 2 is d Euc ( v 1, v 2), η is the weighting coefficient, then the edge e Weight w e ( v 1, v 2) is:

[0081]

[0082] η The value is η 0 multiplied by the SSD of all adjacent pairs / d Euc Mean, preferred, η 0 = 0.1;

[0083] Use Dijkstra algorithm to calculate the geodesic distance between all point pairs in ROI and establish the geodesic distance matrix W;

[0084] Spectral clustering is performed based on the geodesic distance matrix W to obtain K high-resolution cerebral cortical partitions.

[0085] like Figure 2 , use Dijkstra algorithm to calculate the geodesic distance between all pairs of points in the ROI, for any vertex in the ROI v The specific process

[0086] Initialize the distance of all vertices in the ROI to the maximum floating point number, mark them as unvisited, and initialize the vertices v The distance is 0, marked as visited;

[0087] Place the vertices v Insert priority queue Q ;

[0088] pop up Q The front vertex u ,right u All adjacent vertices of n , if d( u ) + w e ( u , n )< d ( n ),Will n insert Q tail, mark as visited, distance d ( n ) is updated to d(u ) + w e ( u , n );

[0089] Repeat steps until Q is empty, d( u )for v arrive u The geodesic distance is recorded in the geodesic distance matrix W , W i,j For the i Vertex to j The geodesic distance of the vertices;

[0090] Based on the geodesic distance matrix W Perform spectral clustering, assuming D for W The degree matrix of W The Laplacian matrix of :

[0091] L = D - W

[0092] Laplacian Matrix L To standardize:

[0093] L sym = D -1 / 2 LD -1 / 2

[0094] right L sym Perform feature decomposition and take the largest k The eigenvalues ​​corresponding to the eigenvectors u 1, u 2,⋯, u k , as column vectors to form a matrix R ;

[0095] right R The row vector of is clustered by K means, then i The vertex corresponds to i The clustering results of the rows;

[0096] k The value is K Multiply the volume of the structural partition / total cortical volume, preferably, K = 2048.

[0097] S102: Perform representation learning on high-resolution cerebral cortex partitions to extract deep local imaging features and brain network connection features, and extract imaging omics and connectomics features, generate a spatial adjacency graph of high-resolution partition ROI based on spatial adjacency relationships, and merge brain network nodes based on a greedy algorithm to generate a hierarchical brain network dendrogram;

[0098] The extraction process of high-resolution partition ROI:

[0099] Taking the partition ROI on the gray-white matter interface as the seed point, the cluster segmentation results were mapped to the cortical area using a multi-seed point region growing algorithm, and then expanded 2 mm into the white matter area to obtain the Voronoi partition of the cortical gray matter and the superficial white matter under the cortex;

[0100] Inputting the T1WI, FLAIR and PET images of the cortex Voronoi partition and the dMRI images of the white matter Voronoi partition into a convolutional autoencoder for representation learning, preferably, the convolutional autoencoder includes a 3-layer VGG module, and taking the latent features of the convolutional autoencoder as local depth image features;

[0101] Extract the radiomic features of the subregions, including morphological features such as cortical thickness, sulcal depth, and cortical curvature, grayscale features such as T1WI, FLAIR, and PET signal mean, fMRI local consistency and low-frequency fluctuation amplitude, dMRI fractional anisotropy and mean diffusivity, as well as texture features and wavelet features;

[0102] Functional brain network and structural brain network were established with cortical Voronoi partitions as nodes. The weight of the functional brain network edge was the Pearson correlation of the node BOLD signal, and Fisher Z transformation was performed. The weight of the structural brain network edge was the fiber bundle connection density:

[0103]

[0104] |S ij |To connect i and j The number of fiber bundle curves in the cortical Voronoi partition, g i and g j is the number of partition voxels; the connection vector is input into the autoencoder, and the latent features of the autoencoder are taken as the deep connection features;

[0105] For functional brain networks and structural brain networks, the connectomic features of nodes were extracted, including node strength, node clustering coefficient, rich club coefficient, assortativity coefficient, average shortest path length of nodes, betweenness centrality, and eigenvector centrality.

[0106] The extracted ROI local deep image features, imaging omics features, deep connection features and connectomics features are spliced ​​and stored in the corresponding network nodes.

[0107] The steps for constructing a hierarchical brain network dendrogram include:

[0108] The high-resolution cerebral cortex partition ROI is used as the leaf node of the hierarchical brain network dendrogram;

[0109] According to the spatial adjacency relationship of leaf node ROI, a weighted undirected graph G is established. S , G S The vertex of is a leaf node ROI; if and only if two ROIs are adjacent in space, their corresponding G S There are edges between vertices, and the weight of the edge is the cosine similarity of the feature vector stored in the leaf node ROI;

[0110] Aggregate node ROI based on greedy algorithm:

[0111] Select the node with the smallest area to join the queue Q ;

[0112] Pop Queue Q The front node is used as the center to search for the other three unmerged nodes adjacent to it, so that the sum of the shortest distances of the four nodes in the spatial adjacency graph is minimized;

[0113] The selected four nodes are merged into nodes of a higher-level brain network, and the feature vectors are average-pooled;

[0114] Search for unmerged nodes that are spatially adjacent to the four merged nodes, and add them to the queue in ascending order. Q ;

[0115] Iterate and repeat steps until the queue Q is empty;

[0116] Construct a new level of graph based on the spatial adjacency relationship of the merged ROIs , l Indicates l Merge again, preferably, merge and construct levels with ROI numbers of 512, 128, and 32 ROIs;

[0117] For each layer of node partitions, functional brain networks and structural brain networks were established respectively. The functional brain network connectivity was the Pearson correlation after Fisher Z transformation between node BOLD signals, and the structural brain network connectivity was the nerve fiber connection density between nodes.

[0118] S103: Establishing a graph convolution kernel based on the circular harmonic function on the sphere, aggregating the local node information of the spatial adjacency graph, and establishing a graph convolution kernel based on the spherical harmonic function, aggregating the remote node information connected through the brain network, and performing pooling and de-pooling on the aggregated nodes according to the hierarchical brain network dendrogram to obtain a residual graph convolution neural network model for graph classification and a UNet++ graph convolution neural network model for node classification;

[0119] The graph convolution kernel for aggregating spatial adjacent node information specifically includes:

[0120] Spatial adjacency graph formed by node partition ROI , establish a graph convolution kernel based on circular harmonic function to aggregate the information of spatially adjacent ROIs;

[0121] The gray-white matter interface surface is topologically deformed into a spherical surface, and the distance between the surface vertices is kept unchanged during the deformation process;

[0122] For the node partition S on the sphere, extract the spherical great circle C0 passing through the center point of S and the North Pole of the sphere; extract the great circle passing through the center point of S and the center point of the adjacent node N, and calculate the angle θ between it and C0 in the clockwise direction;

[0123] Define the graph convolution coefficient of the adjacent nodes as a function of the azimuth angle θ, ignoring the spatial distance between the adjacent nodes and the central node, then the graph convolution kernel can be expressed as a function defined on a ring;

[0124] Using circular harmonic functions as orthogonal basis, the graph convolution kernel on the ring w To express:

[0125]

[0126] in and is a circular harmonic function, k=0,1,2,⋯,∞; in order to reduce the amount of training parameters, preferably, a circular harmonic function with k≤4 is used;

[0127] set up h i For the i The local feature vector of a node, For the i The spatial neighboring nodes of a node, w 0 ’ is the weight coefficient of the node to be learned. The graph convolution of the circular harmonic function can aggregate the feature vectors of spatially adjacent nodes to the current node:

[0128]

[0129] like Figure 3As shown in the figure, this convolution kernel is similar to the 2D image convolution kernel, which aggregates information from spatially adjacent nodes (blue triangles) to the central node (red square); through multiple layers of convolution, information within a certain neighborhood can be perceived (green disk).

[0130] The graph convolution kernel that aggregates the information of brain network connection nodes includes:

[0131] The gray-white matter interface surface is further deformed into a unit sphere;

[0132] For the connections within the ipsilateral cerebral hemisphere and between the cerebral hemispheres, two spherical functions f are established respectively. intra (θ, φ) and f inter (θ,φ); let g i For the i The brain network connection feature vector of nodes is a vector, then the graph convolution that aggregates the information of brain network connection nodes can be expressed as:

[0133]

[0134] in, For the j The coordinates of the nodes on the unit sphere, θ and φ are the elevation and azimuth angles, respectively, e ij is the brain network connection between the current node and the jth node in the ipsilateral hemisphere, Indicates the contralateral hemisphere j The brain network of nodes is connected. and Represents the ipsilateral and contralateral cerebral hemispheres j The feature vector of a node, For Node i Its own weight coefficient;

[0135] like Figure 4 As shown, any square integrable function f(θ,φ) defined on the unit sphere can be expressed as a linear combination of spherical harmonic functions, namely:

[0136]

[0137] is the linear combination coefficient, Y l,m are spherical harmonics:

[0138]

[0139] in,

[0140]

[0141] Convolution kernel S defined on the sphere conv (g i ) can be expressed as a linear combination of spherical harmonics,

[0142]

[0143] To reduce the amount of calculation, it is preferred to use the highest order l = 4 spherical harmonics, and the constraint Make the range of spherical functions the real number domain;

[0144] Graph convolution operations are performed on functional brain networks and structural brain networks respectively.

[0145] Graph pooling and unpooling based on hierarchical brain network dendrogram, including:

[0146] like Figure 5 As shown, the graph pooling process merges network nodes into nodes at a higher level according to the hierarchical brain network dendrogram, while changing the brain network structure;

[0147] During the node merging process, the feature vectors in the nodes are pooled, and maximum pooling or average pooling can be selected;

[0148] De-pooling splits the network nodes into nodes at a lower level according to the tree structure, and de-pools the feature vectors in the nodes. De-maximum pooling or de-average pooling can be selected.

[0149] S104: Process the image based on the residual graph convolutional neural network model for graph classification and the UNet++ graph convolutional neural network model for node classification to achieve a deep representation of the hierarchy of brain network modules.

[0150] Furthermore, the residual graph convolutional neural network for graph classification specifically includes:

[0151] like Figure 6 As shown, based on the graph convolution kernel C conv and S conv , graph pooling and fully connected layers to establish a residual graph convolutional neural network for graph classification;

[0152] The network uses the residual neural network (ResNet) as the basic structure, which includes 6 graph convolution layers, 3 graph pooling layers, 1 global average pooling layer and 1 fully connected layer. It uses the Leaky ReLU function as the intermediate layer activation function and the Sigmoid function or Softmax function as the output layer activation function.

[0153] The graph convolutional layer includes C conv and S conv Two convolution operations are performed on the feature vectors h and g stored in the nodes respectively.

[0154] Furthermore, the UNet++ graph convolutional neural network for node classification specifically includes:

[0155] like Figure 7 As shown, based on the graph convolution kernel C conv and S conv , as well as graph pooling and unpooling operations to establish the UNet++ graph convolutional neural network;

[0156] The network uses ResNet as the backbone network, which includes 8 graph convolution layers, 3 graph pooling layers and 3 upsampling layers, and uses dense skip connections and deep supervision mechanisms.

[0157] Embodiment 2: Figure 8 As shown, the image processing system based on hierarchical brain network graph convolution deep learning is characterized by comprising:

[0158] A partitioning and segmentation module 11 is used to receive and preprocess the three-dimensional brain multimodal image data to obtain the preprocessed three-dimensional brain multimodal image data, and segment and obtain high-resolution cerebral cortex partitions according to the voxel fiber bundle tracking results of the gray-white matter interface of the preprocessed three-dimensional brain multimodal image data;

[0159] The feature extraction module 12 is used to perform representation learning on the high-resolution cerebral cortex partitions to extract deep local image features and brain network connection features, and to extract imaging omics and connectomics features, generate a spatial adjacency graph of the high-resolution partition ROI based on the spatial adjacency relationship, and merge brain network nodes based on a greedy algorithm to generate a tree diagram of the hierarchical brain network;

[0160] The model generation module 13 is used to establish a graph convolution kernel based on a circular harmonic function on a sphere, aggregate the local node information of the spatial adjacency graph, and establish a graph convolution kernel based on a spherical harmonic function to aggregate the remote node information connected through the brain network, and perform pooling and de-pooling on the aggregated nodes according to the hierarchical brain network dendrogram to obtain a residual graph convolution neural network model for graph classification and a UNet++ graph convolution neural network model for node classification;

[0161] The image processing module 14 is used to process the image according to the residual graph convolutional neural network model for graph classification and the UNet++ graph convolutional neural network model for node classification, so as to realize the in-depth representation of the hierarchy of brain network modules.

[0162] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0163] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0164] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0165] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. An image processing method based on hierarchical brain network graph convolution deep learning, characterized in that: The method comprises the following steps: Receiving three-dimensional brain multimodal image data and performing preprocessing to obtain preprocessed three-dimensional brain multimodal image data, and segmenting and obtaining high-resolution cerebral cortex partitions according to the voxel fiber bundle tracking results of the gray-white matter interface of the preprocessed three-dimensional brain multimodal image data; Perform representation learning on high-resolution cerebral cortex partitions to extract deep local imaging features and brain network connection features, and extract imaging omics and connectomics features. Generate a spatial adjacency graph of high-resolution partition ROI based on spatial adjacency relationships, and merge brain network nodes based on a greedy algorithm to generate a hierarchical brain network dendrogram. A graph convolution kernel is established based on circular harmonic functions on the sphere to aggregate local node information of the spatial adjacency graph. A graph convolution kernel based on spherical harmonic functions is established to aggregate remote node information connected through the brain network. The aggregated nodes are pooled and unpooled according to the hierarchical brain network dendrogram to obtain a residual graph convolution neural network model for graph classification and a UNet++ graph convolution neural network model for node classification. By processing images using the residual graph convolutional neural network model for graph classification and the UNet++ graph convolutional neural network model for node classification, a deep hierarchical representation of brain network modules is achieved.

2. The image processing method based on hierarchical brain network graph convolution deep learning according to claim 1 is characterized in that: The process of segmenting and obtaining high-resolution cerebral cortical partitions based on the gray-white matter interface voxel fiber bundle tracing results of the preprocessed three-dimensional brain multimodal imaging data: The voxels on the gray-white matter interface surface are used as seed points for probabilistic fiber bundle tracing, and a fiber bundle density map is established. A weighted undirected graph is established based on the similarity of the fiber bundle density map, and the connection vertices are set v 1 and v The side of 2 is e ,by v 1 and v 2 is the seed point for fiber bundle tracing and the fiber density maps obtained are TDI 1 and TDI 2, TDI 1 and TDI The sum of squares of 2 is SSD ( TDI 1 , TDI 2), v 1 and v The Euclidean distance between 2 is d Euc ( v 1, v 2), η is the weighting coefficient, then the edge e Weight w e ( v 1, v 2) is: η The value is η 0 multiplied by the SSD of all adjacent pairs / d Euc Mean; Use Dijkstra algorithm to calculate the geodesic distance between all point pairs in ROI and establish the geodesic distance matrix W ; Based on the geodesic distance matrix W Perform spectral clustering and obtain K High-resolution parcellation of the cerebral cortex.

3. The image processing method based on hierarchical brain network graph convolution deep learning according to claim 1 is characterized in that: The extraction process of the high-resolution partition ROI: Taking the ROI on the gray-white matter interface surface as the seed point, the seed point region growing algorithm is used to project the partitioning result to the cortex and subcortical superficial white matter to obtain the Voronoi partition; The Voronoi partitions of multimodal images are input into the convolutional autoencoder to extract local deep image features and image omics features; Functional and structural brain networks were established based on gray matter Voronoi partitioning, and the deep connection features were obtained by inputting the autoencoder. Connectomics features were extracted and all the extracted features were combined into high-resolution partition ROI features.

4. The image processing method based on hierarchical brain network graph convolution deep learning according to claim 1, characterized in that: The generation process of the tree diagram of the hierarchical brain network: The high-resolution cerebral cortex partition ROI is used as the leaf node of the hierarchical brain network dendrogram; A weighted undirected graph is established based on the spatial adjacency relationship of node ROIs, and adjacent similar ROIs are continuously aggregated based on a greedy algorithm to construct a higher-level ROI; Each layer of nodes is partitioned to establish functional brain networks and structural brain networks respectively, and a tree diagram of the hierarchical brain network is obtained.

5. The image processing method based on hierarchical brain network graph convolution deep learning according to claim 1, characterized in that: The process of aggregating the graph convolution kernel based on the circular harmonic function on the sphere: The gray-white matter interface surface is topologically deformed into a spherical surface, and the distance between the vertices of the surface is kept unchanged during the deformation process; Partitioning nodes on the sphere S , extract through S Great circle of the sphere with the center point and the North Pole C 0; Extraction through S Center and adjacent nodes N The great circle with the center point is calculated in a clockwise direction. C Angle of 0 θ ; Graph convolution kernel on a ring w Using circular harmonic functions: in and is a circular harmonic function, k=0,1,2,⋯,∞ set up h i For the i The local feature vector of a node, For the i The spatial neighboring nodes of a node, w 0 ’ is the weight coefficient of the node to be learned. The graph convolution of the circular harmonic function can aggregate the feature vectors of spatially adjacent nodes to the current node: 。 6. The image processing method based on hierarchical brain network graph convolution deep learning according to claim 1, characterized in that: The graph convolution kernel based on spherical harmonics aggregates the information of remote nodes connected through the brain network: The gray-white matter interface surface is further deformed into a unit sphere; For the connections within the ipsilateral cerebral hemisphere and between the cerebral hemispheres, two spherical functions are established respectively. and ; set up g i For the i The brain network connection feature vector of nodes is represented by the graph convolution of the aggregated brain network connection node information: in, For the j The coordinates of the nodes on the unit sphere, θ and φ are the elevation and azimuth angles, respectively, e ij is the brain network connection between the current node and the jth node in the ipsilateral hemisphere, Indicates the contralateral hemisphere j The brain network of nodes is connected. and Represents the ipsilateral and contralateral cerebral hemispheres j The feature vector of each node, For Node i Its own weight coefficient; The graph convolution kernel is further expressed using spherical harmonic functions as: in is the linear combination coefficient, Y l,m are spherical harmonics: 。 7. The image processing method based on hierarchical brain network graph convolution deep learning according to claim 1, characterized in that: The process of pooling and unpooling the aggregated nodes according to the hierarchical brain network dendrogram: The pooling process merges network nodes into nodes at a higher level according to the hierarchical brain network dendrogram, while changing the brain network structure; During the node merging process, the feature vectors in the nodes are pooled, and maximum pooling or average pooling can be selected; De-pooling splits the network nodes into nodes at a lower level according to the dendrogram structure, and de-pools the feature vectors in the nodes, selecting de-maximum pooling or de-average pooling.

8. The image processing method based on hierarchical brain network graph convolution deep learning according to claim 1, characterized in that: The residual graph convolutional neural network model for graph classification is based on the graph convolution kernel C conv and S conv , graph pooling and fully connected layers to establish a residual graph convolutional neural network for graph classification; The network is based on the ResNet structure, which includes 6 graph convolution layers, 3 graph pooling layers, 1 global average pooling layer and 1 fully connected layer. The Leaky ReLU function is used as the intermediate layer activation function, and the Sigmoid function or Softmax function is used as the output layer activation function.

9. The image processing method based on hierarchical brain network graph convolution deep learning according to claim 1, characterized in that: The UNet++ graph convolutional neural network model for node classification is based on the graph convolution kernel C conv and S conv , as well as graph pooling and unpooling operations to establish the UNet++ graph convolutional neural network; The network uses ResNet as the backbone network, including 8 layers of graph convolution layers, 3 layers of graph pooling layers and 3 layers of upsampling layers, using dense skip connections and deep supervision mechanism.

10. An image processing system based on hierarchical brain network graph convolution deep learning, characterized in that: include: A partitioning and segmentation module is used to receive and preprocess the three-dimensional brain multimodal image data to obtain the preprocessed three-dimensional brain multimodal image data, and segment and obtain high-resolution cerebral cortex partitions according to the voxel fiber bundle tracking results of the gray-white matter interface of the preprocessed three-dimensional brain multimodal image data; The feature extraction module is used to perform representation learning on high-resolution cerebral cortex partitions to extract deep local imaging features and brain network connection features, and to extract imaging omics and connectomics features. It generates a spatial adjacency graph of high-resolution partition ROI based on spatial adjacency relationships, and merges brain network nodes based on a greedy algorithm to generate a tree diagram of a hierarchical brain network. The model generation module is used to establish a graph convolution kernel based on circular harmonic functions on a sphere, aggregate local node information of a spatial adjacency graph, and establish a graph convolution kernel based on spherical harmonic functions to aggregate remote node information connected through a brain network. The aggregated nodes are pooled and unpooled according to a hierarchical brain network dendrogram to obtain a residual graph convolution neural network model for graph classification and a UNet++ graph convolution neural network model for node classification. The image processing module is used to process images based on the residual graph convolutional neural network model for graph classification and the UNet++ graph convolutional neural network model for node classification, so as to achieve a deep representation of the hierarchy of brain network modules.

Images