An Adaptive Multi-channel Graph Convolutional Network for Joint Graph Contrastive Learning

Abstract

The present invention discloses an adaptive multi-channel graph convolutional network method for joint graph contrast learning. Multimodal big data is often composed of a variety of data in different structural forms, and these data are often complementary, mutually verified, and fusible. How to accurately and efficiently extract complementary information between multimodal data is the main goal of multimodal research. However, most of the current methods for multimodal fusion only focus on the complementary information between multimodalities, but often ignore the specific information under a single modality. In addition, how to use graph convolutional networks to extract rich and discriminative expressions from multimodalities has been rarely studied. To this end, the present invention discloses an adaptive multi-channel graph convolutional network method for joint graph contrast learning. This framework is based on brain network research and can not only mine the temporal and spatial information of brain networks, but also effectively fuse the unique and shared features of multimodalities.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning, and in particular relates to graph-based convolutional networks and graph contrast learning. Background Art

[0002] The brain is the most complex organ in the human biological system, and brain network technology is currently an important method for studying brain diseases, providing a powerful representation of human brain interaction patterns. In order to conduct non-invasive studies on brain function, a variety of brain functional imaging methods have been used. In neuroscience, brain networks can usually be represented by different structural patterns (such as diffusion tensor imaging DTI) and functional patterns (such as resting state functional magnetic resonance imaging rs-fMRI). These network data represent the complex structure of human brain connections. For example, in rs-fMRI networks, edge connections represent the correlation between brain regions and functional stimuli, while DTI can reveal the physical connections between functionally related gray matter regions. Therefore, they are of great research value for understanding the biological mechanisms of brain function. Previous studies have shown that different patterns of brain networks convey complementary information to each other, and the fusion of multiple patterns can lead to continuous improvements in brain analysis, so that more effective features can be extracted to improve the classification and diagnosis of brain diseases.

[0003] In addition, recent studies have found that even in a resting state, the functional connections of the brain are not static, but show complex spontaneous spatiotemporal changes. These changes in brain connections are related to neural activity and, to a certain extent, to changes in cognition or alertness. In addition, studies have shown that the time-varying properties of these functional connections can serve as biological targets for disease diagnosis, and are of great significance in studying the spatiotemporal changes in brain activity in disease diagnosis. Therefore, in recent years, dynamic functional network analysis has been used in the diagnosis of various brain diseases. It not only provides new insights into the dynamic spatiotemporal organization of resting brain activity, but also captures changes in the functional connectivity group caused by disease pathology.

[0004] Recently, with the development of deep learning in natural language and natural images, more and more studies have also introduced deep learning into medical image analysis. Traditional convolutional neural networks often obtain sample features by translating the convolution kernel on the graph, but this method is only suitable for obtaining features in Euclidean space. For some irregular graph structures, such as brain, social network, protein structure, effective results cannot be obtained. However, as a powerful representation of graph data, graph convolutional network (GCN) provides a fusion strategy of topological structure and node features to learn node embedding, which is very suitable for studying brain networks that usually have complex nonlinear structures. The working principle of GCN is to make the nodes on the graph aggregate feature information from the topological neighbors in each convolutional layer. In this way, the feature information is propagated to the node embedding through the network topology, and then the learned node embedding is used for classification tasks. GCN research on multimodal brain networks usually regards brain networks under different modalities as multiple views of the brain and extracts complementary features between views. However, there are still some problems in the existing use of GCN to study multimodal brain networks:

[0005] (1) Most studies only focus on the common features among multiple modalities during multimodal fusion, while ignoring the discriminative features of each modality. However, the specific features of each modality and the common features between modalities are complementary, and adaptive fusion is needed to obtain deeper relevant information in order to better describe the brain network.

[0006] (2) The embedded expression obtained by encoding the brain network using GCN may not necessarily reflect the richness of the original brain network data and structure, especially during the encoding process, as the number of network layers increases, information is easily lost.

[0007] Therefore, how to effectively obtain the features of multimodal brain networks and adaptively combine them has become a major difficulty in this field. To address the above problems, we proposed an adaptive multi-channel graph convolutional network with joint graph contrast learning, which can be used for the diagnosis of individual brain diseases. Summary of the invention

[0008] Purpose of the invention: The automatic recognition technology of medical images plays an important auxiliary role in the work of doctors. The invention and application of a good technology in related fields can greatly improve the diagnosis level of hospitals. At present, the automatic recognition method of brain images cannot effectively utilize the information of functional magnetic resonance imaging (FMRI) and diffusion tensor imaging (DTI), and the dynamic characteristics of brain networks are often ignored in the process of constructing brain networks. In addition, the features usually extracted from brain networks cannot reflect the richness of the original brain network data and structure. Therefore, based on the above problems, we propose an adaptive multi-channel graph convolutional network with joint graph contrast learning, in order to extract the discriminative and complementary features of multimodal brain networks, so as to better represent the brain for the diagnosis of various brain diseases.

[0009] Technical solution: To achieve the above-mentioned purpose, the present invention proposes an adaptive multi-channel graph convolutional network for joint graph contrast learning. First, we use a sliding time window to divide the time series and construct corresponding fMRI images and DTI images in each window. Then we use an adaptive multi-channel graph convolutional network to extract spatial features in each window. Finally, we use LSTM to explore the temporal variability of brain regions between windows as the temporal features of the brain. We extracted the spatial and temporal features of each subject's brain network and used them for the diagnosis of individual subject brain diseases. The specific technical method includes the following steps:

[0010] Step 1: Construction of multimodal images: Use overlapping fixed-size time windows to divide the rs-fMRI corresponding to each brain area of ​​interest, and construct fMRI-based and DTI-based images in each window, where the adjacency matrix of each image is the functional connection brain network and structural connection brain network corresponding to each window, and the feature matrix is ​​the time series information of each window.

[0011] Step 2: Adaptive multi-channel graph convolutional network to extract spatial information features of multimodal brain networks: The multi-channel graph convolutional network consists of a three-module graph convolutional network, two of which are GCN encoders used to extract the most discriminative features of a single modality, and the other module's GCN encoder is used to obtain multi-modal shared features. Finally, the attention mechanism is used to fuse the features.

[0012] Step 3: Graph contrast learning optimizes features: In order to ensure that we can obtain the shared features of multiple modalities and the most discriminative features of each modality, we combine graph contrast learning to optimize the extracted features

[0013] Step 4: Extracting the temporal characteristics of the brain network: Using a long short-term memory artificial neural network (LSTM) to explore the temporal variability of brain regions between each window and use it as the temporal characteristics of the brain network

[0014] Step 5: Classify the output features using a multi-layer perceptron to obtain the final recognition result

[0015] Beneficial effects: The present invention aims at the problem of brain disease diagnosis and proposes an adaptive multi-channel graph convolutional network with joint graph contrast learning. Compared with the prior art, the present invention has the following technical effects:

[0016] 1) This invention combines multimodal information and effectively mines the temporal and spatial characteristics of dynamic brain networks, making the feature representation of each sample discriminative, thereby improving classification performance

[0017] 2) The multi-channel graph convolutional network designed in the present invention can effectively combine the complementary features of multimodal fusion and the most specific features of each modality. Therefore, compared with traditional multimodal fusion methods, more spatial features of each sample can be mined and utilized.

[0018] 3) The present invention adopts a graph contrast learning approach to maximize the mutual information between the graph embedding expression after the encoder and the original brain network, and minimize the information contained in each graph under different tasks, so that the obtained features can reflect the global and local characteristics of the original brain network. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is an overall framework diagram of an adaptive multi-channel graph convolutional network for joint graph contrast learning;

[0020] Figure 2 It is a workflow diagram of an adaptive multi-channel graph convolutional network for joint graph contrast learning;

[0021] Figure 3 It is to construct functional connection brain network diagram and structural connection brain network diagram;

[0022] Figure 4 It uses the attention mechanism to adaptively fuse the features extracted by multi-channel GCN;

[0023] Figure 5 It is a comparison strategy between different modalities; DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings.

[0025] The overall framework diagram of the present invention is as follows: Figure 1 shown. Figure 2 Shown is a flow chart of the method implementation. Figure 3 Shown is the process of constructing functional brain network and structural connectivity brain network maps for each window. Figure 4 It uses the attention mechanism to adaptively fuse the features extracted by multi-channel GCN. Figure 5Comparative strategies for optimizing features between different modalities are demonstrated.

[0026] The following is based on Figure 2 The implementation architecture shown illustrates the specific implementation process of the adaptive multi-channel graph convolutional network method for joint graph contrast learning provided by the present invention. In practical applications, it is not limited to the learning of multimodal brain networks.

[0027] First, we use fixed-size overlapping sliding time windows to segment the rs-fMRI data, and then construct the corresponding rs-fMRI and DTI images in each window. The adjacency matrix of the graph is the functional connection brain network and structural connection network corresponding to the window, and the feature matrix of the node is the time series in each window. Then, we use multi-channel GCN to extract features from the multi-modality in each window and fuse the features with the attention mechanism. The multi-channel GCN is composed of three modules of GCN encoders, two of which are GCNs to obtain the most discriminative feature Z in the fMRI modality and DTI modality. F and Z D , the GCN of another module uses a shared weight approach to obtain multi-modal complementary feature embedding Z C Each encoder is composed of three layers of GCN stacked together, and a layered pooling method is used to obtain Figure 1 If f in and f out As the input node feature matrix and output of GCN, we can formulate GCN as:

[0028]

[0029] Where D is the degree matrix and W represents the learnable weight matrix of those connected vertices. Therefore, for a specific convolutional module, the output of its first layer can be formulated as:

[0030]

[0031] in represents the graph embedding expression obtained through the channel. For the common convolution module, the fMRI image under the i-th window The embedding of layer 1 can be expressed as:

[0032]

[0033] The DTI image under the i-th window The first layer embedding of can be expressed as:

[0034]

[0035] The shared weight matrix The complementary information of the two modalities can be filtered out. Finally, we embed the two graphs into and Combined as the embedding extracted by the entire common convolutional module

[0036]

[0037] In addition, in order to make the obtained features reflect the richest features of the original brain network graph as much as possible, we introduce the strategy of graph comparison learning. We compare the obtained graph representation with the node structure of the original graph, so that the multimodal fusion features can reflect the global and local information of the brain network and realize the InfoMax of information fusion. In order to obtain the most discriminative information of each modality, we compare the same graph structure under different modules node to node to realize the InfoMin of modal information under different channels. Finally, we regard the features obtained through adaptive fusion as the spatial information of each window.

[0038] Since only the spatial features of each window are obtained here, and there are interactions between brain regions between windows, it is necessary to calculate the temporal variability of brain regions as temporal feature information. To this end, we introduced LSTM to capture the connection between features between windows. Specifically, the present invention uses stacked LSTM layers to encode the overall functional feature representation to learn the temporal dynamics of the brain network under the time step, and each LSTM is followed by batch processing and tanh activation. Finally, we use the obtained embedding that combines spatial and temporal information as the feature of the entire subject for the classification diagnosis of individual brain diseases.

[0039] Combine the following Figure 3 As shown, the construction of fMRI images and DTI images in the present invention is described in detail.

[0040] We construct corresponding fMRI images and DTI images for each time window, corresponding to the functional connection brain network map and structural connection brain network map under each window. Among them, the adjacency matrix of each image is the functional connection brain network and the structural connection brain network, which is obtained by calculating the similarity between two brain regions through the Pearson correlation coefficient. The Pearson correlation coefficient can be formulated as:

[0041]

[0042] Where Cov(x, y) is the covariance between x and y. Var(x) and Var(y) represent the variance of x and y respectively. In order to prevent redundant connections from making the model too complicated, we choose to use the kNN algorithm to select K nearest neighbors of each node, retain the topological connections between them, and use the resulting connection matrix as the adjacency matrix of the graph. The feature matrix of the graph, that is, the features of each node on the graph, is composed of the time series of the window.

[0043] Combine the following Figure 4 , the features extracted from multiple channels are explained by adaptively fusing the attention mechanism in the present invention.

[0044] Through multi-channel GCN, we can obtain the embedding Z of fMRI modality respectively F , the embedding Z of the DTI modality D , and the embedding Z under multimodal fusion C Directly connecting all the obtained graph representation embeddings as the features of the subject's brain network is not easy to highlight the most important feature information. In order to obtain the most relevant embedding Z to represent each subject, the present invention uses an attention mechanism to fuse features. The attention mechanism can be expressed as

[0045] (a f , a d , a c )=att(Z F , Z D , Z C )

[0046] where a f , a d , a c Represents Z F , Z D , Z C Specifically, Z F As an example, we first transform the embedding through a nonlinear transformation and use a shared attention vector to obtain the following attention value

[0047]

[0048] Here W is the weight matrix and b is the bias vector. Similarly, we can get and The attention value is then normalized using the softmax function to obtain the final weight:

[0049]

[0050] The larger the value, the higher the ZF The more important. Similarly, we can get So we learn the weights and And expressed as a F =diag(a f ), a D =dia9(a d )and a C =diag(a c ). Then we combine these three embeddings to get the final embedding Z to represent the spatial information of the entire window:

[0051] Z=a f *Z F +a d *Z D +a c *Z C

[0052] Combine the following Figure 5 As shown, the extracted features are optimized using graph contrast learning in the present invention.

[0053] In order to make the extracted features most specific and able to reflect the global and local features of the original graph, we introduce graph contrast learning to evaluate the learned features. Specifically, we introduce two different graph contrast learning functions according to the different purpose of feature extraction.

[0054] (1) Multimodal Fusion Information Maximization (InfoMax): From the perspective of extracting common information from multiple modalities, in order to obtain the shared features of the two modalities as much as possible and make the extracted features reflect the global and local characteristics of the original graph as much as possible, and to maximize the information of multimodal fusion, we compare the point representation of one view with the graph representation of another view. Specifically, Figure 5 As shown in (a), we maximize the graph representation Z CD and Z CF With the original graph structure H CD and H CF Therefore, the contrast function of multimodal fusion InfoMax is designed as:

[0055]

[0056] However, we need to use an estimator to calculate the mutual information. Here we use the Jason-ShannonDivergence (JSD) estimator, so the calculation of the mutual information can be expressed as:

[0057]

[0058] Where sp(x) = log(1+e x ), d is the discriminator function which takes the inner product with a sigmoid activation function.

[0059] (2) Minimization of information between different channels (InfoMin): Since each modality contains its own most abundant discriminative information that is different from the common features under multimodal fusion, it is equally important to retain the specific information of a single modality. Although the original graph structure is the same as that under the multimodal fusion task, the encoding purpose of the encoder is different, that is, the task is different, so the graph obtained after encoding should also be different. To this end, we perform node-to-node comparative learning on the graph structure after encoding under different tasks. We regard the points on each view and all the nodes on the other view as negative samples to minimize the mutual information between nodes under different tasks in the same graph. Figure 5 As shown in (b), we compute the graph view H F The node i and the graph view are H CF The mutual information between:

[0060]

[0061] where k represents the remaining nodes, τ is the temperature parameter, and we define θ(u, v) = s(p(u), p(v)), s(·, ·) is implemented using a simple cosine similarity, and p(·) is a nonlinear expression for enhanced criticism. Since the two graphs are symmetric, we can conclude that L SF for:

[0062]

[0063] In general, the loss function of the invention is composed of three parts, including: classification cross entropy loss, multimodal fusion information maximization constraint, and information minimization constraint between different tasks. Therefore, the total constraint L d It can be expressed as:

[0064] L d =L t +αL SF +βL SD

[0065] Among them, α and β are parameters for maximizing multimodal fusion information and minimizing information between different tasks. Under the guidance of labeled data, we can optimize the model through back propagation and learn the embedding of nodes for classification.

[0066] In summary, the present invention proposes an adaptive multi-channel graph convolutional network method of joint graph contrast learning for the diagnosis of brain diseases, and effectively combines the spatial information and temporal features of multimodal brain networks. The proposed multi-channel graph convolutional network structure can obtain complementary features between multiple modalities, as well as the discriminative features of a single modality, and effectively fuse them by using an attention mechanism. In addition, the strategy of graph contrast learning is introduced, which can not only enable the obtained graph representation to better reflect the global and local structure of the original graph, but also make the extracted features discriminative. Therefore, the present invention can make the extracted information better represent the brain structure and state of the subject, which is conducive to the diagnosis of individual diseases.

[0067] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0068] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A brain image classification and recognition method using an adaptive multi-channel graph convolutional network with joint graph contrast learning, characterized in that: The following steps are involved: (i) Construct fMRI-based and DTI-based maps: Use overlapping fixed-size time windows to divide the rs-fMRI corresponding to each brain region of interest, and construct fMRI-based and DTI-based maps in each window, where the adjacency matrix of each map is the functional connection brain network and structural connection brain network corresponding to each window, and the feature matrix is ​​the time series information of each window; (ii) Design of adaptive multi-channel graph convolutional network: A multi-channel graph convolutional network is used to extract spatial information features of multimodal brain networks. Two graph convolutional neural network encoders are used to extract the most discriminative features of a single modality, and another GCN encoder with shared weights is used to obtain shared features of multiple modalities. Finally, the attention mechanism is used to adaptively fuse the features. (III) Designing the objective function of graph contrast learning to optimize features: In order to ensure that the shared features of multiple modalities and the most discriminative features of each modality can be obtained, the extracted features are optimized by joint graph contrast learning; (iv) LSTM extracts sample temporal features: Long short-term memory artificial neural network is used to explore the temporal variability of brain regions between each window as the temporal features of the brain network; (V) Classify the output features using a multi-layer perceptron to obtain the final recognition result.

2. The brain image classification and recognition method of the adaptive multi-channel graph convolutional network combined with graph contrast learning according to claim 1 is characterized by: The construction of the fMRI-based image and the DTI-based image in (a) specifically includes: using the Pearson correlation coefficient to calculate the correlation between each pair of brain regions to obtain the functional connection brain network and the structural connection brain network under each window, the corresponding time series under each window is regarded as a feature matrix, and the feature matrix is ​​combined with the adjacency matrix to construct the fMRI-based image and the DTI-based image under each window respectively.

3. The brain image classification and recognition method of the adaptive multi-channel graph convolutional network combined with graph contrast learning according to claim 1 is characterized by: The design of the multi-channel graph convolutional network in the step (ii): the multi-channel graph convolutional network is composed of three different GCN encoders, two of which are used to obtain the most discriminative features based on fMRI images and DTI images, and the other encoder is used to obtain complementary features of fMRI and DTI to achieve multimodal fusion, wherein each encoder is composed of three layers of GCN stacking, and a hierarchical pooling method is adopted to obtain the first-order, second-order, and third-order information of the graph. Through the multi-channel graph convolutional network, functional feature embedding and structural feature embedding are respectively obtained, as well as an embedding that combines functional features and structural features, and then an attention mechanism is introduced to adaptively fuse the obtained embedded features.

4. The brain image classification and recognition method of the adaptive multi-channel graph convolutional network combined with graph contrast learning according to claim 1 is characterized by: The step (iii) designs two graph contrast learning objective functions to optimize the extracted features: Due to the multi-channel GCN designed, each channel encodes different information of the original graph. In order to ensure that the information is maximized when multimodal fusion is obtained and the obtained graph representation can better reflect the local and global information of the original brain network, and obtain the most discriminative features of each modality, two objective functions are designed in combination with the concept of graph contrast learning, specifically including: (1) Multimodal fusion information maximization: The purpose of this objective function is to extract the complementary features of the two modalities so that the obtained graph embedding expression can reflect the global and local information of each other's original brain network graph, thereby maximizing the multimodal fusion information. The graph embedding expression is compared with the graph structure nodes, and the InfoMax formula is designed as: Among them, H A and H B Represents two different mode diagrams, Z A and Z B Represents the graph embedding of modal graph A and modal graph B after passing through the encoder, is the node i on the graph A, N is the total number of nodes, I(A; B) represents the calculation of the mutual information between A and B, and the mutual information adopts the Jason-Shannon Divergence divergence estimator: Where sp(x) = log(1+e x ), d is the discriminator function, which uses the inner product with a sigmoid activation function; (2) Information minimization between different channels: Since each channel has different tasks, two of the channels are used to obtain discriminative features under multimodal conditions, and the other channel is used to obtain complementary features of multimodal fusion. In order to enable each channel to obtain discriminative features, the two graphs under different channels are compared node by node, thereby minimizing the information under different tasks on the same graph. The InfoMin function is designed as follows: Where k represents the remaining nodes, τ is the temperature parameter, and θ(u, v) = s(p(u), p(v)), s(·,·) is implemented using simple cosine similarity, p(·) is a nonlinear expression for enhanced criticism, the two graphs are symmetrical, and the L InfoMin for: The loss function of the brain image classification and recognition method consists of three parts: (1) classification cross entropy loss (2) multimodal fusion information maximization constraint (3) information minimization constraint between different tasks.

5. The brain image classification and recognition method of the adaptive multi-channel graph convolutional network combined with graph contrast learning according to claim 1, characterized in that: The step (iv) uses LSTM to obtain the temporal characteristics of the brain network. Since the brain is essentially a dynamic system, there is temporal variability in brain regions between windows. A long short-term memory artificial neural network is used to obtain the dynamic connections between brain regions as the temporal characteristics of the brain, so that the embedded features can more effectively describe the brain state over a period of time. The stacked LSTM layers are used to encode the overall functional feature representation to learn the temporal dynamics of the brain network under the time step, and each LSTM is followed by batch processing and tanh activation.

Images