Brain image feature analysis method and system based on multi-temporal and spatial fusion of graph neural network

Abstract

The present disclosure provides a brain image feature analysis method and system based on multi-temporal and spatial fusion of graph neural network, including: obtaining images to be classified corresponding to several target objects, wherein the images to be classified are generated based on the technology of resting-state functional magnetic resonance imaging, extracting features of the images to be classified to obtain multi-temporal and spatial features, wherein the multi-temporal and spatial features include dynamic local features, dynamic brain state features and dynamic functional connection features, constructing graph data based on multi-temporal and spatial features and non-image information of several target objects obtained in advance, wherein the graph data includes nodes and edges, one node represents the multi-temporal and spatial features and non-image information of one target object, and the edge connecting two nodes represents the relationship between the two nodes, and inputting the graph data into a graph convolutional neural network GCN model to perform binary classification of the images to be classified based on the GCN model to obtain classification results. The efficiency, accuracy and reliability of image classification can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a brain image feature analysis method and system based on multi-temporal and spatial fusion of graph neural networks. Background Art

[0002] With the development of artificial intelligence technology, neural network models can be used to distinguish different types of images.

[0003] For example, a convolutional neural network (CNN) model is used as a basic network model. The basic network model is trained based on training data (such as sample images) to train the ability to classify sample images, thereby obtaining a network model for classifying images to be classified.

[0004] However, the above method has the problem of low accuracy and reliability of image classification.

[0005] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the invention

[0006] The present disclosure provides a brain image feature analysis method and system based on multi-temporal and spatial fusion of graph neural networks to solve at least one of the above-mentioned technical problems.

[0007] In a first aspect, the present disclosure provides a brain image feature analysis method based on multi-temporal and spatial fusion of graph neural networks, comprising:

[0008] Obtaining images to be classified corresponding to a plurality of target objects, wherein the images to be classified are generated based on resting-state functional magnetic resonance imaging technology;

[0009] Extracting features from the image to be classified to obtain multi-temporal and spatial features, wherein the multi-temporal and spatial features include dynamic local features, dynamic brain state features, and dynamic functional connection features;

[0010] Graph data is constructed based on the multi-temporal and spatial features and the pre-acquired non-image information of the plurality of target objects, wherein the graph data includes nodes and edges, a node represents the multi-temporal and spatial features and non-image information of a target object, and an edge connecting two nodes represents the relationship between the two nodes;

[0011] The graph data is input into a graph convolutional neural network (GCN) model to perform binary classification on the image to be classified based on the GCN model to obtain a classification result.

[0012] In some embodiments, extracting features from the image to be classified to obtain multi-temporal and spatial features includes:

[0013] The sliding window correlation SWC analysis and the dynamic functional connectivity dFC analysis are performed on the image to be classified in sequence to extract features of the image to be classified and obtain the multi-temporal and spatial features.

[0014] In some embodiments, the image to be classified is a preprocessed image, and the preprocessing includes: removing the first multiple time points, time layer correction, head movement correction, standardization, denoising, smoothing and bandpass filtering.

[0015] In some embodiments, the image to be classified includes blood oxygen level-dependent BOLD signals of whole-brain voxels; the dynamic local features are obtained by dividing the BOLD signals of whole-brain voxels into several segments and calculating them through a sliding time window method.

[0016] In some embodiments, the multi-spatiotemporal features are: calculating the explained variance contribution rates on different dimensionality reduction feature dimensions, generating a principal component analysis (PCA) cumulative explained variance contribution rate curve, and selecting features whose cumulative explained variance contribution rates reach a preset contribution rate threshold.

[0017] In some embodiments, the relationship between the two nodes can be represented by an adjacency matrix; the adjacency matrix is ​​determined by the feature similarity of the two nodes; the feature similarity includes the similarity of the multi-spatiotemporal features and the similarity of the non-image information.

[0018] In some embodiments, the GCN model includes two convolutional layers, each of which is connected to a ReLU activation function; the GCN model also includes an output layer, and the output layer uses a softmax function.

[0019] In a second aspect, the present disclosure provides a brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks, comprising:

[0020] An acquisition unit, used for acquiring images to be classified corresponding to a plurality of target objects, wherein the images to be classified are generated based on a resting state functional magnetic resonance imaging technology;

[0021] A feature extraction unit, used for extracting features from the image to be classified to obtain multi-temporal and spatial features, wherein the multi-temporal and spatial features include dynamic local features, dynamic brain state features and dynamic functional connectivity features;

[0022] A construction unit, configured to construct graph data based on the multi-temporal and spatial features and the pre-acquired non-image information of the plurality of target objects, wherein the graph data includes nodes and edges, a node represents the multi-temporal and spatial features and the non-image information of a target object, and an edge connecting two nodes represents the relationship between the two nodes;

[0023] The classification unit is used to input the graph data into a graph convolutional neural network GCN to perform binary classification on the image to be classified based on the GCN to obtain a classification result.

[0024] In some embodiments, the feature extraction unit is used to perform sliding window correlation SWC analysis and dynamic functional connectivity dFC analysis on the image to be classified in sequence to extract features from the image to be classified and obtain the multi-temporal and spatial features.

[0025] In some embodiments, the image to be classified is a preprocessed image, and the preprocessing includes: removing the first multiple time points, time layer correction, head movement correction, standardization, denoising, smoothing and bandpass filtering.

[0026] In some embodiments, the image to be classified includes blood oxygen level-dependent BOLD signals of whole-brain voxels; the dynamic local features are obtained by dividing the BOLD signals of whole-brain voxels into several segments and calculating them through a sliding time window method.

[0027] In some embodiments, the multi-spatiotemporal features are: calculating the explained variance contribution rates on different dimensionality reduction feature dimensions, generating a principal component analysis (PCA) cumulative explained variance contribution rate curve, and selecting features whose cumulative explained variance contribution rates reach a preset contribution rate threshold.

[0028] In some embodiments, the relationship between the two nodes can be represented by an adjacency matrix; the adjacency matrix is ​​determined by the feature similarity of the two nodes; the feature similarity includes the similarity of the multi-spatiotemporal features and the similarity of the non-image information.

[0029] In some embodiments, the GCN model includes two convolutional layers, each of which is connected to a ReLU activation function; the GCN model also includes an output layer, and the output layer uses a softmax function.

[0030] In a third aspect, the present disclosure provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, wherein the computer program is used to enable the processor to execute the method described in the first aspect above.

[0031] In a fourth aspect, the present disclosure provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0032] The present disclosure provides a brain image feature analysis method and system based on multi-temporal and spatial fusion of a graph neural network, including: obtaining images to be classified corresponding to each of several target objects, wherein the images to be classified are generated based on the technology of resting-state functional magnetic resonance imaging, performing feature extraction on the images to be classified to obtain multi-temporal and spatial features, wherein the multi-temporal and spatial features include dynamic local features, dynamic brain state features and dynamic functional connection features, constructing graph data based on the multi-temporal and spatial features and non-image information of several target objects obtained in advance, wherein the graph data includes nodes and edges, one node represents the multi-temporal and spatial features and non-image information of one target object, and an edge connecting two nodes represents the relationship between the two nodes, and inputting the graph data into a graph convolutional neural network GCN model to perform binary classification on the images to be classified based on the GCN model to obtain classification results. In this embodiment, on the one hand, by extracting multi-dimensional features of the image to be classified (including multi-temporal and spatial features of dynamic local features, dynamic brain state features, and dynamic functional connection features), the diversity and flexibility of feature extraction can be achieved, so that the image to be classified can be fully, reliably and effectively characterized by the extracted features; on the other hand, by combining multi-temporal and spatial features and non-image information (equivalent to combining image information and non-image information) to construct graph data, the graph data can represent both image information and non-image information, thereby improving the effectiveness and reliability of the graph data; on the other hand, by using the GCN model to classify the input graph data to classify the image to be classified, the image classification efficiency and accuracy can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0034] Figure 1 A schematic diagram of the process of the brain image feature analysis method based on multi-temporal and spatial fusion of graph neural network provided by the present invention;

[0035] Figure 2 A schematic diagram of the principle of the brain image feature analysis method based on multi-temporal and spatial fusion of graph neural network provided in the present disclosure;

[0036] Figure 3 A schematic diagram of an experimental comparison provided for an embodiment of the present disclosure;

[0037] Figure 4 A schematic diagram of an experiment comparison provided for another embodiment of the present disclosure;

[0038] Figure 5 A schematic diagram of a brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks provided by the present disclosure;

[0039] Figure 6 Schematic diagram of an electronic device provided for the present disclosure.

[0040] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of systems and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0042] It should be understood that the terms "including" and "having" and any variations thereof in the embodiments of the present disclosure are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0043] In the embodiments of the present disclosure, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0044] The term "plurality" in the embodiments of the present disclosure refers to two or more than two, and other quantifiers are similar thereto.

[0045] The terms "first", "second", "third", etc. in the present disclosure are used to distinguish similar or similar objects or entities, and are not necessarily meant to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of the present disclosure.

[0046] The term "unit / module" as used in this disclosure refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or a combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0047] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0048] This disclosure provides a brain image feature analysis method based on multi-temporal and spatial fusion of graph neural networks. Figure 1 , Figure 1 The flowchart of the brain image feature analysis method based on multi-temporal and spatial fusion of graph neural network provided by the present disclosure is as follows. Figure 1 As shown, the method includes:

[0049] S101: Obtaining images to be classified corresponding to a plurality of target objects, wherein the images to be classified are generated based on resting-state functional magnetic resonance imaging (rs-fMRI) technology.

[0050] Exemplarily, the execution subject of this embodiment may be a brain image feature analysis system based on multi-temporal and spatial fusion of graph neural network. The brain image feature analysis system based on multi-temporal and spatial fusion of graph neural network may be a server, a processor, a chip, etc., which is not limited in this embodiment. The brain image feature analysis system based on multi-temporal and spatial fusion of graph neural network is described elsewhere in this specification.

[0051] The several target objects may be one target object or multiple target objects, and usually multiple target objects. For each target object, an image to be classified generated based on resting state functional magnetic resonance imaging technology of the target object is obtained. The image to be classified is an image that needs to be classified to determine the type.

[0052] In some embodiments, Figure 2 As shown, the image to be classified is a preprocessed image, and the preprocessing includes: removing at least one of the first multiple time points, time layer correction, head motion correction, standardization, denoising, smoothing and bandpass filtering.

[0053] Exemplarily, after obtaining the image to be classified, the brain image feature analysis system based on multi-temporal and spatial fusion of graph neural network can preprocess the image to be classified to improve the accuracy and reliability of the image to be classified. Among them, the multiple time points can be determined based on demand, historical records, experiments, etc., and this embodiment does not limit it. For example, the multiple time points can be 5 time points.

[0054] The brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks can use one of the above methods to preprocess the images to be classified, or can use multiple of the above methods to preprocess the images to be classified, which is not limited in this embodiment.

[0055] For example:

[0056] The brain image feature analysis system based on multi-temporal fusion of graph neural network can first perform slice time correction to eliminate the time error between adjacent slices. Then perform head motion correction. Then the image after head motion correction is registered, segmented and normalized with the high-resolution T1 image (a type of image in magnetic resonance imaging MRI) of the target object to the standard MNI space (a standard space for neuroimaging, mainly used to standardize and compare brain images of different individuals) to facilitate comparative analysis of different brain data. Then perform spatial smoothing to eliminate the influence of hardware instability and interference signals generated by physiological movement and improve the signal-to-noise ratio. Then, in the frequency domain, the signal obtained by the above processing is band-pass filtered to retain the signal in the relevant frequency range (it is generally believed that the frequency range of blood oxygen level dependent BOLD signal is 0.01-0.1 Hz). At this point, the time series signal of each voxel after preprocessing can be obtained.

[0057] S102: Extract features from the image to be classified to obtain multi-temporal and spatial features, wherein the multi-temporal and spatial features include dynamic local features, dynamic brain state features, and dynamic functional connection features.

[0058] For example, the dynamic local features can be understood as the activity patterns of local brain regions in different time windows. Relatively speaking, the dynamic local features reflect the changes of local brain regions over time.

[0059] Dynamic brain state characteristics can be understood as the overall state of the brain in different time windows. Relatively speaking, dynamic brain state characteristics reflect the overall activity pattern of the brain at different time points.

[0060] Dynamic functional connectivity features can be understood as the functional connections between different brain regions in different time windows. Relatively speaking, dynamic functional connectivity features reflect the interaction patterns between brain regions in time.

[0061] This embodiment does not limit the way in which the brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks extracts features from the images to be classified. Figure 2 It can be known that S102 may include: performing sliding window correlation (SWC) analysis and dynamic functional connectivity (dFC) analysis on the image to be classified in sequence to extract features of the image to be classified and obtain multi-temporal and spatial features.

[0062] For example, SWC analysis can be understood as a method of calculating the correlation of time series by sliding time windows. It divides the time series into multiple overlapping windows, calculates the correlation within each window, and thus obtains a dynamic correlation matrix.

[0063] dFC analysis can be understood as the changes in functional connectivity between different brain regions within different time windows. dFC analysis can capture the dynamic changes in functional connectivity over time.

[0064] Correspondingly, through SWC analysis, the local correlation features within each window can be extracted to reflect the dynamic changes of local brain regions over time, and can be used to analyze the dynamic correlation of time series and study the interaction patterns between brain regions.

[0065] Through dFC analysis, brain states within different time windows can be identified and the overall activity patterns of the brain at different time points can be studied.

[0066] In other words, SWC analysis can capture dynamic correlation changes in time series, provide more detailed brain region interaction information, and extract features within local time windows, which helps to discover short-term brain region activity patterns.

[0067] Through dFC analysis, the dynamic changes of brain functional connections are revealed, providing a more comprehensive brain function map. Different brain states can be identified, which helps to understand the functional patterns of the brain in different tasks and states.

[0068] Therefore, in this embodiment, the brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks can improve the effectiveness, accuracy and reliability of feature extraction by performing feature extraction on images to be classified through SWC analysis and dFC analysis.

[0069] In combination with the above analysis, it can be seen that in some embodiments, the image to be classified includes the blood oxygen level-dependent BOLD signal of the whole brain voxels; the dynamic local features are obtained by dividing the BOLD signal of the whole brain voxels into several segments and calculating them through a sliding time window method.

[0070] Exemplarily, a brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks can use SWC analysis to calculate five types of rs-fMRI dynamic local features.

[0071] For example, a brain imaging feature analysis system based on multi-temporal and spatial fusion of graph neural networks can use a sliding time window method to divide the BOLD signal based on whole-brain voxels into several segments. The window width of the sliding time window is 30 repetition times (TR) and the step size is 1 TR. Then, five rs-fMRI local indicators are calculated in each time window to obtain five rs-fMRI dynamic local features.

[0072] In an embodiment, by dividing the BOLD signal of the whole brain voxels into several segments and calculating the dynamic local features, the diversity and reliability of the dynamic local features can be achieved. On this basis, the obtained dynamic local features can be used to fully and effectively characterize the image to be classified. In turn, the accuracy and reliability of image classification can be improved.

[0073] In some embodiments, in combination Figure 2 It can be seen that the multi-temporal and spatial features are: calculating the explained variance contribution rate on different dimensionality reduction feature dimensions, generating the principal component analysis (PCA) cumulative explained variance contribution rate curve, and selecting the features whose cumulative explained variance contribution rate reaches the preset contribution rate threshold.

[0074] In other words, the multi-temporal and spatial features are the features after PCA dimensionality reduction processing.

[0075] Exemplarily, after the brain image feature analysis system based on graph neural network multi-temporal and spatial fusion obtains the multi-temporal and spatial features without dimensionality reduction, the explained variance contribution rate of the multi-temporal and spatial features without dimensionality reduction at different dimensionality reduction feature dimensions can be calculated, and then the PCA cumulative explained variance contribution rate curve with the number of features retained after dimensionality reduction as the horizontal axis is obtained, and the features with an explained variance contribution rate of 90% are retained to select the best number of feature dimensionality reduction, thereby obtaining the multi-temporal and spatial features after dimensionality reduction.

[0076] In this embodiment, the brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks can not only effectively reduce the dimension of the data through the above-mentioned PCA dimensionality reduction processing, but also retain most of the useful information, thereby improving the efficiency and accuracy of subsequent analysis.

[0077] S103: Graph data is constructed based on multi-temporal and spatial features and non-image information of several target objects obtained in advance, wherein the graph data includes nodes and edges, a node represents the multi-temporal and spatial features and non-image information of a target object, and an edge connecting two nodes represents the relationship between the two nodes.

[0078] For example, the brain image feature analysis system based on multi-temporal and spatial fusion of graph neural network can obtain the non-image information corresponding to each target object in advance. Among them, the non-image information can be understood as the user information of the target object, such as age, gender, education level, etc.

[0079] Accordingly, combined Figure 2 It can be seen that the brain image feature analysis system based on multi-temporal and spatial fusion of graph neural network can be constructed into sparse graph data based on multi-temporal and spatial features (i.e. dynamic local features after dimensionality reduction, dynamic brain state features, dynamic functional connection features) and non-image information. Among them, the graph data can be an undirected sparse graph.

[0080] In some embodiments, the relationship between two nodes can be represented by an adjacency matrix; the adjacency matrix is ​​determined by feature similarity of the two nodes; the feature similarity includes similarity of multi-temporal and spatiotemporal features and similarity of non-image information.

[0081] For example, graph data can be obtained by To represent. Among them, Represents nodes in graph data, each of which represents a set of feature vectors of a target object (including multi-temporal and spatial features and non-image information). Representing an edge, which describes the relationship between the two nodes it connects, can be obtained by integrating image information (such as multi-temporal features) and non-image information. Represents the adjacency matrix in graph data, which describes the connections or relationships between nodes.

[0082] In some embodiments, the adjacency matrix It can be calculated by formula 1:

[0083]

[0084] in, Represents nodes in graph data and nodes The similarity of multiple spatiotemporal features, Representation Node and nodes The similarity of non-image information, Indicates the number of types of non-image information.

[0085] In some embodiments, It can be calculated by formula 2:

[0086]

[0087] in, Representation Node and nodes The distance of multiple spatiotemporal features, Represents the width of the kernel.

[0088] In some embodiments, It can be calculated by formula 3:

[0089]

[0090] The non-image information includes age, gender and education level, namely Take 3 as an example. Formula 3 can be understood as, when the node and nodes If the age, gender or education level of and nodes The similarity of non-image information is 1; otherwise, the node and nodes The similarity of non-image information is 0.

[0091] In an embodiment, by determining the adjacency matrix from two dimensions of multi-temporal and spatial features (ie, image information) and non-image information, the accuracy and reliability of the constructed graph data can be improved.

[0092] S104: Input the graph data into a graph convolutional network (GCN) model to perform binary classification on the image to be classified based on the GCN model to obtain a classification result.

[0093] The GCN model is obtained by end-to-end training based on training samples (image samples). Since the training logic is roughly the same as the principle of classification application, it will not be repeated here.

[0094] In this embodiment, the GCN model is used as the binary classification model. Compared with the CNN model, the GCN model has a simple structure, fewer parameters, and a lower time cost for training the model. In addition, the GCN model can contain node pairwise interaction information and integrate image information and non-image information of the data.

[0095] Accordingly, the graph data (which can represent multiple spatiotemporal features and non-image information) is input into the GCN model, so that the GCN model can perform binary classification on the image to be classified to obtain the classification probability (i.e., the classification result). For example, the classification result can represent the probability that the image to be classified is a normal image, and / or the classification result can represent the probability that the image to be classified is an abnormal image.

[0096] Combined with the above analysis, it can be seen that there may be multiple types of multi-temporal and spatial features. For example, in the above example, there may be 5 types of dynamic local features. Correspondingly, there may be 5 types of dynamic brain state features and dynamic functional connection features respectively. That is, there may be 15 types of multi-temporal and spatial features. Alternatively, it can be understood that the 15 different multi-temporal and spatial features can be divided into 3 categories: dynamic local features, dynamic brain state features, and dynamic functional connection features.

[0097] Exemplarily, the brain image feature analysis system based on graph neural network multi-temporal and spatial fusion can input the pairwise combination of three types of multi-temporal and spatial features and the features of a single type into the multi-dimensional feature classification framework based on the GCN model to perform binary classification and obtain classification results.

[0098] In some embodiments, the GCN model includes two convolutional layers, each of which is connected to a ReLU activation function; the GCN model also includes an output layer, which uses a softmax function.

[0099] For example, the GCN model consists of two convolutional layers, and each convolutional layer is followed by a ReLU activation function, and the output layer of the GCN model uses a softmax function.

[0100] In some embodiments, the input of the GCN model can be expressed based on Formula 4:

[0101]

[0102] in, is the first The input of the layer, is the activation function, is the degree matrix and can be obtained by express, Adjacency matrix representing the relationship between nodes Added self-join , is the first The output of the layer, is the weight parameter matrix.

[0103] Combined with the above analysis of S101 to S104, it can be seen that the brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks can achieve diversity and flexibility in feature extraction by extracting multi-dimensional features of the images to be classified (including multi-temporal and spatial features of dynamic local features, dynamic brain state features and dynamic functional connection features), so as to fully, reliably and effectively characterize the images to be classified through the extracted features; the brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks constructs graph data by combining multi-temporal and spatial features and non-image information (equivalent to combining image information and non-image information), so that the graph data can represent both image information and non-image information, thereby improving the effectiveness and reliability of the graph data; the brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks uses the GCN model to classify the input graph data to be classified, which can improve the efficiency and accuracy of image classification.

[0104] Based on the above analysis, it can be seen that the classification result can be the probability that the image to be classified is a normal image and / or an abnormal image. For different application scenarios, normal images and abnormal images may have different understandings.

[0105] For example, bipolar disorder (BD) is a common chronic mental illness. The normal image may be an image of a healthy subject, and the abnormal image may be an image of a BD patient.

[0106] It is worth noting that, based on the analysis of the above technical solutions, the focus of the technical solution of the present disclosure is on how to classify different types of images. That is, the technical solution of the present disclosure mainly acts on the processing of images, and specifically on the classification of images. The classification results of images can be used to distinguish the types of images of different objects.

[0107] In order to verify the effectiveness and advantages of the brain image feature analysis method based on multi-temporal and spatial fusion of graph neural network based on the GCN model provided by the present disclosure, we compared it with other classification models, such as the support vector machine (SVM) model and the ridge regression (RR) model in the regression model. We set the parameters of the three classification models as follows:

[0108] In the GCN model, the recall rate is 0.3 and the weight of the L2 regularization term is 10 -5 , the learning rate is 0.002, all networks are trained for 400 iterations (epochs), and in the SVM model, we use the linear kernel function and set the maximum number of iterations to 400.

[0109] In this experiment, the three classification models were compared by a 10-fold cross-validation strategy. In addition, we used accuracy (ACC), sensitivity (SEN), specificity (SPE), and the area under the receiver operating characteristic (ROC) curve (AUC) to evaluate the performance of the three models.

[0110] The accuracy ACC can be calculated by formula 5:

[0111]

[0112] in, The number of abnormal images correctly classified is The number of normal images correctly classified is is the number of normal images classified as abnormal images, is the number of abnormal images classified as normal images.

[0113] The sensitivity SEN can be calculated by formula 6:

[0114]

[0115] In other words, sensitivity SEN is used to characterize the proportion of abnormal images correctly classified to all abnormal images, which can reflect the ability of the classification model to identify abnormal images.

[0116] The specific SPE can be calculated by formula 7:

[0117]

[0118] In other words, the specific SPE is used to characterize the proportion of correctly classified normal images to all normal images, which can reflect the ability of the classification model to identify normal images.

[0119] Figure 3 and Figure 4 Three classification models were demonstrated respectively, and the evaluation effects were evaluated using accuracy ACC, sensitivity SEN, specificity SPE, and area under the curve AUC as measurement indicators.

[0120] Combination Figure 3It can be seen that compared with SVM and RR, the GCN model has higher accuracy ACC, sensitivity SEN and area under the curve AUC indicators. This shows that the GCN model can effectively distinguish abnormal images from correct images by integrating the image information (multi-temporal and spatial features) and non-image information of the target object, thereby effectively identifying BD patients corresponding to abnormal images and healthy subjects corresponding to correct images.

[0121] Combination Figure 4 It can be seen that we use one or two or three of the dynamic local features, dynamic brain state features, and dynamic functional connection features for binary classification. The results show that compared with considering only a single or two multi-temporal and spatial features, when all multi-temporal and spatial features (dynamic local features, dynamic brain state features, and dynamic functional connection features) are considered for binary classification, the GCN model can more effectively identify abnormal images and normal images.

[0122] It should be noted that in the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information (such as images to be classified, non-image information, etc.) involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0123] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the brain image feature analysis method based on multi-temporal and spatial fusion of graph neural networks as described above.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0125] According to an embodiment of the present disclosure, the present disclosure also provides a brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks. Figure 5 , Figure 5 This is a schematic diagram of a brain image feature analysis system based on multi-temporal and spatial fusion of graph neural networks provided by the present disclosure. Figure 5 As shown, the brain image feature analysis system 500 based on multi-temporal and spatial fusion of graph neural network includes:

[0126] An acquisition unit 501 is used to obtain images to be classified corresponding to a plurality of target objects, wherein the images to be classified are generated based on a resting state functional magnetic resonance imaging technology;

[0127] A feature extraction unit 502 is used to extract features from the image to be classified to obtain multi-temporal and spatial features, wherein the multi-temporal and spatial features include dynamic local features, dynamic brain state features and dynamic functional connectivity features;

[0128] A construction unit 503 is used to construct graph data based on the multi-temporal and spatial features and the pre-acquired non-image information of the plurality of target objects, wherein the graph data includes nodes and edges, a node represents the multi-temporal and spatial features and the non-image information of a target object, and an edge connecting two nodes represents the relationship between the two nodes;

[0129] The classification unit 504 is used to input the graph data into a graph convolutional neural network GCN to perform binary classification on the image to be classified based on the GCN to obtain a classification result.

[0130] In some embodiments, the feature extraction unit is used to perform sliding window correlation SWC analysis and dynamic functional connectivity dFC analysis on the image to be classified in sequence to extract features from the image to be classified and obtain the multi-temporal and spatial features.

[0131] In some embodiments, the image to be classified is a preprocessed image, and the preprocessing includes: removing the first multiple time points, time layer correction, head movement correction, standardization, denoising, smoothing and bandpass filtering.

[0132] In some embodiments, the image to be classified includes blood oxygen level-dependent BOLD signals of whole-brain voxels; the dynamic local features are obtained by dividing the BOLD signals of whole-brain voxels into several segments and calculating them through a sliding time window method.

[0133] In some embodiments, the multi-spatiotemporal features are: calculating the explained variance contribution rates on different dimensionality reduction feature dimensions, generating a principal component analysis (PCA) cumulative explained variance contribution rate curve, and selecting features whose cumulative explained variance contribution rates reach a preset contribution rate threshold.

[0134] In some embodiments, the relationship between the two nodes can be represented by an adjacency matrix; the adjacency matrix is ​​determined by the feature similarity of the two nodes; the feature similarity includes the similarity of the multi-spatiotemporal features and the similarity of the non-image information.

[0135] In some embodiments, the GCN model includes two convolutional layers, each of which is connected to a ReLU activation function; the GCN model also includes an output layer, and the output layer uses a softmax function.

[0136] According to the disclosed embodiment, the present disclosure also provides an electronic device for executing the above-mentioned brain image feature analysis method based on multi-temporal and spatial fusion of graph neural network. Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0137] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0138] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0139] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a brain image feature analysis method based on multi-temporal fusion of a graph neural network. For example, in some embodiments, a brain image feature analysis method based on multi-temporal fusion of a graph neural network may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the brain image feature analysis method based on multi-temporal fusion of a graph neural network described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured in any other appropriate manner (for example, by means of firmware) to execute a brain image feature analysis method based on multi-temporal and spatial fusion of a graph neural network.

[0140] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0142] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0145] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0146] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0147] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A brain image feature analysis method based on multi-temporal and spatial fusion of graph neural network, characterized in that: The method comprises: Obtaining images to be classified corresponding to a plurality of target objects, wherein the images to be classified are generated based on resting-state functional magnetic resonance imaging technology; Extracting features from the image to be classified to obtain multi-temporal and spatial features, wherein the multi-temporal and spatial features include dynamic local features, dynamic brain state features and dynamic functional connectivity features, the dynamic local features are activity patterns of local brain regions in different time windows, the dynamic brain state features are the states of the brain as a whole in different time windows, and the dynamic functional connectivity features are the functional connections between different brain regions in different time windows. Extracting features from the image to be classified to obtain multi-temporal and spatial features includes: sequentially performing sliding window correlation SWC analysis and dynamic functional connectivity dFC analysis on the image to be classified, so as to extract local correlation features in each window through the sliding window correlation SWC analysis, and identifying brain states in different time windows through the dynamic functional connectivity dFC analysis to obtain the multi-temporal and spatial features; Graph data is constructed based on the multi-temporal and spatial features and the pre-acquired non-image information of the plurality of target objects, wherein the graph data includes nodes and edges, a node represents the multi-temporal and spatial features and non-image information of a target object, and an edge connecting two nodes represents the relationship between the two nodes; The graph data is input into the graph convolutional neural network GCN model to perform binary classification on the image to be classified based on the GCN model to obtain a classification result. The GCN model includes two convolutional layers, each convolutional layer is connected to a ReLU activation function, and the GCN model also includes an output layer, which uses a softmax function. The input of the GCN model ,in, is the first The input of the layer, is the activation function, is a degree matrix and can be obtained by express, Adjacency matrix representing the relationship between nodes Added self-join , is the first The output of the layer, is the weight parameter matrix.

2. The method according to claim 1, characterized in that The image to be classified is a preprocessed image, and the preprocessing includes: removing the first multiple time points, time layer correction, head movement correction, standardization, denoising, smoothing and bandpass filtering.

3. The method according to claim 1, characterized in that The image to be classified includes blood oxygen level-dependent BOLD signals of whole-brain voxels; the dynamic local features are obtained by dividing the BOLD signals of whole-brain voxels into several segments and calculating them through a sliding time window method.

4. The method according to claim 1, characterized in that: The relationship between the two nodes is represented by an adjacency matrix; the adjacency matrix is ​​determined by the feature similarity of the two nodes; The feature similarity includes the similarity of the multi-temporal and spatial features and the similarity of the non-image information.

5. A brain image feature analysis system based on multi-temporal and spatial fusion of graph neural network, characterized in that: include: An acquisition unit, used for acquiring images to be classified corresponding to a plurality of target objects, wherein the images to be classified are generated based on a resting state functional magnetic resonance imaging technology; A feature extraction unit is used to extract features from the image to be classified to obtain multi-temporal and spatial features, wherein the multi-temporal and spatial features include dynamic local features, dynamic brain state features and dynamic functional connection features, the dynamic local features are activity patterns of local brain regions in different time windows, the dynamic brain state features are the states of the brain as a whole in different time windows, and the dynamic functional connection features are the functional connections between different brain regions in different time windows. The feature extraction from the image to be classified to obtain multi-temporal and spatial features comprises: sequentially performing sliding window correlation SWC analysis and dynamic functional connection dFC analysis on the image to be classified, so as to extract local correlation features in each window through the sliding window correlation SWC analysis, and identifying brain states in different time windows through the dynamic functional connection dFC analysis to obtain the multi-temporal and spatial features; A construction unit, configured to construct graph data based on the multi-temporal and spatial features and the pre-acquired non-image information of the plurality of target objects, wherein the graph data includes nodes and edges, a node represents the multi-temporal and spatial features and the non-image information of a target object, and an edge connecting two nodes represents the relationship between the two nodes; A classification unit is used to input the graph data into a graph convolutional neural network GCN to perform binary classification on the image to be classified based on the GCN to obtain a classification result. The GCN model includes two convolutional layers, each convolutional layer is connected to a ReLU activation function, and the GCN model also includes an output layer, the output layer uses a softmax function, and the input of the GCN model ,in, is the first The input of the layer, is the activation function, is a degree matrix and can be obtained by express, Adjacency matrix representing the relationship between nodes Added self-join , is the first The output of the layer, is the weight parameter matrix.

6. A processor-readable storage medium, characterized in that: The processor-readable storage medium stores a computer program, and the computer program is used to cause a processor to execute the method according to any one of claims 1 to 5.

7. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images