A heterogeneous graph-based ADHD intelligent decision support method and system

Abstract

This invention proposes a heterogeneous graph-based intelligent decision support method and system for ADHD, including: simultaneously collecting motion data of subjects during an ADHD test using multiple motion sensors; transposing the single-channel continuous time series data of each motion sensor within the same time window as the features of each node; using the mutual information between the data channels of each motion sensor as the weight of the edge between nodes; saving the features of each node and the weight of the edge between nodes to form a heterogeneous graph of the motion data in the current time window; splicing the heterogeneous graphs of each time window to construct a heterogeneous graph sequence in the time and frequency domains; extracting representation vectors of the nodes in the heterogeneous graph sequence using a heterogeneous graph recursive neural network; and classifying the node representation vectors using a classifier to obtain an intelligent decision support result for ADHD. This invention utilizes multiple aspects of motion data to achieve more accurate classification and identification of ADHD, providing more valuable and accurate auxiliary decision support.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to an ADHD intelligent decision support method and system based on heterogeneous graph fusion motion sensor data. Background Art

[0002] Attention Deficit Hyperactivity Disorder (ADHD) is a common neurodevelopmental disorder in school-age children. Compared to their peers, children with ADHD may exhibit inappropriate inattention, impulsivity, or hyperactivity, but the symptoms are nonspecific and therefore difficult to detect in the early stages.

[0003] Among the existing diagnostic methods, scale-based assessment and diagnosis methods are the most widely used. The scale is a table of behavioral characteristics summarized based on the doctor's clinical experience. At present, there are a variety of scales for ADHD assessment, among which the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) is the most authoritative and universally recognized diagnostic standard in the world. However, the scale only uses descriptive language and lacks quantitative standards. The scoring mainly depends on the doctor's subjective experience and judgment and is susceptible to human interference. DSM-5 lists 18 typical ADHD symptoms, which are composed of "attention deficit" and "hyperactivity / impulsivity". Because these symptoms may cause abnormal behavior in children to a certain extent, the movement patterns and fine-grained movement characteristics of children with ADHD are different from those of their normally developing peers.

[0004] Guided by this idea, a possible quantitative evaluation method for ADHD is based on the discrimination of exercise volume. This method has a significant diagnostic effect on ADHD-HI (hyperactive-impulsive type) and ADHD-C (mixed type) with abnormal behavior as the main symptom. For example, the existing technology collects acceleration data of the wrist and ankle joints, and uses statistical theory to compare the energy consumption of the ADHD group and the non-ADHD group to evaluate the difference in exercise ability. Another method is to directly obtain movement and body posture data through wearable motion sensor devices (such as accelerometers, gyroscopes, and magnetometers), and then use them to extract behavior-related features. Combining these features with general artificial intelligence algorithms can achieve auxiliary identification of children with ADHD and normally developing children.

[0005] The second type of ADHD auxiliary diagnosis, based on motion sensor data and artificial intelligence algorithms, is objective and accurate, and has great potential. Accelerometers worn on the arms or ankles collect limb movement characteristics of ADHD patients, and then classify them using a multivariate logistic regression model to achieve auxiliary assessment and diagnosis of ADHD. Alternatively, two inertial measurement units (IMUs) are used to conduct a continuous performance test (CPT) on subjects in five restricted scenarios. Support vector machine classification is used to assess movement differences between children with neurodevelopmental disorders and normal children. Alternatively, four sensors are worn on children's wrists and ankles, converting triaxial acceleration signals into two-dimensional acceleration images. Convolutional neural networks (CNNs) are then trained to identify differences between children with ADHD and controls.

[0006] Existing ADHD medical auxiliary diagnosis methods based on motion sensor data and artificial intelligence algorithms have the following significant problems:

[0007] Motion data collected by multiple sensors is directly connected as basic features, and feature calculation and selection are performed before inputting into a machine learning classification model. Alternatively, three-axis acceleration signals are converted into two-dimensional acceleration images and input into a convolutional neural network. These methods ignore the heterogeneity and correlation between motion data collected by different types of sensors at different locations, and fail to simultaneously consider the motion data's information in the time, frequency, and spatial domains. This can easily lead to poor fusion effects and low classification accuracy. Summary of the Invention

[0008] In view of the problem that existing auxiliary diagnosis methods ignore the heterogeneity and correlation between different types of sensor data at different parts of the body, which easily leads to poor information fusion effect, we choose to introduce heterogeneous graph neural network to represent data, model the heterogeneity and correlation between data, and realize the information fusion of motion sensor data;

[0009] Aiming at the problem that the existing auxiliary diagnosis methods do not consider the information of time domain, frequency domain and space domain at the same time, a spatiotemporal heterogeneous graph sequence and a spatio-frequency heterogeneous graph sequence are constructed at the same time to make full use of the complementarity of time-space domain information and frequency-space domain information to maximize the utilization of motion data.

[0010] Specifically, the present invention proposes an ADHD intelligent decision support method based on heterogeneous graph, which includes

[0011] Step 1: Using multiple motion sensors to simultaneously collect the subject's motion data during the ADHD test;

[0012] Step 2: Transpose the single-channel continuous time series data of each motion sensor in the same time window as the feature of each node, use the mutual information between the data channels of each motion sensor as the weight of the edge between nodes, save the features of each node and the weight of the edge between nodes, and form the heterogeneous graph of the motion data in the current time window;

[0013] Step 3: Splice the heterogeneous graphs of each time window to construct a sequence of heterogeneous graphs in the time domain and frequency domain;

[0014] Step 4: Extract the representation vectors of the nodes in the heterogeneous graph sequence through the heterogeneous graph recursive neural network;

[0015] Step 5: Classify the node representation vector through the classifier to obtain the ADHD intelligent decision support result.

[0016] The ADHD intelligent decision support method based on heterogeneous graphs, wherein the motion sensor includes an accelerometer, a gyroscope and a magnetometer, wherein the accelerometer detects three-axis acceleration in space; the gyroscope detects three-axis angular acceleration in space; and the magnetometer detects three-axis magnetic field in space, and each axis of each sensor is used as a separate signal channel.

[0017] The ADHD intelligent decision support method based on heterogeneous graph, wherein the heterogeneous graph G = (V, E), wherein V represents a node set and E represents an edge set; the pattern of the graph is defined as T G =(T V ,T E ), where T V is a set of node types, T E is the set of edge types; define φ:V→T V As a node type mapping, define ψ:E→T E As an edge type mapping; for a heterogeneous graph G = (V, E, φ, ψ), |T V |+|T E |>2;

[0018] All motion data within the same time window constitute a heterogeneous graph;

[0019] V=(v1,v2,…,v N ) as a node set, where node v i Represents the transpose of the single-axis signal data value sequence of a single motion sensor within a specified time window; the total number of nodes N = the number of body parts wearing sensors × the total number of signal channels of multiple types of sensors; node type T V = number of sensor types;

[0020] The edge represents the connection between two different channels; according to the value of mutual information, the threshold and weight are set for comparison to determine whether the edge is connected, or the top-k strategy is directly used to select the top k edges with the largest mutual information for connection; E = (e1, e2, ..., e m ) is defined as an edge set, where m is the number of edges.

[0021] The ADHD intelligent decision support method based on heterogeneous graph, wherein the time domain and frequency domain heterogeneous graph sequence G=(G1, G2, ..., G n ), where n is the total amount of heterogeneous graphs;

[0022] This step 3 includes: constructing a spatiotemporal graph sequence Where T represents the number of windows after the window is divided, and the frequency-space diagram sequence is constructed. Where B is the number of frequency bands;

[0023] Calculate the correlation between different channels in the motion data and use mutual information to establish the relationship between channels. Given a channel pair (j, k), the correlation γ j,k :

[0024]

[0025] Where 1≤j≤N, 1≤k≤N, represents the signal of channel j, represents the signal of channel k, T represents the Tth time window, and N is the number of channels or the total number of nodes; after calculating the correlation of all channel pairs, the adjacency matrix of the time-space graph and the frequency-space graph is obtained

[0026] Calculate single time window features in is the transpose of the signal sequence of the i-th channel in the motion signal in the t-th time window, 1≤t≤T;

[0027] Combined with time domain feature vector and adjacency matrix A to form a heterogeneous space-time graph T heterogeneous spatiotemporal graphs are obtained from the sample; all heterogeneous spatiotemporal graphs are stacked in chronological order to form the spatiotemporal graph sequence G T =(G1,G2,…,G T-1 ,G T );

[0028] Extract differential entropy features from the frequency bands to construct the frequency-space map sequence: These feature vectors of N channels extracted from the bth frequency band are and the sample adjacency matrix A form a heterogeneous frequency-space graph Each sample obtains B heterogeneous frequency-space maps in B frequency bands; all of these heterogeneous frequency-space maps are stacked into the frequency-space map sequence GB =(G1,G2,…,G B );

[0029] The ADHD intelligent decision support method based on heterogeneous graph, wherein the heterogeneous graph recurrent neural network is composed of a graph transformation network GTN, a graph neural network GCN and a gated recurrent unit GRU;

[0030] GTN automatically extracts meta-paths between channels from heterogeneous graphs;

[0031] GCN uses this meta-path to perform graph convolution: for the same node features, different weights are set for the results of using different meta-paths and weighted summation is performed to reconstruct the spatiotemporal graph sequence and the frequency-space graph sequence:

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] GRU uses the reconstructed spatiotemporal graph sequence and frequency-space graph sequence to model different time points: for the extracted graph sequence, the graph of each time point is sent to the corresponding unit of the GRU layer, and the information of all extracted units is spliced ​​together and sent to the classifier for classification.

[0038] The present invention also proposes an ADHD intelligent decision support system based on heterogeneous graph, which includes

[0039] Multiple motion sensors are used to simultaneously collect the subject's motion data during the ADHD test;

[0040] A data processing module is used to transpose the single-channel continuous time series data of each motion sensor in the same time window of the motion data as the feature of each node, use the mutual information between the data channels of each motion sensor as the weight of the edge between the nodes, and save the features of each node and the weight of the edge between the nodes as the heterogeneous graph of the motion data in the current time window;

[0041] The classification module is used to splice the heterogeneous graphs of each time window to construct a heterogeneous graph sequence in the time domain and frequency domain; extract the representation vectors of the nodes in the heterogeneous graph sequence through the heterogeneous graph recursive neural network; and classify the node representation vectors through the classifier to obtain the ADHD intelligent decision support results.

[0042] The ADHD intelligent decision support system based on heterogeneous graph, wherein the motion sensor includes an accelerometer, a gyroscope and a magnetometer, wherein the accelerometer detects three-axis acceleration in space; the gyroscope detects three-axis angular acceleration in space; and the magnetometer detects three-axis magnetic field in space, and each axis of each sensor is used as a separate signal channel;

[0043] The heterogeneous graph G = (V, E), where V represents the node set and E represents the edge set; the graph schema is defined as T G =(T V ,T E ), where T V is a set of node types, T E is the set of edge types; define φ:V→T V As a node type mapping, define ψ:E→T E As an edge type mapping; for a heterogeneous graph G = (V, E, φ, ψ), |T V |+|T E |>2;

[0044] All motion data within the same time window constitute a heterogeneous graph;

[0045] V=(v1,v2,…,v N ) as a node set, where node v i Represents the transpose of the single-axis signal data value sequence of a single motion sensor within a specified time window; the total number of nodes N = the number of body parts wearing sensors × the total number of signal channels of multiple types of sensors; node type T V = number of sensor types;

[0046] The edge represents the connection between two different channels; according to the value of mutual information, the threshold and weight are set for comparison to determine whether the edge is connected, or the top-k strategy is directly used to select the top k edges with the largest mutual information for connection; E = (e1, e2, ..., e m ) is defined as an edge set, where m is the number of edges.

[0047] The ADHD intelligent decision support system based on heterogeneous graph, wherein the time domain and frequency domain heterogeneous graph sequence G=(G1, G2, ..., G n ), where n is the total amount of heterogeneous graphs;

[0048] The classification module includes: constructing a spatiotemporal graph sequence Where T represents the number of windows after the window is divided, and the frequency-space diagram sequence is constructed. Where B is the number of frequency bands;

[0049] Calculate the correlation between different channels in the motion data and use mutual information to establish the relationship between channels. Given a channel pair (j, k), the correlation γ j,k :

[0050]

[0051] Where 1≤j≤N, 1≤k≤N, represents the signal of channel j, represents the signal of channel k, T represents the Tth time window, and N is the number of channels or the total number of nodes; after calculating the correlation of all channel pairs, the adjacency matrix of the time-space graph and the frequency-space graph is obtained

[0052] Calculate single time window features in is the transpose of the signal sequence of the i-th channel in the motion signal in the t-th time window, 1≤t≤T;

[0053] Combined with time domain feature vector and adjacency matrix A to form a heterogeneous space-time graph T heterogeneous spatiotemporal graphs are obtained from the sample; all heterogeneous spatiotemporal graphs are stacked in chronological order to form the spatiotemporal graph sequence G T =(G1,G2,…,G T-1 ,G T );

[0054] Extract differential entropy features from the frequency bands to construct the frequency-space map sequence: These feature vectors of N channels extracted from the bth frequency band are and the sample adjacency matrix A form a heterogeneous frequency-space graph Each sample obtains B heterogeneous frequency-space maps in B frequency bands; all of these heterogeneous frequency-space maps are stacked into the frequency-space map sequence G B =(G1,G2,…,G B );

[0055] The heterogeneous graph recurrent neural network consists of a graph transformation network (GTN), a graph neural network (GCN), and a gated recurrent unit (GRU).

[0056] GTN automatically extracts meta-paths between channels from heterogeneous graphs;

[0057] GCN uses this meta-path to perform graph convolution: for the same node features, different weights are set for the results of using different meta-paths and weighted summation is performed to reconstruct the spatiotemporal graph sequence and the frequency-space graph sequence:

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] GRU uses the reconstructed spatiotemporal graph sequence and frequency-space graph sequence to model different time points: for the extracted graph sequence, the graph of each time point is sent to the corresponding unit of the GRU layer, and the information of all extracted units is spliced ​​together and sent to the classifier for classification.

[0064] The present invention also proposes a storage medium for a program of any one of the heterogeneous graph-based ADHD intelligent decision support methods.

[0065] The present invention also proposes a client for use in any one of the heterogeneous graph-based ADHD intelligent decision support systems.

[0066] From the above scheme, it can be seen that the advantages of the present invention are:

[0067] The present invention deploys motion sensors at multiple locations and performs auxiliary assessment of ADHD through a data fusion method based on heterogeneous graphs, thereby providing auxiliary decision support to relevant personnel such as doctors, patients, medical teachers and students. It can more fully utilize the multi-faceted information of motion data, achieve more accurate classification and identification of ADHD, and provide more reference-worthy and accurate auxiliary decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Flowchart of auxiliary diagnostic methods for attention deficit hyperactivity disorder;

[0069] Figure 2 A diagram of the process of building a heterogeneous graph for motion data;

[0070] Figure 3 A process diagram for constructing a sequence of heterogeneous spatiotemporal graphs;

[0071] Figure 4 Computing node representation graphs for heterogeneous graph recurrent neural networks. DETAILED DESCRIPTION

[0072] The poor auxiliary diagnosis effect in the existing technology is due to the neglect of the heterogeneity and correlation of different types of motion sensor data in different parts, and the failure to integrate the information of motion data in the time domain, frequency domain, and spatial domain. This defect can be solved by using the heterogeneous graph method.

[0073] The invented auxiliary diagnosis method for attention deficit hyperactivity disorder based on heterogeneous graph fusion motion data has as input the motion sensor data collected from the subject over a period of time, and as output the intelligent decision support result of the subject, namely the ADHD classification result and its corresponding probability, for reference by relevant personnel. The ADHD classification result includes the probability that the motion sensor data belongs to each type of ADHD. According to the symptom and cognitive state detection paradigm covered by the DSM-5 scale, ten diagnostic assessment tasks were designed, such as Schulte grid, color-shape interference, etc. The quantification of the amount of motion during the test is helpful for the identification of ADHD. A variety of high-precision motion sensors are selected to collect the motion data of the subject in the process of completing the task. In the example of the present invention, three motion sensors, namely accelerometer, gyroscope and magnetometer, are used.

[0074] The single-channel continuous time series data from different types of motion sensors on six body parts within the same time window is transposed and used as node features. The mutual information between the motion sensor data channels is used as the edge weight, transforming the motion data into a heterogeneous graph (different types of motion sensor features have different meanings in each dimension). This method explores the heterogeneity and correlation of different types of motion sensor data from different body parts, integrating motion data information in the time, frequency, and spatial domains to achieve a comprehensive perception of motion characteristics, thereby enabling auxiliary diagnosis of attention deficit hyperactivity disorder.

[0075] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0076] The specific process of the method is as follows Figure 1 As shown, the main steps include:

[0077] (1) The motion sensor collects the subject's motion data during the test in the task scenario;

[0078] (2) Convert motion data signals and their relationships into nodes and connecting edges on a heterogeneous graph;

[0079] (3) Multiple heterogeneous images are spliced ​​together to construct heterogeneous image sequences in the time domain and frequency domain;

[0080] (4) Calculate the node representation vector through heterogeneous graph recursive neural network;

[0081] (5) Classification and identification of ADHD, providing auxiliary intelligent decision support results.

[0082] 1.1 Motion data signal acquisition

[0083] The wearable motion sensors used in this study primarily include accelerometers, gyroscopes, and magnetometers. The accelerometer can detect the magnitude of three-axis acceleration in space; the gyroscope can detect the magnitude of three-axis angular acceleration in space; and the magnetometer can detect the magnitude of three-axis magnetic field in space. Each axis of each sensor is treated as a separate signal channel. In this study, six wearable composite motion sensors (integrating three motion sensors: an accelerometer, a gyroscope, and a magnetometer) were selected and worn on the subject's head (one), waist (one), wrists (one on each hand, two in total), and ankles (one on each foot, two in total). Each sensor can record the magnitude of acceleration, angular acceleration, and magnetic field at the location where it is worn, thereby comprehensively sensing the subject's whole-body posture and motion changes.

[0084] Based on the commonly used clinical DSM-5 scales and related testing paradigms, ten diagnostic tasks were designed based on ADHD symptoms, such as the Schulte grid and color-shape interference. The quantification of movement during these tests can reflect abnormalities in the subject's attention and limb movements, which helps in the assessment and identification of ADHD. The task process served as the data collection scenario, with most scenarios having three difficulty levels: high, medium, and low. The length of time series data collected for each difficulty level was approximately 30 seconds.

[0085] 1.2 Conversion of motion data into heterogeneous graphs

[0086] like Figure 2 As shown, the ADHD auxiliary diagnosis method based on heterogeneous graph fusion motion data proposed by the present invention uses the motion sensor data of the subject when completing the above test task. Where L represents the total time length of the collected time series motion data, and N represents the total number of signal channels of various types of sensors on each body part. The time window is divided into fixed lengths, and the single-channel continuous time series data of different types of motion sensors on six body parts in the same time window are transposed as the features of each node. For example, the feature of the i-th node in the heterogeneous graph of the t-th time window is expressed as The mutual information between the data channels of each motion sensor is used as the weight of the edge to construct a heterogeneous graph (different types of motion sensor features have different meanings in each dimension)

[0087] The definition of a heterogeneous graph is as follows: G = (V, E) is defined as a graph, where V represents the node set and E represents the edge set. The schema of the graph is defined as T G =(T V ,T E ), where T V is a set of node types, T E is the set of edge types; define φ:V→T V As a node type mapping, define ψ:E→T EAs an edge type mapping. Therefore, a graph can be refined to be defined as G = (V, E, φ, ψ). For a heterogeneous graph G = (V, E, φ, ψ), |T V |+|T E |>2, that is, there are at least two types of points or edges.

[0088] The motion sensor data is converted into a heterogeneous graph. The heterogeneous graph G = (V, E, φ, ψ) of the present invention is constructed based on the motion data. With a fixed time window size, all motion data within the same time window constitute a heterogeneous graph.

[0089] Define V = (v1, v2, ..., v N ) as a node set, where node v i Represents the transpose of a sequence of single-axis signal data values ​​from a single motion sensor on a certain body part within a specified time window. The total number of nodes N = the number of body parts with sensors * the total number of signal channels of multiple types of sensors. In this example, N = 6 * (3 * 3) = 54. Node type T V =The number of sensor types. In this example, the number of node types is equal to 3.

[0090] An edge represents the connection between two different channels. Here, the weight of the edge is calculated using mutual information. A threshold is set based on the mutual information value and compared with the weight to determine whether the edge is connected, or a top-k strategy is used to directly select the top k edges with the largest mutual information for connection. E = (e1, e2, ..., e m ) is defined as an edge set, where m is the number of edges.

[0091] Define node type mapping φ:V→T V and edge type mapping ψ:E→T E , construct a heterogeneous network G = (V, E, φ, ψ). In addition, the heterogeneous graph can also be defined as G = (X F ,A), where X F represents the features of all nodes in the graph, and A represents the adjacency matrix of G.

[0092] 1.3 Splicing and constructing heterogeneous graph sequences

[0093] G=(G1,G2,…,G n ) is defined as a heterogeneous graph sequence, where n is the number of heterogeneous graphs. For example, construct a spatiotemporal graph sequence Where T represents the number of windows after the sample is divided into windows. Similarly, construct the frequency-space diagram sequence Where B is the number of frequency bands. The frequency range is divided into several subranges, each of which is a frequency band. The frequency range for human motion is generally 0.3 to 15 Hz, and different intensities of exercise tend to be concentrated in different frequency bands. For example, walking is concentrated in the 1-2 Hz band, while more intense running is concentrated in the 2-3 Hz band.

[0094] like Figure 3 As shown in Figure 1, for each sample (here, a single person and a single scene), a heterogeneous spatiotemporal graph sequence and a heterogeneous spatiofrequency graph sequence are constructed. These heterogeneous graph sequences are used to describe the spatial distribution of the time domain and frequency domain information of the motion signal. The heterogeneous graph sequence consists of multiple heterogeneous graphs. The general heterogeneous graph is usually defined as G = (X F ,A), where X F represents the graph node features, and A represents the graph adjacency matrix describing the channel relationship.

[0095] The process of constructing a heterogeneous graph sequence is as follows: First, a time-space graph and a frequency-space graph are constructed respectively. The correlation between different channels in the sample is calculated, and the mutual information is used to establish the relationship between the channels. Formally, given a channel pair (j, k), their correlation γ j,k It can be expressed as follows:

[0096]

[0097] Where 1≤j≤N, 1≤k≤N, represents the signal of channel j, Denotes the signal of channel k, T denotes the Tth time window, and N is the number of channels or the total number of nodes. After calculating the correlation of all channel pairs, we obtain the adjacency matrix of the space-time graph and the frequency-space graph

[0098] In order to construct a heterogeneous spatiotemporal graph sequence, single time window features are calculated in It is the transpose of the signal sequence of the i-th channel in the motion signal in the t-th time window, 1≤t≤T.

[0099] Then, combined with the time domain feature vector and adjacency matrix A to form a heterogeneous space-time graph Therefore, T heterogeneous spatiotemporal graphs are obtained from the sample. These heterogeneous spatiotemporal graphs are stacked in time order to form a heterogeneous spatiotemporal graph sequence G T =(G1,G2,…,G T-1 ,G T ).

[0100] In order to construct a heterogeneous frequency-space map sequence, similar to the process of generating a heterogeneous space-time map sequence, differential entropy features are extracted from the frequency bands. These features extracted from different channels and the correlation between channels are converted into a heterogeneous frequency-space map. Specifically, these feature vectors of N channels extracted from the bth frequency band are and the sample adjacency matrix A form a heterogeneous frequency-space graph Therefore, each sample obtains B heterogeneous frequency-space maps in B frequency bands. Then, these heterogeneous frequency-space maps are stacked into a heterogeneous frequency-space map sequence G B =(G1,G2,…,G B ).

[0101] 1.4 Computational Node Representation of Heterogeneous Graph Recurrent Neural Networks

[0102] Since the spatiotemporal graph sequence and the frequency-space graph sequence of the model have the same structure, both are heterogeneous graph recurrent neural networks, so the spatiotemporal graph sequence is used as an example. Figure 4 As shown in the figure, the heterogeneous graph recurrent neural network consists of a graph transition network (GTN), a graph neural network (GCN) and a gated recurrent unit (GRU).

[0103] The functions of the three parts are as follows:

[0104] GTN mainly models the heterogeneity of multimodal data by automatically extracting some meta-paths (i.e., the relationships between different channels) from the heterogeneous graph.

[0105]

[0106] Among them, x i represents the input node features, represents an adjacency matrix with self-loops, represents the degree matrix, W is the weight matrix, b is the bias vector, σ is the activation function, Represents the node features after graph convolution.

[0107] GCN uses the meta-paths extracted by GTN to perform graph convolution. For the same node features, different weights are set for the results of using different meta-paths and weighted summation is performed to reconstruct the extracted spatiotemporal graph sequence.

[0108]

[0109]

[0110]

[0111]

[0112] Among them, ⊙ represents the corresponding multiplication of matrix elements, R i Represents the reset gate in the GRU unit, which controls the forgetting of the previous GRU unit information; Represents the features of the i-th image in the image sequence, represents the output of the i-1th GRU unit, Represents the output after scaling by the activation function, Z i Represents the update gate in the GRU unit, controlling the extent to which information from the previous unit and the current unit is passed on. W, V, and b are all learnable parameters, where W and V are weight matrices and b is the bias vector.

[0113] The GRU uses the graph sequence obtained by GCN to model different time points. For the extracted graph sequence, the graph at each time point is fed into the corresponding unit of the GRU layer. The information of all extracted units is concatenated as the representation vector of the node in the heterogeneous graph sequence and fed into the classifier for classification.

[0114]

[0115] Y=softmax(ReLU(H3W3+b3)W4+b4)#(8)

[0116] Among them, || represents the splicing operation, and They represent the final features of the spatiotemporal graph sequence and the frequency-space graph sequence, W is the weight matrix, and b is the bias vector.

[0117] The classifier is trained offline to learn the mapping function between the graph sequence and the ADHD diagnosis result label. Given a spatiotemporal graph sequence G T Sum frequency space graph sequence G B , the ADHD diagnostic problem can be defined as Y=F(G T ,G B ), where Y represents the ADHD classification auxiliary diagnosis result. The classification in the example of the present invention is the four classifications of ADHD subtypes, and the four categories of results are normal, ADHD-I (attention deficit type), ADHD-H (hyperactivity-impulsive type), and ADHD-C (mixed type), and F represents the mapping function.

[0118] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0119] The present invention also proposes an ADHD intelligent decision support system based on heterogeneous graph, which includes

[0120] Multiple motion sensors are used to simultaneously collect the subject's motion data during the ADHD test;

[0121] A data processing module is used to transpose the single-channel continuous time series data of each motion sensor in the same time window of the motion data as the feature of each node, use the mutual information between the data channels of each motion sensor as the weight of the edge between the nodes, and save the features of each node and the weight of the edge between the nodes as the heterogeneous graph of the motion data in the current time window;

[0122] The classification module is used to splice the heterogeneous graphs of each time window to construct a heterogeneous graph sequence in the time domain and frequency domain; extract the representation vectors of the nodes in the heterogeneous graph sequence through the heterogeneous graph recursive neural network; and classify the node representation vectors through the classifier to obtain the ADHD intelligent decision support results.

[0123] The ADHD intelligent decision support system based on heterogeneous graph, wherein the motion sensor includes an accelerometer, a gyroscope and a magnetometer, wherein the accelerometer detects three-axis acceleration in space; the gyroscope detects three-axis angular acceleration in space; and the magnetometer detects three-axis magnetic field in space, and each axis of each sensor is used as a separate signal channel;

[0124] The heterogeneous graph G = (V, E), where V represents the node set and E represents the edge set; the graph schema is defined as T G =(T V ,T E ), where T V is a set of node types, T E is the set of edge types; define φ:V→T V As a node type mapping, define ψ:E→T E As an edge type mapping; for a heterogeneous graph G = (V, E, φ, ψ), |T V |+|T E |>2;

[0125] All motion data within the same time window constitute a heterogeneous graph;

[0126] V=(v1,v2,…,v N ) as a node set, where node v i Represents the transpose of the single-axis signal data value sequence of a single motion sensor within a specified time window; the total number of nodes N = the number of body parts wearing sensors × the total number of signal channels of multiple types of sensors; node type T V = number of sensor types;

[0127] The edge represents the connection between two different channels; according to the value of mutual information, the threshold and weight are set for comparison to determine whether the edge is connected, or the top-k strategy is directly used to select the top k edges with the largest mutual information for connection; E = (e1, e2, ..., e m ) is defined as an edge set, where m is the number of edges.

[0128] The ADHD intelligent decision support system based on heterogeneous graph, wherein the time domain and frequency domain heterogeneous graph sequence G=(G1, G2, ..., G n ), where n is the total amount of heterogeneous graphs;

[0129] The classification module includes: constructing a spatiotemporal graph sequence Where T represents the number of windows after the window is divided, and the frequency-space diagram sequence is constructed. Where B is the number of frequency bands;

[0130] Calculate the correlation between different channels in the motion data and use mutual information to establish the relationship between channels. Given a channel pair (j, k), the correlation γ j,k :

[0131]

[0132] Where 1≤j≤N, 1≤k≤N, represents the signal of channel j, represents the signal of channel k, T represents the Tth time window, and N is the number of channels or the total number of nodes; after calculating the correlation of all channel pairs, the adjacency matrix of the time-space graph and the frequency-space graph is obtained

[0133] Calculate single time window features in is the transpose of the signal sequence of the i-th channel in the motion signal in the t-th time window, 1≤t≤T;

[0134] Combined with time domain feature vector and adjacency matrix A to form a heterogeneous space-time graph T heterogeneous spatiotemporal graphs are obtained from the sample; all heterogeneous spatiotemporal graphs are stacked in chronological order to form the spatiotemporal graph sequence G T =(G1,G2,…,G T-1 ,G T );

[0135] Extract differential entropy features from the frequency bands to construct the frequency-space map sequence: These feature vectors of N channels extracted from the bth frequency band are and the sample adjacency matrix A form a heterogeneous frequency-space graph Each sample obtains B heterogeneous frequency-space maps in B frequency bands; all of these heterogeneous frequency-space maps are stacked into the frequency-space map sequence G B =(G1,G2,…,G B );

[0136] The heterogeneous graph recurrent neural network consists of a graph transformation network (GTN), a graph neural network (GCN), and a gated recurrent unit (GRU).

[0137] GTN automatically extracts meta-paths between channels from heterogeneous graphs;

[0138] GCN uses this meta-path to perform graph convolution: for the same node features, different weights are set for the results of using different meta-paths and weighted summation is performed to reconstruct the spatiotemporal graph sequence and the frequency-space graph sequence:

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] GRU uses the reconstructed spatiotemporal graph sequence and frequency-space graph sequence to model different time points: for the extracted graph sequence, the graph of each time point is sent to the corresponding unit of the GRU layer, and the information of all extracted units is spliced ​​together and sent to the classifier for classification.

[0145] The present invention also proposes a storage medium for a program of any one of the heterogeneous graph-based ADHD intelligent decision support methods.

[0146] The present invention also proposes a client for use in any one of the heterogeneous graph-based ADHD intelligent decision support systems.

Claims

1. An ADHD intelligent decision support system based on heterogeneous graph, characterized by: include Multiple motion sensors are used to simultaneously collect the subject's motion data during the ADHD test; The data processing module is used to transpose the single-channel continuous time series data of each motion sensor in the same time window of the motion data to construct the features of each node, construct the weight of the edge between nodes based on the mutual information between the data channels of each motion sensor, save the features of each node and the weight of the edge between nodes, and construct a heterogeneous graph of the motion data in the current time window; The classification module is used to splice the heterogeneous graphs of each time window to construct a heterogeneous graph sequence in the time domain and frequency domain; extract the representation vectors of the nodes in the heterogeneous graph sequence through the heterogeneous graph recursive neural network; and classify the node representation vectors through the classifier to obtain the ADHD intelligent decision support results.

2. The ADHD intelligent decision support system based on heterogeneous graph according to claim 1, characterized in that: The motion sensor includes an accelerometer, a gyroscope, and a magnetometer, wherein the accelerometer detects three-axis acceleration in space; the gyroscope detects three-axis angular acceleration in space; The magnetometer detects the three-axis magnetic field in space, treating each axis of each sensor as a separate signal channel; The heterogeneous graph G = (V, E), where V represents the node set and E represents the edge set; the graph schema is defined as T G =(T V ,T E ), where T V is a set of node types, T E is the set of edge types; define φ:V→T V As a node type mapping, define ψ:E→T E As an edge type mapping; for a heterogeneous graph G = (V, E, φ, ψ), |T V |+|T E |>2; All motion data within the same time window constitute a heterogeneous graph; V=(v1,v2,…,v N ) as a node set, where node v i Represents the transpose of the single-axis signal data value sequence of a single motion sensor within a specified time window; the total number of nodes N = the number of body parts wearing sensors × the total number of signal channels of multiple types of sensors; node type T V = number of sensor types; The edge represents the connection between two different channels; according to the value of mutual information, the threshold and weight are set for comparison to determine whether the edge is connected, or the top-k strategy is directly used to select the top k edges with the largest mutual information for connection; E = (e1, e2, ..., e m ) is defined as an edge set, where m is the number of edges.

3. The ADHD intelligent decision support system based on heterogeneous graph according to claim 1, characterized in that: The time domain and frequency domain heterogeneous graph sequence G=(G1,G2,…,G n ), where n is the total amount of heterogeneous graphs; The classification module includes: constructing a spatiotemporal graph sequence Where T represents the number of windows after the window is divided, and the frequency-space diagram sequence is constructed. Where B is the number of frequency bands; Calculate the correlation between different channels in the motion data and use mutual information to establish the relationship between channels. Given a channel pair (j, k), the correlation γ j,k : Where 1≤j≤N, 1≤k≤N, represents the signal of channel j, represents the signal of channel k, T represents the Tth time window, and N is the number of channels or the total number of nodes; after calculating the correlation of all channel pairs, the adjacency matrix of the time-space graph and the frequency-space graph is obtained Calculate single time window features in is the transpose of the signal sequence of the i-th channel in the motion signal in the t-th time window, 1≤t≤T; Combined with time domain feature vector and adjacency matrix A to form a heterogeneous space-time graph T heterogeneous spatiotemporal graphs are obtained from the sample; all heterogeneous spatiotemporal graphs are stacked in chronological order to form the spatiotemporal graph sequence G T =(G1,G2,…,G T-1 ,G T ); Extract differential entropy features from the frequency bands to construct the frequency-space map sequence: These feature vectors of N channels extracted from the bth frequency band are and the sample adjacency matrix A form a heterogeneous frequency-space graph Each sample obtains B heterogeneous frequency-space maps in B frequency bands; all of these heterogeneous frequency-space maps are stacked into the frequency-space map sequence G B =(G1,G2,…,G B ); The heterogeneous graph recurrent neural network consists of a graph transformation network (GTN), a graph neural network (GCN), and a gated recurrent unit (GRU). GTN automatically extracts meta-paths between channels from heterogeneous graphs; GCN uses this meta-path to perform graph convolution: for the same node features, different weights are set for the results of using different meta-paths and weighted summation is performed to reconstruct the spatiotemporal graph sequence and the frequency-space graph sequence: GRU uses the reconstructed spatiotemporal graph sequence and frequency-space graph sequence to model different time points: for the extracted graph sequence, the graph of each time point is sent to the corresponding unit of the GRU layer, and the information of all extracted units is spliced ​​together and sent to the classifier for classification.

4. A client, used in the heterogeneous graph-based ADHD intelligent decision support system according to any one of claims 1 to 3.

Images