Human behavior rapid recognition method and system based on multi-dimensional skeleton spatiotemporal relationship graph network

Abstract

The present invention belongs to the technical field of human behavior and action recognition, and specifically relates to a method and system for rapid human behavior recognition based on a multidimensional skeletal spatiotemporal relationship graph network. The method comprises: constructing a teacher network model, using a multidimensional skeletal spatiotemporal relationship graph network MM-STGCN to construct the teacher network model, MM-STGCN comprises a plurality of basic blocks, each of which comprises two modeling types: multidimensional adaptive spatial topology dynamic modeling MDAST and multidimensional temporal feature modeling MLTF; MDAST: constructing a special multidimensional adaptive spatial topology graph to extract spatial features from different levels and enhance semantic feature representation; constructing a recognition model, providing additional supervised training to the student network model based on a knowledge distillation strategy through the loss function provided by the teacher network model; using a recognition function to calculate the distillation loss of the student network model; and using the trained student network model as a recognition model for rapid human behavior recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of human behavior and action recognition, and in particular relates to a method and system for rapid recognition of human behavior based on a multi-dimensional skeletal spatiotemporal relationship graph network. Background Art

[0002] Human behavior recognition technology is similar to the human visual system. After observing human behavior, it generates corresponding behavior labels. Its goal is to identify human behavior and thus serve human life and production. Its application scenarios are very wide, such as self-driving cars, video surveillance, and human-computer interaction. Integrating human behavior recognition technology into various intelligent devices can play an important role in a variety of application scenarios and can significantly reduce the labor costs of manual guidance and monitoring.

[0003] Compared with traditional methods using RGB videos, skeleton-based action recognition methods have received increasing attention in recent years due to their strong adaptability to complex backgrounds and dynamic environments. In addition, the use of skeleton data can significantly improve the efficiency of human action classification and recognition. At present, most studies have modeled the spatial structure and temporal dynamics of human behavior by extracting rich and discriminative features from skeleton data. However, these methods use a predefined spatial topology graph structure, which is fixed during the training process and cannot achieve a more reasonable topological expansion according to different actions. It lacks the flexibility and ability to model multi-level semantics across multiple layers, so it is not good at extracting key features of specific actions. Secondly, this spatial topology does not consider the common dependency between joints and the linkage effect of various parts of the human body, which is insufficient in recognizing complex actions. Finally, the parameters of the existing spatiotemporal convolutional model are too large and complex, and the recognition efficiency is low. Summary of the invention

[0004] The present invention is made to solve the above-mentioned problems, and its purpose is to provide a method and system for rapid recognition of human behavior based on a multidimensional skeletal spatiotemporal relationship graph network, which can effectively mine from multiple levels to extract the key features of the action, and while ensuring the recognition accuracy, greatly reduce the network parameters and execute quickly with a low computing cost.

[0005] In order to achieve the above purpose, the present invention adopts the following scheme:

[0006] <Method>

[0007] The present invention provides a method for rapid recognition of human behavior based on a multidimensional skeleton spatiotemporal relationship graph network, comprising:

[0008] Construct the teacher network model, and use the multi-dimensional skeleton spatio-temporal relationship graph network MM-STGCN to construct the teacher network model. MM-STGCN includes N basic blocks, N is a positive integer, and each basic block contains two modeling types: multi-dimensional adaptive spatial topology dynamic modeling MDAST and multi-dimensional time domain feature modeling MLTF; multi-dimensional adaptive spatial topology dynamic modeling MDAST: construct a multi-dimensional adaptive spatial topology graph Extract spatial features from different levels and enhance semantic feature representation, defined as:

[0009]

[0010] In the formula, is the weight matrix, is the input human motion skeleton sequence, For batches, For channels, For frame, For joints, is the set of adjacent edges; is the adjacency matrix; is the initial standard adjacency matrix, which is used to represent the self-connected graph of the human skeleton; is the parameterized adjacency matrix; is a multi-granularity adjacency matrix used to mine contextual relationships between joints; the parameter Initialized to 0, parameters During the training process, the value is updated continuously with the number of training times. In order to automatically adjust the update, the uncertain value (which does not change regularly with the number of training times, for example, it may also be 0, or 2, 3, etc.) in the subsequent training process is responsible for adjusting weights to adapt to different actions; multi-granularity adjacency matrix Defined as:

[0011]

[0012]

[0013]

[0014]

[0015]

[0016] Indicates two joints; Indicates joints The relationship characteristic function between is the number of iterations, Iterations, by multi-layer perceptron Mapping and splicing joints The features between them are calculated; is the order, with values ​​of 0, 1, and 2; For the corresponding A multi-order trainable weight matrix; Diagonal matrix transformation;

[0017] Construct a recognition model, and provide additional supervision and guidance training for the student network model based on the knowledge distillation strategy through the loss function provided by the teacher network model; use the following recognition function to calculate the distillation loss of the student network model:

[0018]

[0019] In the formula, and denote the first The confidence level of each category, is the distillation temperature parameter;

[0020] Student Network Model Use Learn from real samples and use It learns to match the prediction structure of the teacher network model and uses the trained student network model as a recognition model for rapid recognition of human behaviors.

[0021] Preferably, in the method for rapid human behavior recognition based on a multidimensional skeletal spatiotemporal relationship graph network provided by the present invention, MLTF includes three branches: ~ , used to maintain the speed of time convolution, the first branch First execute a Convolutional layer, followed by a BN plus The activation function is then used with a kernel size of The depth-wise separable convolutional layer learns temporal context information, and then uses the convolutional layer to aggregate features; the second branch Perform the same operation, but with an additional MaxPool layer; the third branch It only contains one convolutional layer and one BN layer; finally, the results of the three branches are connected together ;The MLTF module is deployed after the MDASTGC module.

[0022]

[0023]

[0024]

[0025] .

[0026] Preferably, in the method for rapid recognition of human behavior based on a multidimensional skeletal spatiotemporal graph network provided by the present invention, the teacher network model is a 10-layer multidimensional skeletal spatiotemporal graph network MM-STGCN, N=10, and the ends of the teacher network model are a global pooling layer GAP and a Classifier; the temporal dimension of the 5th and 8th basic blocks uses the temporal convolution of the 2nd module.

[0027] Preferably, in the human behavior rapid recognition method based on the multidimensional skeletal spatiotemporal relationship graph network provided by the present invention, the student network model is finally converted into a 5-layer multidimensional skeletal spatiotemporal relationship graph network after knowledge distillation, comprising 5 basic blocks, 1 global pooling layer GAP and a softmax classifier for predicting action labels.

[0028] Preferably, in the method for rapid recognition of human behavior based on a multidimensional skeletal spatiotemporal relationship graph network provided by the present invention, , For the existing natural connections between joints, by calculating the adjacency matrix ( ) and the identity matrix (self-connection matrix) and we get, for The degree matrix of The difference is, The components in are learned from the training data and updated and optimized together with other parameters, where each element is initialized to 0. It is unique for each layer, so the various semantics contained in different layers can be personalized.

[0029] Preferably, in the method for rapid human behavior recognition based on a multidimensional skeletal spatiotemporal relationship graph network provided by the present invention, the multi-granularity adjacency matrix The build process is:

[0030] 1) Capture the existence and strength of the connection between two joints and aggregate the context feature information, which can be expressed as:

[0031]

[0032]

[0033] After that, integrate the joints and All feature information between the two joints is calculated :

[0034]

[0035] 2) Calculate the linkage relationship diagram between multiple joints To capture the linkage between multiple joints:

[0036]

[0037] 3) Calculate the spatial feature relationship diagram between different parts of the human body ,Since each part of the human body is composed of multiple adjacent joints, the relationship between parts is similar to the relationship between multiple joints, so the adjacency matrix is ​​also used to capture the relationship between parts, which is specifically defined as:

[0038]

[0039] 4) Combining the above three steps, we can finally complete the construction of complex human joint topology diagrams for different behaviors. is the characteristic rule function, which is specifically defined as:

[0040] .

[0041] It is a multi-granularity adjacency matrix used to mine the contextual relationship between joints. It is an independent graph that changes according to the data in the spatial dimension and learns a unique topology for each behavior. The topological relationship of different behaviors can be divided into two granularities, namely ① the relationship between human joints, which captures the dependency between adjacent nodes; ② the relationship between human parts, which captures the dependency between parts, where ① includes the relationship between two joints and the relationship between multiple linkage joints.

[0042] Preferably, in the method for rapid human behavior recognition based on a multidimensional skeletal spatiotemporal relationship graph network provided by the present invention, in MM-STGCN, except for the first basic block, all basic blocks contain residual connections, which can further improve the training efficiency.

[0043] Preferably, in the method for rapid human behavior recognition based on a multidimensional skeletal spatiotemporal relationship graph network provided by the present invention, the teacher model loss function , is the sample category in one-hot encoding form, The network model predicts The probability of each category; the loss function of the recognition model during training is: , is the balance weight between the two losses, estimated by cross-validation.

[0044] <System>

[0045] Furthermore, the present invention also provides a human behavior rapid identification system based on a multidimensional skeletal spatiotemporal relationship graph network that can automatically implement the above <method>, including a memory, a computer program stored in the memory, and a processor, and the processor executes the computer program to implement the above <method>.

[0046] Preferably, the human behavior rapid identification system based on the multidimensional skeletal spatiotemporal relationship graph network provided by the present invention may also include: an input display, which is communicatively connected to the memory and the processor, and displays the data of the memory and the processor accordingly according to the operation instructions input by the operator.

[0047] Functions and Effects of the Invention

[0048] The invention relates to a human behavior rapid recognition method and system based on a multi-dimensional skeletal spatiotemporal relationship graph network, which uses MM-STGCN to build a teacher network model. Each basic block of MM-STGCN contains two modeling types, MDAST and MLTF, based on a special multi-granularity adjacency matrix. By aggregating three layers of parallel spatial topological structures to learn the dynamic relationship between joints and extracting unique features at different levels, the overall performance and flexibility of the network are improved. By extracting time domain information at multiple scales, the network's perception in the time domain is further expanded, and the network's behavior recognition ability is improved. Then, based on the teacher network model and special recognition function Calculate the loss, train the student network model, and after the training is completed, use the lightweight student network model to quickly and accurately identify the newly input (to be identified) human behavior. Based on the above process, the present invention greatly reduces the network parameters while ensuring the recognition accuracy, and can be executed quickly at a low computing cost; in addition, it can also adaptively learn the spatial topological graph structure of different layers, explore the potential dependencies between joints from multiple levels, extract diverse spatiotemporal features, and additionally enhance the similarity of positive samples, maximize the enhancement of semantic feature representation, highlight the characteristics of different actions, improve recognition accuracy, and obtain an efficient and real-time lightweight behavior recognition model with stronger generalization capabilities.

[0049] Furthermore, MLTF captures skeleton time information through three branch structures, learns the dynamic changes of the positions of skeleton joints from the time dimension, enhances the feature weights of the main joints of different actions, and models the dependencies between various actions at different times, which can further improve the recognition performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] FIG1 is a flow chart of a method for rapid recognition of human behavior based on a multi-dimensional skeleton spatiotemporal relationship graph network according to an embodiment of the present invention;

[0051] FIG2 is a general architecture diagram of a fast knowledge distillation behavior recognition model according to an embodiment of the present invention;

[0052] FIG3 is a schematic diagram of the modeling process of MDAST involved in an embodiment of the present invention;

[0053] FIG4 is a schematic diagram of a modeling process of an MLTF according to an embodiment of the present invention;

[0054] FIG. 5 is a diagram showing the final recognition output result of the “walking” behavior case according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The specific implementation scheme of the method and system for rapid recognition of human behavior based on a multi-dimensional skeletal spatiotemporal relationship graph network according to the present invention is described in detail below in conjunction with the accompanying drawings.

[0056] <Example 1>

[0057] like Figure 1 As shown, the method for rapid recognition of human behavior based on a multidimensional skeleton spatiotemporal relationship graph network provided in this embodiment includes the following steps:

[0058] S1 Prepare experimental dataset.

[0059] Two large public datasets, NTU RGB+D and NTU RGB+D 120, were selected for training and used to verify the application effect.

[0060] S2 Experimental environment setup.

[0061] All experiments were conducted on the PyTorch deep learning framework. The network was trained using stochastic gradient descent (SGD) with momentum 0.9 and weight decay 0.0004 for 60 epochs. During the distillation process, the temperature T was set to 5, the initial learning rate was set to 0.1, the learning rate was reduced to 0.0001 through a cosine annealing scheduler, and the batch size was set to 64. Label smoothing with a coefficient of 0.1 was used for all experiments. The traditional cross entropy loss function was used for the teacher model. Training, for the target student model using the proposed loss function Come to train.

[0062] S3 data preprocessing.

[0063] The entire skeleton sequence extracted from the data set is evenly divided into 20 segments, and a frame is randomly selected from each segment as a new 20-frame sequence. The human skeleton joint data information in the frame sequence is used as the input of the behavior recognition network, which can be expressed as a four-dimensional matrix Description, where Indicates the number of videos in a processing batch, represents the feature channel, Indicates the frame sequence size, Represents the number of recognized joints.

[0064] S4 model training.

[0065] The present invention adopts the knowledge distillation strategy to train an efficient and real-time lightweight behavior recognition model. First, a powerful teacher network model is trained. The teacher network model provides additional supervision and guidance to the lightweight student model during the distillation recognition process through the provided loss function. Among them, the teacher model loss function , is the sample category in one-hot encoding form, The network model predicts The student network is trained by transferring the learned teacher network knowledge through the knowledge distillation strategy, so a recognition function is designed to calculate the distillation loss, which is defined as follows:

[0066]

[0067] In the formula, and denote the first The confidence level of each category, is the distillation temperature parameter.

[0068] Therefore, the loss function of the recognition model during training is the sum of the above two losses, expressed as:

[0069]

[0070] In the formula, is the balance between the two losses, estimated by cross-validation. On the one hand, the target student network uses Learning from real samples, on the other hand, uses Learn to match the prediction structure of the teacher model.

[0071] In practical applications, the student network model is deployed, and the highly complex teacher model MM-STGCN is abandoned after the training task of guiding the student model is completed, because the target student model has learned the knowledge of the teacher model to form the recognition model FKDR-Net, which can be directly used to recognize new or to-be-recognized human behavior actions.

[0072] In step S4 of this embodiment, the teacher network model (TRM) is first trained; then, at a high temperature T, the knowledge of the TRM is distilled into the student network model (SRM). Figure 2As shown in Figure 1, the teacher network model constructed in this embodiment is a 10-layer multi-dimensional skeleton spatiotemporal graph network (MM-STGCN) for model reasoning, which contains ten basic blocks (basic modules or blocks) that are stacked on top of each other to extract high-level discriminative features. The end of the network is a global pooling layer GAP and a classifier. The number of feature channels of the ten basic blocks is 64-64-64-64-128-128-128-256-256-256. The temporal dimension of the 5th and 8th basic blocks is downsampled by using the temporal convolution of the 2nd basic block. After knowledge distillation, the student network model is finally a 5-layer multi-dimensional skeletal spatiotemporal relationship graph network, which contains 5 basic blocks, a global pooling layer GAP and a Classifier, the number of feature channels of the five basic blocks is 32-64-64-128-128.

[0073] Each basic block in MM-STGCN contains two types of modeling. (A) Multidimensional Adaptive Spatial Topology Dynamic Modeling (MDAST), which learns the dynamic relationship between joints by aggregating three layers of parallel spatial topological structures, extracts unique features at different levels, and improves the performance and flexibility of the network as a whole. (B) Multidimensional Time Domain Feature Modeling (MLTF), which further expands the network's perception in the time domain and improves the network's behavior recognition ability by extracting time domain information at multiple scales. In addition, except for the first basic block, all basic blocks contain residual connections.

[0074] (A) Multidimensional Adaptive Spatial Topology Dynamic Modeling (MDAST)

[0075] In order to aggregate different features of different layers and enhance the flexibility of the network, a multi-dimensional adaptive spatial topology graph is constructed instead of a predefined fixed graph, which can extract more diverse features and enhance the semantic feature representation. It is specifically defined as:

[0076]

[0077] In the formula, is the weight matrix, (batch ,aisle ,frame and joints ) is the input skeleton sequence, Represents the set of adjacent edges. is the adjacency matrix, where Initialized to 0, so Plays a leading role in early training. Parameters It is also constantly updated during the training process and is responsible for adjusting weights to suit different actions.

[0078] Each basic block performs two module operations:

[0079] The first step is multidimensional adaptive spatial topology dynamic modeling (MDAST). First, three different adjacency matrices are used to construct a multidimensional adaptive spatial topology relationship graph, such as Figure 3 As shown, the steps are as follows:

[0080] A-1: Use the formula calculate , get the initial adjacency matrix :

[0081]

[0082] A-2: Initialize the parameter adjacency matrix All 0s.

[0083] A-3: Computing multi-granular adjacency matrices , integrating the relationship between human joints and parts to obtain :

[0084]

[0085] A-4: Summing element by element, and then performing matrix multiplication with the input matrix, substituting into the formula Weighted summation is performed and the result is passed to Convolutional layer. Finally, the results obtained by residual connection are added element by element to obtain the spatial topological feature map of the human body.

[0086] The second step is multi-dimensional time domain feature modeling (MLTF), such as Figure 4 As shown, the steps are as follows:

[0087] B-1: Calculate the first branch and execute a Convolutional layer, followed by a BN plus The activation function is then used with a kernel size of The depthwise separable convolutional layers learn the temporal context information, and finally the convolutional layers are used to aggregate the features.

[0088] B-2: Calculate the second branch, and also execute one first Convolutional layer, followed by a BN plus The activation function is operated, and then the MaxPool layer is used for pooling and dimensionality reduction.

[0089] B-3: Calculate the third branch, perform a convolution layer and a BN layer.

[0090] B-4: Use concatenation to connect the results of the above three branches.

[0091] B-5: Output Figure 5 The “walking” behavior recognition prediction results are shown.

[0092] Compared with the most advanced technology, the method of the present invention has the following improvement effects:

[0093] 1. As can be seen from Table 1 below, the parameters of the action recognition lightweight model proposed by the present invention are significantly better than those of other models, which can effectively reduce the number of parameters and the amount of calculation (computational complexity), thereby reducing the computational cost. More importantly, the present invention does not reduce the accuracy of the model.

[0094] Table 1 Performance comparison with the top 5% state-of-the-art on the NTU RGB+D dataset (recognition accuracy % and parameters)

[0095]

[0096] In Table 1, MM-STGCN represents the teacher network model constructed by the present invention, and FKDR-Net represents the recognition model finally adopted by the present invention. It can be seen from Table 1 that the recognition model accuracy of the present invention can reach 91.3%, which is about 2.9% lower than the 94.2% of AS-GCN, but the model parameters of the present invention are about 10 times lower than ASGCN. In addition, it can be seen from Table 1 that the present invention is superior to ST-GCN in terms of model accuracy and complexity. This shows that the present invention not only saves computing costs, but also ensures good performance and accuracy of the model.

[0097] It can be seen from Table 2 below that if the knowledge distillation method of the present invention is not adopted but the student network model is trained using the existing technology method, the recognition accuracy of the student network model is lower than that of the recognition model FKDR-Net of the present invention, and the computational complexity is much higher than that of the method of the present invention.

[0098] Table 2 Comparison of ablation experiments on NTU RGB+D dataset (recognition accuracy%)

[0099]

[0100] <Example 2>

[0101] The second embodiment provides a human behavior rapid recognition system based on a multi-dimensional skeletal spatiotemporal relationship graph network that can automatically implement the above-mentioned method of the present invention. The system includes a memory, a computer program stored in the memory, a processor, and an input display.

[0102] The computer program stored in the memory is an execution program of the method in <Example 1>. The processor can execute the computer program to implement the method for rapid recognition of human behavior in <Example 1>.

[0103] The input display is used to display the input, output and intermediate processing data of the memory and processor in the form of text, list, or two-dimensional / three-dimensional static / dynamic graph or model according to the operation instructions input by the operator.

[0104] The above embodiments are merely examples of the technical solutions of the present invention. The method and system for rapid identification of human behavior based on a multidimensional skeletal spatiotemporal relationship graph network involved in the present invention are not limited to the contents described in the above embodiments, but are subject to the scope defined in the claims. Any modification, supplement or equivalent replacement made by a technician in the field of the present invention based on the embodiment is within the scope of protection required by the claims of the present invention.

Claims

1. A human behavior rapid recognition method based on a multi-dimensional skeleton spatiotemporal relationship graph network, characterized in that: include: Construct the teacher network model, and use the multi-dimensional skeleton spatio-temporal graph network MM-STGCN to construct the teacher network model. MM-STGCN includes N basic blocks, N is a positive integer, and each basic block contains two modeling types: multi-dimensional adaptive spatial topology dynamic modeling MDAST and multi-dimensional time domain feature modeling MLTF; Multidimensional adaptive spatial topology dynamic modeling MDAST: building a multidimensional adaptive spatial topology graph Extract spatial features from different levels and enhance semantic feature representation, defined as: In the formula, is the weight matrix, is the input human motion skeleton sequence, For batches, For channels, For frame, For joints, is the set of adjacent edges; is the adjacency matrix; is the initial standard adjacency matrix, which is used to represent the self-connected graph of the human skeleton; is the parameterized adjacency matrix; is a multi-granularity adjacency matrix used to mine contextual relationships between joints; the parameter Initialized to 0, parameters Continuously updated during training, responsible for adjusting weights to suit different actions; Multi-granularity adjacency matrix Defined as: Indicates two joints; Indicates joints The relationship characteristic function between is the number of iterations, Iterations, by multi-layer perceptron Mapping and splicing joints The features between them are calculated; is the order, with values ​​of 0, 1, and 2; For the corresponding A multi-order trainable weight matrix; Diagonal matrix transformation; Construct a recognition model, and provide additional supervision and guidance training for the student network model based on the knowledge distillation strategy through the loss function provided by the teacher network model; use the following recognition function to calculate the distillation loss of the student network model: In the formula, and denote the first The confidence level of each category, is the distillation temperature parameter; Student Network Model Use Learn from real samples and use Learning to match the prediction structure of the teacher network model, using the trained student network model as a recognition model for rapid human behavior recognition; Among them, MLTF includes three branches ~ , and connect the results of the three branches together ; 。 2. The method for rapid human behavior recognition based on a multidimensional skeleton spatiotemporal relationship graph network according to claim 1, characterized in that: in, The teacher network model is a 10-layer multi-dimensional skeleton spatiotemporal graph network MM-STGCN, N=10. The end of the teacher network model is the global pooling layer GAP and the action label prediction layer Classifier.

3. The method for rapid human behavior recognition based on a multidimensional skeleton spatiotemporal relationship graph network according to claim 2, characterized in that: in, After knowledge distillation, the student network model is finally converted into a 5-layer multi-dimensional skeletal spatiotemporal graph network, which contains 5 basic blocks, 1 global pooling layer GAP and a Classifier.

4. The method for rapid human behavior recognition based on a multidimensional skeleton spatiotemporal relationship graph network according to claim 1, characterized in that: in, , For the existing natural connections between joints, by calculating the adjacency matrix and the identity matrix The sum is obtained, for The degree matrix of The components in are learned from the training data.

5. The method for rapid human behavior recognition based on a multidimensional skeleton spatiotemporal relationship graph network according to claim 1, characterized in that: in, Multi-granularity adjacency matrix The build process is: 1) Capture the existence and strength of the connection between two joints and aggregate the context feature information, which can be expressed as: After that, integrate the joints and All feature information between the two joints is calculated : 2) Calculate the linkage relationship diagram between multiple joints To capture the linkage between multiple joints: 3) Calculate the spatial feature relationship diagram between different parts of the human body : 4) Combining the above three steps, we can finally complete the construction of complex human joint topology diagrams for different behaviors. is the characteristic rule function, which is specifically defined as: 。 6. The method for rapid human behavior recognition based on a multidimensional skeleton spatiotemporal relationship graph network according to claim 1, characterized in that: in, In MM-STGCN, all basic blocks except the first one contain residual connections.

7. The method for rapid human behavior recognition based on a multidimensional skeleton spatiotemporal relationship graph network according to claim 1, characterized in that: in, Teacher model loss function , is the sample category in one-hot encoding form, The network model predicts The probability of each category; The loss function of the recognition model during training is: , is the balance weight between the two losses, estimated by cross-validation.

8. A human behavior rapid recognition system based on a multi-dimensional skeletal spatiotemporal relationship graph network, comprising a memory, a computer program stored in the memory, and a processor, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 7.

9. The human behavior rapid recognition system based on multi-dimensional skeleton spatiotemporal relationship graph network according to claim 8 is characterized in that: Also includes: The input display is connected to the memory and the processor for communication, and displays the data of the memory and the processor accordingly according to the operation instruction input by the operator.

Images