Skeleton behavior recognition method based on selectable multi-scale image convolutional network

Abstract

The invention provides a skeleton behavior recognition method based on a selectable multi-scale image convolutional network, which can improve the accuracy of skeleton behavior recognition, especially can obtain excellent recognition accuracy on a large-scale skeleton data set, and has good generalization performance. According to the method, in a spatial domain, a multi-scale channel topology refining adjacent matrix is constructed and participates in graph convolution, so that more representative joint features are fully extracted; in a time domain, a selectable multi-scale time sequence convolution module is provided to realize full attention to useful time information; and then, in combination with a multi-scale image convolution module and a selectable multi-scale time sequence convolution module, a selectable multi-scale image convolution model is constructed, end-to-end training is performed under a multi-stream network, and more joint features and useful time information for learning are effectively captured.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a skeleton behavior recognition method based on a selectable multi-scale graph convolutional network. Background Art

[0002] Compared with depth image data and RGB video data, skeleton data is more robust to the influence of factors such as camera perspective changes, background occlusion, and lighting. Therefore, behavior recognition research based on skeleton data can be widely used in human-computer interaction, smart medical care and other fields. Since the human skeleton can be represented in the form of a topological graph, compared with traditional deep learning methods such as convolutional neural networks and recurrent neural networks, graph convolutional networks have been cited in skeleton behavior recognition with their superior performance and have achieved excellent results. For example, the patent with publication number CN111814719B discloses a skeleton behavior recognition method based on 3D spatiotemporal graph convolution, which is based on 3D graph convolution construction and adaptive adjacency matrix structure to recognize skeleton behavior. However, in actual applications, it is found that it still has shortcomings in the accuracy of skeleton behavior, which makes it unsuitable for use in scenarios with high recognition accuracy. Summary of the invention

[0003] In order to solve the problem of insufficient accuracy of existing skeleton behavior recognition methods, the present invention provides a skeleton behavior recognition method based on a selectable multi-scale graph convolutional network, which can improve the accuracy of skeleton behavior recognition, especially can achieve excellent recognition accuracy on large skeleton data sets, and has good generalization performance.

[0004] The technical solution of the present invention is as follows: a skeleton behavior recognition method based on a selectable multi-scale graph convolutional network comprises the following steps:

[0005] S1: obtaining an original video sample, preprocessing the original video sample, and obtaining skeleton information data in the original video sample;

[0006] S2: construct a spatiotemporal graph representing the skeleton information data, including C number of channels, T number of frames and V number of joints;

[0007] S3: constructing a human skeleton behavior recognition model based on the spatiotemporal graph;

[0008] S4: Training the human skeleton behavior recognition model based on a multi-stream network to obtain the trained human skeleton behavior recognition model, specifically including the following operations:

[0009] Determine skeleton behavior feature data representing static features and dynamic features of skeleton behavior, and model the skeleton behavior feature data; the skeleton behavior feature data includes: joints, bones, joint movements, and bone movements;

[0010] During training, the input spatiotemporal graph is trained separately for four information branches, namely, joints, bones, joint movements and bone movements, based on the multi-stream network to obtain four trained human skeleton behavior recognition branch models, and the outputs of the four human skeleton behavior recognition branch models are fused as the output of the multi-stream graph convolutional network;

[0011] S5: Acquire the video data to be identified, extract the skeleton information data in the video data group to be identified, convert the skeleton information data into a spatiotemporal graph, and send it into the trained human skeleton behavior recognition model to obtain a final recognition result;

[0012] Features:

[0013] The construction of the human skeleton behavior recognition model includes the following steps:

[0014] S3-1: Divide the neighborhood of all joint points in the spatiotemporal graph into different skeleton node subsets, including: the root joint point itself, the centripetal neighbor nodes close to the skeleton center of gravity, and the centrifugal neighbor nodes far from the skeleton center of gravity;

[0015] S3-2: Obtain the skeleton node subsets one by one, recorded as: skeleton node subset to be calculated;

[0016] Construct a multi-scale graph convolution module, take the skeleton node subset to be calculated as input features, and perform spatial feature extraction on all skeleton node subsets based on the multi-scale graph convolution module. The specific operations include:

[0017] For two nodes v in the subset of skeleton nodes to be calculated, i and v j The dependency relationship between them is modeled to obtain the pairwise joint adjacency matrix, where i,j∈{1,…,V}, V is the number of joints in the skeleton node subset to be calculated; at the same time, for a single node v i and multiple nodes v j 、v j+1 、v j+2 ……v j+k-1 The dependency relationship of (i∈{1,…,V},j∈{1,…,V+1-k},k∈{2,…,V}) is modeled to obtain a multi-scale channel topology refinement adjacency matrix corresponding to each skeleton node subset, and feature extraction is performed on the spatiotemporal graph in combination with a graph convolution operation;

[0018] S3-3: constructing a selectable multi-scale temporal convolution, extracting the temporal features of the spatiotemporal graph based on the spatial features extracted by the multi-scale graph convolution module, and constructing a temporal convolution layer, and recording the temporal convolution layer as: a selectable multi-scale temporal convolution module;

[0019] S3-4: Construct selectable multi-scale graph convolution blocks;

[0020] The selectable multi-scale graph convolution block includes: a multi-scale graph convolution module and a selectable multi-scale temporal convolution module connected in sequence;

[0021] S3-5: Constructing a selectable multi-scale graph convolutional network;

[0022] The selectable multi-scale graph convolution network includes: 10 selectable multi-scale graph convolution blocks, a GAP layer and an FC layer connected in sequence;

[0023] S3-6: Construct the human skeleton behavior recognition model based on a selectable multi-scale graph convolutional network.

[0024] It is further characterized by:

[0025] In step S3-2, in the multi-scale graph convolution module, two nodes v representing the subset of skeleton nodes to be calculated are i and v j The dependency relationship between them is modeled, which includes the following steps:

[0026] a1: Take the subset of skeleton nodes to be calculated as input features, denoted as X in , whose size is C×T×V, representing the number of input channels, frames and joints corresponding to the spatiotemporal graph respectively; two 1×1 convolutions with a reduction ratio of r are used to respectively in While performing channel dimension reduction, learn the paired joints v i and v j The features of the , get two features with feature dimensions of C / r×T×V;

[0027] a2: After using the time pooling operation, the two feature dimensions of C / r×T×V are converted into C / r×V×1 and C / r×1×V features respectively through the unsqueeze function;

[0028] While preserving the channel dimension, the two features of C / r×V×1 and C / r×1×V are subtracted to obtain the differential topology to obtain the correlation of paired joints in each channel;

[0029] a3: The differential topology after subtraction is calculated by tanh activation function and 1×1 convolution to obtain the pairwise joint adjacency matrix Q1:

[0030]

[0031] Among them, the size of Q1 is C'×V×V, where C' represents the number of output channels; ω(·) represents a 1×1 convolution operation, TP represents a temporal pooling operation, and θ(·) and Represents two 1×1 convolution operations;

[0032] In step S3-2, in the multi-scale graph convolution module, a single node v is established i and multiple nodes v j 、v j+1 、v j+2 ……v j+k-1 The dependencies include the following steps:

[0033] b1: The input feature X is processed by a 1×k convolution with a reduction ratio of r. in Perform channel dimensionality reduction and aggregate the features of k joints;

[0034] b2: The input feature X is reduced by a 1×1 convolution with a reduction ratio of r. in Perform channel dimension reduction and learn the features of a single joint;

[0035] b3: After performing temporal pooling operations on the aggregated multi-joint features and the learned single joint features, the unsqueeze function is used to convert the dimensions of the aggregated multi-joint features to C / r×V×1, and the feature dimensions of the learned single joint features to C / r×1×V;

[0036] While preserving the channel dimension, the C / r×V×1 and C / r×1×V features are subtracted to obtain the differential topology to obtain the correlation of a single joint relative to multiple joints in each channel;

[0037] b4: The differential topology after subtraction is calculated by tanh activation function and 1×1 convolution to obtain the multi-joint adjacency matrix Q2:

[0038] Q2=τ(tanh(TP(μ(X in ))-TP(λ(X in ))));

[0039] Among them, the size of Q2 is C'×V×V, τ(·) represents a 1×1 convolution operation, μ(·) and λ(·) represent two 1×1 convolution operations;

[0040] In step S3-2, in the multi-scale graph convolution module, feature extraction is performed on the spatiotemporal graph in combination with the graph convolution operation, specifically including the following steps:

[0041] c1: Establish multi-scale dependencies between pairs of joints and single joints relative to multiple joints. The specific process is as follows:

[0042] Refine the function based on topology Based on the pairwise joint adjacency matrix Q1 and the multi-joint adjacency matrix Q2, the shared prior topology A∈R V×V Refine and obtain the multi-scale channel topology refinement adjacency matrix R∈R C′×V×V :

[0043]

[0044] Among them, α and β are trainable parameters used to adjust the refinement strength;

[0045] c2: for input feature X in ∈R C×T×V A 1×1 convolution operation is used to perform feature transformation to generate high-order features X′∈R C′×T×V ;

[0046] The multi-scale channel topology refinement adjacency matrix R for the c-th channel (c∈{1,…,C'}) c and high-order features X c ′ is aggregated to realize the graph convolution operation, and the final output X is obtained by connecting the outputs of all channel topological features in series. out ∈R C ′×T×V , the calculation formula is as follows:

[0047]

[0048] Among them, ψ(·) represents a 1×1 convolution operation, represents channel aggregation operation, and || represents concatenation operation;

[0049] In step S3-3, the step of constructing the selectable multi-scale temporal convolution module includes:

[0050] S3301: Given an input feature X in , whose size is C'×T×V, which respectively represent the number of channels, input frames and joints corresponding to the spatiotemporal graph; the selectable multi-scale temporal convolution module includes four parallel branches, each of which uses 1×1 convolution to transform the input feature X in The number of channels is reduced to C' / 4; the two convolution branches use temporal convolutions with different expansion coefficients to extract features of temporal context information of different scales, and the maximum pooling branch realizes the extraction of key frames. Finally, the outputs X1, X2, X3 and X4 of the four branches are obtained. The calculation formula is as follows:

[0051]

[0052] Among them, the size of the output features X1, X2, X3 and X4 is C' / 4×T'×V, where T' represents the number of output frames of the feature; and The convolution weights are expansion coefficients d1 and d2 respectively, and the size of the convolution kernel is k. d ×1,W 1×1 is a 1×1 convolution weight, and Maxpool represents the maximum pooling operation;

[0053] S3302: Use the time selection mechanism to select the appropriate feature map from the features extracted by the time series convolution of different scales, concatenate the features X1 and X2 obtained by the convolution kernels of different scales, and then use the permute function to convert the dimension to T'×C' / 2×V to obtain the feature X p :

[0054]

[0055] in, is the permute function, Concat(·) represents the concatenation operation;

[0056] S3303: For feature X p Extract global average pooling feature X gap And the global maximum pooling feature X gmp ;

[0057]

[0058] Among them, δ represents the softmax function, GMP and GAP represent the global maximum pooling and global average pooling operations respectively;

[0059] S3304: Based on the global maximum pooling feature X gmp And the global average pooling feature X gap Two corresponding time-selective masks are calculated and The calculation formula is as follows:

[0060]

[0061] Where Split(·) represents the separation function, σ is the sigmoid activation function;

[0062] S3305: The output features X1 and X2 of different scales obtained in the feature extraction stage are combined with the corresponding time selection masks and Weighted feature S∈R C′ / 4×T ' ×V , and then use 1×1 convolution to fuse and upgrade the dimension to C' / 2×T'×V; finally, the upgraded features The final output X is obtained by concatenating the features X3 and X4 obtained in the feature extraction stage. out ∈RC′×T'×V , the calculation formula is as follows:

[0063]

[0064] Among them, ⊙ represents the element-by-element product operation;

[0065] In step S4, the skeleton behavior feature data is modeled, which specifically includes the following steps:

[0066] S4-1: The joint point itself is modeled as joint information. Assume that the joint information of the tth frame is represented by J t ;

[0067] S4-2: The joint point close to the center of gravity of the bone is the source joint point The joint point far away from the center of gravity of the bone is the target joint point Skeleton information represents the vector from the source joint point to the target joint point

[0068] S4-3: The motion information of joints and bones is expressed as the coordinate difference between two adjacent frames, and the motion information of joints is expressed as

[0069] The motion information of bones is represented as

[0070] In step S4, stochastic gradient descent is used as the optimization strategy during training, and cross entropy is selected as the loss function of gradient back propagation;

[0071] In step S1, the step of obtaining the skeleton information data in the original video sample includes:

[0072] S1-1: performing frame processing on the collected original video samples, and decomposing continuous video clips into a picture sequence including static frames;

[0073] S1-2: Calculation based on Openpose posture estimation algorithm, including the following:

[0074] The calculation parameters of the Openpose algorithm are set, and the static frame image obtained by decomposing the video is input into Openpose, and the human skeleton data corresponding to the number of joints in the static frame is obtained, and the calculation parameters include: the number of human joints and the number of human bodies;

[0075] S1-3: constructing a connection relationship of human skeleton data to represent the morphological characteristics of the human body according to the numbers of human joints and corresponding joints in the Openpose algorithm, that is, obtaining the skeleton information data;

[0076] In step S2, the step of constructing a spatiotemporal graph representing the skeleton sequence includes:

[0077] S2-1: Let the spatiotemporal graph of a skeleton sequence with N joints and T frames be G = (V, E), and its joint set be V = {Vti|t = 1, ..., T, i = 1, ..., N};

[0078] S2-2: Connect each joint point of the same frame in the spatiotemporal graph G=(V, E) into an edge;

[0079] S2-3: Connect the same joint points in two consecutive frames in the spatiotemporal graph G=(V, E) into edges.

[0080] The skeleton behavior recognition method based on the selectable multi-scale graph convolution network provided by the present application first constructs a multi-scale channel topology refinement adjacency matrix in the spatial domain and participates in graph convolution, so as to build a multi-scale graph convolution module to establish multi-scale dependencies between paired joints and single joints and multiple joints, so as to fully extract more representative joint features; secondly, in the time domain, a selectable multi-scale temporal convolution module is proposed, which extracts and fuses temporal context information of different scales by selecting appropriate multi-scale temporal convolution kernels, so as to fully pay attention to useful temporal information; then, a selectable multi-scale graph convolution model is constructed by combining the multi-scale graph convolution module and the selectable multi-scale temporal convolution module, and end-to-end training is performed under a multi-stream network. In the process of graph convolution, the technical solution of the present application can fully capture more representative joint features in both the spatial domain and the temporal domain, and learn useful temporal context information, so that the model can achieve excellent recognition accuracy and good generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a flowchart of the skeleton behavior recognition method based on a selectable multi-scale graph convolutional network in the present invention;

[0082] Figure 2 Schematic diagram of the structure of the multi-scale graph convolution module proposed in the present invention;

[0083] Figure 3 A schematic diagram of a model of a selectable multi-scale temporal convolution module proposed in the present invention;

[0084] Figure 4 Schematic diagram of the overall structure of the model based on the selectable multi-scale graph convolutional network proposed in the present invention.

[0085] Figure 5 This is a schematic diagram of the module based on the selectable multi-scale graph convolutional network proposed in the present invention. DETAILED DESCRIPTION

[0086] like Figure 1 to Figure 5 As shown, the present application includes a skeleton behavior recognition method based on a selectable multi-scale graph convolutional network, which includes the following steps.

[0087] S1: Obtain an original video sample, preprocess the original video sample, and obtain skeleton information data in the original video sample.

[0088] The steps for obtaining skeleton information data in the original video sample are as follows.

[0089] S1-1: Perform frame processing on the collected original video samples, and decompose the continuous video clips into a picture sequence including static frames;

[0090] S1-2: Calculation based on Openpose posture estimation algorithm, including the following:

[0091] Set the calculation parameters of the Openpose algorithm, input the static frame image obtained by decomposing the video into Openpose, and extract the human skeleton data corresponding to the number of joints in the static frame. The calculation parameters include: the number of human joints and the number of human bodies;

[0092] S1-3: Based on the human joints and the numbers of the corresponding joints in the Openpose algorithm, the connection relationship of the human skeleton data is constructed to represent the morphological characteristics of the human body, that is, the skeleton information data is obtained.

[0093] S2: Construct a spatiotemporal graph representing the skeleton information data. The spatiotemporal graph contains C channels, T frames, and V joints. The specific steps for constructing a spatiotemporal graph representing the skeleton sequence are as follows:

[0094] S2-1: Let the spatiotemporal graph of a skeleton sequence with N joints and T frames be G = (V, E), and its joint point set be V = {Vti|t = 1,…,T,i = 1,...,N}.

[0095] S2-2: Connect each joint point of the same frame in the spatiotemporal graph G = (V, E) into an edge.

[0096] S2-3: Connect the same joint points in two consecutive frames in the spatiotemporal graph G = (V, E) into edges.

[0097] S3: Constructing a human skeleton behavior recognition model based on the spatiotemporal graph; including the following steps.

[0098] S3-1: Divide the neighborhood of all joint points in the space-time graph into different skeleton node subsets, including: the root joint point itself, the centripetal neighbor nodes close to the skeleton center of gravity, and the centrifugal neighbor nodes far from the skeleton center of gravity.

[0099] S3-2: Obtain skeleton node subsets one by one, recorded as: skeleton node subset to be calculated.

[0100] Construct a multi-scale graph convolution module, take the skeleton node subset to be calculated as the input feature, and extract spatial features from all skeleton node subsets based on the multi-scale graph convolution module; Figure 2 As shown, specifically including:

[0101] The two nodes v in the subset of skeleton nodes to be calculated are represented by i and v j The dependency relationship between them is modeled to obtain the pairwise joint adjacency matrix, where i,j∈{1,…,V}; at the same time, for a single node v i and multiple nodes v j 、v j+1 、v j+2 ……v j+k-1 The dependency relationship of (i∈{1,…,V},j∈{1,…,V+1-k},k∈{2,…,V}) is modeled to obtain the multi-scale channel topology refinement adjacency matrix corresponding to each skeleton node subset, and the feature extraction of the spatiotemporal graph is performed in combination with the graph convolution operation.

[0102] The multi-scale graph convolution module models the dependency relationships between paired joints and between a single joint and multiple joints, and generates paired joint and multi-joint adjacency matrices; the two adjacency matrices are combined with the shared prior topology to obtain a multi-scale channel topology refined adjacency matrix, which is then graph convolved and a spatial graph convolution layer is constructed together with a subset of skeleton nodes, completing the operations in the multi-scale graph convolution module.

[0103] In the technical solution of the present invention, the generated multi-scale channel topology refinement adjacency matrix can not only capture the dependency relationship of paired joints, but also capture the dependency relationship of a single joint relative to multiple joints, which is conducive to learning more representative joint features. The obtained multi-scale graph convolution module enhances the graph convolution network's ability to extract the spatial structural features of the human body, and further improves the accuracy of the graph convolution network's skeleton behavior recognition.

[0104] In specific applications, in the multi-scale graph convolution module, two nodes v in the subset of skeleton nodes to be calculated are i and v j The dependency relationship between them is modeled, which includes the following steps:

[0105] a1: Take the subset of skeleton nodes to be calculated as input features, denoted as X in , whose size is C×T×V, representing the number of input channels, frames and joints corresponding to the spatiotemporal graph respectively; two 1×1 convolutions with a reduction ratio of r are used to respectively in While performing channel dimension reduction, learn the paired joints v i and vj The features of the , get two features with feature dimensions of C / r×T×V;

[0106] a2: After using the temporal pooling operation, the two C / r×T×V feature dimensions are converted into C / r×V×1 and C / r×1×V features respectively through the unsqueeze function;

[0107] While preserving the channel dimension, the two features of C / r×V×1 and C / r×1×V are subtracted to obtain the differential topology to obtain the correlation of paired joints in each channel;

[0108] a3: The differential topology after subtraction is calculated by tanh activation function and 1×1 convolution to obtain the pairwise joint adjacency matrix Q1:

[0109]

[0110] The size of Q1 is C'×V×V, where C' represents the number of output channels; ω(·) represents a 1×1 convolution operation, TP represents a temporal pooling operation, and θ(·) and Represents two 1×1 convolution operations.

[0111] In the multi-scale graph convolution module, a single joint v is established i and multiple joints v j 、v j+1 、v j+2 ……v j+k-1 The dependencies include the following steps:

[0112] b1: The input feature X is processed by a 1×k convolution with a reduction ratio of r. in Perform channel dimensionality reduction and aggregate the features of k joints;

[0113] b2: The input feature X is processed by a 1×1 convolution with a reduction ratio of r in Perform channel dimension reduction and learn the features of a single joint;

[0114] b3: After performing temporal pooling operations on the aggregated multi-joint features and the learned single joint features, the unsqueeze function is used to convert the dimensions of the aggregated multi-joint features to C / r×V×1, and the feature dimensions of the learned single joint features to C / r×1×V;

[0115] While preserving the channel dimension, the C / r×V×1 and C / r×1×V features are subtracted to obtain the differential topology to obtain the correlation of a single joint relative to multiple joints in each channel;

[0116] b4: The differential topology after subtraction is calculated by tanh activation function and 1×1 convolution to obtain the multi-joint adjacency matrix Q2:

[0117] Q2=τ(tanh(TP(μ(X in ))-TP(λ(X in ))));

[0118] Among them, the size of Q2 is C'×V×V, τ(·) represents a 1×1 convolution operation, and μ(·) and λ(·) represent two 1×1 convolution operations.

[0119] In the multi-scale graph convolution module, the spatiotemporal graph is feature extracted by combining the graph convolution operation, which specifically includes the following steps:

[0120] c1: Establish multi-scale dependencies between pairs of joints and single joints relative to multiple joints. The specific process is as follows:

[0121] To establish the multi-scale dependencies between pairs of joints and single joints relative to multiple joints, the topological refinement function Based on the shared prior topology A∈R of the pairwise joint adjacency matrix Q1 and the multi-joint adjacency matrix Q2 V×V Refine and obtain the multi-scale channel topology refinement adjacency matrix R∈R C′×V×V :

[0122]

[0123] Among them, α and β are trainable parameters used to adjust the refinement strength;

[0124] c2: for input feature X in ∈R C×T×V A 1×1 convolution operation is used to perform feature transformation to generate high-order features X′∈R C′×T×V ;

[0125] The multi-scale channel topology refinement adjacency matrix R for the c-th channel (c∈{1,…,C'}) c and high-order features X c ′ is aggregated to realize the graph convolution operation, and the final output X is obtained by connecting the outputs of all channel topological features in series. out ∈R C ′×T×V , the calculation formula is as follows:

[0126]

[0127] Among them, ψ(·) represents a 1×1 convolution operation, represents a channel aggregation operation, and || represents a concatenation operation.

[0128] like Figure 2 As shown in the figure, it is the multi-scale graph convolution module in this technical solution. in and X out represents input and output features; θ(1×1), μ(1×k), λ(1×1), ψ(1×1), ω(1×1) and τ(1×1) represent 1×1 and 1×k convolution operations, which are used to calculate the similarity between paired joint points and the similarity of a single joint relative to multiple joints, so as to obtain paired joint adjacency matrix Q1 and multi-joint adjacency matrix Q2; C / r×T×V represents the size of the input feature after the 1×1 and 1×k convolution operations; Temporal pooling represents the temporal pooling operation; Unsqueeze represents the unsqueeze function; Tanh represents the tanh activation function. The multi-scale channel topology refinement adjacency matrix in the present invention can effectively capture the dependency of paired joints and the dependency of a single joint relative to multiple joints, realize the extraction of more representative joint features, and help improve the recognition accuracy of the model.

[0129] S3-3: Construct a selectable multi-scale temporal convolution. Based on the spatial features extracted by the multi-scale graph convolution module, after extracting the temporal features of the spatiotemporal graph, a temporal convolution layer is constructed, and the temporal convolution layer is recorded as: a selectable multi-scale temporal convolution module.

[0130] like Figure 3 As shown, it is an optional multi-scale temporal convolution module in the technical solution of the present invention, X in and X out are the input and output features of the selectable multi-scale temporal convolution module. The steps to construct the selectable multi-scale temporal convolution module are as follows.

[0131] S3301: Given an input feature X in , whose size is C'×T×V, representing the number of channels, input frames and joints corresponding to the spatiotemporal graph respectively; the optional multi-scale temporal convolution module contains four parallel branches, each of which uses 1×1 convolution to transform the input feature X in The number of channels is reduced to C' / 4; the two convolution branches use temporal convolutions with different expansion coefficients to extract features of temporal context information of different scales, and the maximum pooling branch realizes the extraction of key frames. Finally, the outputs X1, X2, X3 and X4 of the four branches are obtained. The calculation formula is as follows:

[0132]

[0133] Among them, the size of the output features X1, X2, X3 and X4 is C' / 4×T'×V, where T' represents the number of output frames of the feature; and The convolution weights are expansion coefficients d1 and d2 respectively, and the size of the convolution kernel is k. d ×1,W 1×1 is a 1×1 convolution weight, and Maxpool represents the maximum pooling operation.

[0134] S3302: To enhance the network's ability to focus on useful temporal context information to identify long- and short-term action features, a temporal selection mechanism is used to select appropriate feature maps from the features extracted by temporal convolutions of different scales, concatenate the features X1 and X2 obtained by convolution kernels of different scales, and then use the permute function to convert the dimension to T'×C' / 2×V to obtain the feature X p :

[0135]

[0136] in, is the permute function, and Concat(·) represents the concatenation operation.

[0137] S3303: For feature X p Extract global average pooling features and global maximum pooling features;

[0138] For feature X p , the global maximum pooling (GMP) operation is used to extract the key time features of the behavior information, the permute function is used for dimension conversion and the softmax function is used to calculate the global maximum pooling feature X gmp ∈R 1×T ' ×1 ;

[0139] For feature X p The global average pooling (GAP) operation is used to extract the global temporal context features, and the permute function is used for dimension conversion and the softmax function is used to calculate the global average pooling feature X. gap ∈R 1×T ' ×1 :

[0140]

[0141] Among them, δ represents the softmax function, GMP and GAP represent the global maximum pooling and global average pooling operations respectively.

[0142] S3304: To allow global maximum pooling feature X gmp And the global average pooling feature X gapThe information interaction between them is achieved by concatenating the two features and using 1×1 convolution to convert the pooled features into a temporal attention map. After sigmoid function calculation and separation operation, two corresponding time selection masks are obtained. and The calculation formula is as follows:

[0143]

[0144] Among them, Split(·) represents the separation function and σ is the sigmoid activation function.

[0145] S3305: The output features X1 and X2 of different scales obtained in the feature extraction stage are combined with their corresponding time selection masks and Weighted feature S∈R C′ / 4×T'×V , and then use 1×1 convolution to fuse and upgrade the dimension to C' / 2×T'×V; finally, the upgraded features The final output X is obtained by concatenating the features X3 and X4 obtained in the feature extraction stage. out ∈R C′×T ' ×V , the calculation formula is as follows:

[0146]

[0147] Where ⊙ represents an element-wise product operation.

[0148] like Figure 3 As shown in the figure, from the perspective of the processing process, the selectable multi-scale temporal convolution modules are divided into:

[0149] Feature extraction stage, time selection stage, feature fusion stage;

[0150] The multi-scale temporal convolution module can be selected to use convolution kernels of different scales to extract temporal features in the feature extraction stage. In the time selection stage, the time selection mechanism is used to generate the corresponding time selection mask for the extracted time features, so as to select the appropriate convolution kernel according to the long and short time action features. Finally, in the feature fusion stage, each output feature is fused to obtain the final output, effectively extracting useful temporal context information.

[0151] S3-4: Construct selectable multi-scale graph convolution blocks;

[0152] The selectable multi-scale graph convolution block includes: a multi-scale graph convolution module connected in sequence and a selectable multi-scale temporal convolution module.

[0153] like Figure 4 As shown, X in and Xout In order to select the input and output features of the multi-scale graph convolution block, the multi-scale graph convolution module and the selectable multi-scale temporal convolution module are combined to construct a selectable multi-scale graph convolution block; the selectable multi-scale graph convolution block includes a multi-scale graph convolution module (marked in the figure: MS-GC) and a selectable multi-scale temporal convolution module (marked in the figure: SMS-TC) connected in sequence, and the residual connection is used to stabilize the training effect of the model and accelerate convergence. BN is the batch normalization (BN) layer, and ReLU is the activation function.

[0154] In the optional multi-scale graph convolution block, three subsets of skeleton nodes to be calculated are used as input features X in After the multi-scale graph convolution module extracts the spatial features respectively, the three sets of spatial features are concatenated, processed by BN and ReLU, and then combined with the input feature X in After the residual connection, it is sent to the optional multi-scale temporal convolution module. After the temporal feature extraction by the optional multi-scale temporal convolution module, the obtained temporal context information is combined with the input feature X in After the residual connection, it is processed by ReLU and finally the output feature X is obtained. out .

[0155] S3-5: Constructing a selectable multi-scale graph convolutional network;

[0156] like Figure 5 As shown in the figure, the selectable multi-scale graph convolution network includes 10 selectable multi-scale graph convolution blocks (marked as SMS-GCN blocks in the figure), a GAP layer, and an FC layer connected in sequence. The number of channels of the first four of the 10 graph convolution blocks is 64, the number of channels of the middle three is 128, and the number of channels of the last three is 256.

[0157] S3-6: Construct a human skeleton behavior recognition model based on a selectable multi-scale graph convolutional network.

[0158] S4: Training the human skeleton behavior recognition model based on the multi-stream network to obtain a trained human skeleton behavior recognition model, specifically including the following operations:

[0159] Determine skeleton behavior feature data representing static features and dynamic features of skeleton behavior, and model the skeleton behavior feature data; the skeleton behavior feature data includes: joints, bones, joint movements, and bone movements;

[0160] During training, the input spatiotemporal graph is trained based on the multi-stream network for the four information branches of joints, bones, joint movements and bone movements, and four trained human skeleton behavior recognition branch models are obtained. The outputs of the four human skeleton behavior recognition branch models are fused as the output of the multi-stream graph convolutional network. Figure 5 As shown in the figure, the skeleton sequence is processed into multi-branch data and input into the network. Then, at the end of the network, the output size of the features is unified through the GAP layer. Finally, the final recognition result is obtained through the FC layer.

[0161] In step S4, the skeleton behavior feature data is modeled, which specifically includes the following steps:

[0162] S4-1: The joint point itself is modeled as joint information. Assume that the joint information of the tth frame is represented by J t ;

[0163] S4-2: The joint point close to the center of gravity of the bone is the source joint point The joint point far away from the center of gravity of the bone is the target joint point Skeleton information represents the vector from the source joint point to the target joint point

[0164] S4-3: The motion information of joints and bones is expressed as the coordinate difference between two adjacent frames, and the motion information of joints is expressed as

[0165] The motion information of bones is represented as

[0166] In step S4, stochastic gradient descent is used as the optimization strategy during training, and cross entropy is selected as the loss function for gradient back propagation.

[0167] S5: Obtain the video data to be identified, extract the skeleton information data in the video data group to be identified, convert the skeleton information data into a spatiotemporal graph, and send it into a trained human skeleton behavior recognition model to obtain the final recognition result.

[0168] Based on the pytorch deep learning framework, the skeleton behavior recognition method in this method is implemented. The structure of the selectable multi-scale graph convolutional network model in this embodiment is shown in Table 1. The optimization strategy adopts stochastic gradient descent (SGD), the Nesterov momentum is set to 0.9, the iteration cycle is set to 65, the initial learning rate is set to 0.1, and in order to overcome the problem of overfitting during training, the cross entropy is selected as the loss function and the weight decay is set to 0.0004, the decay cycle is set at the 35th and 55th cycles, and the batch size is set to 64.

[0169] We will choose cross entropy as the loss function for gradient backpropagation:

[0170]

[0171] Among them, i represents the sample, c represents the category, and p ic represents the predicted probability that sample i belongs to category c, y ic represents an indicator variable (0 or 1), which is 1 if the behavior category is the same as the category of sample i (i.e., each frame skeleton sequence) in the training set, otherwise it is 0; M represents the number of behavior categories.

[0172] Table 1: Selectable multi-scale graph convolutional network model structure

[0173] Model Layer Number of input and output channels stride Output data size enter [3,300,25] <![CDATA[1 st ]]> in_channels=3,out_channels=64 1 [64,300,25] <![CDATA[2 nd ]]> in_channels=64,out_channels=64 1 [64,300,25] <![CDATA[3 rd ]]> in_channels=64,out_channels=64 1 [64,300,25] <![CDATA[4 th ]]> in_channels=64,out_channels=64 1 [64,300,25] <![CDATA[5 th ]]> in_channels=64,out_channels=128 2 [128,150,25] <![CDATA[6 th ]]> in_channels=128,out_channels=128 1 [128,150,25] <![CDATA[7 th ]]> in_channels=128,out_channels=128 1 [128,150,25] <![CDATA[8 th ]]> in_channels=128 out_channels=256 2 [256,7525] <![CDATA[9 th ]]> in_channels=256,out_channels=256 1 [256,75,25] <![CDATA[10 th ]]> in_channels=256,out_channels=256 1 [256,75,25] GAP [480000] FC [480000,60] or [480000,120]

[60] or

[120]

[0174] Table 1 is an example of the structure of a multi-scale graph convolutional network model that can be selected in this application. Based on the network structure in Table 1, the input data is processed by 10 sub-network structure blocks and then enters the GAP layer. In the GAP layer, the 3-dimensional data output by the sub-network structure block is converted into 1-dimensional data, and then the data is reduced from 480,000 to 60 (120) dimensions through the FC layer, and finally mapped to 60 (120) dimensions for prediction. In order to verify the effectiveness and practicality of the human behavior recognition method in the technical solution of the present invention, the NTU-RGB+D and NTU-RGB+D120 data sets were selected as experimental data sets for experiments.

[0175] NTU-RGB+D is collected using three Microsoft Kinect v2 sensors, which are placed at angles of -45°, 0°, and 45°, respectively. It contains 60 types of actions in total. Actions 1-49 are single-person actions, and actions 50-60 are two-person interactive actions. This dataset has two evaluation methods: cross-object (X-Sub) and cross-view (X-View).

[0176] NTU-RGB+D120 has more samples performed by more subjects in more different camera views than the NTU-RGB+D dataset. This dataset contains 120 actions performed by 106 different subjects. This dataset provides two benchmarks for evaluation, namely cross-subject (X-Sub) and cross-set (X-Set).

[0177] Table 2: Recognition accuracy under X-View and X-Sub in NTU-RGB+D dataset (%)

[0178] scene Joint Flow Skeleton Flow Joint Movement Flow Skeletal Motion Flow Shuangliu Multi-stream X-View 95.1 95.0 93.3 92.4 96.5 96.9 X-Sub 89.7 90.4 87.7 87.5 91.9 92.6

[0179] From the data in Table 2, it can be seen that on the NTU-RGB+D dataset divided into X-View and X-Sub, the technical solution of the present invention is tested under joint flow, skeleton flow, joint motion flow, skeleton motion flow, dual-stream model and multi-stream model. It can be seen that the accuracy of the multi-stream model is improved compared with the single-stream and dual-stream models.

[0180] Table 3: Comparison of recognition accuracy under different models on the NTU-RGB+D dataset (%)

[0181] Model X-Sub X-View CNC-LSTM 83.3 91.8 LAGA-Net 87.1 93.2 ST-GCN 81.5 88.3 2s-AGCN 88.5 95.1 STFE-GCN 89.8 96.0 CTR-GCN 92.4 96.8 HC-GCN 91.2 96.7 MADT-GCN 90.4 96.5 LMA-GCN 88.2 95.0 SMS-GCN 92.6 96.9

[0182] From the data in Table 3, it can be seen that on the NTU dataset divided by X-View and X-Sub, the technical solution SMS-GCN of the present invention achieves the highest recognition accuracy, which is 96.9% and 92.6% respectively, which fully demonstrates the advanced nature of the technical solution of the present invention.

[0183] Table 4: Comparison of recognition accuracy under different models on the NTU-RGB+D120 dataset (%)

[0184] Model X-sub X-setup GCA-LSTM 58.3 59.2 LAGA-Net 81.0 82.2 2s-AGCN 82.9 84.9 CTR-GCN 88.9 90.6 STFE-GCN 84.1 86.3 HC-GCN 87.6 89.0 MADT-GCN 86.5 88.2 LMA-GCN 83.2 84.1 SMS-GCN (joint flow) 85.3 86.6 SMS-GCN (dual stream) 88.8 90.0 SMS-GCN 89.3 90.7

[0185] In order to further verify the generalization performance of the model, tests were conducted on the NTU-RGB+D120 dataset. From the data in Table 4, it can be seen that on the NTU-RGB+D120 dataset divided by X-sub and X-setup, the technical solution SMS-GCN of the present application achieved the highest recognition accuracy, which were 89.3% and 90.7% respectively, further verifying the good recognition accuracy and generalization performance of the model.

[0186] The skeleton behavior recognition method based on the selectable multi-scale graph convolution network provided by the present application can fully and effectively capture more joint features and learn useful time information. At the same time, it can achieve excellent recognition accuracy on large skeleton data sets and has good generalization performance. The technical solution of the present invention includes a multi-scale graph convolution module and a selectable multi-scale temporal convolution module. First, a multi-scale channel topology refinement adjacency matrix is ​​constructed in the spatial domain and participates in graph convolution to construct a multi-scale graph convolution module to establish multi-scale dependencies between paired joints and single joints and multiple joints, thereby achieving full extraction of more representative joint features. Secondly, a selectable multi-scale temporal convolution module is proposed in the time domain, which extracts and fuses temporal context information of different scales by selecting a suitable multi-scale temporal convolution kernel, thereby achieving full attention to useful time information. Finally, a selectable multi-scale graph convolution model is constructed by combining the multi-scale graph convolution module and the selectable multi-scale temporal convolution module, and end-to-end training is achieved based on a multi-stream model on a public large-scale data set, effectively capturing more joint features and learning useful time information.

Claims

1. A skeleton behavior recognition method based on a selectable multi-scale graph convolutional network, comprising the following steps: S1: obtaining an original video sample, preprocessing the original video sample, and obtaining skeleton information data in the original video sample; S2: construct a spatiotemporal graph representing the skeleton information data, including C number of channels, T number of frames and V number of joints; S3: constructing a human skeleton behavior recognition model based on the spatiotemporal graph; S4: Training the human skeleton behavior recognition model based on a multi-stream network to obtain the trained human skeleton behavior recognition model, specifically including the following operations: Determining skeleton behavior feature data representing static features and dynamic features of skeleton behavior, and modeling the skeleton behavior feature data; The skeleton behavior characteristic data includes: joints, bones, joint movements and bone movements; During training, the input spatiotemporal graph is trained separately for four information branches, namely, joints, bones, joint movements and bone movements, based on the multi-stream network to obtain four trained human skeleton behavior recognition branch models, and the outputs of the four human skeleton behavior recognition branch models are fused as the output of the multi-stream graph convolutional network; S5: Acquire the video data to be identified, extract the skeleton information data in the video data group to be identified, convert the skeleton information data into a spatiotemporal graph, and send it into the trained human skeleton behavior recognition model to obtain a final recognition result; Features: The construction of the human skeleton behavior recognition model includes the following steps: S3-1: Divide the neighborhood of all joint points in the spatiotemporal graph into different skeleton node subsets, including: the root joint point itself, the centripetal neighbor nodes close to the skeleton center of gravity, and the centrifugal neighbor nodes far from the skeleton center of gravity; S3-2: Obtain the skeleton node subsets one by one, recorded as: skeleton node subset to be calculated; Construct a multi-scale graph convolution module, take the skeleton node subset to be calculated as input features, and perform spatial feature extraction on all skeleton node subsets based on the multi-scale graph convolution module. The specific operations include: For two nodes v in the subset of skeleton nodes to be calculated, i and v j The dependency relationship between them is modeled to obtain the pairwise joint adjacency matrix, where i,j∈{1,…,V}, V is the number of joints in the subset of skeleton nodes to be calculated; at the same time, for a single node v i and multiple nodes v j 、v j+1 、v j+2 ……v j+k-1 The dependency relationship of (i∈{1,…,V},j∈{1,…,V+1-k},k∈{2,…,V}) is modeled to obtain a multi-scale channel topology refinement adjacency matrix corresponding to each skeleton node subset, and feature extraction is performed on the spatiotemporal graph in combination with a graph convolution operation; S3-3: constructing a selectable multi-scale temporal convolution, extracting the temporal features of the spatiotemporal graph based on the spatial features extracted by the multi-scale graph convolution module, and constructing a temporal convolution layer, and recording the temporal convolution layer as: a selectable multi-scale temporal convolution module; S3-4: Construct selectable multi-scale graph convolution blocks; The selectable multi-scale graph convolution block includes: a multi-scale graph convolution module and a selectable multi-scale temporal convolution module connected in sequence; S3-5: Constructing a selectable multi-scale graph convolutional network; The selectable multi-scale graph convolution network includes: 10 selectable multi-scale graph convolution blocks, a GAP layer and an FC layer connected in sequence; S3-6: Construct the human skeleton behavior recognition model based on a selectable multi-scale graph convolutional network.

2. The skeleton behavior recognition method based on a selectable multi-scale graph convolutional network according to claim 1, characterized in that: In step S3-2, in the multi-scale graph convolution module, two nodes v representing the subset of skeleton nodes to be calculated are i and v j The dependency relationship between them is modeled, which includes the following steps: a1: Take the subset of skeleton nodes to be calculated as input features, denoted as X in , whose size is C×T×V, representing the number of input channels, frames and joints corresponding to the spatiotemporal graph respectively; two 1×1 convolutions with a reduction ratio of r are used to respectively in While performing channel dimension reduction, learn the paired joints v i and v j The features of the , get two features with feature dimensions of C / r×T×V; a2: After using the time pooling operation, the two feature dimensions of C / r×T×V are converted into C / r×V×1 and C / r×1×V features respectively through the unsqueeze function; While preserving the channel dimension, the two features of C / r×V×1 and C / r×1×V are subtracted to obtain the differential topology to obtain the correlation of paired joints in each channel; a3: The differential topology after subtraction is calculated by tanh activation function and 1×1 convolution to obtain the pairwise joint adjacency matrix Q1: Among them, the size of Q1 is C'×V×V, where C' represents the number of output channels; ω(·) represents a 1×1 convolution operation, TP represents a temporal pooling operation, and θ(·) and Represents two 1×1 convolution operations.

3. The skeleton behavior recognition method based on a selectable multi-scale graph convolutional network according to claim 2, characterized in that: In step S3-2, in the multi-scale graph convolution module, a single node v is established i and multiple nodes v j 、v j+1 、v j+2 ……v j+k-1 The dependencies include the following steps: b1: The input feature X is processed by a 1×k convolution with a reduction ratio of r. in Perform channel dimensionality reduction and aggregate the features of k joints; b2: The input feature X is reduced by a 1×1 convolution with a reduction ratio of r. in Perform channel dimension reduction and learn the features of a single joint; b3: After performing temporal pooling operations on the aggregated multi-joint features and the learned single joint features, the unsqueeze function is used to convert the dimensions of the aggregated multi-joint features to C / r×V×1, and the feature dimensions of the learned single joint features to C / r×1×V; While preserving the channel dimension, the C / r×V×1 and C / r×1×V features are subtracted to obtain the differential topology to obtain the correlation of a single joint relative to multiple joints in each channel; b4: The differential topology after subtraction is calculated by tanh activation function and 1×1 convolution to obtain the multi-joint adjacency matrix Q2: Q2=τ(tan(TP(μ(X in ))-TP(λ(X in )))); Among them, the size of Q2 is C'×V×V, τ(·) represents a 1×1 convolution operation, and μ(·) and λ(·) represent two 1×1 convolution operations.

4. The skeleton behavior recognition method based on a selectable multi-scale graph convolutional network according to claim 3, characterized in that: In step S3-2, in the multi-scale graph convolution module, the spatiotemporal graph is subjected to feature extraction in combination with the graph convolution operation. Specifically, The following steps are involved: c1: Establish multi-scale dependencies between pairs of joints and single joints relative to multiple joints. The specific process is as follows: Refine the function based on topology Based on the pairwise joint adjacency matrix Q1 and the multi-joint adjacency matrix Q2, the shared prior topology A∈R V×V Refine and obtain the multi-scale channel topology refinement adjacency matrix R∈R C′×V×V : Among them, α and β are trainable parameters used to adjust the refinement strength; c2: for input feature X in ∈R C×T×V A 1×1 convolution operation is used to perform feature transformation to generate high-order features X′∈R C ′×T×V ; The multi-scale channel topology refinement adjacency matrix R for the c(c∈{1,…,C'})th channel c and high-order features X c ′ is aggregated to realize the graph convolution operation, and the final output X is obtained by connecting the outputs of all channel topological features in series. out ∈R C′×T×V , the calculation formula is as follows: Among them, ψ(·) represents a 1×1 convolution operation, represents a channel aggregation operation, and || represents a concatenation operation.

5. The skeleton behavior recognition method based on a selectable multi-scale graph convolutional network according to claim 1, characterized in that: In step S3-3, the step of constructing the selectable multi-scale temporal convolution module includes: S3301: Given an input feature X in , whose size is C'×T×V, which respectively represent the number of channels, input frames and joints corresponding to the spatiotemporal graph; the selectable multi-scale temporal convolution module includes four parallel branches, each of which uses 1×1 convolution to transform the input feature X in The number of channels is reduced to C' / 4; the two convolution branches use temporal convolutions with different expansion coefficients to extract features of temporal context information of different scales, and the maximum pooling branch realizes the extraction of key frames. Finally, the outputs X1, X2, X3 and X4 of the four branches are obtained. The calculation formula is as follows: Among them, the size of the output features X1, X2, X3 and X4 is C' / 4×T'×V, where T' represents the number of output frames of the feature; and The convolution weights are expansion coefficients d1 and d2 respectively, and the size of the convolution kernel is k. d ×1,W 1×1 is a 1×1 convolution weight, and Maxpool represents the maximum pooling operation; S3302: Use the time selection mechanism to select the appropriate feature map from the features extracted by the time series convolution of different scales, concatenate the features X1 and X2 obtained by the convolution kernels of different scales, and then use the permute function to convert the dimension to T'×C' / 2×V to obtain the feature X p : in, is the permute function, Concat(·) represents the concatenation operation; S3303: For feature X p Extract global average pooling feature X gap And the global maximum pooling feature X gmp ; Among them, δ represents the softmax function, GMP and GAP represent the global maximum pooling and global average pooling operations respectively; S3304: Based on the global maximum pooling feature X gmp And the global average pooling feature X gap Two corresponding time-selective masks are calculated and The calculation formula is as follows: Where Split(·) represents the separation function, σ is the sigmoid activation function; S3305: The output features X1 and X2 of different scales obtained in the feature extraction stage are combined with the corresponding time selection masks and Weighted feature S∈R C′ / 4×T'×V , and then use 1×1 convolution to fuse and upgrade the dimension to C' / 2×T'×V; finally, the upgraded features The final output X is obtained by concatenating the features X3 and X4 obtained in the feature extraction stage. out ∈R C′×T'×V , the calculation formula is as follows: Where ⊙ represents an element-wise product operation.

6. The skeleton behavior recognition method based on a selectable multi-scale graph convolutional network according to claim 1, characterized in that: In step S4, the skeleton behavior feature data is modeled, which specifically includes the following steps: S4-1: The joint point itself is modeled as joint information. Assume that the joint information of the tth frame is represented by J t ; S4-2: The joint point close to the center of gravity of the bone is the source joint point The joint point far away from the center of gravity of the bone is the target joint point Skeleton information represents the vector from the source joint point to the target joint point S4-3: The motion information of joints and bones is expressed as the coordinate difference between two adjacent frames, and the motion information of joints is expressed as The motion information of bones is represented as 7. The skeleton behavior recognition method based on a selectable multi-scale graph convolutional network according to claim 6, characterized in that: In step S4, stochastic gradient descent is used as the optimization strategy during training, and cross entropy is selected as the loss function for gradient back propagation.

8. The skeleton behavior recognition method based on a selectable multi-scale graph convolutional network according to claim 1, characterized in that: In step S1, the step of obtaining the skeleton information data in the original video sample includes: S1-1: performing frame processing on the collected original video samples, and decomposing continuous video clips into a picture sequence including static frames; S1-2: Calculation based on Openpose posture estimation algorithm, including the following: The calculation parameters of the Openpose algorithm are set, and the static frame image obtained by decomposing the video is input into Openpose, and the human skeleton data corresponding to the number of joints in the static frame is obtained, and the calculation parameters include: the number of human joints and the number of human bodies; S1-3: Based on the human body joints and the numbers of the corresponding joints in the Openpose algorithm, a connection relationship of the human body skeleton data is constructed to represent the morphological characteristics of the human body, that is, the skeleton information data is obtained.

9. The skeleton behavior recognition method based on a selectable multi-scale graph convolutional network according to claim 1, characterized in that: In step S2, the step of constructing a spatiotemporal graph representing the skeleton sequence includes: S2-1: Let the spatiotemporal graph of a skeleton sequence with N joints and T frames be G = (V, E), and its joint set be V = {Vti|t = 1, ..., T, i = 1, ..., N}; S2-2: Connect each joint point of the same frame in the spatiotemporal graph G=(V, E) into an edge; S2-3: Connect the same joint points in two consecutive frames in the spatiotemporal graph G=(V, E) into edges.

Images