Space-time diagram convolution method and device for spatio-temporal data

Abstract

The invention provides a spatio-temporal diagram convolution method and device for spatio-temporal data. The method comprises the following steps: acquiring a video to be processed, acquiring a corresponding frame sequence from the video to be processed, generating a corresponding spatio-temporal data sequence based on the frame sequence, and representing the spatio-temporal data sequence as a diagram structure; and inputting the graph structure into a trained spatio-temporal data classification model to obtain a classification result. The spatio-temporal data classification model comprises a first batch processing module, a spatio-temporal feature extraction module and a classification module which are connected in sequence; the first batch processing module inputs features corresponding to the standardized graph structure into the spatial-temporal feature extraction module; and the spatio-temporal feature extraction module fuses the features corresponding to the standardized graph structure, the output of the time convolution sub-module and the output of the first branch, and inputs the fused result into the classification module. According to the method, time sequence characteristics can be better understood, the time dependence relation of long sequence data is processed, and information loss is reduced.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a spatiotemporal graph convolution method and device for spatiotemporal data. Background Art

[0002] Behavior recognition based on spatiotemporal data is the process of analyzing, understanding and classifying specific behaviors using data containing temporal and spatial information. It aims to extract useful features from rich spatiotemporal information to reveal the trend, intention or emotional state of the behavior.

[0003] The spatiotemporal graph convolutional network is a deep learning algorithm based on skeleton behavior recognition. It is a graph convolutional neural network used to process spatiotemporal data. It goes beyond the limitations of previous methods and can capture the correlation and periodicity in spatiotemporal data. It automatically learns spatial and temporal data features, models dynamic skeletons based on the time series representation of human joint positions, and expands to a spatiotemporal graph convolutional network to capture spatiotemporal change relationships. With the in-depth study of ST-GCN, more and more scholars are applying it to other recognition tasks.

[0004] However, with the continuous research of spatiotemporal graph convolution in different fields, the shortcomings of ST-GCN have also been raised. In the existing popular TCN-GCN (ST-GCN) paradigm, it is difficult for TCN to effectively capture long-range dependencies. For temporal modeling, most simply stack multiple temporal convolution layers to extract motion features. The temporal receptive field increased in this way is quite limited. When reaching deeper layers, detailed motion information may disappear to a large extent, thus hindering the interaction between distant input frames, that is, long-term feature extraction cannot be performed, and a relatively correct recognition effect cannot be achieved for long sequences of spatiotemporal data. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a spatiotemporal graph convolution method and device for spatiotemporal data.

[0006] According to a first aspect of the present invention, a spatiotemporal graph convolution method for spatiotemporal data is provided, the method comprising the following steps:

[0007] Step S1: obtaining a video to be processed, obtaining a corresponding frame sequence from the video to be processed, generating a corresponding spatiotemporal data sequence based on the frame sequence, and representing the spatiotemporal data sequence as a graph structure G={V,E}, wherein V is a node matrix set, each node is an object in the video, and E is an edge set;

[0008] Step S2: inputting the graph structure into the trained spatiotemporal data classification model to obtain a classification result;

[0009] The spatiotemporal data classification model includes a first batch processing module, a spatiotemporal feature extraction module and a classification module connected in sequence;

[0010] The first batch processing module standardizes the graph structure, and inputs the features corresponding to the standardized graph structure into the spatiotemporal feature extraction module;

[0011] The spatiotemporal feature extraction module includes a spatial convolution submodule and a residual submodule, a temporal convolution submodule, and a first branch connected in parallel with the spatial convolution submodule; the first branch includes a second batch processing submodule and a TGE submodule connected in sequence; the residual submodule fuses the features corresponding to the standardized graph structure, the output of the temporal convolution submodule, and the output of the first branch, and inputs the fused result into the classification module;

[0012] The classification module includes a global pooling layer, a fully connected layer, and an activation function layer which are connected in sequence.

[0013] Preferably, the step S1: obtains a video to be processed, obtains a corresponding frame sequence from the video to be processed, generates a corresponding spatiotemporal data sequence based on the frame sequence, and represents the spatiotemporal data sequence as a graph structure G={V, E}, wherein V is a node matrix set, each node is an object in the video, and E is an edge set, wherein:

[0014] Node matrix set V = {v ti |t=1,2,…,T; i=1,2,…,N},

[0015] t is the time corresponding to each frame in the frame sequence, T is the last time corresponding to the frame sequence, i is the node number, and N is the total number of nodes;

[0016] The edge set E = {E s ,E F},

[0017] E s = {v ti v tj |t=1,…,T;i,j∈H},

[0018] E F = {v t1i v (t1+1)i |t1=1,…,T-1; i=1,…,N},

[0019] H={1,2,…,N}

[0020] j is the node number, E s is the first edge set, E F is the second edge set, H is the node set, v ti and vtj All are nodes.

[0021] Preferably, the spatial convolution submodule performs a graph convolution operation on each node in the node matrix set, wherein:

[0022] The graph convolution formula is:

[0023]

[0024] f out (v ti ) is the node v ti The feature vector output, B(v ti ) is the node v ti The adjacent node set, Z ti (v tj ) is the normalization term, f in (v tj ) is the graph convolution at node v tj The feature vector input, ω is the weight function, l ti (v tj ) is the node v ti The adjacent nodes v tj The mapping label to which it belongs.

[0025] Preferably, the temporal convolution submodule is out (v ti ) performs dilated convolution to obtain local features in the time domain, where the sampling function for dilated convolution is:

[0026]

[0027] Among them, the node v tj When the value of t is q, the corresponding node is recorded as v qj , d(v tj ,v ti ) is the node v tj To node v ti The shortest distance, K represents the length of the label representation of each time frame, and Γ is the time span, that is, the size of the convolution kernel that controls the time domain;

[0028] The weight function for dilated convolution is:

[0029]

[0030] Among them, l ST (v qj ) is v qj The corresponding mapping label, l ti (v tj ) is the adjacent node v tj Label mapping for .

[0031] Preferably, the first branch comprises a second batch processing submodule and a TGE submodule connected in sequence, wherein the second batch processing submodule performs out (v ti ) for batch processing, and input the obtained results into the TGE submodule; the TGE submodule includes a plurality of multi-head attention submodules and a fully connected feedforward network layer connected in sequence; every two adjacent multi-head attention submodules form a group, and the outputs of the multi-head attention submodules in each group are skip-connected, and the results obtained after the skip connection are layer-normalized through the fully connected feedforward network layer, and the layer normalization results corresponding to each group are fused to obtain the output of the TGE submodule as the fused result.

[0032] Preferably, the residual submodule adds the features corresponding to the standardized graph structure, the output of the temporal convolution submodule and the output of the TGE submodule bit by bit to form a residual output.

[0033] According to a second aspect of the present invention, a spatiotemporal graph convolution device for spatiotemporal data is provided, the device comprising:

[0034] Initialization module: configured to obtain a video to be processed, obtain a corresponding frame sequence from the video to be processed, generate a corresponding spatiotemporal data sequence based on the frame sequence, and represent the spatiotemporal data sequence as a graph structure G={V,E}, where V is a node matrix set, each node is an object in the video, and E is an edge set;

[0035] Classification module: configured to input the graph structure into the trained spatiotemporal data classification model to obtain a classification result;

[0036] The spatiotemporal data classification model includes a first batch processing module, a spatiotemporal feature extraction module and a classification module connected in sequence;

[0037] The first batch processing module standardizes the graph structure, and inputs the features corresponding to the standardized graph structure into the spatiotemporal feature extraction module;

[0038] The spatiotemporal feature extraction module includes a spatial convolution submodule and a residual submodule, a temporal convolution submodule, and a first branch connected in parallel with the spatial convolution submodule; the first branch includes a second batch processing submodule and a TGE submodule connected in sequence; the residual submodule fuses the features corresponding to the standardized graph structure, the output of the temporal convolution submodule, and the output of the first branch, and inputs the fused result into the classification module;

[0039] The classification module includes a global pooling layer, a fully connected layer, and an activation function layer which are connected in sequence.

[0040] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0041] A processor, which is used to execute multiple instructions;

[0042] A memory for storing a plurality of instructions;

[0043] The plurality of instructions are used to be stored by the memory and loaded and executed by the processor to implement the method as described above.

[0044] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the aforementioned method.

[0045] The present invention proposes a spatiotemporal graph convolution method and device for spatiotemporal data, which integrates Transformer into the spatiotemporal graph convolution network and applies it to the behavior recognition task of long-sequence spatiotemporal data. An improved spatiotemporal graph convolution-Transformer behavior recognition method for long-sequence spatiotemporal data is proposed. The network is improved on the basis of the GCN-TCN paradigm, and adopts a method of fusing global modeling and local modeling for the entire spatiotemporal data. That is, after the output of GCN, the TCN module and the Transformer Graph Encoder (TGE) module are respectively incorporated to capture the local time domain features and global time domain features in the sequence respectively. The respective outputs are fused and finally passed through the fully connected layer in sequence through the residual connection, and the softmax obtains the final recognition result.

[0046] The present invention has the following technical effects:

[0047] Enhanced time series modeling capabilities: By combining Transformer and TCN, the model can better understand time series features, process the time dependencies of long sequence data, and reduce information loss.

[0048] Improve recognition accuracy: By fusing Transformer and TCN, local features and global features in long-sequence spatiotemporal data can be effectively integrated, thereby improving recognition accuracy.

[0049] Adaptive feature extraction: Through the design of residual connections and fully connected layers, the model can design features and combinations at different levels to adaptively complete recognition tasks of different data sets (such as video surveillance, traffic flow detection, etc.), further improving the generalization ability of the model.

[0050] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, and the present invention is described by providing the following accompanying drawings. In the accompanying drawings:

[0052] Figure 1 This is a schematic diagram of a process flow of a spatiotemporal graph convolution method for spatiotemporal data according to an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the architecture of a spatiotemporal graph convolution method for spatiotemporal data according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the structure of a TGE module according to an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a multi-head attention mechanism according to an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of a self-attention network structure according to an embodiment of the present invention;

[0057] Figure 6 This is a structural block diagram of a spatiotemporal graph convolution device for spatiotemporal data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] First combine Figure 1-Figure 2 The following describes a spatiotemporal graph convolution method for spatiotemporal data according to an embodiment of the present invention. Figure 1-Figure 2 As shown, the method comprises the following steps:

[0059] Step S1: obtaining a video to be processed, obtaining a corresponding frame sequence from the video to be processed, generating a corresponding spatiotemporal data sequence based on the frame sequence, and representing the spatiotemporal data sequence as a graph structure G={V,E}, wherein V is a node matrix set, each node is an object in the video, and E is an edge set;

[0060] Step S2: inputting the graph structure into the trained spatiotemporal data classification model to obtain a classification result;

[0061] The spatiotemporal data classification model includes a first batch processing module, a spatiotemporal feature extraction module and a classification module connected in sequence;

[0062] The first batch processing module standardizes the graph structure, and inputs the features corresponding to the standardized graph structure into the spatiotemporal feature extraction module;

[0063] The spatiotemporal feature extraction module includes a spatial convolution submodule and a residual submodule, a temporal convolution submodule, and a first branch connected in parallel with the spatial convolution submodule; the first branch includes a second batch processing submodule and a TGE submodule connected in sequence; the residual submodule fuses the features corresponding to the standardized graph structure, the output of the temporal convolution submodule, and the output of the first branch, and inputs the fused result into the classification module;

[0064] The classification module includes a global pooling layer, a fully connected layer, and an activation function layer which are connected in sequence.

[0065] The step S1: obtaining a video to be processed, obtaining a corresponding frame sequence from the video to be processed, generating a corresponding spatiotemporal data sequence based on the frame sequence, and representing the spatiotemporal data sequence as a graph structure G={V, E}, wherein V is a node matrix set, each node is an object in the video, and E is an edge set, wherein:

[0066] Node matrix set V = {v ti |t=1,2,…,T; i=1,2,…,N},

[0067] t is the time corresponding to each frame in the frame sequence, T is the last time corresponding to the frame sequence, i is the node number, and N is the total number of nodes;

[0068] The edge set E = {E s ,E F},

[0069] E s = {v ti v tj |t=1,…,T;i,j∈H},

[0070] E F = {v t1i v (t1+1)i |t1=1,…,T-1; i=1,…,N},

[0071] H={1,2,…,N}

[0072] j is the node number, E s is the first edge set, E F is the second edge set, H is the node set, v ti and v tj All are nodes.

[0073] In the present invention, time series data is abstracted into a graph. In this graph, each node represents an object, and each edge represents the time relationship between two moments. This graph can be regarded as a space-time graph, in which spatial information and time information are represented uniformly. Subsequently, convolution operations are performed on the nodes and edges on this space-time graph to extract the features of the nodes and edges. The graph structure of the present invention has connections within the group and between frames. The first edge set describes the connection of each node in each frame, and the second edge set expresses the connection edges of the same nodes between adjacent frames.

[0074] Furthermore, the spatial convolution submodule performs a graph convolution operation on each node in the node matrix set, wherein:

[0075] The graph convolution formula is:

[0076]

[0077] f out (v ti ) is the node v ti The feature vector output, B(v ti ) is the node v ti The adjacent node set, Z ti (v tj ) is the normalization term, f in (v tj ) is the graph convolution at node v tj The feature vector input, ω is the weight function, l ti (v tj ) is the node v ti The adjacent nodes v tj The mapping label to which it belongs.

[0078] In the present invention, the reasoning process of the graph convolution formula is as follows.

[0079] The spatial convolution output for a certain position X can be expressed as:

[0080]

[0081] The sampling function p(X, h, ω) represents the position X and its neighborhood, and the weight function w(h, ω) represents the weight vector in the dimension space provided by the weight ω, which is used to calculate the inner product of the sampled input feature vector with dimension . Define the node v on the graph ti The sampling function of the neighborhood set B(v ti )={v tj |d(v ti ,v tj )≤D}. Among them, d(v ti ,v tj ) is v ti to v tjThe minimum path. For D = 1, the sampling function is:

[0082] p(v ti ,v tj )=v tj

[0083] Define the mapping ti Mapping points in the neighborhood to corresponding subset labels, the weight function can be expressed as:

[0084] ω(v ti ,v tj )=ω(l ti (v tj ))

[0085] Among them l ti (v tj ) indicates v tj In v ti The label of the molecular set label for the central node, ω(l ti (v tj )) indicates label l ti (v tj ) corresponds to multiple weight values.

[0086] The improved sampling function and weight function are used to express the graph convolution formula, and we get:

[0087]

[0088] The normalized term is:

[0089] Z ti (v tj )=|{v tk |l ti (v tk )=l ti (v tj )}|

[0090] The normalization term is equivalent to the basis of the corresponding subset. Then update the sampling function and weight function to the formula and get the formula:

[0091]

[0092] The residual submodule adds the features corresponding to the standardized graph structure, the output of the time convolution submodule and the output of the TGE submodule bit by bit to form a residual output. As a fusion result, the residual output is used as an aggregation of spatiotemporal dimension information.

[0093] The temporal convolution submodule performs out (v ti) performs dilated convolution to obtain local features in the time domain, where the sampling function for dilated convolution is:

[0094]

[0095] Among them, the node v tj When the value of t is q, the corresponding node is recorded as v qj , d(v tj ,v ti ) is the node v tj To node v ti The shortest distance, K represents the length of the label representation of each time frame, and Γ is the time span, that is, the size of the convolution kernel that controls the time domain;

[0096] The weight function for dilated convolution is:

[0097]

[0098] Among them, l ST (v qj ) is v qj The corresponding mapping label, l ti (v tj ) is the adjacent node v tj Label mapping for .

[0099] Furthermore, if Figure 3-Figure 5 As shown, the first branch includes a second batch processing submodule and a TGE submodule connected in sequence, and the second batch processing submodule performs a out (v ti ) for batch processing, and input the obtained results into the TGE submodule; the TGE submodule includes a plurality of multi-head attention submodules and a fully connected feedforward network layer connected in sequence; every two adjacent multi-head attention submodules form a group, and the outputs of the multi-head attention submodules in each group are connected by skip connection, and the results obtained after the skip connection are normalized by the fully connected feedforward network layer, and the layer normalization results corresponding to each group are fused to obtain the output of the TGE submodule.

[0100] The multi-head attention submodule includes two self-attention heads, and the operation of the multi-head attention submodule is expressed as:

[0101]

[0102] in is a connection operator, Att1 and Att2 are both self-attention heads, FC is a fully connected layer, which projects the connection of the self-attention head into a 128-dimensional feature space, mhAtt(x) is the output of the multi-head attention submodule, and x is the spatial feature after graph convolution.

[0103] The present invention performs temporal modeling on the entire segment of spatiotemporal data, improving deeper models and making them easier and faster to optimize without sacrificing their performance. Multi-Head Attention is a self-attention layer that models the inter-frame relationship between nodes. An example of the multi-head attention module used is in Figure 4 Indicated in.

[0104] in is the concatenation operator Att num is a self-attention head, num=1,2, each head is initialized differently. In theory, each head is allowed to extract different features from the spatiotemporal graph sequence. FC is a fully connected layer that projects the connections of the self-attention head into a 128-dimensional feature space.

[0105] Self-Attention HeadAtt num In each node and all the frames it is in, it is represented as:

[0106]

[0107] Among them, R is a set of real numbers, γ represents the sequence length, λ represents the number of nodes constituting the graph, and d e V num The characteristic size, d k is the matrix K num and Q num The characteristic size of Q num ,K num ,V num is the independent projection matrix of the spatial feature vector corresponding to the node in the graph structure, W Qi , is the weight matrix and is calculated as follows:

[0108]

[0109] d K = 32 is also used as a scaling factor, i.e. Q i .K i Multiply To scale large dot product values, the operations involved in the calculation method are scaled dot products, which are used for local operations between connectors. Figure 5 An example of a self-attention head is shown in .

[0110] like Figure 6 As shown, the present invention provides a spatiotemporal graph convolution device for spatiotemporal data, the device comprising:

[0111] Initialization module: configured to obtain a video to be processed, obtain a corresponding frame sequence from the video to be processed, generate a corresponding spatiotemporal data sequence based on the frame sequence, and represent the spatiotemporal data sequence as a graph structure G={V,E}, where V is a node matrix set, each node is an object in the video, and E is an edge set;

[0112] Classification module: configured to input the graph structure into the trained spatiotemporal data classification model to obtain a classification result;

[0113] The spatiotemporal data classification model includes a first batch processing module, a spatiotemporal feature extraction module and a classification module connected in sequence;

[0114] The first batch processing module standardizes the graph structure, and inputs the features corresponding to the standardized graph structure into the spatiotemporal feature extraction module;

[0115] The spatiotemporal feature extraction module includes a spatial convolution submodule and a residual submodule, a temporal convolution submodule, and a first branch connected in parallel with the spatial convolution submodule; the first branch includes a second batch processing submodule and a TGE submodule connected in sequence; the residual submodule fuses the features corresponding to the standardized graph structure, the output of the temporal convolution submodule, and the output of the first branch, and inputs the fused result into the classification module;

[0116] The classification module includes a global pooling layer, a fully connected layer, and an activation function layer which are connected in sequence.

[0117] The embodiment of the present invention further provides an electronic device, including:

[0118] A processor, which is used to execute multiple instructions;

[0119] A memory for storing a plurality of instructions;

[0120] The plurality of instructions are used to be stored by the memory and loaded and executed by the processor to implement the method as described above.

[0121] The embodiment of the present invention further provides a computer-readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the method as described above.

[0122] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0123] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0124] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0126] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a physical server, or a network cloud server, etc., and the Ubuntu operating system must be installed) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0127] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A spatiotemporal graph convolution method for spatiotemporal data, characterized in that: The method comprises the following steps: Step S1: obtaining a video to be processed, obtaining a corresponding frame sequence from the video to be processed, generating a corresponding spatiotemporal data sequence based on the frame sequence, and representing the spatiotemporal data sequence as a graph structure G={V,E}, wherein V is a node matrix set, each node is an object in the video, and E is an edge set; Step S2: inputting the graph structure into the trained spatiotemporal data classification model to obtain a classification result; The spatiotemporal data classification model includes a first batch processing module, a spatiotemporal feature extraction module and a classification module connected in sequence; The first batch processing module standardizes the graph structure, and inputs the features corresponding to the standardized graph structure into the spatiotemporal feature extraction module; The spatiotemporal feature extraction module includes a spatial convolution submodule and a residual submodule, a temporal convolution submodule, and a first branch connected in parallel with the spatial convolution submodule; the first branch includes a second batch processing submodule and a TGE submodule connected in sequence; the residual submodule fuses the features corresponding to the standardized graph structure, the output of the temporal convolution submodule, and the output of the first branch, and inputs the fused result into the classification module; The classification module includes a global pooling layer, a fully connected layer, and an activation function layer which are connected in sequence.

2. The method according to claim 1, characterized in that The step S1: obtaining a video to be processed, obtaining a corresponding frame sequence from the video to be processed, generating a corresponding spatiotemporal data sequence based on the frame sequence, and representing the spatiotemporal data sequence as a graph structure G={V, E}, wherein V is a node matrix set, each node is an object in the video, and E is an edge set, wherein: Node matrix set V = {v ti |t=1,2,…,T; i=1,2,…,N}, t is the time corresponding to each frame in the frame sequence, T is the last time corresponding to the frame sequence, i is the node number, and N is the total number of nodes; The edge set E = {E s ,E F }, E s ={v ti v tj |t=1,…,T;i,j∈H}, From F ={v t1i v (t1+1)i |t1=1,…,T-1;i=1,…,N}, H={1,2,…,N} j is the node number, E s is the first edge set, E F is the second edge set, H is the node set, v ti and v tj All are nodes.

3. The method according to claim 1, characterized in that The spatial convolution submodule performs a graph convolution operation on each node in the node matrix set, where: The graph convolution formula is: f out (v ti ) is the node v ti The feature vector output, B(v ti ) is the node v ti The adjacent node set, Z ti (v tj ) is the normalization term, f in (v tj ) is the graph convolution at node v tj The feature vector input, ω is the weight function, l ti (v tj ) is the node v ti The adjacent nodes v tj The mapping label to which it belongs.

4. The method according to claim 3, characterized in that The temporal convolution submodule performs out (v ti ) performs dilated convolution to obtain local features in the time domain, where the sampling function N(v ti )for: Among them, the node v tj When the value of t is q, the corresponding node is denoted as v qj , d(v tj ,v ti ) is the node v tj To node v ti The shortest distance, K represents the length of the label representation of each time frame, and Γ is the time span, that is, the size of the convolution kernel that controls the time domain; The weight function for dilated convolution is: Among them, l ST (v qj ) is v qj The corresponding mapping label, l ti (v tj ) is the adjacent node v tj Label mapping for .

5. The method according to claim 3, characterized in that The first branch includes a second batch processing submodule and a TGE submodule connected in sequence, wherein the second batch processing submodule performs a out (v ti ) for batch processing, and input the obtained results into the TGE submodule; the TGE submodule includes a plurality of multi-head attention submodules and a fully connected feedforward network layer connected in sequence; every two adjacent multi-head attention submodules form a group, and the outputs of the multi-head attention submodules in each group are connected by skip connection, and the results obtained after the skip connection are normalized by the fully connected feedforward network layer, and the layer normalization results corresponding to each group are fused to obtain the output of the TGE submodule.

6. The method according to any one of claims 3 to 4, characterized in that: The residual submodule adds the features corresponding to the standardized graph structure, the output of the temporal convolution submodule, and the output of the TGE submodule bit by bit to form a residual output as a fused result.

7. A spatiotemporal graph convolution device for spatiotemporal data, characterized in that: The device comprises: Initialization module: configured to obtain a video to be processed, obtain a corresponding frame sequence from the video to be processed, generate a corresponding spatiotemporal data sequence based on the frame sequence, and represent the spatiotemporal data sequence as a graph structure G={V,E}, where V is a node matrix set, each node is an object in the video, and E is an edge set; Classification module: configured to input the graph structure into the trained spatiotemporal data classification model to obtain a classification result; The spatiotemporal data classification model includes a first batch processing module, a spatiotemporal feature extraction module and a classification module connected in sequence; The first batch processing module standardizes the graph structure, and inputs the features corresponding to the standardized graph structure into the spatiotemporal feature extraction module; The spatiotemporal feature extraction module includes a spatial convolution submodule and a residual submodule, a temporal convolution submodule, and a first branch connected in parallel with the spatial convolution submodule; the first branch includes a second batch processing submodule and a TGE submodule connected in sequence; the residual submodule fuses the features corresponding to the standardized graph structure, the output of the temporal convolution submodule, and the output of the first branch, and inputs the fused result into the classification module; The classification module includes a global pooling layer, a fully connected layer, and an activation function layer which are connected in sequence.

8. An electronic device comprising: A processor, which is used to execute multiple instructions; A memory for storing a plurality of instructions; The plurality of instructions are used to be stored in the memory and loaded and executed by the processor according to any one of claims 1 to 6.

9. A computer-readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the method as claimed in any one of claims 1 to 6.

Images