Human body behavior recognition method and system based on space-time fusion network

Abstract

The invention discloses a human body behavior recognition method and system based on a space-time fusion network, and relates to the technical field of behavior recognition, and the method comprises the steps: obtaining a human body skeleton sequence, carrying out the joint position coding of the sequence, taking the coded sequence as an input feature, inputting the input feature into the space-time fusion network based on graph convolution and Transform, and outputting a human body behavior recognition result; wherein the space-time fusion network comprises a plurality of groups of graph convolution Transform space fusion modules and multi-scale time fusion modules which are connected in sequence, after input features are input, local structure features and global space features are extracted through the parallel adaptive graph convolution module and Transform module respectively, feature fusion is carried out through the correlation fusion module, and the local structure features and the global space features are fused through the correlation fusion module. And the fused features are input into a multi-scale time fusion module, and are subjected to feature extraction through a multi-scale convolution branch and a Transform branch and then spliced, so that a more accurate human behavior recognition result is output.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior recognition, and in particular to a method and system for human behavior recognition based on a spatiotemporal fusion network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The wide application prospects of human behavior recognition technology are inseparable from its importance in daily life. For example, in the field of video surveillance, human behavior recognition technology can realize intelligent monitoring, automatically detect abnormal events and issue reminders, thereby further ensuring public safety. With the continuous advancement of technology, the extraction of human behavior data has become more convenient and faster. At present, the existing behavior recognition methods mainly include two behavior recognition models based on RGB video data and skeleton data. Compared with RGB video data, skeleton data does not depend on ambient lighting conditions or background, and has stronger robustness. In addition, skeleton data directly represents the skeleton structure and motion state of the human body, and has a higher level of semantic information. In addition, skeleton data only contains human joint information and has better privacy. Therefore, skeleton data has become an important data representation form in behavior recognition, and skeleton-based human behavior recognition methods are widely used in video surveillance, smart home, health detection and other fields.

[0004] In the skeleton-based action recognition task, convolutional neural network and recurrent neural network methods were used for training in the early stage. However, due to the non-Euclidean spatial structure of skeleton data, the early methods had certain limitations. With the development of deep learning, more model methods have been proposed and applied to skeleton action recognition tasks. At present, graph convolution and Transformer models have shown strong performance in processing skeleton data. As the research in the field of action recognition continues to deepen, different types of graph convolutional networks and Transformer models have been designed to extract skeleton features. Among them, for graph convolutional networks, it extracts spatiotemporal features by modeling the relationship between joints, which can well capture spatial topological information, and the network updates joint features by aggregating the information of the adjacency matrix, which has good local correlation; however, the graph convolutional network is limited by the size of the receptive field, and the adjacency matrix changes based on the inherent connection of the skeleton, which makes it impossible to calculate the joints that are not directly connected, resulting in the loss of some key information, affecting the accuracy of the final recognition. For the Transformer network, it can capture the global dependencies between different positions in the sequence, and has good global feature extraction capabilities. In addition, the Transformer is not affected by the position of the joints and can extract features of non-adjacent joints. However, the Transformer has weak spatial perception capabilities and pays too much attention to global information, which leads to the loss of key local information.

[0005] Therefore, for behavior recognition tasks, existing human behavior recognition methods based on deep learning algorithms such as graph convolution and Transformer models have the problem of local and global information imbalance, which affects the accuracy of the final behavior recognition. Summary of the invention

[0006] To address the deficiencies of the above-mentioned prior art, the present invention provides a method and system for human behavior recognition based on a spatiotemporal fusion network. In the spatial dimension, the long-range dependency relationship between human joints is identified through the self-attention mechanism of the Transformer, and the local structural features are identified by using the adjacency matrix of the adaptive graph convolutional network. A correlation fusion method is introduced to extract highly correlated features from local information and global information for fusion, thereby avoiding information loss and obtaining more comprehensive spatial information. In the temporal dimension, effective features are extracted at different scales and local areas through multi-scale temporal convolution branches, and the long-range dependency relationship in time series data is processed through the Transformer branch, thereby making up for the deficiency of multi-scale dilated convolution in capturing long-range dependency relationships. The feature combination of different branches enables the model to more comprehensively understand the complex behavior patterns in skeleton movements and more accurately identify human behaviors.

[0007] In a first aspect, the present invention provides a method for human behavior recognition based on a spatiotemporal fusion network.

[0008] A human behavior recognition method based on spatiotemporal fusion network, comprising: Obtain a pedestrian motion video, extract the human skeleton of each frame in the video and represent it in a graph to obtain a human skeleton sequence; Encode the joint positions of the human skeleton sequence to obtain the characteristics of the human skeleton sequence; Input the human skeleton sequence features into the spatiotemporal fusion network based on graph convolution and Transformer, and output the human behavior recognition results; Among them, the spatiotemporal fusion network includes multiple groups of graph convolution Transformer spatial fusion modules and multi-scale time fusion modules connected in sequence. The input features are input into the graph convolution Transformer spatial fusion module. After the local structural features and global spatial features are extracted respectively by the parallel adaptive graph convolution module and Transformer module, the features are fused through the correlation fusion module. The fused features are then input into the multi-scale time fusion module. After the features are extracted respectively by the parallel multi-scale convolution branch and Transformer branch and then spliced, the human behavior recognition results are output based on the spliced ​​features.

[0009] In a second aspect, the present invention provides a human behavior recognition system based on a spatiotemporal fusion network.

[0010] A human behavior recognition system based on spatiotemporal fusion network, comprising: The data acquisition module is used to acquire pedestrian motion videos, extract the human skeleton of each frame in the video and represent it in a graph to acquire a human skeleton sequence; A data preprocessing module is used to encode the joint positions of the human skeleton sequence and obtain the characteristics of the human skeleton sequence; The human behavior recognition module is used to input the human skeleton sequence features into the spatiotemporal fusion network based on graph convolution and Transformer, and output the human behavior recognition results; Among them, the spatiotemporal fusion network includes multiple groups of graph convolution Transformer spatial fusion modules and multi-scale time fusion modules connected in sequence. The input features are input into the graph convolution Transformer spatial fusion module. After the local structural features and global spatial features are extracted respectively by the parallel adaptive graph convolution module and Transformer module, the features are fused through the correlation fusion module. The fused features are then input into the multi-scale time fusion module. After the features are extracted respectively by the parallel multi-scale convolution branch and Transformer branch and then spliced, the human behavior recognition results are output based on the spliced ​​features.

[0011] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the method described in the first aspect are completed.

[0012] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the method described in the first aspect.

[0013] One or more of the above technical solutions have the following beneficial effects: 1. The present invention provides a method and system for human behavior recognition based on a spatiotemporal fusion network, which effectively extracts local and global spatiotemporal features. In the spatial dimension, a graph convolution Transformer spatial fusion module is designed to identify the long-range dependencies between human joints through the self-attention mechanism of the Transformer, and the local structural features are identified by using the adjacency matrix of the adaptive graph convolution network to ensure the extraction of local and global information. At the same time, a correlation fusion method is introduced to extract highly correlated features from local information and global information for fusion, so as to better combine the two features, avoid information loss, and obtain more comprehensive spatial information, thereby improving the overall accuracy of the fusion process; in the temporal dimension, a multi-scale time fusion module is designed, which combines multi-scale convolution and Transformer to effectively enhance the expressiveness of the model. It extracts effective features at different scales and local areas through multi-scale temporal convolution branches, and processes the long-range dependencies in time series data through Transformer branches, thereby making up for the shortcomings of multi-scale hole convolution in capturing long-range dependencies. The model can more comprehensively understand the complex behavior patterns in skeleton movements and more accurately identify human behaviors through the combination of features from different branches.

[0014] 2. In the present invention, in order to improve the sensitivity of the adaptive graph convolution network to local features, a prior information matrix is ​​added to the adaptive graph convolution module, which is combined with the shortest path algorithm to smoothly adjust the connectivity between different joint pairs in the form of exponential scaling, so that the model can better capture more complex and comprehensive spatial relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0016] Figure 1 The overall algorithm framework diagram of the human behavior recognition method based on the spatiotemporal fusion network according to the embodiment of the present invention; Figure 2 Schematic diagram of the structure of the Transformer module in an embodiment of the present invention; Figure 3 Schematic diagram of the structure of an adaptive graph convolution module in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a multi-correlation fusion module in an embodiment of the present invention; Figure 5 Schematic diagram of the structure of a multi-scale time fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0018] Embodiment 1 This embodiment provides a method for human behavior recognition based on a spatiotemporal fusion network. Figure 1 As shown, the specific steps include: Step S1, obtaining a pedestrian motion video, extracting the human skeleton of each frame in the video and performing a graph representation to obtain a human skeleton sequence; Step S2, encoding the joint positions of the human skeleton sequence to obtain the features of the human skeleton sequence; Step S3, inputting the human skeleton sequence features into the spatiotemporal fusion network based on graph convolution and Transformer, and outputting the human behavior recognition results; Among them, the spatiotemporal fusion network includes multiple groups of graph convolution Transformer spatial fusion modules and multi-scale time fusion modules connected in sequence. The input features are input into the graph convolution Transformer spatial fusion module. After the local structural features and global spatial features are extracted respectively by the parallel adaptive graph convolution module and Transformer module, the features are fused through the correlation fusion module. The fused features are then input into the multi-scale time fusion module. After the features are extracted respectively by the parallel multi-scale convolution branch and Transformer branch and then spliced, the human behavior recognition results are output based on the spliced ​​features.

[0019] The following content introduces the human behavior recognition method based on spatiotemporal fusion network proposed in this embodiment in more detail.

[0020] In step S1, a pedestrian motion video is obtained, and the human skeleton of each frame in the video is extracted, and the extracted human skeleton is represented by a graph to obtain a sequence of human skeleton graph representations, which is recorded as a human skeleton sequence. Among them, for the human skeleton extracted from each frame of the video image, the human skeleton is represented by a graph, with human joints as nodes and human skeletons as edges, to construct a human skeleton graph representation, which is recorded as , represents a node set, Indicates i nodes, Indicates the number of nodes in the graph; Represents edge set, adjacency matrix represents the connection of nodes. Then, the human skeleton graph representation of consecutive frames constitutes a human skeleton graph representation sequence, recorded as human skeleton sequence, which can be expressed as , where 3 is the input dimension, is the time frame, It is the number of joints in a frame, that is, the number of nodes.

[0021] In step S2, in order to facilitate subsequent data processing, the human skeleton sequence is first preprocessed by joint position encoding, that is, firstly, the human skeleton sequence is projected into a high-dimensional feature space through linear transformation; then, the position feature of each node is automatically obtained by using a trainable parameter matrix to generate node position embedding; then, the node position embedding is combined with the linear transformation to obtain the human skeleton sequence feature after joint position embedding, which can be specifically expressed as: ; ; in, is the processed feature (i.e. human skeleton sequence feature), where represents the input dimension, represents a linear transformation, represents the normal distribution initialization, Indicates that the matrix is ​​a trainable parameter, which also means that during the training process, It will be adjusted during the optimization process to learn the node position embedding that best suits the model.

[0022] In step S3, the human skeleton sequence features are input into the spatiotemporal fusion network based on graph convolution and Transformer, and the human behavior classification and recognition results are output.

[0023] like Figure 1As shown, the spatiotemporal fusion network includes multiple groups of graph convolution Transformer spatial fusion modules and multi-scale time fusion modules connected in sequence. Each group is used as a basic block. In this embodiment, a total of 9 basic blocks are set, each of which is composed of graph convolution and Transformer spatial fusion modules and multi-scale time fusion modules. By adopting 9 basic blocks, it is ensured that the network has sufficient depth when extracting spatiotemporal features, and at the same time, the complexity of the model is controlled to improve the training effect and generalization ability.

[0024] The specific content and ideas of the above graph convolution Transformer spatial fusion module are as follows: In the skeleton-based action recognition task, the graph convolutional network uses the adjacency matrix of the skeleton graph to define the connection relationship between nodes, thereby capturing the interaction and dependency between adjacent nodes. Since the graph convolutional network is modeled and learned based on the connection mode of the adjacency matrix, it can efficiently extract the local relationship between nodes in the skeleton graph. The graph convolutional network can generate feature representations that are strongly related to the node neighborhood structure. These features can provide detailed spatial information in the action recognition task. For example, the relative position of the arm and shoulder are important expressions of local motion in the human skeleton. However, although this network is powerful in processing local spatial features, it lacks the ability to process long-range dependencies between remote nodes. This limitation makes adaptive graph convolutions perform poorly when facing complex behavior patterns that require comprehensive global information. In addition, deep graph convolutions may face gradient vanishing or over-smoothing problems, making it difficult for information to be effectively transmitted.

[0025] Similarly, in the skeleton-based action recognition task, the Transformer can connect each node with all other nodes through the self-attention mechanism, thereby integrating and propagating information globally. The Transformer can capture long-range dependencies in the skeleton, such as the coordination between hand movements and foot movements. However, although the Transformer is strong in modeling global dependencies, it tends to focus on the overall feature pattern, which ignores some key local structural information, especially when it comes to fine movements or small posture changes, and brings redundant information to the node features.

[0026] Taking into account the above-mentioned drawbacks of graph convolutional networks and Transformer networks, in order to capture richer features, in the spatial dimension, in this embodiment, a graph convolutional Transformer spatial fusion module is constructed. The module adopts a dual-branch structure, which is composed of an adaptive graph convolution module and a Transformer module, such as Figure 1As shown in the figure, the Transformer module is used to model the complex global spatial long-distance dependencies between human joints with the help of the self-attention mechanism and extract global spatial features; the adaptive graph convolution module is used to model the correlation between human joints, and to capture the complex local dependencies between joints in the skeleton sequence and extract local structural features by improving adaptive learning, that is, combining prior knowledge to adaptively adjust the graph topology structure; in addition, the graph convolution Transformer spatial fusion module also includes a correlation fusion module for fusing these two types of features at different levels. Among them, the local feature representation of the graph convolution network can provide the model with detailed local information, making up for the shortcomings of the Transformer in processing local details; at the same time, the global feature representation of the Transformer can enhance the global perspective of the adaptive graph convolution, making the feature representation more comprehensive and balanced; this complementarity makes the final fused feature representation have both local accuracy and global consistency, thereby effectively improving the expressive power of the model.

[0027] like Figure 2 As shown in the figure, the Transformer module includes a self-attention mechanism, that is, in order to effectively capture the global dependencies of nodes, the self-attention mechanism in the Transformer is used to extract features. The specific process of implementing the self-attention mechanism can be expressed as: ; ; in, represents the input features, is the feature obtained after the Transformer module; denote query, key, and value matrices respectively; Represents the key matrix Dimensions; express Learning weights for convolution; It is a matrix with spatial information, which is used to enhance the global representation ability; Represents the feedforward layer, which consists of a Convolution composition; represents the residual connection, Convolution operation and Operation composition; represents the attention function; Represents the activation function.

[0028] Through the operation of the above Transformer module, the global dependency between the nodes in the input features is captured. Regardless of the positions of the two nodes in the input data, the module can understand the relationship between the two nodes through the attention mechanism.

[0029] like Figure 3 As shown in the figure, the adaptive graph convolution module adaptively adjusts the topological structure of the graph by modeling the correlation between joints and combining prior knowledge, so as to better capture the local information in the input data, that is, to extract local structural features. Specifically, for the input feature First, the input features are transformed through two different linear transformation functions (i.e., convolution) to generate initial features for constructing the relationship matrix; second, the two initial features are activated by the activation function (Tanh function) and then the difference operation is performed to obtain the relationship matrix used to represent the relationship between nodes; then, the relationship matrix is ​​combined with the prior knowledge matrix to generate the weight matrix of the graph convolution. Among them, the above-mentioned prior verification knowledge matrix (referred to as the prior knowledge matrix) consists of an adjacency matrix and an exponential scaling matrix with the shortest path property The specific process of the above relationship modeling can be expressed as: ; ; in, represents the generated relationship matrix; represents the activation function; and are two different transformation functions, consisting of two The convolution is composed of convolutions, each of which projects the input features into different dimensional spaces; the prior knowledge matrix is ​​composed of the adjacency matrix Scaling the matrix with exponential Add together.

[0030] While modeling relationships, input features After linear transformation (i.e., convolution), multi-scale spatial features are extracted through dilated convolution without increasing the number of parameters; finally, the weight matrix and multi-scale spatial features are combined through the aggregation function to obtain local structural features, which can be expressed as follows: ; in, Features output by the adaptive graph convolution module; represents a dilated convolution; It is used to extract features convolution; is an aggregation function, which is used to combine the relationship matrix obtained after relationship modeling with the multi-scale spatial features.

[0031] The adaptive graph convolution module can effectively integrate the spatial information of the graph structure into the feature map and capture local feature information. This local feature extraction helps to capture fine-grained joint information, thereby improving the model's sensitivity to subtle changes and enhancing the model's performance in complex scenarios.

[0032] like Figure 3 As shown in the figure, the prior knowledge matrix can also be called the prior information matrix. The prior information matrix uses the shortest path algorithm to extract the distance information between nodes and introduces nonlinear scaling to more accurately and smoothly adjust the connection strength between different nodes. Adding the prior information matrix to the adaptive graph convolution module can better capture the local spatial relationship between joints.

[0033] The construction of the above prior knowledge matrix includes: First, the adjacency matrix based on the human skeleton feature graph is (Its dimensions are ), build a distance matrix , and initialize it, the distance matrix is: ; In the above formula, express( ) position, for any two nodes in the adjacency matrix , the value on the diagonal represents the distance from the node to itself, which is set to 0; if there is no connection between two nodes, use Indicates that the two nodes are unreachable.

[0034] Secondly, in order to enhance the capture of local features while maintaining the overall spatial structure, the Floyd-Warshall algorithm is used to calculate the shortest path matrix between all joints. , which represents the shortest distance between any two joints, and also provides the model with distance information between joints. , by checking whether there is a Can make joints arrive If there is such a path, the initialized distance matrix is ​​updated. , the specific process can be expressed as: ; in, Represents a slave node Via Node Arrival Node If the path is shorter than the original path length, update , and record the intermediate nodes passed .

[0035] Finally, we get the shortest path matrix After that, an exponential scaling operation is introduced to convert the shortest path matrix Each element in is passed through the parameter Perform exponential operation to get a new matrix , that is, the exponential scaling matrix , so that the connection weights between joints can be adjusted by scaling the distance, especially emphasizing local features. The above exponential scaling matrix The calculation can be expressed as: ; In this embodiment, The value of is 0.7, and the specific value can be known from the test results.

[0036] Furthermore, the obtained prior information matrix is ​​combined with the adjacency matrix to perform graph convolution operations. The exponential matrix can smoothly adjust the connection weights between joints to avoid overly aggressive pruning or strengthening; the introduced nonlinear scaling can adjust the connection strength between different nodes and capture complex spatial relationships. Compared with simple linear scaling or traditional adjacency matrices, the prior information matrix can provide smoother weight adjustments and make feature representations more delicate, which helps the model converge more stably during the learning process and improves overall performance.

[0037] like Figure 4 As shown in the figure, the correlation fusion module includes a depth-separable convolutional layer, a pooling layer, a convolutional layer and an activation layer connected in sequence. The correlation fusion module effectively combines the two types of features extracted at different levels, so that the model can simultaneously have the ability to perceive local details and understand global information.

[0038] Specifically, first, the extracted local structural features and global spatial features are input into the correlation fusion module, and then input into the depthwise separable convolutional layer after splicing operation. Dimensionality reduction is performed to further extract spatial information while reducing the amount of calculation, which can be expressed as: ; in, and Represent the features extracted from the Transformer module and the adaptive graph convolution module respectively; It is the feature obtained after dimensionality reduction by deep separable convolution; Represents a join operation.

[0039] Then, the reduced-dimensional features are average-pooled by the pooling layer and input into the convolution layer. The attention weights are generated by the convolution operation and activated by the activation layer to selectively enhance the features of different channels. The specific formula is: ; ; in, is the obtained correlation coefficient, that is, the attention weight, is the activation function, is the average pooling operation.

[0040] Finally, the generated attention weight matrix is ​​used to weight the two input features and then concatenate them, and the final fusion feature is output after nonlinear processing. , which can be expressed as: ; in, To obtain relevant fused features, the global information extracted by the Transformer module is fused with the local information extracted by the adaptive graph convolution module. is the activation function.

[0041] In the above way, the fused feature space is more expressive and can better describe the complexity of the behavior. Considering that some behaviors may depend on specific local joint changes, while others require analysis of the overall motion trajectory, the fusion of these two types of information can complement each other and enhance the generalization ability of the model.

[0042] like Figure 5 As shown in the figure, the multi-scale time fusion module includes a multi-scale convolution branch and a Transformer branch, wherein the multi-scale convolution branch includes multiple parallel convolution branches and pooling branches with different hole rates, which are used to extract deep features in different scales and local areas; the Transformer branch uses the self-attention mechanism to learn the global relationship between each moment in the input feature sequence, adaptively captures the long-distance dependency between time points, and extracts time series features.

[0043] In this embodiment, the multi-scale time fusion module consists of four branches, including a multi-scale convolution branch based on three branches and a Transformer branch, wherein the three convolution branches respectively include: a dilated convolution branch with a dilation rate of 1 , used to capture fine-grained temporal features; while maintaining computational efficiency, a dilated convolution branch with a dilation rate of 2 is used To capture the long-term dependencies in local time series, it can be expressed as: ; in, These are the features extracted by the two dilated convolution branches respectively.

[0044] In addition, a pooling branch is included, which is used to select the local maximum feature in the time dimension, thereby helping the model to extract key features from a longer time scale, which can be expressed as: ; in, The features extracted by the pooling branch, Represents the maximum pooling operation. The pooling operation reduces redundant information and highlights the important time points in the behavior.

[0045] like Figure 5 As shown in the figure, the global attention branch is the Transformer branch, which uses the Transformer's self-attention mechanism to learn the global relationship between each moment in the sequence, and adaptively captures the long-distance dependency between time points to extract time series features, which can be expressed as: ; ; ; in, Represents the features extracted from the Transformer branch; Indicates segmentation by channel, used for channel compression; It is a dimensional transformation that can better achieve feature conversion; Represents a linear transformation.

[0046] Finally, in order to ensure that the features are fully utilized, the features extracted from different branches are spliced ​​to obtain the output features , the formula is: .

[0047] Although the atrous convolution branch effectively expands the range of the receptive field, the receptive field is still predetermined and still depends on the local structure of the convolution kernel, making it difficult to capture complex global relationships. Therefore, using the global attention branch to dynamically select relevant time points can capture complex dependencies across the entire sequence, rather than just being limited to a fixed local time range, thereby making up for the problem of insufficient global time feature extraction.

[0048] In step S3, feature extraction is performed through the above-mentioned spatiotemporal fusion network based on graph convolution and Transformer, and then Softmax is used to perform behavior classification on the extracted features to obtain the final human behavior recognition result.

[0049] This embodiment uses the existing NTU RGBD dataset, which has 60 types of human behaviors, of which 40 are daily behaviors, 9 are health-related behaviors, and 11 are two-person interactions. Preferably, the NTU RGBD120 dataset can also be used, which is an expansion of the NTU RGBD dataset and has 120 behaviors. After dividing the adopted dataset into a training set and a validation set, the above method is used for training and validation to achieve higher accuracy human behavior recognition.

[0050] Embodiment 2 This embodiment provides a human behavior recognition system based on a spatiotemporal fusion network, which specifically includes: The data acquisition module is used to acquire pedestrian motion videos, extract the human skeleton of each frame in the video and represent it in a graph to acquire a human skeleton sequence; A data preprocessing module is used to encode the joint positions of the human skeleton sequence and obtain the characteristics of the human skeleton sequence; The human behavior recognition module is used to input the human skeleton sequence features into the spatiotemporal fusion network based on graph convolution and Transformer, and output the human behavior recognition results; Among them, the spatiotemporal fusion network includes multiple groups of graph convolution Transformer spatial fusion modules and multi-scale time fusion modules connected in sequence. The input features are input into the graph convolution Transformer spatial fusion module. After the local structural features and global spatial features are extracted respectively by the parallel adaptive graph convolution module and Transformer module, the features are fused through the correlation fusion module. The fused features are then input into the multi-scale time fusion module. After the features are extracted respectively by the parallel multi-scale convolution branch and Transformer branch and then spliced, the human behavior recognition results are output based on the spliced ​​features.

[0051] Embodiment 3 This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps in the human behavior recognition method based on the spatiotemporal fusion network as described above are completed.

[0052] Embodiment 4 This embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the human behavior recognition method based on a spatiotemporal fusion network as described above are completed.

[0053] The steps involved in the above embodiments 2 to 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0054] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0055] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not a limitation of the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A human behavior recognition method based on spatiotemporal fusion network, characterized in that: include: Obtain a pedestrian motion video, extract the human skeleton of each frame in the video and represent it in a graph to obtain a human skeleton sequence; Encode the joint positions of the human skeleton sequence to obtain the characteristics of the human skeleton sequence; Input the human skeleton sequence features into the spatiotemporal fusion network based on graph convolution and Transformer, and output the human behavior recognition results; Among them, the spatiotemporal fusion network includes multiple groups of graph convolution Transformer spatial fusion modules and multi-scale time fusion modules connected in sequence. The input features are input into the graph convolution Transformer spatial fusion module. After the local structural features and global spatial features are extracted respectively by the parallel adaptive graph convolution module and Transformer module, the features are fused through the correlation fusion module. The fused features are then input into the multi-scale time fusion module. After the features are extracted respectively by the parallel multi-scale convolution branch and Transformer branch and then spliced, the human behavior recognition results are output based on the spliced ​​features.

2. A human behavior recognition method based on a spatiotemporal fusion network as claimed in claim 1, characterized in that: For the human skeleton extracted from each frame of video image, a human skeleton graph is constructed with human joints as nodes and human skeleton as edges to obtain a human skeleton sequence; Encode the joint positions of human skeleton sequences, including: Through linear transformation, the human skeleton sequence is projected into a high-dimensional feature space; Using a trainable parameter matrix, the position features of each node are automatically obtained to generate node position embedding; The node position embedding is combined with the linear transformation to obtain the human skeleton sequence features after the joint position is embedded.

3. A human behavior recognition method based on a spatiotemporal fusion network as claimed in claim 1, characterized in that: The graph convolution and Transformer spatial fusion module includes an adaptive graph convolution module, a Transformer module and a correlation fusion module; The adaptive graph convolution module is used to model the correlation between human body joints, adaptively adjust the graph topology structure based on prior knowledge, and extract local structural features; The Transformer module is used to model the complex global spatial long-distance dependencies between human joints with the help of the self-attention mechanism and extract global spatial features; The correlation fusion module is used to fuse two types of features at different levels.

4. A human behavior recognition method based on a spatiotemporal fusion network as claimed in claim 3, characterized in that: The input features are fed into the adaptive graph convolution module to extract local structural features, including: The input features are transformed through two different linear transformation functions to generate initial features for constructing the relationship matrix. The two initial features are interpolated through the activation function to obtain the relationship matrix used to represent the relationship between nodes. The relationship matrix is ​​combined with the prior knowledge matrix to generate the weight matrix of the graph convolution. At the same time, after the input features are linearly transformed, multi-scale spatial features are extracted through dilated convolution; The weight matrix and multi-scale spatial features are combined through the aggregation function to obtain local structural features.

5. A human behavior recognition method based on a spatiotemporal fusion network as claimed in claim 4, characterized in that: The prior knowledge matrix uses the shortest path algorithm to extract the distance information between nodes, and introduces nonlinear scaling to smoothly adjust the connection strength between different nodes and optimize the local spatial relationship between the captured joints; The construction of the prior knowledge matrix includes: Based on the adjacency matrix represented by the human skeleton feature map, an initialization distance matrix is ​​constructed; The Floyd-Warshall algorithm is used to calculate the shortest paths between all joints, and the shortest path matrix is ​​determined in combination with the initialized distance matrix; Based on the shortest path matrix, exponential scaling is introduced, and each element in the shortest path matrix is ​​subjected to exponential operation to obtain an exponential scaling matrix; Based on the addition of the adjacency matrix and the exponential scaling matrix, the prior knowledge matrix is ​​obtained.

6. The human behavior recognition method based on spatiotemporal fusion network as claimed in claim 3, characterized in that: The correlation fusion module includes a depth-separable convolutional layer, a pooling layer, a convolutional layer and an activation layer connected in sequence; The extracted local structural features and global spatial features are input into the correlation fusion module. After the splicing operation, they are first input into the depthwise separable convolutional layer for dimensionality reduction. The features after dimensionality reduction are average-pooled by the pooling layer and then input into the convolutional layer. The attention weights are generated by the convolution operation, and then the two input features are weighted-spliced ​​after activation by the activation layer. After nonlinear processing, the final fusion features are output.

7. The human behavior recognition method based on spatiotemporal fusion network as claimed in claim 1, characterized in that: The multi-scale time fusion module includes a multi-scale convolution branch and a Transformer branch; The multi-scale convolution branch includes multiple parallel convolution branches and pooling branches with different aperture ratios, which are used to extract deep features in different scales and local areas; The Transformer branch uses the self-attention mechanism to learn the global relationship between each moment in the input feature sequence, adaptively captures the long-distance dependency between time points, and extracts time series features.

8. A human behavior recognition system based on spatiotemporal fusion network, characterized in that: include: The data acquisition module is used to acquire pedestrian motion videos, extract the human skeleton of each frame in the video and represent it in a graph to acquire a human skeleton sequence; A data preprocessing module is used to encode the joint positions of the human skeleton sequence and obtain the characteristics of the human skeleton sequence; The human behavior recognition module is used to input the human skeleton sequence features into the spatiotemporal fusion network based on graph convolution and Transformer, and output the human behavior recognition results; Among them, the spatiotemporal fusion network includes multiple groups of graph convolution Transformer spatial fusion modules and multi-scale time fusion modules connected in sequence. The input features are input into the graph convolution Transformer spatial fusion module. After the local structural features and global spatial features are extracted respectively by the parallel adaptive graph convolution module and Transformer module, the features are fused through the correlation fusion module. The fused features are then input into the multi-scale time fusion module. After the features are extracted respectively by the parallel multi-scale convolution branch and Transformer branch and then spliced, the human behavior recognition results are output based on the spliced ​​features.

9. An electronic device, characterized in that: The invention comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of a human behavior recognition method based on a spatiotemporal fusion network as described in any one of claims 1 to 7 are completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of a human behavior recognition method based on a spatiotemporal fusion network as described in any one of claims 1-7.

Images