A human skeleton data behavior recognition method and system based on an improved Transformer
By dividing human skeleton data into multiple spatiotemporal groups and using a spatiotemporal grouping self-attention model to model key relationships, the problems of high computational cost and overfitting in existing technologies are solved, achieving efficient behavior recognition.
Patent Information
- Application Number
- CN202310358481.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-06
- Publication Date
- 2025-12-09
- Estimated Expiration
- 2043-04-06
AI Technical Summary
Existing skeleton-based action recognition methods struggle to effectively model long-term sequence dependencies and global correlations of spatiotemporal joints, resulting in high computational costs and a tendency to overfit.
The original human skeleton data is divided into multiple parts in the time dimension, each part being a spatiotemporal group. A self-attention model is used to combine the spatiotemporal groups, and inter-group features are aggregated. The spatiotemporal grouping self-attention model is used to model the relationship between key points, and classification is performed through a global average pooling layer and a fully connected layer.
It effectively reduces computational complexity and the number of parameters, reduces overfitting problems, and improves the ability to extract and recognize spatiotemporal joint semantic information.
Smart Images
Figure CN116343340B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computer vision behavior recognition, and particularly to a human skeleton data behavior recognition method and system based on improved Transformer. BACKGROUND
[0002] In the past few decades, human action recognition has been widely applied, including security monitoring, healthcare and self-driving cars. Generally, according to the type of data processed, human action recognition existing work mainly develops in two directions, namely, RGB video-based action recognition and skeleton-based action recognition. Compared with the method based on RGB video, each skeleton is composed of a small number of joints, so it contains higher level semantic information in a small amount of data, and has strong adaptability to dynamic environment.
[0003] The previous skeleton-based action recognition method converts the original skeleton sequence into a grid-like structure, such as pseudo-image or graph, using graph topology or artificially designed rules, and then extracts features using deep network. The method based on RNN processes the sequence recursively, while the method based on CNN performs local operation on a fixed size window. The method based on GCN uses the topology structure of human skeleton to aggregate the features of related skeleton nodes and time series. Generally, the above methods cannot effectively model the long-term dependence of the sequence and the global correlation of the spatio-temporal joints, which limits the universality and performance of the previous work.
[0004] The method based on Transformer does not depend on the human structure, can process very long sequences, and models the relationship between all joints. Considering this advantage, Transformer is applied to the task of skeleton-based action recognition. There are mainly three difficulties in using Transformer for this task: (1) The spatio-temporal joints of skeleton data are related, so how to use Transformer to model the spatio-temporal correlation is crucial. Specifically, the skeleton sequence has different semantic information in the spatial dimension and the temporal dimension. The relationship between the joints in the spatial dimension reflects the interaction between the parts of the human body, while the same joints between the time dimensions represent the movement trajectory of a part of the human body, for example, in the action "wave hand", the arm is in a specific position relative to the torso, which is the relative position relationship between the arm and the torso in the spatial dimension, which is the feature in the spatial dimension, while the left and right swing of the hand is the feature of its movement in the time dimension.(2) At the same time, different joints in the continuous frames of the skeleton sequence are also related, for example, in the action "jogging", the arm in the previous frame is related to the leg in the next frame, because the joints of these parts will move together when the action moves.(3) Computing the self-attention between all joints of the skeleton sequence requires a large amount of computational cost, and it is easy to learn noise and cause overfitting. SUMMARY
[0005] To solve the problems mentioned in the background, the purpose of the present application is to provide a human skeleton data behavior recognition method and system based on improved Transformer.
[0006] The purpose of the present application can be achieved by the following technical solutions: a human skeleton data behavior recognition method based on improved Transformer, the method comprising the following steps:
[0007] Receiving original human skeleton data, dividing the original human skeleton data into multiple parts in the time dimension, each part being a space-time group, and the space-time group containing an equal number of human skeleton data frames;
[0008] Encoding the spatial position of each space-time group to obtain encoded human skeleton data;
[0009] Inputting the encoded human skeleton data into a pre-established space-time grouping self-attention model to obtain the relationship between the joints;
[0010] Group feature aggregation is performed on the obtained joint relationship features to obtain aggregated features;
[0011] The aggregated features are input into a global average pooling layer and a fully connected layer to obtain a classification score recognition result.
[0012] Preferably, the original human skeleton data has T frames, and each frame contains V joints.
[0013] Preferably, the process of encoding the spatial position of each space-time group uses sine and cosine functions of different frequencies as encoding functions to represent the position information of each human joint of the human skeleton data, as follows:
[0014]
[0015]
[0016] where PE represents the final position encoding information of each human joint, p represents the position of the joint, i represents the dimension of the position encoding vector, C in represents the number of channels of the input human skeleton data.
[0017] Preferably, all joints in the same frame of the original human skeleton data have the same encoding, and the same joints in different frames are sequentially encoded.
[0018] Preferably, the original human skeleton data X∈R C×T×V After dividing in the time dimension, the space-time group encoding X G =CG X M X V n ,
[0019] Preferably, the spatio-temporal grouping self-attention model comprises a time grouping self-attention model and a space grouping self-attention model.
[0020] Preferably, the space grouping self-attention model calculates the relationship of all joints in each spatio-temporal group with other joints in the same frame by using a self-attention mechanism, and the obtained spatial attention map of all frames in the spatio-temporal group is averaged and shared:
[0021]
[0022] wherein is the spatial attention map of the spatio-temporal group G i , is the skeleton sequence of the jth frame in the spatio-temporal group G i , and only the joints in the same frame are considered to calculate the spatial attention map, but the spatial attention map of all frames in the spatio-temporal group G i is averaged and shared.
[0023] Preferably, the time grouping self-attention model calculates the relationship of all joints in each spatio-temporal group G i with the joints at the same position in different frames, and the obtained time attention map of all frames in the spatio-temporal group is also averaged and shared:
[0024]
[0025] wherein is the time attention map of the spatio-temporal group G i . is the skeleton sequence composed of the joints at the same position b in different n frames in the spatio-temporal group G i , and only the joints at the same position in different frames are considered to calculate the time attention map, but the time attention map of all frames in the spatio-temporal group G i is averaged and shared.
[0026] Preferably, the inter-group feature aggregation adopts a convolution operation with a kernel size of Kx1 to realize the aggregation of the sub-actions contained in each spatio-temporal group in the time dimension, and the formula is as follows:
[0027]
[0028] X STGA represents the human skeleton data processed by the spatio-temporal grouping self-attention module, X IGFA represents the result obtained by processing the inter-group feature aggregation module, Conv 2D(K×1) represents a convolution operation with a kernel size of Kx1.
[0029] Preferably, a human skeleton data behavior recognition system based on improved Transformer comprises:
[0030] A space-time grouping encoding module is configured to receive original human skeleton data, divide the original human skeleton data into multiple parts in a time dimension, each part being a space-time group, and the space-time group containing an equal number of human skeleton data frames;
[0031] A space-time position encoding module is configured to encode a space-time group for a spatial position to obtain encoded human skeleton data;
[0032] A space-time grouping self-attention module is configured to input the encoded human skeleton data into a pre-established space-time grouping self-attention model to obtain the relationship between the joints;
[0033] An inter-group feature aggregation module is configured to aggregate the relationship features between the joints to obtain aggregated features;
[0034] A result generation module is configured to input the aggregated features into a global average pooling layer and a fully connected layer to obtain a classification score.
[0035] The present application has the following beneficial effects:
[0036] The present application helps to extract semantic information with different meanings in the spatial and temporal dimensions, and can also model different joints between consecutive frames in each space-time group. Since the skeleton data is divided into multiple space-time groups for intra-group modeling instead of global modeling, and the spatial attention map is averaged and shared within the space-time group, the method significantly reduces the number of parameters and has lower computational complexity, effectively reducing the problem of overfitting. BRIEF DESCRIPTION OF DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings;
[0038] Figure 1 The structural diagram of the method of the present application;
[0039] Figure 2 The schematic diagram of the space-time grouping encoding module of the present application;
[0040] Figure 3 The structural diagram of the space-time grouping attention module of the present application. DETAILED DESCRIPTION
[0041] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.
[0042] As shown in the figure, an improved human skeleton data behavior recognition method based on Transformer, the method comprises the following steps: Figure 1
[0043] Receiving original human skeleton data, dividing the original human skeleton data into multiple parts in the time dimension, each part being a space-time group, and the space-time group containing an equal number of human skeleton data frames;
[0044] Encoding the spatial position of each space-time group to obtain encoded human skeleton data;
[0045] Inputting the encoded human skeleton data into a pre-established space-time grouping self-attention model to obtain the relationship between the joints;
[0046] Performing inter-group feature aggregation on the obtained joint relationship features to obtain aggregated features;
[0047] Inputting the aggregated features into a global average pooling layer and a fully connected layer to obtain classification scores.
[0048] The original human skeleton data has T frames, and each frame contains V joints.
[0049] The process of encoding the spatial position of each space-time group uses sine and cosine functions of different frequencies as encoding functions to represent the position information of each human joint of the human skeleton data, as follows:
[0050]
[0051]
[0052] where PE represents the final position encoding information of each human joint, p represents the position of the joint, i represents the dimension of the position encoding vector, C in represents the number of channels of the input human skeleton data.
[0053] In the original human skeleton data, all joints within the same frame have the same code, and the same joints in different frames are encoded sequentially. In a specific implementation, the temporal position encoding is analogous: joints within the same frame have the same code, and the same joints in different frames are encoded sequentially. In this way, the originally unified encoding in the spatiotemporal dimensions is decoupled, and the differences between time and space are well expressed. Each element has a unique label to represent the relationship between joints.
[0054] like Figure 2 As shown, the original human skeleton data X∈R C×T×V After dividing the data along the time dimension, the spatiotemporal group code X is obtained. G =C G ×M×V n φ(·) consists of two convolutional layers with BatchNorm and ReLU functions. It is implemented by a linear layer with a ReLU function.
[0055] The spatiotemporal grouping self-attention model includes a temporal grouping self-attention model and a spatial grouping self-attention model.
[0056] The spatial grouping self-attention model uses a self-attention mechanism to calculate the relationship between all joints in each spatiotemporal group and other joints in the same frame. The resulting spatial attention map for all frames within the spatiotemporal group is averaged and shared.
[0057]
[0058] Among them is Spacetime Group G i Spatial attention map, It is spacetime group G i The skeleton sequence of frame j is used to calculate the spatial attention map by considering only the joints in the same frame, but this yields the spatiotemporal group G. i The spatial attention maps of all frames within the scope are averaged and shared.
[0059] The time-grouped self-attention model calculates the spatiotemporal group G respectively. i The relationship between all joints in the spatiotemporal group and joints at the same position in different frames is obtained, and the resulting temporal attention map for all frames within the spatiotemporal group is also averaged and shared:
[0060]
[0061] in It is spacetime group G i Time attention graph. It is spacetime group G iThe skeleton sequence composed of joints in the same position b in different n frames, only the joints in the same position in different frames are considered to calculate the time attention map, but the space-time group G is obtained i The time attention map of all frames in the group is averaged and shared.
[0062] It needs to be further explained that in the specific implementation process, Figure 3 The left is the spatial grouping self-attention module, which first adds the input Then it is fed into two linear mapping functions to get And C s Generally smaller than C G , the purpose is to eliminate feature redundancy and reduce calculation. Then calculate the spatial attention map and add the intra-group spatial global regularization. Unlike the Transformer, we use Tanh instead of SoftMax to calculate the attention map, because the output of Tanh is not limited to positive values, so it can produce negative relationships and provide more flexibility. Finally, multiply the attention map with the original input to get the output feature. Like the Transformer, we use multi-head self-attention to make the model jointly focus on information from different representation subspaces, with a total of S heads, the results of all heads are concatenated and mapped to the output space with a linear layer Finally, a point-by-point feedforward layer is added to obtain the final output, using ReLU as the nonlinear function. There are two residual connections in the module to stabilize network training and integrate different functions. Similar to the spatial grouping self-attention module, Figure 3 The right is the temporal grouping self-attention module, the difference is that the input is And no intra-group global regularization is added.
[0063] The inter-group feature aggregation uses a convolution operation with a kernel size of Kx1 to realize the aggregation of sub-actions in the time dimension contained in each space-time group, and the formula is as follows:
[0064] X IGFA = Conv 2D(K×1) (X STGA ).
[0065] X STGA represents the human skeleton data processed by the space-time grouping self-attention module, X IGFA represents the result obtained after processing by the inter-group feature aggregation module, Conv 2D(K×1) represents a convolution operation with a kernel size of Kx1.
[0066] It needs to be explained that an improved human skeleton data behavior recognition system based on Transformer includes:
[0067] The spatio-temporal grouping encoding module is configured to receive the original human skeleton data, divide the original human skeleton data in a time dimension into a plurality of parts, each part being a spatio-temporal group, and the spatio-temporal group containing an equal number of human skeleton data frames.
[0068] The spatio-temporal position encoding module is configured to encode a spatial position of each spatio-temporal group to obtain encoded human skeleton data.
[0069] The spatio-temporal grouping self-attention module is configured to input the encoded human skeleton data into a pre-established spatio-temporal grouping self-attention model to obtain relationships between the joints.
[0070] The inter-group feature aggregation module is configured to aggregate the obtained relationship features between the joints to obtain aggregated features.
[0071] The result generation module is configured to input the aggregated features into a global average pooling layer and a fully connected layer to obtain a classification score.
[0072] The application embodiment adopts the human skeleton data behavior recognition algorithm based on the improved Transformer in the first embodiment, and applies it to the open source data sets NTU-RGB+D 60 and NTU-RGB+D 120 data sets to verify its effectiveness. The experimental results and ablation experiments are shown in Tables 1 and 2.
[0073] Precision comparison: Compared with the methods based on CNN and RNN, the algorithm in this paper has a significant advantage. Compared with the method based on GCN, the algorithm in this paper is 1% higher than the best GCN method on X-Sub of NTU-RGB+D 60, and 0.7% better than the best GCN method on X-Set of NTU-RGB+D 120, respectively. In terms of methods based on Transformer, our method also achieves the best performance.
[0074] Model size and computational complexity comparison: The number of parameters is used to evaluate the model size, and the number of FLOPs (floating point multiplication and addition) is used to evaluate the computational complexity. The computational complexity of the algorithm in this paper is 2 times lower than that of DC-GCN+ADG, while the model size is similar. Compared with the method based on Transformer, the best result is achieved with the smallest computational complexity and a small number of parameters.
[0075] Table 1 Performance of the algorithm in this paper on the NTU-RGB+D 60 / 120 data set and comparison of accuracy, parameter quantity and computational complexity with other models.
[0076]
[0077] As shown in Table 2, the effect of spatio-temporal position encoding is studied. The accuracy of the model using the ordinary position encoding strategy is lower than that of the model using the spatio-temporal position encoding strategy, indicating that spatio-temporal position encoding can significantly improve performance. The main reason is that spatio-temporal position encoding distinguishes the position information of different joints in the spatial and temporal dimensions, expressing the difference of joints in the spatio-temporal position.
[0078] Table 2 Ablation experiments of spatio-temporal position encoding for joint patterns of the NTU-RGB+D 60 skeleton dataset. PE represents the ordinary position encoding strategy, and STPE represents the spatio-temporal position encoding strategy proposed in this paper.
[0079]
[0080] As shown in Table 3, in order to verify the effectiveness of our spatio-temporal grouping self-attention module, we set the number of consecutive frames n included in each spatio-temporal group to 6, which means that the relationship between different joints in the time dimension and the space dimension in 6 consecutive frames is modeled at the same time. It is obvious that the proposed spatio-temporal grouping self-attention module can significantly improve the performance of the model. The main advantage is that the proposed spatio-temporal grouping self-attention module can capture the correlation of joints in the spatial dimension and the temporal dimension within several consecutive frames, respectively, and distinguish the semantic information of different spatio-temporal dimensions. By adding intra-group spatial global regularization, the performance is improved significantly, but if we regularize the time dimension at the same time, the performance will decrease. This is reasonable because the time dimension has no specific meaning, and forcing to learn a uniform pattern will lead to the gap between the training set and the test set.
[0081] Table 3 Ablation experiments of spatio-temporal grouping self-attention module for joint patterns of the NTU-RGB+D 60 skeleton dataset. USTGA represents the strategy of using uniform spatio-temporal dimension to obtain attention map, SGA represents using only spatial self-attention module, TGA represents using only temporal self-attention module, and STGA represents using the spatio-temporal self-attention mechanism strategy proposed in this paper. WSGR and WTGR represent spatial intra-group global regularization and temporal intra-group global regularization, respectively.
[0082] Table 3
[0083]
[0084] Based on the same inventive concept, the present application further provides a computer device, which comprises one or more processors and a memory for storing one or more computer programs; the program comprises program instructions, and the processor is configured to execute the program instructions stored in the memory. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal and are configured to implement one or more instructions, and are specifically configured to load and execute one or more instructions in the computer storage medium to implement the above method.
[0085] It needs to be further explained that, based on the same inventive concept, the present application further provides a computer storage medium, which stores a computer program, and the computer program is executed by the processor to perform the above method. The storage medium can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include: electrical connections having one or more wires, portable computer disks, hard drives, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.
[0086] In the description of the present application, the description of the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0087] The foregoing presents and describes the basic principles, main features and advantages of the present disclosure. It should be understood by those skilled in the art that the present disclosure is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, various changes and improvements can be made to the present disclosure, and all these changes and improvements fall within the scope of the present disclosure.
Claims
1. A human skeleton data behavior recognition method based on Transformer improvement, characterized in that, The method comprises the following steps: Receiving original human skeleton data, dividing the original human skeleton data into multiple parts in the time dimension, each part being a space-time group, and the space-time group containing an equal number of human skeleton data frames; Encoding the spatial position of each space-time group to obtain encoded human skeleton data; Inputting the encoded human skeleton data into a pre-established space-time grouping self-attention model to obtain the relationship between the joints; The space-time grouping self-attention model comprises a time grouping self-attention model and a space grouping self-attention model; The space grouping self-attention model calculates the relationship between all joints in each space-time group and other joints in the same frame using a self-attention mechanism, and the spatial attention map of all frames in the space-time group obtained is averaged and shared: wherein is spatiotemporal group spatial attention map, is a spatiotemporal group the skeleton sequence of the frame, only considering the joints in the same frame to calculate the spatial attention map, but obtaining the spatial attention map of all frames in the spatiotemporal group is averaged and shared; The time grouping self-attention model respectively calculates the spatiotemporal group The relationship between all joints in the spatiotemporal group and the same position joints in different frames, and the time attention graph of all frames in the spatiotemporal group obtained is also averaged and shared: wherein is a spatiotemporal group of time attention maps, is a spatiotemporal group of joint groups at the same position in different frames, a skeleton sequence composed of joint groups at the same position in different frames, only considering the same position joints in different frames to compute the time attention maps, but obtaining the time attention maps for all frames within a spatiotemporal group are averaged and shared; Group feature aggregation is performed on the obtained relationship features between the joints to obtain aggregated features; Inputting the aggregated features into a global average pooling layer and a fully connected layer to obtain a classification score recognition result.
2. The method of claim 1, wherein the method is based on an improved human skeleton data behavior recognition method based on a Transformer. The original human skeleton data has frames, each frame containing joints.
3. The method of claim 1, wherein the method is based on an improved human skeleton data behavior recognition method based on a Transformer. The process of encoding the spatial position of each space-time group uses sine and cosine functions of different frequencies as encoding functions to represent the position information of each human joint of the human skeleton data, as follows: wherein represents the position encoding information of each human joint finally obtained, represents the position of the joint, represents the dimension of the position encoding vector, represents the number of channels of the input human skeleton data.
4. The method according to claim 2, wherein, All joints in the same frame in the original human skeleton data have the same encoding, and the same joints in different frames are sequentially encoded.
5. The method of claim 1, wherein the method is based on an improved human skeleton data behavior recognition method based on a Transformer. The original human skeleton data , through the time dimension division, get space-time group coding .
6. The method of claim 1, wherein the method is based on an improved human skeleton data behavior recognition method based on a Transformer. The inter-group feature aggregation employs Convolution operation on the core size, realizing the aggregation of the sub-actions contained in each space-time group in the time dimension, and the formula is as follows: represents the human skeleton data processed by the spatio-temporal grouping self-attention module, represents the result obtained after processing by the inter-group feature aggregation module, represents the convolution operation with a kernel size of a kernel size.
7. A human skeleton data behavior recognition system based on a Transformer improvement, adopting a human skeleton data behavior recognition method based on a Transformer improvement according to any one of claims 1 to 6, characterized in that, It comprises: A space-time grouping encoding module for receiving original human skeleton data, dividing the original human skeleton data into multiple parts in the time dimension, each part being a space-time group, and the space-time group containing an equal number of human skeleton data frames; A space-time position encoding module for encoding the spatial position of each space-time group to obtain encoded human skeleton data; A space-time grouping self-attention module for inputting the encoded human skeleton data into a pre-established space-time grouping self-attention model to obtain the relationship between the joints; A group feature aggregation module for performing group feature aggregation on the obtained relationship features between the joints to obtain aggregated features; A result generation module for inputting the aggregated features into a global average pooling layer and a fully connected layer to obtain a classification score.