Skeleton behavior recognition method based on cross-dimensional interaction attention mechanism graph convolution

Abstract

The application discloses a skeleton behavior recognition method based on cross-dimension interaction attention mechanism graph convolution, and comprises the following steps: acquiring a trained complete human skeleton behavior recognition model, wherein the human skeleton behavior recognition model comprises a cross-dimension interaction attention module, the cross-dimension interaction attention module comprises a space-channel grouping attention module, a time-space displacement attention module and a time-channel difference attention module; acquiring a to-be-detected human skeleton sequence image, and detecting the to-be-detected human skeleton sequence image based on the trained complete human skeleton behavior recognition model to obtain a detection result of the to-be-detected human skeleton sequence image. The application solves the technical problem that a traditional adaptive graph convolution network in the prior art is difficult to model the semantic correlation between skeleton features in the channel dimension, the space dimension and the time dimension in a human skeleton behavior recognition task, thereby leading to low recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of human skeleton behavior recognition technology, specifically to a skeleton behavior recognition method based on graph convolution with a cross-dimensional interactive attention mechanism. Background Technology

[0002] Human behavior recognition technology is a research area with significant application value in computer vision, finding wide applications in video understanding, motion analysis, human interaction, intelligent monitoring, and industrial security. Skeleton sequences, as a lightweight and efficient dynamic spatiotemporal topological data set, possess excellent robustness, multimodal compatibility, and noise resistance, making them highly valuable for research in video semantic understanding and action recognition.

[0003] Currently, Graph Convolutional Networks (GCNs), with their excellent performance in processing non-Euclidean geometric data, are well-suited for skeleton action recognition tasks and have become the mainstream approach. Adaptive Graph Convolutional Networks (AGCNs) utilize adaptive adjacency matrices to enable the network to learn more flexible node adjacencies, designing a flexible graph structure representation in the spatial dimension of skeleton features. Furthermore, various attention mechanisms can also be used to enhance the performance of GCNs in human skeleton action recognition. These methods enhance skeleton feature representation and achieve information association in one or two dimensions. Examples include the Squeeze-and-Excitation Module (SE), the Convolutional Block Attention Module (CBAM), and the Spatial-Temporal-Channel Attention Module (STC-attention) in Attention Adaptive Graph Convolutional Networks (AAGCNs). However, these methods only enhance the attentional representation of skeleton features in one or two dimensions, and have not yet explicitly modeled the cross-dimensional interactive relationships of skeleton sequences in the three dimensions of channel, space and time. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a skeleton behavior recognition method based on cross-dimensional interactive attention mechanism graph convolution. This solves the technical problem that traditional adaptive graph convolutional networks in the prior art have difficulty modeling the semantic association between skeleton features in the channel dimension, spatial dimension and temporal dimension in human skeleton behavior recognition tasks, resulting in low recognition accuracy.

[0005] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a skeleton behavior recognition method based on graph convolution with a cross-dimensional interactive attention mechanism, comprising:

[0007] Obtain a fully trained human skeleton behavior recognition model, wherein the human skeleton behavior recognition model includes a cross-dimensional interactive attention module, and the cross-dimensional interactive attention module includes a spatial-channel grouping attention module, a temporal-spatial displacement attention module, and a temporal-channel differential attention module;

[0008] A sequence of human skeleton images to be detected is acquired, and the human skeleton behavior recognition model is trained to detect the sequence of human skeleton images to obtain the detection results.

[0009] In some embodiments, the spatial-channel grouping attention module includes a first subgraph grouping layer, multiple first channel displacement layers, a second channel displacement layer, a second subgraph grouping layer, an inter-group cross attention layer, and a spatial dimension feature splicing layer;

[0010] The first sub-image grouping layer is used to group the human structural parts of the single frame skeleton image at time t in the human skeleton sequence image to be detected, and obtain local sub-image features of multiple different structural parts.

[0011] The number of the first channel displacement layers is consistent with and corresponds one-to-one with the number of the local subgraph features. The multiple channel displacement layers perform intra-group feature interaction between nodes on their respective local subgraph features to obtain multiple corresponding local channel displacement subgraph features.

[0012] The second channel displacement layer is used to perform intra-group feature interaction between nodes in the single-frame skeleton graph at time t to obtain global channel displacement features;

[0013] The second sub-graph grouping layer is used to group the global channel displacement features by human body structural parts to obtain global channel displacement sub-graph features of multiple different structural parts;

[0014] The inter-group cross-attention layer is used to perform feature interaction on the multiple local channel displacement subgraph features and the multiple global channel displacement subgraph features to obtain multiple inter-group interaction features.

[0015] The spatial dimension feature splicing layer is used to splice the multiple inter-group interaction features in a spatial dimension to obtain the spatial-channel group attention features at time t.

[0016] In some embodiments, the spatial-channel grouped attention module further includes a plurality of first grouped attention convolutional layers and second grouped attention convolutional layers;

[0017] The number of the first group attention convolutional layers is consistent with the number of the first channel displacement layers and corresponds one-to-one. The multiple first convolutional layers are used to perform linear mapping on their respective local channel displacement sub-map features to obtain multiple corresponding local channel displacement mapping results.

[0018] The second group attention convolutional layer is used to perform linear mapping on the global channel displacement submap features to obtain the global channel displacement mapping result.

[0019] In some embodiments, the temporal-spatial displacement attention module includes a temporal displacement layer, multiple convolutional network layers, a first cross-product operation layer, a first normalization layer, an attention weight calculation layer, a second cross-product operation layer, and a second normalization layer.

[0020] The temporal shift layer is used to extract skeleton features from the human skeleton sequence image to be detected, and obtain skeleton features at time t, time t-1 and time t-2.

[0021] The multiple convolutional network layers correspond one-to-one with the skeleton features at time t, time t-1, and time t-2. The multiple convolutional network layers are used to perform linear mapping on the skeleton features at time t, time t-1, and time t-2 to obtain the corresponding value features, key features, and query features.

[0022] The first cross-product operation layer is used to perform a cross-product operation on the query feature and the key feature to obtain the first cross-product result;

[0023] The first normalization layer is used to perform a normalization operation on the first cross product result to obtain the first normalized result;

[0024] The attention weight calculation layer is used to calculate the attention weight of the first normalization result to obtain the query-key weight feature.

[0025] The second cross-product operation layer is used to perform a cross-product operation on the query-key weight feature and the value feature to obtain the second cross-product result;

[0026] The second normalization layer is used to normalize the result of the second cross product to obtain the temporal-spatial displacement attention feature.

[0027] In some embodiments, the time-channel differential attention module includes a time-channel shift layer, a global average pooling layer, a first differential operation layer, a second differential operation layer, and a multilayer perceptron.

[0028] The temporal channel shift layer is used to extract skeleton features from the human skeleton sequence image to be detected, and to obtain the temporal channel skeleton features at time t, time t-1, and time t-2.

[0029] The global average pooling layer is used to perform global pooling calculations on the time-series channel skeleton features at time t, time-1, and time-2, respectively, to obtain the global average pooling skeleton features at time t, time-1, and time-2.

[0030] The first differential operation layer is used to perform differential calculation on the global average pooling skeleton feature at time t and the global average pooling skeleton feature at time t-1 to obtain the first channel differential operator feature;

[0031] The second differential operation layer is used to perform differential calculation on the global average pooling skeleton features at time t and the global average pooling skeleton features at time t-2 to obtain the second channel differential operator features;

[0032] The multilayer perceptron is used to perform nonlinear mapping on the first channel differential operator features and the second channel differential operator features respectively to obtain a first mapping result and a second mapping result;

[0033] The weighted operation layer is used to perform weighted fusion of the first mapping result and the second mapping result to obtain the time-channel differential attention feature.

[0034] In some embodiments, the cross-dimensional interactive attention module further includes a first weighted addition layer, a second weighted addition layer, and an element-wise dot product layer;

[0035] The first weighted summation layer is used to perform weighted summation on the spatial-channel grouping attention features and the temporal-spatial displacement attention features to obtain the first weighted summation result;

[0036] The second weighted summation layer is used to perform a weighted summation of the first weighted summation result and the human skeleton sequence image to be detected, to obtain the second weighted summation result;

[0037] The element-wise multiplication layer is used to perform a multiplication operation on the second weighted sum and the time-channel differential attention feature to obtain the cross-dimensional interactive attention feature.

[0038] In some embodiments, the human skeleton recognition model further includes spatial graph convolution, a first activation layer, a random deactivation layer, temporal convolution, and a second activation layer;

[0039] The spatial graph convolution, the first activation layer, the random deactivation layer, the cross-dimensional interactive attention module, the temporal convolution, and the second activation layer are connected in sequence.

[0040] Secondly, the present invention also provides a skeleton behavior recognition device based on graph convolution with a cross-dimensional interactive attention mechanism, comprising:

[0041] The model acquisition module is used to acquire a fully trained human skeleton behavior recognition model, which includes a spatial-channel grouping attention module, a temporal-spatial displacement attention module, and a temporal-channel differential attention module.

[0042] The human skeleton behavior recognition module is used to acquire a sequence of human skeleton images to be detected, and to detect the sequence of human skeleton images to be detected based on the fully trained human skeleton behavior recognition model, so as to obtain the detection result of the sequence of human skeleton images to be detected.

[0043] Thirdly, the present invention also provides an electronic device, comprising: a processor and a memory;

[0044] The memory stores a computer-readable program that can be executed by the processor;

[0045] When the processor executes the computer-readable program, it implements the steps in the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism as described above.

[0046] Fourthly, the present invention also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the skeleton behavior recognition method based on cross-dimensional interactive attention mechanism graph convolution as described above.

[0047] Compared with existing technologies, the skeleton behavior recognition method based on cross-dimensional interactive attention mechanism graph convolution provided by this invention has the following advantages:

[0048] 1. The cross-dimensional interactive attention mechanism, which integrates the spatial-channel grouping attention module, the temporal-spatial displacement attention module, and the temporal-channel differential attention module, enhances the understanding ability of graph convolutional networks in the field of human skeleton behavior recognition. It fully explores the semantic interaction between skeleton features in the channel dimension, spatial dimension, and temporal dimension. By using different attention mechanisms to explicitly model the interactive semantic association between skeleton features, it effectively improves the performance of skeleton behavior recognition tasks.

[0049] 2. The spatial-channel grouping attention module enhances the adjacency weights of nodes within and between parts, modeling richer dynamic connection relationships between nodes; the temporal-spatial displacement attention module correlates first-order pose features between frames, making it easier to capture contextual dependencies of short and long temporal sequences; the temporal-channel differential attention module integrates second-order differential features between frames, effectively improving the expression of dynamic information in the channel dimension.

[0050] 3. The fusion of spatial-channel grouping attention module, temporal-spatial displacement attention module, and temporal-channel differential attention module is used for skeleton behavior recognition tasks of graph convolutional network models. It is lightweight and does not require recoding within the module during actual use. It has high flexibility and portability, and is plug-and-play. Attached Figure Description

[0051] Figure 1 This is a flowchart of an embodiment of the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism provided by the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of an embodiment of the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism in the graph convolution block, provided by the present invention.

[0053] Figure 3 This is a schematic diagram of an embodiment of the adaptive graph convolutional network structure and the integrated position of the cross-dimensional interactive attention mechanism network in the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism provided by the present invention.

[0054] Figure 4 This is a schematic diagram of the structure of a spatial-channel grouped attention module in the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism provided by the present invention.

[0055] Figure 5 This is a schematic diagram of the structure of a temporal-spatial displacement attention module in the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism provided by the present invention.

[0056] Figure 6 This is a schematic diagram of the structure of a temporal-channel differential attention module in the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism provided by the present invention.

[0057] Figure 7 This is a schematic diagram of the structure of an embodiment of the feature fusion method of the cross-dimensional interactive attention mechanism in the skeleton behavior recognition method based on graph convolution of cross-dimensional interactive attention mechanism provided by the present invention.

[0058] Figure 8 This is a schematic diagram of an embodiment of the skeleton behavior recognition device based on graph convolution with cross-dimensional interactive attention mechanism provided by the present invention;

[0059] Figure 9 This is a schematic diagram of the operating environment of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0061] This invention provides a skeleton behavior recognition method based on graph convolution with a cross-dimensional interactive attention mechanism, which will be described below.

[0062] Figure 1 This is a schematic flowchart of an embodiment of the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism provided by the present invention. Figure 2 This is a schematic diagram of an embodiment of the human skeleton behavior recognition integrated model provided by the present invention. Figure 3 This is a schematic diagram of the integrated location of the adaptive graph convolutional network structure and cross-dimensional interactive attention mechanism network provided by the present invention.

[0063] like Figure 1 , Figure 2 and Figure 3 As shown, the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism includes:

[0064] S101. Obtain a fully trained human skeleton behavior recognition model, wherein the human skeleton behavior recognition model includes a spatial-channel grouping attention module, a temporal-spatial displacement attention module, and a temporal-channel differential attention module;

[0065] S102. Obtain the human skeleton sequence image to be detected, and detect the human skeleton sequence image to be detected based on the fully trained human skeleton behavior recognition model to obtain the detection result of the human skeleton sequence image to be detected.

[0066] Compared with the prior art, the embodiments of the present invention improve the understanding ability of graph convolutional networks in the field of human skeleton behavior recognition by integrating spatial-channel grouping attention module, temporal-spatial displacement attention module and temporal-channel differential attention module to form a cross-dimensional interactive attention mechanism. It fully explores the semantic interaction between skeleton features in channel dimension, spatial dimension and temporal dimension, and explicitly models the interactive semantic association between skeleton features through different attention mechanisms, effectively improving the performance of skeleton behavior recognition task.

[0067] Furthermore, the spatial-channel grouping attention module enhances the adjacency weights of nodes within and between parts, modeling richer dynamic connection relationships between nodes; the temporal-spatial displacement attention module correlates first-order pose features between frames, facilitating the capture of contextual dependencies between long and short temporal sequences; and the temporal-channel differential attention module integrates second-order differential features between frames, effectively improving the expression of dynamic information in the channel dimension.

[0068] Furthermore, the spatial-channel grouping attention module, the temporal-spatial displacement attention module, and the temporal-channel differential attention module are integrated for skeleton behavior recognition tasks of graph convolutional network models. This approach is lightweight and requires no recoding within the modules during practical use, offering high flexibility and portability through plug-and-play functionality.

[0069] It should be noted that a fully trained human skeleton behavior recognition model refers to a model whose average uniform accuracy on a test set of human skeleton sequence images is greater than or equal to 95%. Using average uniform accuracy as a target evaluation metric ensures the detection performance of the human skeleton behavior recognition model.

[0070] It should also be noted that when the mean average precision (mAP) of the detection results on the human skeleton sequence image test set is less than 95%, a human skeleton sequence image sample set is obtained, and the human skeleton behavior recognition model is retrained using this sample set. Specifically, the coordinates of two-dimensional or three-dimensional skeleton points extracted from the RGB videos of the civil service exam dataset are preprocessed, including coordinate normalization and uniform sampling, and then trained using a pre-constructed adaptive graph convolutional network. The skeleton features extracted by the network model are then fed into the classifier after global average pooling to obtain the recognition and classification results for various behaviors, and the recognition accuracy is evaluated.

[0071] Furthermore, in step S102, the human skeleton sequence image features can be represented by F, with dimensions (C, T, V), where C represents the number of feature channels, T represents the number of frames in the feature sequence, and V represents the number of skeleton joints; the human skeleton sequence F, as a sequence with temporal characteristics, can be represented as a set of skeleton images for each frame, which can be described as: F = {F (t) |t=1,2,…,T};where, F (t) Let be the spatial features of the single-frame skeleton graph at time t, with dimensions (C, V).

[0072] In a preferred embodiment of the present invention, such as Figure 4As shown, the spatial-channel grouping attention module includes a first subgraph grouping layer, multiple first channel displacement layers, a second channel displacement layer, a second subgraph grouping layer, an inter-group cross attention layer, and a spatial dimension feature splicing layer;

[0073] The first sub-image grouping layer is used to group the human structural parts of the single frame skeleton image at time t in the human skeleton sequence image to be detected, and obtain local sub-image features of multiple different structural parts.

[0074] The number of the first channel displacement layers is consistent with and corresponds one-to-one with the number of the local subgraph features. The multiple channel displacement layers perform intra-group feature interaction between nodes on their respective local subgraph features to obtain multiple corresponding local channel displacement subgraph features.

[0075] The second channel displacement layer is used to perform intra-group feature interaction between nodes in the single-frame skeleton graph at time t to obtain global channel displacement features;

[0076] The second sub-graph grouping layer is used to group the global channel displacement features by human body structural parts to obtain global channel displacement sub-graph features of multiple different structural parts;

[0077] The inter-group cross-attention layer is used to perform feature interaction on the multiple local channel displacement subgraph features and the multiple global channel displacement subgraph features to obtain multiple inter-group interaction features.

[0078] The spatial dimension feature splicing layer is used to splice the multiple inter-group interaction features in a spatial dimension to obtain the spatial-channel group attention features at time t.

[0079] This invention constructs a spatial-channel grouping attention module to build channel feature interactions between nodes within different human body groups and semantic associations between local and global features between different groups in the spatial and channel dimensions of skeleton features.

[0080] Furthermore, for the single-frame skeleton map F at time t in the skeleton feature sequence image... (t) The human body is divided into sub-images based on its structural parts, specifically the skeletal node indices of the head, hands, and feet. These sub-images are categorized into three groups: Group 1, Group 2, and Group 3, representing local sub-image features. And the single-frame skeleton diagram F at time t (t) Assigned as a global feature Subsequently, the first channel displacement was used to analyze the local sub-image features of group 1, group 2, and group 3 respectively. Perform intra-group feature interactions between nodes, and use second-channel displacement layer pairs and global features. Perform intra-group feature interaction between nodes so that each node feature in the three local subgraph features and the global feature includes the features of all nodes in the group, thus obtaining the local channel displacement subgraph features of group 1, group 2, and group 3 at time t. and global channel displacement characteristics The global channel displacement features are then further subdivided into group 1, group 2, and group 3, resulting in global channel displacement sub-map features. Finally, inter-group cross-attention is used to perform feature interaction between the local channel displacement submap features and the global channel displacement submap features of each group. The inter-group interaction features of group 1, group 2, and group 3 are calculated using a query-key-value structure. Finally, the inter-group interaction features of group 1, group 2, and group 3 are concatenated in the spatial dimension to obtain the spatial-channel grouping attention features at time t.

[0081] It should be noted that the subgraph grouping rule of the first subgraph grouping layer is as follows: Let the single-frame skeleton graph at time t be F. (t) global features Set to F (t) The number of heads, hands, and feet in the human skeleton are v1, v2, and v3 respectively. What are the local subgraph features of group 1, group 2, and group 3? It can be described as:

[0082]

[0083] In the formula, v m This indicates the number of subgraph nodes in group m. Set v0 = 0, and “:” indicates spatial slicing operation.

[0084] Furthermore, the channel displacement rules for the first and second channel displacement layers are as follows: a learnable mask matrix M is set for both local subgraph features and global features. i M g By using the tanh() function and dot product operations, the learnable mask matrix is ​​multiplied with local subgraph features and global features. This allows for flexible implementation of interactive features between different nodes within a group using the learnable mask matrix of each node in the channel dimension. Each node, after being multiplied by the learnable mask matrix, contains feature information from other nodes. Therefore, the local channel displacement subgraph features at time t are... and global channel displacement characteristics It can be described as:

[0085] i =1, 2, 3;

[0086] ;

[0087] In the formula: This is an element-wise dot product.

[0088] Furthermore, the subgraph grouping rule for the second subgraph grouping layer is: to group the global channel displacement features... The first subgraph grouping method, the global channel displacement subgraph features of group 1, group 2, and group 3. It can be described as:

[0089]

[0090] In some embodiments, the spatial-channel grouped attention module further includes a plurality of first grouped attention convolutional layers and second grouped attention convolutional layers;

[0091] The number of the first group attention convolutional layers is consistent with the number of the first channel displacement layers and corresponds one-to-one. The multiple first convolutional layers are used to perform linear mapping on their respective local channel displacement sub-map features to obtain multiple corresponding local channel displacement mapping results.

[0092] The second group attention convolutional layer is used to perform linear mapping on the global channel displacement submap features to obtain the global channel displacement mapping result.

[0093] In this embodiment, the inter-group cross-attention rule of the inter-group cross-attention layer is as follows: the output feature of group i is calculated using a query-key-value structure. Using two additional sets of subgraph displacement features The linear mappings are used as query features and key features, respectively. Attention weights are calculated using the Softmax() function. The global channel displacement subgraph features of group i after subgraph grouping are used. The linear mapping is used as the Value feature to calculate cross-attention, and the output feature of group i is... It can be described as:

[0094]

[0095] In the formula, i, j, k = 1, 2, 3 and i ≠ j ≠ k. T is the matrix transpose. Describes the linear mapping function between groups j and k. The linear mapping function representing the global features is implemented using a first group attention convolutional layer and a second group attention convolutional layer; specifically, the first group attention convolutional layer and the second group attention convolutional layer are 1×1 convolutional layers.

[0096] In some embodiments, please refer to Figure 5The temporal-spatial displacement attention module includes a temporal displacement layer, multiple convolutional network layers, a first cross-product operation layer, a first normalization layer, an attention weight calculation layer, a second cross-product operation layer, and a second normalization layer.

[0097] The temporal shift layer is used to extract skeleton features from the human skeleton sequence image to be detected, and obtain skeleton features at time t, time t-1 and time t-2.

[0098] The multiple convolutional network layers correspond one-to-one with the skeleton features at time t, time t-1, and time t-2. The multiple convolutional network layers are used to perform linear mapping on the skeleton features at time t, time t-1, and time t-2 to obtain the corresponding value features, key features, and query features.

[0099] The first cross-product operation layer is used to perform a cross-product operation on the query feature and the key feature to obtain the first cross-product result;

[0100] The first normalization layer is used to perform a normalization operation on the first cross product result to obtain the first normalized result;

[0101] The attention weight calculation layer is used to calculate the attention weight of the first normalization result to obtain the query-key weight feature.

[0102] The second cross-product operation layer is used to perform a cross-product operation on the query-key weight feature and the value feature to obtain the second cross-product result;

[0103] The second normalization layer is used to normalize the result of the second cross product to obtain the temporal-spatial displacement attention feature.

[0104] In this embodiment, a temporal-spatial displacement attention module is constructed to associate the first-order pose features of adjacent frames and interval frames in the temporal and spatial dimensions of the skeleton features.

[0105] It should be noted that the skeleton features F at times t-2, t-1, and t are obtained using time-series shifting. (t-2) F (t-1) F (t) ; the skeleton features F at times t-2, t-1, t (t-2) F (t-1) F (t) These are linearly mapped to query features, key features, and value features, respectively. Using the query-key-value structure, the attention values ​​output from the query features, key features, and value features are calculated to obtain the temporal-spatial displacement attention feature at time t.

[0106] Furthermore, the skeleton features F at times t-2, t-1, and t are... (t-2) F (t-1) F (t) A 1×1 convolutional layer is used linearly for query features, key features, and value features, respectively; the Softmax() function is used to calculate attention weights, and the temporal-spatial displacement attention features are calculated using the query-key-value structure. The rules are:

[0107]

[0108] Where: Φ n Let n be a linear mapping function, n = 1, 2, 3, which is implemented using a 1×1 convolutional layer.

[0109] In some embodiments, please refer to Figure 6 The time-channel differential attention module includes a time-channel shift layer, a global average pooling layer, a first differential operation layer, a second differential operation layer, and a multilayer perceptron.

[0110] The temporal channel shift layer is used to extract skeleton features from the human skeleton sequence image to be detected, and to obtain the temporal channel skeleton features at time t, time t-1, and time t-2.

[0111] The global average pooling layer is used to perform global pooling calculations on the time-series channel skeleton features at time t, time-1, and time-2, respectively, to obtain the global average pooling skeleton features at time t, time-1, and time-2.

[0112] The first differential operation layer is used to perform differential calculation on the global average pooling skeleton feature at time t and the global average pooling skeleton feature at time t-1 to obtain the first channel differential operator feature;

[0113] The second differential operation layer is used to perform differential calculation on the global average pooling skeleton features at time t and the global average pooling skeleton features at time t-2 to obtain the second channel differential operator features;

[0114] The multilayer perceptron is used to perform nonlinear mapping on the first channel differential operator features and the second channel differential operator features respectively to obtain a first mapping result and a second mapping result;

[0115] The weighted operation layer is used to perform weighted fusion of the first mapping result and the second mapping result to obtain the time-channel differential attention feature.

[0116] In this embodiment, a temporal-channel differential attention module is constructed to fuse second-order differential features of adjacent frames and interval frames in the temporal dimension of skeleton features, thereby improving the dynamic information expression between frames in the channel dimension.

[0117] It should be noted that, firstly, the skeleton features F at times t-2, t-1, and t are obtained using the time-series channel shift. (t-2) F (t -1) F (t) Subsequently, global average pooling is performed on the skeleton features at times t-2, t-1, and t, and the difference features between time t and time t-1, and between time t and t-2 after pooling are calculated to obtain the channel difference operators between time t and t-1, and between time t and t-2. Finally, a multilayer perceptron is used to perform a nonlinear mapping on the two channel difference operators, and the time-channel differential attention features at time t are obtained through weighted fusion.

[0118] Furthermore, by using inter-frame difference to calculate the difference features between time t and time t-1, and between time t and time t-2, global average pooling is used to aggregate the difference features of all joints in the single-frame skeleton graph into the average value of the features of a single node, resulting in two-channel difference operators. The calculation rules are as follows:

[0119]

[0120]

[0121] In the formula, a single joint can be represented by the symbol j It is stated that the human skeleton consists of V Each key point. Represents global average pooling in spatial dimension. , , For a moment t , t -1、 t -2 Joints in Skeletal Features j eigenvectors, .

[0122] Furthermore, the obtained channel difference operator is obtained by using a multilayer perceptron (MLP) and the activation function Sigmoid (Sigmoid). A nonlinear mapping is performed; finally, the output features of the time-channel differential attention module at time t are obtained by fusing the channel difference operators after the two nonlinear mappings through average weighted summation. The calculation rules are as follows:

[0123]

[0124] In some embodiments, please refer to Figure 7 The human skeleton behavior recognition model further includes a cross-dimensional interaction attention module, which includes a first weighted addition layer, a second weighted addition layer, and an element-wise dot product layer.

[0125] The first weighted summation layer is used to perform weighted summation on the spatial-channel grouping attention features and the temporal-spatial displacement attention features to obtain the first weighted summation result;

[0126] The second weighted summation layer is used to perform a weighted summation of the first weighted summation result and the human skeleton sequence image to be detected, to obtain the second weighted summation result;

[0127] The element-wise multiplication layer is used to perform a multiplication operation on the second weighted sum and the time-channel differential attention feature to obtain the cross-dimensional interactive attention feature.

[0128] In this embodiment, the output features F of the spatial-channel grouping attention module, the temporal-spatial displacement attention module, and the temporal-channel difference attention module are fused by weighted summation and element-wise multiplication. S-CGA F T-SSA F T-CDA During fusion, residual branches are introduced to preserve the overall distribution of the original features, resulting in the final output feature F of the cross-dimensional interactive attention module. CDIA .

[0129] It should be noted that the output feature F corresponding to the spatial-channel grouping attention module, the temporal-spatial displacement attention module, and the temporal-channel differential attention module is... S-CGA F T-SSA F T-CDA The set of attention features of the skeleton map of a single frame at all time points in the skeleton sequence can be described as follows: The three attention features are fused using weighted summation and element-wise dot product, while a residual branch is introduced to preserve the original feature F, resulting in the output feature F of the cross-dimensional interactive attention module. CDIA It can be described as:

[0130] ;

[0131] In the formula, This is an element-wise dot product. These are weighting factors for the output features of the spatial-channel grouped attention module, the temporal-spatial displacement attention module, and the temporal-channel differential attention module, respectively, used to control the influence of the output feature distribution of different attention modules.

[0132] In some embodiments, please refer to Figure 2 The human skeleton recognition model further includes spatial graph convolution, a first activation layer, a random deactivation layer, temporal convolution, and a second activation layer;

[0133] The spatial graph convolution, the first activation layer, the random deactivation layer, the cross-dimensional interactive attention module, the temporal convolution, and the second activation layer are connected in sequence.

[0134] In this embodiment, the adaptive graph convolutional network consists of a batch normalization layer, 10 adaptive graph convolutional blocks connected in series, a global average pooling layer at the end, and a fully connected layer. The human skeleton features extracted by the trained adaptive graph convolutional network are fed into a Softmax classifier to obtain the final class confidence score for each behavior category. This score is used to obtain the network's recognition result for a single skeleton behavior sequence and to analyze the recognition accuracy of the overall network model.

[0135] Furthermore, the cross-dimensional interactive attention module is integrated after the random deactivation layer of the graph convolution block and before the temporal convolution to enhance the cross-dimensional association between skeleton features and local human features and whole-body features after spatial graph convolution, and first-order and second-order dynamic information is introduced before the temporal convolution.

[0136] Based on the above-described skeleton behavior recognition method based on cross-dimensional interactive attention mechanism graph convolution, this invention also provides a corresponding skeleton behavior recognition device 800 based on cross-dimensional interactive attention mechanism graph convolution. Please refer to [link to relevant documentation]. Figure 8 The skeleton behavior recognition device 800 based on cross-dimensional interactive attention mechanism graph convolution includes a model acquisition module 810 and a human skeleton behavior recognition module 820.

[0137] The model acquisition module 810 is used to acquire a fully trained human skeleton behavior recognition model, which includes a spatial-channel grouping attention module, a temporal-spatial displacement attention module, and a temporal-channel differential attention module.

[0138] The human skeleton behavior recognition module 820 is used to acquire a sequence of human skeleton images to be detected, and to detect the sequence of human skeleton images to be detected based on the fully trained human skeleton behavior recognition model, so as to obtain the detection result of the sequence of human skeleton images to be detected.

[0139] like Figure 9 As shown, based on the skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism described above, this invention also provides an electronic device, which can be a mobile terminal, desktop computer, laptop, handheld computer, server, or other computing device. The electronic device includes a processor 910, a memory 920, and a display 930. Figure 9Only some components of the electronic device are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0140] In some embodiments, memory 920 may be an internal storage unit of the electronic device, such as a hard disk or memory. In other embodiments, memory 920 may be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Furthermore, memory 920 may include both internal and external storage units. Memory 920 is used to store application software and various types of data installed on the electronic device, such as program code installed on the electronic device. Memory 920 may also be used to temporarily store data that has been output or will be output. In one embodiment, memory 920 stores a skeleton behavior recognition program 940 based on cross-dimensional interactive attention mechanism graph convolution. This skeleton behavior recognition program 940 based on cross-dimensional interactive attention mechanism graph convolution can be executed by processor 910, thereby implementing the skeleton behavior recognition method based on cross-dimensional interactive attention mechanism graph convolution of the various embodiments of this application.

[0141] In some embodiments, processor 910 may be a central processing unit (CPU), microprocessor or other data processing chip, used to run program code stored in memory 920 or process data, such as executing a skeleton behavior recognition method based on graph convolution of a cross-dimensional interactive attention mechanism.

[0142] In some embodiments, display 930 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 930 is used to display information from the skeleton behavior recognition device based on cross-dimensional interactive attention mechanism graph convolution and to display a visual user interface. Components 910-930 of the electronic device communicate with each other via a system bus.

[0143] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a memory, magnetic disk, optical disk, etc.

[0144] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism, characterized in that, include: Obtain a fully trained human skeleton behavior recognition model, wherein the human skeleton behavior recognition model includes a cross-dimensional interactive attention module, and the cross-dimensional interactive attention module includes a spatial-channel grouping attention module, a temporal-spatial displacement attention module, and a temporal-channel differential attention module; A sequence of human skeleton images to be detected is obtained, and the human skeleton behavior recognition model is trained to detect the sequence of human skeleton images to be detected, thereby obtaining the detection result of the sequence of human skeleton images to be detected. The spatial-channel grouping attention module includes a first subgraph grouping layer, multiple first channel displacement layers, a second channel displacement layer, a second subgraph grouping layer, an inter-group cross attention layer, and a spatial dimension feature splicing layer; The first sub-image grouping layer is used to group the human structural parts of the single frame skeleton image at time t in the human skeleton sequence image to be detected, and obtain local sub-image features of multiple different structural parts. The number of the first channel displacement layers is consistent with and corresponds one-to-one with the number of the local subgraph features. Multiple channel displacement layers perform intra-group feature interaction between nodes on their respective local subgraph features to obtain multiple corresponding local channel displacement subgraph features. The second channel displacement layer is used to perform intra-group feature interaction between nodes in the single-frame skeleton graph at time t to obtain global channel displacement features; The second sub-graph grouping layer is used to group the global channel displacement features by human body structural parts to obtain global channel displacement sub-graph features of multiple different structural parts; The inter-group cross-attention layer is used to perform feature interaction on the multiple local channel displacement subgraph features and multiple global channel displacement subgraph features to obtain multiple inter-group interaction features. The spatial dimension feature splicing layer is used to splice the multiple inter-group interaction features in a spatial dimension to obtain the spatial-channel group attention features at time t.

2. The skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism according to claim 1, characterized in that, The spatial-channel grouped attention module also includes multiple first grouped attention convolutional layers and second grouped attention convolutional layers; The number of the first group attention convolutional layers is consistent with the number of the first channel displacement layers and corresponds one-to-one. Multiple first group attention convolutional layers are used to perform linear mapping on their respective local channel displacement sub-map features to obtain multiple corresponding local channel displacement mapping results. The second group attention convolutional layer is used to perform linear mapping on the global channel displacement submap features to obtain the global channel displacement mapping result.

3. The skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism according to claim 1, characterized in that, The temporal-spatial displacement attention module includes a temporal displacement layer, multiple convolutional network layers, a first cross-product operation layer, a first normalization layer, an attention weight calculation layer, a second cross-product operation layer, and a second normalization layer. The temporal shift layer is used to extract skeleton features from the human skeleton sequence image to be detected, and obtain skeleton features at time t, time t-1 and time t-2. The multiple convolutional network layers correspond one-to-one with the skeleton features at time t, time t-1, and time t-2. The multiple convolutional network layers are used to perform linear mapping on the skeleton features at time t, time t-1, and time t-2 to obtain the corresponding value features, key features, and query features. The first cross-product operation layer is used to perform a cross-product operation on the query feature and the key feature to obtain the first cross-product result; The first normalization layer is used to perform a normalization operation on the first cross product result to obtain the first normalized result; The attention weight calculation layer is used to calculate the attention weight of the first normalization result to obtain the query-key weight feature. The second cross-product operation layer is used to perform a cross-product operation on the query-key weight feature and the value feature to obtain the second cross-product result; The second normalization layer is used to normalize the result of the second cross product to obtain the temporal-spatial displacement attention feature.

4. The skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism according to claim 3, characterized in that, The temporal-channel differential attention module includes a temporal-channel shift layer, a global average pooling layer, a first differential operation layer, a second differential operation layer, a multilayer perceptron, and a weighted operation layer. The temporal channel shift layer is used to extract skeleton features from the human skeleton sequence image to be detected, and to obtain the temporal channel skeleton features at time t, time t-1, and time t-2. The global average pooling layer is used to perform global pooling calculations on the time-series channel skeleton features at time t, time-1, and time-2, respectively, to obtain the global average pooling skeleton features at time t, time-1, and time-2. The first differential operation layer is used to perform differential calculation on the global average pooling skeleton feature at time t and the global average pooling skeleton feature at time t-1 to obtain the first channel differential operator feature; The second differential operation layer is used to perform differential calculation on the global average pooling skeleton features at time t and the global average pooling skeleton features at time t-2 to obtain the second channel differential operator features; The multilayer perceptron is used to perform nonlinear mapping on the first channel differential operator features and the second channel differential operator features respectively to obtain a first mapping result and a second mapping result; The weighted operation layer is used to perform weighted fusion of the first mapping result and the second mapping result to obtain the time-channel differential attention feature.

5. The skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism according to claim 4, characterized in that, The cross-dimensional interactive attention module also includes a first weighted addition layer, a second weighted addition layer, and an element-wise dot product layer; The first weighted summation layer is used to perform weighted summation on the spatial-channel grouping attention features and the temporal-spatial displacement attention features to obtain the first weighted summation result; The second weighted summation layer is used to perform a weighted summation of the first weighted summation result and the human skeleton sequence image to be detected, to obtain the second weighted summation result; The element-wise multiplication layer is used to perform a multiplication operation on the second weighted sum and the time-channel differential attention feature to obtain the cross-dimensional interactive attention feature.

6. The skeleton behavior recognition method based on graph convolution with cross-dimensional interactive attention mechanism according to claim 5, characterized in that, The human skeleton behavior recognition model also includes spatial graph convolution, a first activation layer, a random deactivation layer, temporal convolution, and a second activation layer; The spatial graph convolution, the first activation layer, the random deactivation layer, the cross-dimensional interactive attention module, the temporal convolution, and the second activation layer are connected in sequence.

7. A skeleton behavior recognition device based on graph convolution with cross-dimensional interactive attention mechanism, characterized in that, include: The model acquisition module is used to acquire a fully trained human skeleton behavior recognition model, which includes a spatial-channel grouping attention module, a temporal-spatial displacement attention module, and a temporal-channel differential attention module. The human skeleton behavior recognition module is used to acquire a sequence of human skeleton images to be detected, and to detect the sequence of human skeleton images to be detected based on the fully trained human skeleton behavior recognition model, so as to obtain the detection result of the sequence of human skeleton images to be detected. The spatial-channel grouping attention module includes a first subgraph grouping layer, multiple first channel displacement layers, a second channel displacement layer, a second subgraph grouping layer, an inter-group cross attention layer, and a spatial dimension feature splicing layer; The first sub-image grouping layer is used to group the human structural parts of the single frame skeleton image at time t in the human skeleton sequence image to be detected, and obtain local sub-image features of multiple different structural parts. The number of the first channel displacement layers is consistent with and corresponds one-to-one with the number of the local subgraph features. Multiple channel displacement layers perform intra-group feature interaction between nodes on their respective local subgraph features to obtain multiple corresponding local channel displacement subgraph features. The second channel displacement layer is used to perform intra-group feature interaction between nodes in the single-frame skeleton graph at time t to obtain global channel displacement features; The second sub-graph grouping layer is used to group the global channel displacement features by human body structural parts to obtain global channel displacement sub-graph features of multiple different structural parts; The inter-group cross-attention layer is used to perform feature interaction on the multiple local channel displacement subgraph features and multiple global channel displacement subgraph features to obtain multiple inter-group interaction features. The spatial dimension feature splicing layer is used to splice the multiple inter-group interaction features in a spatial dimension to obtain the spatial-channel group attention features at time t.

8. An electronic device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the skeleton behavior recognition method based on cross-dimensional interactive attention mechanism graph convolution as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the skeleton behavior recognition method based on cross-dimensional interactive attention mechanism graph convolution as described in any one of claims 1-6.

Images