Human skeleton behavior recognition method based on multi-modal enhancement converter and hypergraph structure

Abstract

The invention provides a human skeleton behavior recognition method based on a multi-modal enhancement converter and a hypergraph structure. The method comprises the following steps: S1, acquiring original skeleton data representing human behavior recognition; preprocessing the skeleton data to obtain four skeleton modal data in total; s2, carrying out single-mode feature extraction on the four types of skeleton modal data, and then carrying out multi-mode feature fusion to form unified skeleton features; s3, performing feature refining and three-dimensional dynamic feature reinforcement on the skeleton features of the unified mode in time and space; and S4, the refined skeleton features are subjected to a global average pooling layer and a full connection layer to obtain a behavior identification prediction result, and the result is a discrete action classification label. According to the method, a lightweight multi-modal converter network and a double hypergraph topological structure are constructed to enhance the multi-modal skeleton data fusion performance and capture the cooperative motion relationship between joint groups, so that the recognition accuracy and recognition efficiency of double interaction behaviors are improved.

Description

Technical Field

[0001] The present invention relates to computer vision and computer graphics in the field of artificial intelligence, and particularly relates to a human skeleton behavior recognition method based on a multi-modal enhanced transformer and a hypergraph structure. Background Art

[0002] The human behavior recognition technology based on skeleton data has become the core solution in the fields of intelligent monitoring, human-computer interaction, autonomous driving, etc. due to its strong robustness to light changes and view occlusion.

[0003] Traditional methods mainly rely on graph convolutional networks (GCNs) to process skeleton data, modeling human joints as nodes of a topological graph and bone connections as edges, and extracting local spatial features through hierarchical graph convolution operations. Such methods rely on a predefined human topological graph. Although they can capture the cooperative movement between naturally connected joints, they are limited by the static topological structure and cannot adaptively adjust the joint association strength. In particular, they lack the ability to model non-naturally connected distant joints (such as the interaction between the left hand and the right foot). In addition, the local receptive field characteristic of graph convolution leads to low efficiency in capturing long-range spatio-temporal dependencies and is difficult to represent the global associations across limbs and time in complex actions.

[0004] To improve the model's expressive ability, multi-modal data fusion strategies and Transformer networks have been introduced. They learn skeleton features by parallelly processing heterogeneous modal data such as joint coordinates, bone vectors, and their motion information. Existing fusion methods use independent branch networks to extract features of each modality respectively, and achieve feature fusion through late splicing or weighted summation. This will cause the features of different modalities to be misaligned and out of sync in the high-dimensional space, and the shallow local details and deep abstract semantics cannot be associated, making it difficult for the model to learn the complementary information between different modal data. In addition, existing Transformer networks do not pay special attention to and process key features, which may cause the skeleton features representing specific action behaviors to be ignored, resulting in recognition failure.

[0005] For the two-person interaction scenario, existing methods usually expand the single-person skeleton topology into a two-person joint topology and establish static associations through preset rules. Although such methods can initially capture the spatial proximity between the joints of two people, they rely on artificial prior rules and are difficult to adapt to dynamic interaction patterns. At the same time, their modeling method based on paired joints can only capture low-order spatial dependencies and cannot represent the cooperative movement patterns of multi-joint groups. For example, in the hugging action, multiple joints of the hands and the torso need to be modeled as a whole, which will cause the loss of high-order motion semantics. Summary of the Invention

[0006] The present invention aims to solve at least the technical problems existing in the prior art, and particularly innovatively proposes a human skeleton behavior recognition method based on a multimodal enhanced transformer and a hypergraph structure.

[0007] To achieve the above object of the present invention, the present invention provides a human skeleton behavior recognition method based on a multimodal enhanced transformer and a hypergraph structure, including the following steps:

[0008] S1, obtaining the original skeleton data representing human behavior recognition, and preprocessing the original skeleton data to obtain four types of skeleton modal data, namely skeleton joint points, skeleton vectors, the motion change amount of skeleton joint points relative to the previous frame, and the motion change amount of skeleton vectors relative to the previous frame; wherein, the skeleton joint points are the original data of each frame, represented in the form of three-dimensional coordinates; the skeleton vectors are calculated from the coordinate differences between two adjacent skeleton joint points on the body, and the direction is uniformly specified as outward from the body center of gravity; the motion change amount of skeleton joint points relative to the previous frame is calculated by coordinate differences (i.e., the original skeleton sequence with subscripts 1 to n minus the original skeleton sequence with subscripts 0 to n - 1), and the calculation method of the relative motion change amount of skeleton vectors is the same as that.

[0009] S2, performing single-modal feature extraction on the four types of skeleton modal data first, and then performing multi-modal feature fusion to form unified-modal skeleton features;

[0010] S3, performing feature refinement and three-dimensional dynamic feature enhancement on the unified-modal skeleton features in terms of time and space;

[0011] S4, obtaining the prediction result of behavior recognition by passing the refined skeleton features through global average pooling and a fully connected layer, and the result is a discrete action classification label.

[0012] Preferably, performing single-modal feature extraction on the four types of skeleton modal data first, and then performing multi-modal feature fusion is achieved by sequentially passing the four types of skeleton modal data through a progressive multi-modal enhancement module (PME) and a multi-scale temporal convolutional module (MS-TC);

[0013] The progressive multi-modal enhancement module (PME) includes: a two-person hypergraph self-attention module (TH-SA), a multi-scale temporal convolution module (MS-TC), a multi-head cross-modal attention module (MH-CA), and a global context fusion module (GCF); the data output end of the first two-person hypergraph self-attention module (TH-SA) is connected to the data input end of the multi-scale temporal convolution module (MS-TC), the data output end of the multi-scale temporal convolution module (MS-TC) is respectively connected to the data input end of the global context fusion module (GCF), the second two-person hypergraph self-attention module (TH-SA), and the data input end of the multi-head cross-modal attention module (MH-CA), the data output end of the global context fusion module (GCF) is connected to the data input end of the multi-head cross-modal attention module (MH-CA), the data output end of the multi-head cross-modal attention module (MH-CA) is connected to the data input end of the first layer normalization layer, the data output end of the second two-person hypergraph self-attention module (TH-SA) is connected to the data input end of the second layer normalization layer, the data output ends of the first layer normalization layer and the second layer normalization layer are connected, then weights are assigned to the features output by the first layer normalization layer and the second layer normalization layer, then the features are multiplied by the corresponding weights and added together, and then passed through the GELU activation function and a two-dimensional convolutional layer in sequence;

[0014] Among them, the two-person hypergraph self-attention module (TH-SA) is used to model the spatial dependence of specific modal skeleton features, the multi-scale temporal convolution module (MS-TC) is used to model the temporal dependence of specific modal skeleton features, the multi-head cross-modal attention module (MH-CA) uses global context information to enhance the skeleton features of each single modality, and the global context fusion module (GCF) is used to aggregate the skeleton features of each specific modality in the current layer and convert them into global context information.

[0015] The progressive multi-modal enhancement module is deployed in each layer of the feature extraction and fusion stage. It can not only extract the spatial and temporal features of specific modalities, but also gradually enhance the skeleton features of each single modality using global context.

[0016] Preferably, processing data using the progressive multi-modal enhancement module (PME) includes:

[0017] Its input is four skeletal modalities where the subscript j represents the skeleton joint point (i.e., the original data of each frame, represented by three-dimensional coordinates), the subscript b represents the skeletal vector (obtained by taking the difference between the coordinates of two connected joint points on the body, with the direction uniformly outward), the subscript jm represents the amount of motion change of the skeleton joint point relative to the previous frame, and the subscript bm represents the amount of motion change of the skeletal vector relative to the previous frame.

[0018] Each skeletal modality data will first undergo a dual-person hypergraph self-attention module (TH-SA) and a multi-scale temporal convolutional module (MS-TC) for single-modal skeletal data for preliminary feature extraction. This also makes the model more lightweight and improves computational efficiency. Next, the skeletal features will be input into the global context fusion module (GCF) to extract global context information, and in the subsequent multi-head cross-modal attention module (MH-CA), through to enhance This process is represented as follows:

[0019]

[0020] where represents the global context information obtained by the GCF module at the i-th layer;

[0021] Q, K, and V are the three key matrices in the self-attention mechanism, representing query, key, and value respectively; MultiHead is the standard multi-head cross-attention form, and the fused information is dynamically filtered (Dynamic Filter) with which is obtained by re-extracting features through the dual-person hypergraph self-attention module (TH-SA) again:

[0022]

[0023] where ⊙ represents the Hadamard product;

[0024] W * [i] represents the important ratio of the fused information and the single-modal information calculated by the dynamic filter in the i-th layer;

[0025] indicates that the fused information g enhances a certain single-modal information;

[0026] The weights of each branch can be dynamically determined by the learnable parameters of the Conv2d function. This operation can filter out unimportant features during the single-modal enhancement process.

[0027] Finally, the GELU activation function and a simple convolutional layer with residual connection are used to obtain the final output The reason for choosing the GELU (Gaussian Error Linear Unit) activation function is that it has better smoothness and gradient stability, and has a built-in dynamic soft mask function, making the model calculation more efficient.

[0028] The overall output of each modal stream in the progressive multi-modal enhancement module (in fact, each layer of the PME module needs to run in parallel four times (corresponding to four single modalities), and here is the information flow of a single modality) is as follows:

[0029]

[0030] where \(i\) represents the \(i\)-th layer of the HyperProMT network;

[0031] \(g\) represents the global information obtained by the Global Context Fusion Module (GCF);

[0032] \(g\rightarrow *\) indicates that this module uses global context information to enhance each unimodal skeletal data.

[0033] The PME module does not change the characteristic shape of the skeletal modality, and the input and output sizes remain the same, both being where \(N\) is the batch size, \(C\) is the number of feature channels, \(T\) is the sequence length of the entire video, and \(J\) is the number of human joints.

[0034] Preferably, processing the data using the Global Context Fusion Module (GCF) includes:

[0035] The Global Context Fusion Module GCF aggregates the skeletal features of each specific modality in the current layer and converts them into global context for input to the subsequent MH-CA module for cross-modal information interaction. This method is more efficient than pairwise information exchange between different skeletal modalities.

[0036] The input of the Global Context Fusion Module includes the global context of the current layer and the features of four skeletal modalities which are first converted into the feature \(z\) through the Conv2d and Tanh functions j , \(z\) b , \(z\) jm , \(z\) bm and \(z\) g , whose value range is \([-1, 1]\). The magnitude of the feature values of different skeletal modalities also defines their importance. Then, the Softmax function is applied to these features to obtain the weights of each skeletal modality, followed by group normalization and GELU activation operations; finally, the features of each modality are multiplied by the corresponding weights and added together to obtain the final global context information of the \(i\)-th layer

[0037] The calculation formula for the above process is as follows:

[0038]

[0039] where \(\odot\) represents the Hadamard product;

[0040] \(e\) represents the natural base;

[0041] respectively represent the features of the four skeletal modalities: skeletal joint points, skeletal vectors, the amount of motion change of skeletal joint points relative to the previous frame, and the amount of motion change of skeletal vectors relative to the previous frame;

[0042] w j 、w b 、w jm 、w bm respectively represent the weights corresponding to the four skeletal modality features: skeletal joint points, skeletal vectors, the amount of motion change of skeletal joint points relative to the previous frame, and the amount of motion change of skeletal vectors relative to the previous frame;

[0043] w g represents the global context corresponding weight;

[0044] This process is a variant of the attention mechanism, which fuses the important features of different skeletal modalities and the global context of the current layer into the global context of the next layer.

[0045] Finally, the final global context information of the i-th layer is input into the GELU activation function and a convolutional layer with a residual connection to obtain the output of the global context fusion module

[0046] Preferably, using the dual-person hypergraph self-attention module TH-SA to process data includes:

[0047] To better model spatial features, TH-SA embeds two types of skeletal structure information on the basis of the standard self-attention (Vanilla SA), namely the dual-person skeletal topology and the dual-person hypergraph representation. They respectively represent the pairwise relationship and the high-order relationship between skeletal joints.

[0048] The input of this module is the unimodal skeletal feature dual-person topological graph structure information R k and dual-person hypergraph feature E hyper . First, calculate the attention index A, which consists of three parts, namely the standard self-attention (Vanilla SA), the dual-person skeletal topology graph embedding, and the dual-person hypergraph (2p-Hypergraph) embedding. Given the input The attention index calculation formula is as follows:

[0049]

[0050] Q, K, and V respectively represent query, key, and value;

[0051] represents the unimodal skeletal feature;

[0052] The dual-person skeleton topology embedding and the dual-person hypergraph embedding are respectively implemented by multiplying the structural information R k and the dual-person hypergraph feature E hyper by the skeleton feature; the design purpose of these two embedding methods is to enable the Transformer to perceive the specific spatial semantics of each joint. By performing matrix multiplication on the attention value A and V, the output of the dual-person hypergraph self-attention module can be obtained

[0053]

[0054] where Softmax is the activation function;

[0055] d k represents the dimension of the vector of the key K.

[0056] Preferably, the unified skeleton feature learned from S2 is refined in terms of features and strengthened with three-dimensional dynamic features in time and space; this is achieved by refining the high-order cooperative motion features of the joint groups through the dual-person hypergraph self-attention module (TH-SA) and the multi-scale temporal convolutional module (MS-TC).

[0057] Preferably, before the refined skeleton features in step S4 are input to the global average pooling, they are comprehensively strengthened in terms of the expression of key features through the spatio-temporal channel enhancement module (STC-Booster), and the spatio-temporal channel enhancement module includes spatio-temporal joint enhancement and channel feature enhancement;

[0058] Spatio-temporal joint enhancement: Two one-dimensional pooling layers are respectively used to compress the time and space dimensions, and then they are concatenated and jointly learned through a two-dimensional convolutional layer and a linear layer, so that there is a channel for the important information of the time and space dimensions to interact; another function of the linear layer is to perform channel dimensionality reduction, reducing the number of channels of the skeleton feature by r times (in the present invention, r = 4), and this step is to reduce the computational overhead of the model. The learned skeleton feature is then re-segmented into two separate feature vectors that only retain the spatial and temporal feature information, and they are respectively passed through a one-dimensional convolutional layer and the Tanh function to map their values to the interval [-1, 1]; finally, these two vectors in the form of C×T×1 and C×1×J can be regarded as two coefficients, and they can enhance the skeleton features at specific key points and time frames point by point;

[0059] Channel feature enhancement: The skeleton feature is divided into multiple subsets according to the dual-person hypergraph partitioning method, each subset contains a certain number of joint points, and each subset is mapped into a coefficient in the form of C×1×1 representing the strength of the channel feature through an adaptive two-dimensional pooling layer and the Tanh function, and then it is multiplied point by point back to the corresponding skeleton feature subset to achieve the purpose of channel feature enhancement.

[0060] In summary, due to the adoption of the above technical solutions, the method of the present invention is based on the progressive multi-modal feature fusion model HyperProMT of the designed Transformer, which optimizes the drawbacks of the existing fusion framework, performs progressive feature fusion in the middle layer of the model, so that the model can learn the complementary information and high-order semantic information of multi-modal data. On this basis, a class-attention method is also used to lightly and explicitly model the dependencies of skeletal features in the channel, spatial, and temporal dimensions, thereby strengthening the important features meaningful for action recognition and improving the performance of the model;

[0061] (2) Construct a learnable hypergraph that conforms to human prior knowledge, dynamically convert the traditional static two-person topology into a high-order representation containing multi-level hyperedges, break through the limitations of pairwise joint modeling, accurately capture the cooperative motion patterns of cross-limb joint groups in interactive behaviors such as back patting and hugging, and improve the recognition accuracy of two-person interactive behaviors.

[0062] The additional aspects and advantages of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and / or additional aspects and advantages of the present invention will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0064] Figure 1 is a schematic diagram of the overall framework of the hypergraph progressive multi-modal conversion algorithm model (HyperProMT) of the present invention.

[0065] Figure 2 is a structural diagram of the progressive multi-modal enhancement module (PME) in the algorithm of the present invention.

[0066] Figure 3 is a structural diagram of the global information fusion module (GIF) in the algorithm of the present invention.

[0067] Figure 4 is a structural diagram of the two-person hypergraph self-attention module (TH-SA) in the algorithm of the present invention.

[0068] Figure 5 is the visualization effect of the single-person skeletal topology map and the two-person skeletal topology map.

[0069] Figure 6 is the visualization comparison between the learnable two-person hypergraph partitioning strategy proposed by the present invention and the predefined two-person hypergraph partitioning strategy.

[0070] Figure 7 is a structural diagram of the spatio-temporal channel enhancement module (STC-Booster) proposed by the present invention.

[0071] Figure 8 It is a visualization comparison chart of the confusion matrix of the recognition results between the present invention and other algorithms. Detailed implementation manners

[0072] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0073] For the given video data with labeled skeletons, the goal of the method of the present invention is to identify what action the person in the video data is performing. The core idea is to model the dual-person interaction behavior by constructing a dual-person hypergraph, and then perform feature extraction, fusion, and refinement of the interaction behavior through a multi-modal feature fusion network based on Transformer. In this way, the collaborative motion patterns of joint groups can be captured and the data of different skeleton modalities can be efficiently utilized, thereby improving the recognition accuracy.

[0074] The method of the present invention includes the following steps:

[0075] S1. Obtain the original skeleton data representing human behavior recognition. Preprocess the skeleton data to obtain a total of four types of skeleton modality data, which are respectively skeleton joint points, skeleton vectors, the amount of motion change of skeleton joint points relative to the previous frame, and the amount of motion change of skeleton vectors relative to the previous frame;

[0076] S2. Use the four types of skeleton modality data obtained in S1 to first perform single-modal feature extraction and then perform multi-modal feature fusion to form unified skeleton features;

[0077] S3. Perform feature refinement and three-dimensional dynamic feature enhancement on the unified skeleton features learned from S2 in terms of time and space;

[0078] S4. Obtain the prediction result of behavior recognition by passing the refined skeleton features through global average pooling and a fully connected layer, and the result is a discrete action classification label.

[0079] The method of the present invention is implemented based on the designed hypergraph progressive multi-modal transformation algorithm model (HyperProMT). The overall architecture of HyperProMT is as Figure 1 shown, and its core architecture is based on the Transformer framework. Specifically, the network model of HyperProMT includes five modules and a topological structure. Except for MS-TC which is a module proposed for other algorithms, the remaining modules are all designed by ourselves:

[0080] This network first generates multi-modal data from skeletal sequence data and then adopts a two-stage processing flow: in the first stage, four skeletal modalities such as joint coordinates and skeletal vectors are processed through parallel data streams. Each modal stream is equipped with a Progressive Multi-modal Enhancement module (PME), a Multi-scale Temporal Convolution module (MS-TC), and a Global Context Fusion module (GCF) to dynamically extract and fuse the features of each modality, while achieving intra-modal spatio-temporal feature extraction and cross-modal global context fusion; in the second stage, the fused unified modal input features are fed into a refinement network. The Two-person Hypergraph Self-attention module (TH-SA) and the Multi-scale Temporal Convolution module (MS-TC) are used to refine the high-order cooperative motion features of joint groups, and then the Spatial-Temporal Channel Booster module (STC-Booster) is used to enhance the expression of key features in all directions. Finally, the action prediction results are output through pooling and fully connected layers. In addition, a two-person hypergraph topology is designed to be embedded in the self-attention calculation of the TH-SA module to represent the multi-level dependency relationships between joints in the skeleton. For the two-person interaction scenario, through a learnable method, a high-order two-person hypergraph containing multi-hyperedge connections is dynamically generated from the basic skeleton topology. This structure is embedded in the self-attention calculation of the TH-SA module, which can break through the limitations of traditional pairwise joint modeling and achieve multi-level feature capture from local limb cooperation to whole-body interaction dependence.

[0081] Progressive Multi-modal Enhancement module PME: This module is a large module that contains multiple other modules. The PME module is deployed in each layer of the feature extraction and fusion stage. It can not only extract the spatial and temporal features of a specific modality, but also gradually enhance the skeleton features of each single modality using global context. The PME module consists of four parts: the Two-person Hypergraph Self-attention module (TH-SA), the Multi-scale Temporal Convolution module (MS-TC), the Multi-head Cross-modal Attention module (MH-CA), and the Global Context Fusion module (GCF). The TH-SA module is used to model the spatial dependence of the skeleton features of a specific modality, the MS-TC module is used to model the temporal dependence of the skeleton features of a specific modality, the MH-CA module uses global context information to enhance the skeleton features of each single modality, and the GCF module is used to aggregate the skeleton features of each specific modality in the current layer and convert them into global context information;

[0082] The technical details of the PME module are as Figure 2 shown. Its input is four skeletal modalities where the subscript j represents the skeletal joint points (i.e., the original data of each frame, represented by three-dimensional coordinates), the subscript b represents the skeletal vector (obtained by subtracting the coordinates of two connected joint points on the body, with the direction uniformly outward), the subscript jm represents the motion change amount of the skeletal joint point relative to the previous frame, and the subscript bm represents the motion change amount of the skeletal vector relative to the previous frame. Compared with the previous methods, we believe that directly fusing the original data of different modalities will lead to data format misalignment and excessive redundant information. The data of different modalities need to first establish their own dependency relationships, so that feature fusion with other modalities will be more efficient. Therefore, the data stream of each modality will first pass through a TH-SA module and an MS-TC module for single-modal skeletal data for preliminary feature extraction, which will also make the model more lightweight and improve the computational efficiency. Next, the skeletal features will be input into the GCF module to extract global context information, and in the subsequent MH-CA module through to enhance This process is shown as follows:

[0083]

[0084] where represents the global context information obtained by the GCF module in the i-th layer; MultiHead is the standard multi-head cross-attention form, which will not be elaborated here. The fused information and the which is obtained by extracting features again through the two-person hypergraph self-attention module (TH-SA) later are subjected to dynamic filtering (Dynamic Filter) processing:

[0085]

[0086] where ⊙ represents the Hadamard product. The weights of each branch can be dynamically determined by the learnable parameters of the Conv2d function. This operation can filter out the unimportant features in the single-modal enhancement process. Finally, we use the GELU activation function and a simple convolutional layer with a residual connection to obtain the final output The reason for choosing the GELU (Gaussian Error Linear Unit) activation function is that it has better smoothness and gradient stability, and has a built-in dynamic soft mask function, making the model calculation more efficient. The overall output of each modality stream in the PME module is as follows:

[0087]

[0088] where \(i\) represents the \(i\)-th layer of the HyperProMT network, \(g\) represents the global information obtained by the Global Context Fusion Module (GCF), and \(g\rightarrow*\) indicates that this module utilizes the global context information to enhance each single-modal skeletal data. The PME module does not change the characteristic shape of the skeletal modality, and the input and output sizes remain the same, both being where \(N\) is the batch size, \(C\) is the number of feature channels, \(T\) is the sequence length of the entire video, and \(J\) is the number of human joints.

[0089] Global Context Fusion Module GCF: This module aggregates the skeletal features of each specific modality in the current layer and converts them into global context for input to the subsequent MH-CA module for cross-modal information interaction. This approach is more efficient than pairwise information exchange between different skeletal modalities. The technical details of the GCF module are as Figure 3 shown. Its input includes the global context of the current layer and four skeletal modalities which are first converted into features \(z\) j , \(z\) b , \(z\) jm , \(z\) bm and \(z\) g through the Conv2d and Tanh functions, and their value range is \([-1, 1]\). The eigenvalue magnitudes of different skeletal modalities also define their importance. Then, we apply the Softmax function to these features to obtain the weights of each skeletal modality, followed by group normalization and GELU activation operations. Finally, the features of each modality are multiplied by their corresponding weights and summed to obtain the global context The calculation process is formulated as follows:

[0090]

[0091] This process is a variant of the attention mechanism, which fuses the important features of different skeletal modalities and the global context of the current layer into the global context of the next layer. Finally, we input the global context information into the GELU activation function and a convolutional layer with a residual connection to obtain the output of the GCF module

[0092] Two-Player Hypergraph Self-Attention Module TH-SA: This module exists in two stages of the HyperProMT network model, and its function is to model the spatial dependence of skeletal features. To better model spatial features, TH-SA embeds two types of skeletal structure information on the basis of the standard self-attention (Vanilla SA), namely the two-player skeletal topology and the two-player hypergraph representation. They represent the pairwise relationships and higher-order relationships between skeletal joints respectively. The technical details of the TH-SA block are as Figure 4As shown. The input of this module is unimodal skeletal features The structural information R of the two-person topological graph structure k and the two-person hypergraph feature E hyper . This module first calculates the attention index A, which consists of three parts, namely Vanilla Self-Attention (Vanilla SA), two-person skeletal topological graph embedding, and two-person hypergraph (2p-Hypergraph) embedding. Given the input The calculation formula of the attention index is as follows:

[0093]

[0094] The two-person skeletal topological graph embedding and the two-person hypergraph embedding are respectively realized by multiplying the structural information R k and the two-person hypergraph feature E hyper by the skeletal feature. The methods for obtaining them will be described in detail in the part introducing the two-person hypergraph topological structure. The design purpose of these two embedding methods is to enable the Transformer to perceive the specific spatial semantics of each joint. By performing matrix multiplication on the attention value A and V, the output of TH-SA can be obtained

[0095]

[0096] Two-person hypergraph structure: The present invention first introduces the structural information encoding method of the two-person skeletal topology, and then introduces the construction method of the two-person hypergraph.

[0097] Structural information encoding of the two-person skeletal topology: The joints of the human body are naturally connected by bones to form a biomechanical model. The movement of each joint during an action will have a chain effect on other joints. Therefore, it is necessary to embed the human body structure information into each position of the skeletal feature.

[0098] The traditional independent single-person skeletal topology cannot capture the interaction information between two people. Therefore, we use a simple but effective two-person skeletal topology. The difference between the independent single-person skeletal topology and the two-person skeletal topology is that we regard the skeletal topologies of two people as a whole and connect the limbs and heads of the two people. Through the skeletal topology embedding method, the skeletal features belonging to two people can interact through these additional connections. Their visualizations are as shown in Figure 5 (a) and 5(b). For the sake of clarity, the number of skeletal joint points mentioned later is still expressed by J, but in fact it is 2J.

[0099] On this basis, we refer to the existing Relative Position Embedding (RPE) method to obtain the structural information R between each joint point of the skeleton k . We first use the Shortest Path Distance (SPD) algorithm to obtain the set where p i, j represents the distance or the number of hops between joint i and joint point j. Next, we define a learnable parameter to represent the weights corresponding to different distances (the maximum number of hops is k, and if the distance between two joint points exceeds k, the weight is fixed to 0). Finally, each element in set P is assigned the corresponding weights in, obtaining the topological information of the skeleton

[0100]

[0101] Construction of the dual hypergraph: In a standard graph, each edge can only connect two vertices. However, a hypergraph is not restricted by this. Its hyperedges can connect multiple vertices. A hypergraph is defined as G=(V, E, W), where V is the set of vertices, E is the set of hyperedges, and W is a symmetric matrix representing the weights of the hyperedges. The hypergraph G can be represented by an incidence matrix H of size |V|×|E|, and its elements are defined as follows:

[0102]

[0103] The degree d(v) of a vertex is defined as d(v)=∑ e∈E w(e)h(v, e), and the degree d(e) of a hyperedge =∑ v∈V h(v, e). We use diagonal matrices D v and D e to represent the degree matrices of vertices and hyperedges respectively, where the elements on the diagonal are the degrees of each vertex or hyperedge.

[0104] To achieve efficient computation, we set the degree d(v)=1 for each vertex. In this specific case, the human joints can be divided into |E| discrete subsets, and the incidence matrix H is equivalent to a partition matrix. Each row is a one - hot vector representing the group to which each joint belongs. Therefore, how to formulate the partitioning strategy to obtain the hypergraph representation is crucial. We show two common skeleton topological partitioning strategies based on human prior knowledge in Figure 6 (a) and 6(b). The body part partitioning strategy divides the skeleton topology into limbs and torso, while the upper - lower partitioning strategy divides the skeleton topology into upper body and lower body. Although the above - mentioned partitioning strategies can reflect the relationships between multiple joints to a certain extent, these artificial methods are not the optimal solutions and have poor generalization ability. Different from these methods, in the TH - SA module, we use a learnable partition matrix that can automatically convert the dual - person skeleton topology into a dual - person hypergraph representation. Although the binary incidence matrix can be regarded as a partition matrix, its values are discrete. We parameterize it by applying the Softmax function along the column axis to make its values continuous:

[0105]

[0106] Continuous partitioning matrix can be jointly optimized with other Transformer parameters. At the end of the optimization, we re-convert the values of the partitioning matrix from continuous to discrete by applying the argmax function along each row of :

[0107]

[0108] Learned partitioning matrix is visualized as Figure 6 (c). Then, we use to obtain the two-person hypergraph representation and embed it into the skeleton features. and the skeleton features Hyperedge set representation can be calculated by the following formula:

[0109]

[0110] where the inverse degree matrix D of the hyperedge e is used for normalization, and W e is a learnable parameter matrix. Finally, we assign the weight of each hyperedge to its corresponding attached joint to obtain the two-person hypergraph feature

[0111]

[0112] Spatio-temporal channel enhancement module STC-Booster: This module is placed in the feature refinement and enhancement stage of the overall model. Its main role is to distinguish and enhance the information in the skeleton features that is more effective for action recognition at low cost. Its technical details are as Figure 7 shown. STC-Booster enhances the specific information of the skeleton features in the spatial, temporal, and channel dimensions.

[0113] Skeleton features are first Figure 7Spatio-temporal joint enhancement of the left part. We use two one-dimensional pooling layers to compress the temporal and spatial dimensions respectively, and then concatenate them and perform joint learning through a two-dimensional convolutional layer and a linear layer, so that there is an interaction channel for the important information in the temporal and spatial dimensions. Another function of the linear layer is to perform channel dimensionality reduction, reducing the number of channels of the skeletal features by r times (in this invention, r = 4). This step is to reduce the computational overhead of the model. The learned skeletal features are then re-segmented into two separate feature vectors that only retain the spatial and temporal feature information, and they are passed through a one-dimensional convolutional layer and a Tanh function in sequence to map their values to the interval [-1, 1]. Finally, these two vectors in the form of C×T×1 and C×1×J can be regarded as two coefficients, which can enhance the skeletal features at specific key points and time frames point by point.

[0114] Figure 7 The right part is channel feature enhancement. Compared with traditional channel feature enhancement methods, this invention takes into account the situation of some collaborative work of various parts of the human body during actual movement, and applies the way of dual-person hypergraph partitioning learned in Figure 6 (c) to the channel feature enhancement process, which can more efficiently distinguish the contributions of different channels or different convolutional kernels in action recognition, so as to strengthen the convolutional kernels with good specific learning effects to improve the action recognition accuracy. Specifically, we divide the skeletal features into multiple subsets according to the dual-person hypergraph partitioning method, and each subset contains a certain number of joints. We map each subset into a coefficient representing the strength of the channel feature in the form of C×1×1 through an adaptive two-dimensional pooling layer and a Tanh function, and then multiply it point by point back to the corresponding skeletal feature subset to achieve the purpose of channel feature enhancement.

[0115] After spatio-temporal joint enhancement and channel feature enhancement, the learning ability of the model will be greatly enhanced. This method is applicable to the human interaction behavior recognition method and system for skeleton data, and can be applied to scenarios such as bionic robot control, intelligent security monitoring, and human-computer interaction systems.

[0116] To prove the technical effects and their effectiveness, relevant experiments were carried out.

[0117] (1) Dataset

[0118] We used the publicly available NTU-RGB+D 60 and NTU-RGB+D 120 datasets provided by Nanyang Technological University.

[0119] NTU-RGB+D 60 (NTU-60) is a large-scale dataset for RGB+D human action recognition, containing over 56,000 video samples and 4 million frames of data, collected from 40 different subjects. This dataset contains 60 different action categories, covering daily behaviors, interactive behaviors, and health-related actions. This invention specifically focuses on the recognition of two-person interaction behaviors, so only the data of 11 action categories are used, which are marked as NTU-60* in subsequent experiments. The specific action categories include: punch / slap, pat on the back, giving object, walking towards, kicking, point finger, touch pocket, walking apart, pushing, hugging, handshaking.

[0120] NTU-RGB+D 120 (NTU-120) is an extended dataset of NTU-60, with 15 new two-person interaction action categories added: hit with object, wield knife, knock over, grab stuff, shoot with gun, step on foot, high-five, cheers and drink, carry object, take a photo, follow, whisper, exchange things, support somebody, rock–paper–scissors. It is marked as NTU-120* in subsequent experiments.

[0121] (2) Experimental settings

[0122] The experimental environment of this article is as follows: Windows 11 operating system, NVIDIA RTX 3090 GPU, Python 3.8, PyTorch 1.13.1, CUDA 11.8. During the data processing stage, we set the training batch size to 16, and uniformly scaled the number of frames per single sample to 64 frames. Other preprocessing procedures are implemented with reference to the preprocessing parts of the CTR-GCN method and the ST-GCN method. The model hierarchy depth of the present invention is 7 layers, the number of channels in each layer is configured to 192, and the attention mechanism module adopts an 8-head parallel computing structure. The model training strategy is as follows: the total number of training epochs is 65, among which the first 5 epochs are the learning rate warm-up stage; the loss function is the standard cross-entropy; the optimizer is momentum stochastic gradient descent (SGD), with a momentum coefficient of 0.9 and a weight decay of 0.0005; the base learning rate (LR) is 0.1, the decay rate is 0.1, and the decay nodes are the 30th and 50th epochs;

[0123] (3) Evaluation metrics

[0124] Similar to most skeleton-based action recognition tasks, for quantitative comparison, we use the recognition accuracy (%) as the metric to evaluate the performance of our method. Specifically, the NTU-60 database uses two dataset partitioning protocols: the cross-subject (X-Sub) protocol (the training set comes from 20 subjects, and the test set comes from another 20 subjects) and the cross-view (X-View) protocol (two cameras are used to collect training data, and the third is used for testing). The NTU-120 database also uses two dataset partitioning protocols: the cross-subject (X-Sub) protocol (53 people in the training set / the remaining people in the test set) and the cross-device configuration (X-Setup) protocol (half of the device configurations are used for training). In addition, we also show the confusion matrix of the recognition results of the present invention and other advanced methods.

[0125] (4) Experimental results and comparison

[0126] We tested and evaluated the proposed method on the NTU-60 and NTU-120 datasets. To prove the improvement effect of the proposed method, we compared our method with other methods based on RNN, GCN, and Transformer networks. Table 1 below reports the comparison of each method on different evaluation metrics, where "-" indicates information that cannot be queried in the official documentation. From Table 1, we can see that our method achieves the best performance on four metrics, which indicates that our method HyperProMT is effective in the skeleton-based dual-person action recognition task.

[0127] Table 1 Comparison of Experimental Result Metrics

[0128]

[0129]

[0130] To further demonstrate the performance of our method, we show Figure 8 the normalized confusion matrices of the recognition results of the EfficientGCN-B1 model (baseline model) and our proposed ME-Former model under the cross-subject (X-Sub) protocol in the NTU-60* dataset. The confusion matrix contains 11 action classes. It can be seen that our model outperforms the EfficientGCN-B1 model in almost all action classes. Among them, the action classes with the most significant improvement in recognition accuracy are point finger and hugging, both of which are 1.7% higher than the baseline model. This once again proves the superior performance of the present invention in the task of dual-person behavior recognition.

[0131] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A human skeleton behavior recognition method based on a multi-modal enhanced transformer and a hypergraph structure, characterized in that It includes the following steps: S1. Obtain the original skeleton data representing human behavior recognition, and preprocess the original skeleton data to obtain four types of skeletal modal data, namely skeleton joint points, skeletal vectors, the amount of motion change of skeleton joint points relative to the previous frame, and the amount of motion change of skeletal vectors relative to the previous frame; S2. First perform single-modal feature extraction on the four types of skeletal modal data, and then perform multi-modal feature fusion to form skeletal features in a unified modality; S3. Refine the features of the unified-modal skeleton in terms of time and space and strengthen the three-dimensional dynamic features; S4. Obtain the prediction result of behavior recognition by passing the refined skeletal features through global average pooling and a fully connected layer, and the result is a discrete action classification label.

2. The human skeleton behavior recognition method based on a multi-modal enhanced transformer and a hypergraph structure according to claim 1, wherein, Performing single-modal feature extraction on the four types of skeletal modal data first and then multi-modal feature fusion is achieved by passing the four types of skeletal modal data through a progressive multi-modal enhancement module and a multi-scale temporal convolutional module in sequence; The progressive multi-modal enhancement module includes: a dual-person hypergraph self-attention module, a multi-scale temporal convolutional module, a multi-head cross-modal attention module, and a global context fusion module; the data output end of the first dual-person hypergraph self-attention module is connected to the data input end of the multi-scale temporal convolutional module, the data output end of the multi-scale temporal convolutional module is respectively connected to the data input end of the global context fusion module, the second dual-person hypergraph self-attention module, and the data input end of the multi-head cross-modal attention module, the data output end of the global context fusion module is connected to the data input end of the multi-head cross-modal attention module, the data output end of the multi-head cross-modal attention module is connected to the data input end of the first layer normalization layer, the data output end of the second dual-person hypergraph self-attention module is connected to the data input end of the second layer normalization layer, the data output ends of the first layer normalization layer and the second layer normalization layer are connected, then weights are assigned to the features output by the first layer normalization layer and the second layer normalization layer, then the features are multiplied by the corresponding weights and added together, and then passed through a GELU activation function and a two-dimensional convolutional layer in sequence; Among them, the dual-person hypergraph self-attention module is used to model the spatial dependence of specific-modal skeleton features, the multi-scale temporal convolutional module is used to model the temporal dependence of specific-modal skeleton features, the multi-head cross-modal attention module uses global context information to enhance the skeleton features of each single modality, and the global context fusion module is used to aggregate the skeleton features of each specific modality in the current layer and convert them into global context information.

3. A human skeleton behavior recognition method based on a multimodal enhanced transformer and a hypergraph structure according to claim 2, characterized in that, Processing data using the progressive multi-modal enhancement module includes: Each type of skeletal modality data will first undergo a dual-person hypergraph self-attention module and a multi-scale temporal convolutional module for single-modal skeletal data for preliminary feature extraction. Next, the skeletal feature Y * [i] will be input into the global context fusion module to extract global context information, and in the subsequent multi-head cross-modal attention module, through to enhance Y * [i] , and this process is shown as follows: wherein represents the global context information obtained by the i-th layer through the GCF module; Q, K, and V represent query, key, and value respectively; MultiHead is in the standard multi-head cross-attention form, and the fused information is processed dynamically with the features re-extracted by the subsequent dual-person hypergraph self-attention module for dynamic filtering: where ⊙ represents the Hadamard product; W * [i] represents the important proportion of the fusion information and the unimodal information calculated by the dynamic filter in the i-th layer; Indicates that the fused information g enhances a certain unimodal information; Finally, use the GELU activation function and a simple convolutional layer with a residual connection to obtain the final output Y * [i+1] ; The overall output of each modality stream in the progressive multi-modal enhancement module is as follows: where i represents the i-th layer of the network; g represents the global information obtained by the global context fusion module; g→* means that this module uses global context information to enhance each single-modal skeletal data.

4. A human skeleton behavior recognition method based on a multi-modal enhanced transformer and a hypergraph structure according to claim 3, characterized in that, Processing data using the global context fusion module includes: The input of the global context fusion module includes the global context of the current layer and the features Y of four skeletal modalities * [i] , * ∈ {j, b, jm, bm}; First, it is converted into the feature z through the Conv2d and Tanh functions j , z b , z jm , z bm and z g , Then, the Softmax function is applied to these features to obtain the weights of each skeletal modality, and then group normalization and GELU activation operations are performed; Finally, the features of each modality are multiplied and added with the corresponding weights to obtain the final global context information of the i-th layer The calculation formula for the above process is as follows: z * = Tanh(Conv2d(Y * [i] )) where ⊙ represents the Hadamard product; e represents the natural base; respectively represent the features of four skeletal modalities: skeletal joint points, skeletal vectors, the amount of motion change of skeletal joint points relative to the previous frame, and the amount of motion change of skeletal vectors relative to the previous frame; w j 、w b 、w jm 、w bm respectively represent the weights corresponding to the four skeletal modal features of the skeleton joint points, the skeletal vectors, the motion change amount of the skeleton joint points relative to the previous frame, and the motion change amount of the skeletal vectors relative to the previous frame; w g represents the global context the corresponding weight; Finally, the final global context information of the i-th layer is input into the GELU activation function and a convolutional layer with a residual connection to obtain the output of the global context fusion module 5. A human skeleton behavior recognition method based on a multimodal enhanced transformer and a hypergraph structure according to claim 3, characterized in that, Processing data using the dual-person hypergraph self-attention module TH-SA includes: First, calculate the attention index A, which consists of three parts: standard self-attention, dual-person skeletal topology map embedding, and dual-person hypergraph embedding. Given the input Q = K = V = Y * [i] , and the attention index calculation formula is as follows: Q, K, and V represent query, key, and value respectively; Y * [i] represents unimodal skeletal features; Denote E hyper as the transpose of; The dual-person skeleton topology embedding and the dual-person hypergraph embedding are respectively realized by multiplying the structural information R k and the dual-person hypergraph feature E hyper with the skeleton feature; by performing matrix multiplication on the attention value A and V, the output Y of the dual-person hypergraph self-attention module can be obtained * [i] : Among them, Softmax is the activation function; d k Represents the dimension of the vector of key K.

6. The human skeleton behavior recognition method based on a multi-modal enhanced transformer and a hypergraph structure according to claim 1, characterized in that, The feature refinement and three-dimensional dynamic feature enhancement of the unified skeletal features learned from S2 in time and space are realized by refining the high-order cooperative motion features of joint groups through a dual-person hypergraph self-attention module and a multi-scale temporal convolutional module.

7. A human skeleton behavior recognition method based on a multi-modal enhanced transformer and a hypergraph structure according to claim 1, characterized in that, Before the refined skeletal features in step S4 are input into the global average pooling, the expression of key features is strengthened in all directions through a spatio-temporal channel enhancement module, and the spatio-temporal channel enhancement module includes spatio-temporal joint enhancement and channel feature enhancement; Spatio-temporal joint enhancement: Two one-dimensional pooling layers are respectively used to compress the time and space dimensions, and then they are concatenated and jointly learned through a two-dimensional convolutional layer and a linear layer, so that there is a channel for the important information of the time and space dimensions to interact; The learned skeletal features are then re-segmented into two separate feature vectors that only retain the spatial and temporal feature information, and they are successively passed through a one-dimensional convolutional layer and a Tanh function to map their values to the interval [-1, 1]; Finally, these two vectors can be regarded as two coefficients, which can enhance the skeletal features at specific key points and time frames point by point; Channel feature enhancement: The skeletal features are divided into multiple subsets according to the dual-person hypergraph division method, each subset contains a certain number of joint points, and each subset is mapped into a coefficient representing the strength of the channel features through an adaptive two-dimensional pooling layer and a Tanh function, and then it is multiplied point by point back to the corresponding skeletal feature subset to achieve the purpose of channel feature enhancement.

Images