A two-person behavior recognition method based on semantics and interaction features

Abstract

This invention proposes a method for dual-person behavior recognition based on semantic and interaction features. The implementation steps are as follows: 1. Obtain the 3D coordinates of human skeletal joints from a video using the OpenPose algorithm. 2. Determine whether the behavior is dual-person based on the skeletal joint information. 3. Represent the skeletal information of the two individuals using and , respectively, and embed them into a high-dimensional representation through high-dimensional encoding. 4. Define the semantic SPA of the joint type and embed it into a high-dimensional representation through high-dimensional encoding, then fuse it with the skeletal information. 5. Extract the dual-person interaction information g_ram from the skeletal information of the two individuals. 6. Construct a joint-level graph convolutional module and perform feature fusion with the skeletal information fused with the joint type and the dual-person interaction information. 7. Define the semantics of the time frame type and fuse it with the features obtained after joint-level graph convolution. 8. Implement behavior recognition and classification through the constructed frame-level convolutional module. This invention is a dual-person behavior recognition method based on semantic and interaction features, which can effectively express interaction features and recognize dual-person behaviors.

Description

Technical Field

[0001] This invention belongs to the field of human behavior feature classification technology, and in particular relates to a method for dual-person behavior recognition based on semantic and interaction features. Background Technology

[0002] Human behavior recognition refers to the automated analysis of ongoing human actions from unknown video or image sequences. Based on current research both domestically and internationally, interactive behavior recognition can be broadly categorized into template matching-based methods and deep learning-based methods. Template matching-based methods are exemplified by Van Gemeren et al., who combined HOG / HOF descriptors to encode feature frames of interactions between humans; and Ji et al., who used HMM to model the changes of HOG descriptors over time and distinguish the start, middle, and end stages of interactive behavior. Deep learning-based interactive behavior recognition can be further divided into CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), and GCN (Graph Convolutional Network) methods. A representative CNN method is the AlphaAction network proposed by Tang et al., which uses an interaction aggregation structure to uniformly model interactive actions in RGB videos; a representative RNN method is Wang et al., who used LSTM to unify the feature modeling process of single-person actions and interactive actions in videos. GCN-based methods have become the most popular direction for behavior recognition in recent years. Yan et al. first introduced ST GCN into the field of human behavior recognition. They used spatiotemporal graph convolution operations to perform spatiotemporal graph convolution operations on graph topology data constructed from human skeletal key points extracted from videos to complete the recognition, which surpassed the performance of previous networks based on direct RGB image analysis.

[0003] In recent years, with the development of high-precision depth cameras, wearable motion capture systems, and human pose estimation algorithms, we can easily obtain the three-dimensional coordinate information of human joints, and human behavior recognition has also developed rapidly, achieving many results. However, the current recognition accuracy is still not high enough. The main problems are as follows: First, videos contain a lot of redundant information that affects action recognition, such as background noise, and lighting conditions can easily interfere with action recognition. Second, current models cannot adequately express the interaction features of interactive behaviors, so how to obtain the interaction features in interactive behaviors and effectively integrate the skeletal information of two people is also a key research focus. Finally, existing models simply consider the three-dimensional coordinates of key points on the human skeleton and cannot adequately express the interaction features. Summary of the Invention

[0004] The purpose of this invention is to provide a two-person behavior recognition method based on semantic and interaction features, which can effectively express interaction features and integrate the skeletal information of two individuals, thereby improving the accuracy of behavior recognition. To achieve the above objective, this invention provides a two-person behavior recognition method based on semantic and interaction features, the method comprising:

[0005] S1. Obtain the three-dimensional coordinate information of human skeletal joints from the video using the OpenPose algorithm;

[0006] S2. Determine whether it is a two-person action by using skeletal joint information;

[0007] S3, use the skeleton information of the first person... The second person's skeletal information was used To represent, and to embed high-dimensional information through high-dimensional encoding, define the semantic spa of joint types, and to fuse it with skeleton information through high-dimensional encoding and embedding of high-dimensional information.

[0008] S4. Extract the interaction information g_ram between the two people using their skeleton information;

[0009] S5. Construct a joint-level graph convolution module and perform feature fusion by integrating skeletal information that incorporates joint types with dual-person interaction information input;

[0010] S6. Define frame index semantics and fuse it with the features obtained after joint-level graph convolution;

[0011] S7. The constructed frame-level convolutional module is used to achieve behavior recognition and classification.

[0012] Further, step S1 includes: using the OpenPose algorithm to extract the three-dimensional coordinate information of the human skeleton joints in the video to obtain a tensor of size (Bs, 3, N, T), where Bs is the batch size, N is the number of joints, each joint includes 3-dimensional coordinate information, and T is the time frame.

[0013] Further, step S2 specifically involves obtaining the number of key points N in the tensor through step S1, which is used to determine whether the behavior is a two-person behavior.

[0014] Furthermore, the number of key points N=25 represents that the video contains one person, and the number of key points N=50 represents that the video contains two people.

[0015] Further, step S3 includes: the skeletal information of the first person. and the skeleton information of the second person The tensors are all of size (Bs, 3, 25, T), which are transformed into tensors of size (Bs, 64, 25, T) after high-dimensional encoding.

[0016] The semantic SPA definition of the joint type: using a heat vector J k ∈R dj The type of the k-th joint is represented by a dimension of 1, with all other dimensions being 0. The semantic spa dimension of the joint type is (Bs, 25, 25, T). After high-dimensional encoding, the semantic spa is transformed into a tensor of size (Bs, 64, 25, T). Finally, the cat function is used to extract the skeleton information of the first person. The second person's skeletal information The results were obtained by fusing them with joint type semantics respectively. in This represents the features resulting from the fusion of skeletal information and joint semantics of the first person. This represents the features resulting from the fusion of the second person's skeletal information and joint semantics.

[0017] Further, step S4 includes: for a given skeleton information input of two people, calculating interaction feature information, and modeling the edge weights from the i-th joint of the first person to the j-th joint of the second person in the same frame t using similarity in the embedding space as follows:

[0018]

[0019] in This represents the information of the i-th joint in the t-th frame of the first human skeleton. This represents the information of the j-th joint of the t-th frame of the second human skeleton. θ and φ represent two transformation functions, each implemented by a fully connected (FC) layer: θ(X) = W1X + b1, φ(X) = W2X + b2, where W1 and W2 are learnable weight parameters, and b1 and b2 are bias vectors. The interaction feature information is then calculated, and g_ram is a tensor of dimension (Bs, T, 25, 25).

[0020] Further, step S5 includes: first, constructing a joint-level graph convolutional module using a three-layer GCN, setting the number of neurons in the three GCN layers to 128, 256, and 256 respectively; then, using the dual-person skeleton information... and The `cat` function is used for fusion, and the fused skeleton information and two-person interaction information `g_ram` are input into the joint-level graph convolution module for feature fusion.

[0021] Furthermore, the content of one layer of GCN in the joint-level graph convolution module is as follows: First, the input feature X is transposed to obtain a new feature X1; then X1 is multiplied by the input two-person interaction information g, and the product result is transposed to obtain X2; X and X2 are respectively input into a 1*1 convolution and the results are added and fused; finally, the fused result is regularized and then passed through the ReLU activation function to obtain the output, where X and g correspond to the fused skeleton information and the two-person interaction information g_ram, respectively.

[0022] Furthermore, the frame index semantic tem has a dimension of (Bs, T, 25, T). After high-dimensional encoding, the frame index semantic tem is transformed into a tensor of size (Bs, 256, 25, T). The encoded frame index semantic tem is then added and fused with the features obtained after passing through the joint-level graph convolution module.

[0023] The content of one layer of GCN in the joint-level graph convolution module is as follows: First, the input feature x is transposed to obtain a new feature X1; then X1 is multiplied by the input two-person interaction information g, and the product result is transposed to obtain a new X1; x and X1 are respectively input into 1*1 convolution and the results are added and fused; finally, the fused result is regularized and then the output is obtained through the ReLU activation function.

[0024] Further, step S7 includes: constructing a frame-level convolutional module using two CNN layers, setting the number of neurons in the first CNN layer to 256 and the kernel size to 3 in the time dimension, and the number of neurons in the second CNN layer to 512 and the kernel size to 1; inputting the data features obtained in S6 into the frame-level convolutional module to obtain the recognition result; and finally obtaining the classification result through a fully connected layer.

[0025] In a second aspect, the present invention provides a dual-person behavior recognition system based on semantic and interaction features, implementing the method of claims 1-6, comprising:

[0026] The acquisition module is used to obtain the three-dimensional coordinate information of human skeletal joints from videos using the OpenPose algorithm;

[0027] The judgment module is used to determine whether it is a two-person action based on skeletal joint information;

[0028] The high-dimensional fusion module is used to combine the skeletal information of the first person. The second person's skeletal information was used To represent, and to embed high-dimensional information through high-dimensional encoding, define the semantic spa of joint types, and to fuse it with skeleton information through high-dimensional encoding and embedding of high-dimensional information.

[0029] The extraction module is used to extract the interaction information g_ram between the two people based on their skeleton information.

[0030] A fusion module is constructed to build a joint-level graph convolution module and to fuse the skeletal information of joint types with the two-person interaction information input for feature fusion.

[0031] Define a fusion module to define frame index semantics and fuse them with features obtained after joint-level graph convolution;

[0032] The recognition module, through the constructed frame-level convolutional module, is used to achieve behavior recognition and classification.

[0033] The beneficial technical effects of the present invention are at least as follows:

[0034] (1) This invention defines the semantics of joint type and frame index. The semantics of joint type helps to learn the appropriate adjacency matrix in graph convolution, and the semantics of frame index can enhance the representation capability of frames.

[0035] (2) By integrating the skeleton information of the two people well, and being able to express the interactive features in the interactive behavior well, the recognition accuracy of the interactive behavior is improved. Attached Figure Description

[0036] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the process of the dual-person behavior recognition method based on semantic and interaction features of the present invention.

[0038] Figure 2 This is a schematic diagram of the connection relationship of key points in the skeleton of a single person in the dual-person behavior recognition method based on semantic and interaction features of the present invention.

[0039] Figure 3 This is a structural diagram of the high-dimensional coding module of the dual-person behavior recognition method based on semantic and interaction features of the present invention.

[0040] Figure 4 This is a structural diagram of the interaction feature extraction module of the dual-person behavior recognition method based on semantic and interaction features of the present invention.

[0041] Figure 5 This is a human skeleton diagram illustrating the interactive behavior of an embodiment of the present invention. Detailed Implementation

[0042] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0043] Specific implementation method one: See Figure 1 The present invention includes the following steps:

[0044] Step 1: Obtain the 3D coordinate information of human skeletal joints from the video using the OpenPose algorithm. The OpenPose algorithm is used to extract the 3D coordinate information of human skeletal joints from the video, resulting in a tensor of size (Bs, 3, N, T), where Bs is the batch size, N is the number of joints, each joint includes 3D coordinate information, and T is the time frame.

[0045] Step Two: Determine if it's a two-person activity based on skeletal joint information. Each human body includes 25 joints, such as... Figure 2 As shown, the N in the tensor information obtained in step one is used to determine whether the behavior is a two-person behavior. When the number of key points N = 50, it means that there are two people in the video.

[0046] Step 3: Use the skeletal information of the two people separately... To represent, and to embed high-dimensional elements through high-dimensional encoding, such as Figure 3 As shown, a semantic SPA for joint types is defined and embedded into high-dimensional data through high-dimensional encoding, which is then fused with skeleton information. They are all tensors of size (Bs, 3, 25, T), which are transformed into tensors of size (Bs, 64, 25, T) after high-dimensional encoding.

[0047] The definition of joint type semantics in SPA is as follows:

[0048] Using a heat vector J k ∈R dj Let represent the type of the k-th joint (also called type k), where the k-th dimension is 1 and all other dimensions are 0. The joint type semantics SPA has dimensions (Bs, 25, 25, T). After high-dimensional encoding, SPA becomes a tensor of size (Bs, 64, 25, T). Finally, the cat function is used to... The results were obtained by fusing the semantics of joint types separately.

[0049] Step Four: As Figure 4 , 5As shown, the interaction information g_ram of the two people is extracted from their skeleton information. Given the skeleton information of two people as input, the interaction feature information is calculated. For the edge weights from the i-th joint of the first person to the j-th joint of the second person in the same frame t, the similarity in the embedding space is modeled as follows:

[0050]

[0051] in This represents the information of the i-th joint in the t-th frame of the first human skeleton. This represents the information of the j-th joint of the t-th frame of the second human skeleton. θ and φ represent two transformation functions, each implemented by a fully connected (FC) layer: θ(X) = W1X + b1, φ(X) = W2X + b2, where W1 and W2 are learnable weight parameters, and b1 and b2 are bias vectors. Finally, the calculated interaction feature information, g_ram, is a tensor of dimension (Bs, T, 25, 25).

[0052] Step 5: Construct a joint-level graph convolutional module and fuse skeletal information incorporating joint types with two-person interaction information for feature fusion. First, construct the joint-level graph convolutional module using a three-layer GCN, setting the number of neurons in the three GCN layers to 128, 256, and 256 respectively. Then, fuse the two-person skeleton information... The `cat` function is used for fusion. The fused skeleton information and two-person interaction information are input into `g_ram` and then into the joint-level graph convolution module for feature fusion.

[0053] The content of one layer of GCN in the joint-level graph convolution module described in step five is as follows: First, the input feature X is transposed to obtain a new feature X1; then X1 is multiplied by the input two-person interaction information g, and the product result is transposed to obtain X2; X and X2 are respectively input into a 1*1 convolution and the results are added and fused; finally, the fused result is regularized and then the ReLU activation function is used to obtain the output.

[0054] X and g correspond to the fused skeleton information and two-person interaction information g_ram, respectively.

[0055] Step Six: Define and fuse the frame index semantics with the features obtained after joint-level graph convolution. The definition of frame index semantics is similar to that of joint type semantics; the frame index semantic tem has dimensions (Bs, T, 25, T). After high-dimensional encoding, tem becomes a tensor of size (Bs, 256, 25, T). The encoded frame index semantics are then added and fused with the features obtained after joint-level graph convolution.

[0056] Step 7: Implement action recognition and classification using the constructed frame-level convolutional module. The frame-level convolutional module uses two CNN layers. In the time dimension, we set the first CNN layer to have 256 neurons and a kernel size of 3, and the second CNN layer to have 512 neurons and a kernel size of 1. The data features obtained in Step 6 are input into the frame-level convolutional module to obtain the recognition result. Finally, the classification result is obtained after passing through a fully connected layer.

[0057] Specific implementation method two, a dual-person behavior recognition system based on semantic and interaction features includes:

[0058] The acquisition module is used to obtain the three-dimensional coordinate information of human skeletal joints from videos using the OpenPose algorithm;

[0059] The judgment module is used to determine whether it is a two-person action based on skeletal joint information;

[0060] The high-dimensional fusion module is used to combine the skeletal information of the first person. The second person's skeletal information was used To represent, and to embed high-dimensional information through high-dimensional encoding, define the semantic spa of joint types, and to fuse it with skeleton information through high-dimensional encoding and embedding of high-dimensional information.

[0061] The extraction module is used to extract the interaction information g_ram between the two people based on their skeleton information.

[0062] A fusion module is constructed to build a joint-level graph convolution module and to fuse the skeletal information of joint types with the two-person interaction information input for feature fusion.

[0063] Define a fusion module to define frame index semantics and fuse them with features obtained after joint-level graph convolution;

[0064] The recognition module, through the constructed frame-level convolutional module, is used to achieve behavior recognition and classification.

[0065] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for dual-person behavior recognition based on semantic and interaction features, characterized in that, The method includes: S1. Obtain the three-dimensional coordinate information of human skeletal joints from the video using the OpenPose algorithm; S2. Determine whether it is a two-person action by using skeletal joint information; S3, use the skeleton information of the first person... To represent, the skeletal information of the second person is used To represent, and to embed high-dimensional information through high-dimensional encoding, define the semantic spa of joint types, and to fuse it with skeleton information through high-dimensional encoding and embedding of high-dimensional information. S4. Extract the interaction information g_ram between the two people using their skeleton information; S5. Construct a joint-level graph convolution module and perform feature fusion by integrating skeletal information that incorporates joint types with dual-person interaction information input; S6. Define frame index semantics and fuse it with the features obtained after joint-level graph convolution; S7. The constructed frame-level convolutional module is used to achieve behavior recognition and classification; Step S3 includes: the skeletal information of the first person. and the skeleton information of the second person The tensors are all of size (Bs, 3, 25, T), which are transformed into tensors of size (Bs, 64, 25, T) after high-dimensional encoding. The semantic SPA definition of the joint type: using a heat vector The type of the k-th joint is represented by a dimension of 1, with all other dimensions being 0. The semantic spa dimension of the joint type is (Bs, 25, 25, T). After high-dimensional encoding, the semantic spa is transformed into a tensor of size (Bs, 64, 25, T). Finally, the cat function is used to extract the skeleton information of the first person. The second person's skeletal information The results were obtained by fusing them with joint type semantics respectively. , ,in This represents the features resulting from the fusion of skeletal information and joint semantics of the first person. This represents the skeletal information and joint semantics of the second person, after fusion. Step S5 includes: first, constructing a joint-level graph convolutional module, using a three-layer GCN, setting the number of neurons in the three GCN layers to 128, 256, and 256 respectively; then, using the dual-person skeleton information... and The cat function is used for fusion, and the fused skeleton information and two-person interaction information g_ram are input into the joint-level graph convolution module for feature fusion. The content of one layer of GCN in the joint-level graph convolution module is as follows: First, the input features are processed... Transpose to obtain new features ; then Multiply the input two-person interaction information g by a matrix, and then transpose the product to obtain the result. ;Will and The inputs are each a 1x1 convolution, and the results are summed and fused. Finally, the fused result is regularized and then passed through a ReLU activation function to obtain the output. g and g correspond to the fused skeleton information and two-person interaction information g_ram, respectively.

2. The method for dual-person behavior recognition based on semantic and interaction features according to claim 1, characterized in that, Step S1 includes: using the OpenPose algorithm to extract the three-dimensional coordinate information of human skeletal joints in the video to obtain a tensor of size (Bs, 3, N, T), where Bs is the batch size, N is the number of joints, each joint includes 3-dimensional coordinate information, and T is the time frame.

3. The method for dual-person behavior recognition based on semantic and interaction features according to claim 2, characterized in that, Step S2 specifically involves obtaining the number of key points N in the tensor through step S1, which is used to determine whether the behavior is a two-person behavior.

4. The method for dual-person behavior recognition based on semantic and interaction features according to claim 3, characterized in that, The number of key points N=25 means that the video contains one person, and the number of key points N=50 means that the video contains two people.

5. The method for dual-person behavior recognition based on semantic and interaction features according to claim 4, characterized in that, Step S4 includes: for a given skeleton information input of two people, calculating interaction feature information, and modeling the edge weights from the i-th joint of the first person to the j-th joint of the second person in the same frame t using similarity in the embedding space as follows: in This represents the information of the i-th joint in the t-th frame of the first human skeleton. This represents the information of the j-th joint point in the t-th frame of the second human skeleton; θ and φ represent two transformation functions, each implemented by a fully connected (FC) layer. , ;in and These are learnable weight parameters. and The bias vector is used to calculate the interaction feature information. g_ram is a tensor with dimensions (Bs, T, 25, 25).

6. The method for dual-person behavior recognition based on semantic and interaction features according to claim 5, characterized in that, The frame index semantic tem has dimensions (Bs, T, 25, T). After high-dimensional encoding, the frame index semantic tem is transformed into a tensor of size (Bs, 256, 25, T). The encoded frame index semantic tem is then added and fused with the features obtained after passing through the joint-level graph convolution module.

7. The method for dual-person behavior recognition based on semantic and interaction features according to claim 1, characterized in that, Step S7 includes: constructing a frame-level convolutional module using two CNN layers, setting the number of neurons in the first CNN layer to 256 and the kernel size to 3 in the time dimension, and the number of neurons in the second CNN layer to 512 and the kernel size to 1; inputting the data features obtained in S6 into the frame-level convolutional module to obtain the recognition result; and finally obtaining the classification result through a fully connected layer.

8. A dual-person behavior recognition system based on semantic and interaction features, implementing the method of any one of claims 1-5, characterized in that, include: The acquisition module is used to obtain the three-dimensional coordinate information of human skeletal joints from videos using the OpenPose algorithm; The judgment module is used to determine whether it is a two-person action based on skeletal joint information; The high-dimensional fusion module is used to combine the skeletal information of the first person. The second person's skeletal information is used To represent, and to embed high-dimensional information through high-dimensional encoding, define the semantic spa of joint types, and to fuse it with skeleton information through high-dimensional encoding and embedding of high-dimensional information. The extraction module is used to extract the interaction information g_ram between the two people based on their skeleton information. A fusion module is constructed to build a joint-level graph convolution module and to fuse the skeletal information of joint types with the two-person interaction information input for feature fusion. Define a fusion module to define frame index semantics and fuse them with features obtained after joint-level graph convolution; The recognition module, through the constructed frame-level convolutional module, is used to achieve behavior recognition and classification.

Images