Human behavior recognition method and device based on fine-grained topological logic modeling

Abstract

The application discloses a human behavior recognition method and device based on fine-grained topological logic modeling, which comprises the following steps: performing human joint position estimation on a video frame by frame, and constructing a space-time graph by connecting the same joints between continuous frames according to the natural joint connection relationship of the human body inside each frame on the joint position data; normalizing the position of the joint in each frame to two-dimensional coordinates by the space-time graph to obtain skeleton data of the graph structure; inputting the skeleton data into a pre-trained graph convolutional neural network based on fine-grained topological logic modeling; obtaining the probability of the input belonging to each class by performing Softmax; and selecting the class corresponding to the component with the maximum value. By using the technical scheme of the application, the influence of intra-class differences can be effectively resisted, human behavior classification can be accurately performed, high-order time sequence data can be fully utilized, the features in the backbone network can be assisted, and the generalization ability and accuracy can be enhanced.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method and apparatus for human behavior recognition based on fine-grained topological logic modeling. Background Technology

[0002] With the advancement of intelligent robot technology, the demands on robots to "understand" human behavior are increasing. In the field of intelligent robotics, robots need to process and react to human behavior in real time. Traditional feature-based human behavior recognition methods are not only computationally complex but also lack guaranteed accuracy, thus failing to meet current requirements. With the application of deep learning to computer vision, deep learning-based human behavior recognition has emerged. These methods extract features from data in an end-to-end manner and classify actions. Currently, human behavior recognition has become an important research direction in the field of computer vision.

[0003] Deep learning-based human behavior recognition methods can be broadly categorized into two types: video-based and graph-based skeletal data-based. Video-based methods use convolutional neural networks (CNNs) as the backbone, layering various modules to extract different features to enhance the overall feature discrimination capability of the model. Since video data is highly susceptible to background noise and consumes significant storage space, effectively enhancing the model's robustness to background noise and efficiently processing video data has become a recent research hotspot. Graph-based skeletal data-based methods mostly process skeletal information by establishing graph convolutional neural networks (GNNs). However, due to limitations such as weak modeling ability for relationships between non-adjacent nodes, susceptibility to "oversmoothing," and insufficient utilization of high-order skeletal data, GNNs still hold significant research value.

[0004] Compared to video data, graph-structured skeletal data requires less memory, is less affected by background noise, and can effectively protect the privacy of the identified individual, making it more suitable for intelligent robots to recognize human behavior.

[0005] In the process of developing this disclosure, the inventors discovered the following problems with the prior art:

[0006] 1. Significant intra-class variability exists in human behavior recognition problems: different people will perform the same type of action in different ways. For example, when waving, some people wave their left hand, some wave their right hand, some wave for a long time, and some wave for a short time. Such intra-class variability can "confuse" the model to some extent, thereby impairing recognition accuracy.

[0007] 2. Current methods do not fully utilize high-order temporal data such as acceleration and jerk, which reflect higher-order motion trends in the human body. Fully utilizing high-order temporal data can effectively improve the accuracy of motion recognition.

[0008] 3. The semantic information contained in different channels of the input data is different. Using the same topological logic to aggregate this information will lead to insufficient utilization of this channel-specific data. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a method and apparatus for human behavior recognition based on fine-grained topological logic modeling.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A human behavior recognition method based on fine-grained topological logic modeling includes:

[0012] Step 1: Pre-train a graph convolutional neural network based on fine-grained topological logic modeling;

[0013] Step 2: Estimate the positions of human joints frame by frame in the video, and within each frame, construct a spatiotemporal graph by connecting the same joints between consecutive frames based on the joint position data, according to the natural joint connectivity of the human body.

[0014] Step 3: Normalize the positions of the joints in each frame of the spatiotemporal graph into two-dimensional coordinates to obtain the skeletal data of the graph structure; input it into a pre-trained graph convolutional neural network based on fine-grained topological logic modeling.

[0015] Step 4: Perform Softmax on the output of the graph convolutional neural network based on fine-grained topological logic modeling to obtain the probability that the input belongs to each class;

[0016] Step 5: Select the class corresponding to the component with the largest value.

[0017] Preferably, the graph convolutional neural network based on fine-grained topological logic modeling consists of 10 GCN_TCN units and a global average pooling. Each GCN_TCN unit has a three-branch structure: the first branch serves as a skip link; the second branch first groups the skeletal data of the graph structure by channel, and in each group, it uses a fine-tuned attention mechanism and prior topological logic to compute the temporal fine-grained topological logic and applies the topological logic to perform graph convolution followed by temporal convolution; the last branch uses a high-order temporal data processing unit to dynamically infer and model high-order temporal data, and then the data from the three branches are summed to obtain the output.

[0018] Preferably, in step four, if the total number of classifications is j, the output of the convolutional neural network based on fine-grained topological logic modeling is a vector Z of shape 1×j. The Softmax function is then used to process this vector Z to obtain the probability of the sample in each class, as shown in the following formula:

[0019]

[0020] Among them, z i It is the i-th component of Z.

[0021] The present invention also provides a human behavior recognition device based on fine-grained topological logic modeling, comprising:

[0022] The first processing module is used for pre-training a graph convolutional neural network based on fine-grained topological logic modeling;

[0023] The second processing module is used to estimate the human joint position of the video frame by frame, and to construct a spatiotemporal graph within each frame based on the joint position data, according to the natural joint connectivity of the human body, by connecting the same joints between consecutive frames.

[0024] The third processing module is used to normalize the positions of the joints in each frame of the spatiotemporal graph into two-dimensional coordinates to obtain the skeletal data of the graph structure; and input it into a pre-trained graph convolutional neural network based on fine-grained topological logic modeling.

[0025] The fourth processing module is used to perform Softmax on the output of the graph convolutional neural network based on fine-grained topological logic modeling to obtain the probability that the input belongs to each class;

[0026] The fifth processing module is used to select the class corresponding to the component with the largest value.

[0027] Preferably, the graph convolutional neural network based on fine-grained topological logic modeling consists of 10 GCN_TCN units and a global average pooling. Each GCN_TCN unit has a three-branch structure: the first branch serves as a skip link; the second branch first groups the skeletal data of the graph structure by channel, and in each group, it uses a fine-tuned attention mechanism and prior topological logic to compute the temporal fine-grained topological logic and applies the topological logic to perform graph convolution followed by temporal convolution; the last branch uses a high-order temporal data processing unit to dynamically infer and model high-order temporal data, and then the data from the three branches are summed to obtain the output.

[0028] Preferably, if the total number of classifications is j, the output of the convolutional neural network based on fine-grained topological logic modeling is a 1×j dimensional vector Z. The Softmax function is then used to process this vector Z to obtain the probability of the sample in each class, as shown in the following formula:

[0029]

[0030] Among them, z i It is the i-th component of Z.

[0031] This invention estimates the positions of human joints frame-by-frame in video. Within each frame, based on this joint position data, a spatiotemporal graph is constructed by connecting the same joints between consecutive frames according to the natural connectivity of human joints. The spatiotemporal graph normalizes the positions of joints in each frame into two-dimensional coordinates to obtain graph-structured skeletal data. This data is then input into a pre-trained graph convolutional neural network based on fine-grained topological logic modeling. Softmax is applied to obtain the probability that the input belongs to each class. The class corresponding to the component with the largest value is selected. Using the technical solution of this invention, the influence of intra-class variance can be effectively resisted, and accurate human behavior classification can be performed. At the same time, it makes full use of high-order temporal data to provide assistance to the features in the backbone network, enhancing generalization ability and accuracy. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0033] Figure 1 The flowchart shows a human behavior recognition method based on fine-grained topological logic modeling.

[0034] Figure 2 Example of a spatiotemporal graph to be constructed;

[0035] Figure 3 The overall structure of a graph convolutional neural network based on fine-grained topological logic modeling and its main components;

[0036] Figure 4 The structure for temporal fine-grained graph convolution. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Example 1:

[0040] like Figure 1 As shown, this embodiment of the invention provides a human behavior recognition method based on fine-grained topological logic modeling. To avoid processing unnecessary parameters such as shooting angle and background lighting, a method is adopted to extract the positions of human joints from the input video frames in real time and encode them into graph-structured skeletal data. Furthermore, since human movements are limited in scenarios where intelligent robots interact with humans, a fully supervised approach is used to train and test the model to achieve human behavior recognition, including the following steps:

[0041] Step 1: Pre-train a fine-grained topological logic modeling graph convolutional neural network on a large dataset;

[0042] Step 2: Estimate the positions of human joints frame by frame in the video, and within each frame, construct a spatiotemporal graph by connecting the same joints between consecutive frames based on the joint position data, according to the natural joint connectivity of the human body.

[0043] Step 3: Normalize the positions of the joints in each frame of the spatiotemporal graph into two-dimensional coordinates to obtain the skeletal data of the graph structure; input it into a pre-trained graph convolutional neural network based on fine-grained topological logic modeling.

[0044] Step 4: Perform Softmax on the output of the graph convolutional neural network based on fine-grained topological logic modeling to obtain the probability that the input belongs to each class;

[0045] Step 5: Select the class corresponding to the component with the largest value.

[0046] As one embodiment of the present invention, in step one, a fine-grained topological logic modeling graph convolutional neural network is trained in a fully supervised manner on a large dataset, and the model that performs best on the validation set is selected for subsequent action recognition.

[0047] In one embodiment of the present invention, each node in the graph in step two corresponds to an actual human body joint, and each spatial edge represents the spatial connectivity between two nodes. Identical joints between adjacent frames are connected by temporal edges. This representation makes the skeletal data of the graph structure occupy less memory and is computationally efficient. The spatiotemporal graph encoding method in step one can effectively utilize the naturally existing spatiotemporal relationships in actions. Figure 2 This is a typical example of a spacetime graph.

[0048] As one embodiment of the present invention, in step three, if there are m frames in the spatiotemporal graph, and k nodes in each frame, and the position of each node is represented by two-dimensional coordinates, then the final skeletal data of the graph structure is a 2×m×k dimensional vector.

[0049] The graph convolutional neural network based on fine-grained topological logic modeling consists of 10 GCN_TCN units and a global average pooling, such as... Figure 3 As shown in (a), each GCN_TCN unit has a three-branch structure: the first branch acts as a skip link; the second branch first groups the skeletal data of the graph structure by channel, and in each group, it uses a fine-tuned attention mechanism and prior topological logic to compute fine-grained temporal topological logic, and then applies this topological logic to perform graph convolution followed by temporal convolution; the last branch uses a higher-order temporal data processing unit for dynamic inference and modeling of higher-order temporal data. The data from the three branches are then summed to obtain the output. The structure diagram of the GCN_TCN unit is shown below. Figure 3 As shown in (b), (c), and (d), the second and third branches will be described in detail below.

[0050] For the second branch, since the semantic information contained in different channels of the input data is different, using the same topological logic to aggregate this information will lead to insufficient utilization of these channel-specific data. Therefore, the data input to GCN_TCN needs to be evenly grouped along the channels, and a topological logic is assigned to each group before graph convolution is performed to ensure full utilization of these channel-specific features. The grouping process is shown in the following formula:

[0051]

[0052] Where R represents the real number field, T is the number of frames, C is the number of channels, V is the number of nodes, and n is the number of groups along the channels.

[0053] After obtaining the above groupings, perform temporal fine-grained convolution within each group. Temporal fine-grained convolution consists of two steps: temporal fine-grained topological logic solving and graph convolution.

[0054] For solving temporally fine-grained topological logic: applying an attention mechanism frame-by-frame to the input data allows for the modeling of temporally fine-grained topological logic A. temporal The standard formula for attention is as follows:

[0055]

[0056] The formulas for solving Q, K, and V are as follows:

[0057] Q,K,V=XW

[0058] Where X is the input feature and W is the linear transformation matrix, usually implemented using one-dimensional convolution.

[0059] However, directly using conventional attention mechanisms to model topological logic with fine temporal granularity leads to a series of problems:

[0060] 1. The initial topological logic only takes values ​​of 0 and 1. The modeled topological logic with fine-grained time has a value range between 0 and 1, which makes it difficult to express negatively correlated joint relationships.

[0061] 2. The normalization operation in the attention mechanism further increases the computational burden.

[0062] To address the aforementioned issues, some modifications were made to the original attention mechanism: the Tanh activation function was used instead of Softmax, and the normalization operation was removed, as shown in the following formula:

[0063] A temporal =Tanh(QK) T )

[0064] Get A temporal Then, a weighted sum is taken between this and the original topological logic A to obtain the final time-fine-grained topological logic. The formula is as follows:

[0065]

[0066] Where α is A temporal The weight.

[0067] get Then, the topological logic is used to perform graph convolution operations, as shown in the formula:

[0068]

[0069] Here, δ refers to the ReLU activation function.

[0070] The above-described process of fine-grained temporal convolution can be achieved by... Figure 4 As shown.

[0071] After performing temporal fine-grained convolution, the output is subjected to temporal convolution. Temporal convolution is typically implemented using a 1×3 one-dimensional convolution, as shown in the following formula:

[0072] X temporal_conv =Conv 1×3 (X out )

[0073] After the temporal convolution is completed, the operations in the second branch are finished. The third branch, namely the higher-order temporal data processing unit, will be introduced below.

[0074] By performing different degrees of differencing on the input data along the time dimension, time series data of corresponding orders can be obtained, as shown in the following formula:

[0075] X temporal_high_order =X[:,:t+1,:]-X[:,:t,:]

[0076] Where, X∈R C×T×V After obtaining the high-order time-series data, it is processed using a 1×3 one-dimensional convolution, then activated and normalized using the ReLU function, as shown in the following formula:

[0077] X temporal_high_order_out =BN(Relu(Conv) 1×3 (X temporal_high_order )))

[0078] Get the output X temporal_high_order_out After that, the operation of the third branch ends, and the process is as follows: Figure 3 As shown in (d).

[0079] After processing through 10 GCN_TCN units and average pooling, the output of step three is obtained.

[0080] In one embodiment of the present invention, in step four, if the total number of classifications is j, the output of the convolutional neural network based on fine-grained topological logic modeling graph is a vector Z of shape 1×j. This vector Z needs to be processed using the Softmax function to obtain the probability of the sample in each class, as shown in the following formula:

[0081]

[0082] Among them, z i It is the i-th component of Z.

[0083] As one embodiment of the present invention, in step five, for each z i Select the z with the largest value i The corresponding class is the classification result of this data.

[0084] Example 2:

[0085] This invention also provides a human behavior recognition device based on fine-grained topological logic modeling, comprising:

[0086] The first processing module is used for pre-training a graph convolutional neural network based on fine-grained topological logic modeling;

[0087] The second processing module is used to estimate the human joint position of the video frame by frame, and to construct a spatiotemporal graph within each frame based on the joint position data, according to the natural joint connectivity of the human body, by connecting the same joints between consecutive frames.

[0088] The third processing module is used to normalize the positions of the joints in each frame of the spatiotemporal graph into two-dimensional coordinates to obtain the skeletal data of the graph structure; and input it into a pre-trained graph convolutional neural network based on fine-grained topological logic modeling.

[0089] The fourth processing module is used to perform Softmax on the output of the graph convolutional neural network based on fine-grained topological logic modeling to obtain the probability that the input belongs to each class;

[0090] The fifth processing module is used to select the class corresponding to the component with the largest value.

[0091] As one embodiment of the present invention, the graph convolutional neural network based on fine-grained topological logic modeling consists of 10 GCN_TCN units and a global average pooling. Each GCN_TCN unit has a three-branch structure: the first branch serves as a skip link; the second branch first groups the skeletal data of the graph structure by channel, and in each group, it uses a fine-tuned attention mechanism and prior topological logic to calculate the temporal fine-grained topological logic and applies the topological logic to perform graph convolution followed by temporal convolution; the last branch uses a high-order temporal data processing unit to dynamically infer and model high-order temporal data, and then the data from the three branches are summed to obtain the output.

[0092] As one embodiment of the present invention, if the total number of classifications is j, the output of the convolutional neural network based on fine-grained topological logic modeling is a vector Z of shape 1×j. The Softmax function is used to process the vector Z to obtain the probability of the sample in each class, as shown in the following formula:

[0093]

[0094] Among them, z i It is the i-th component of Z.

[0095] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A human behavior recognition method based on fine-grained topological logic modeling, characterized in that, include: Step 1: Pre-train a graph convolutional neural network based on fine-grained topological logic modeling; Step 2: Estimate the positions of human joints frame by frame in the video, and within each frame, construct a spatiotemporal graph by connecting the same joints between consecutive frames based on the joint position data, according to the natural joint connectivity of the human body. Step 3: Normalize the positions of the joints in each frame of the spatiotemporal graph into two-dimensional coordinates to obtain the skeletal data of the graph structure; It is then fed into a pre-trained graph convolutional neural network based on fine-grained topological logic modeling; Step 4: Perform Softmax on the output of the graph convolutional neural network based on fine-grained topological logic modeling to obtain the probability that the input belongs to each class; Step 5: Select the class corresponding to the component with the largest value; The graph convolutional neural network based on fine-grained topological logic modeling consists of 10 GCN_TCN units and a global average pooling. Each GCN_TCN unit has a three-branch structure: the first branch serves as a skip link; the second branch first groups the skeletal data of the graph structure by channel, and then uses a fine-tuned attention mechanism and prior topological logic to compute the temporal fine-grained topological logic in each group, and applies this topological logic to perform graph convolution followed by temporal convolution; the last branch uses a high-order temporal data processing unit to dynamically infer and model high-order temporal data, and then sums the data from the three branches to obtain the output.

2. The human behavior recognition method based on fine-grained topological logic modeling as described in claim 1, characterized in that, In step four, if the total number of classifications is j, the output of the convolutional neural network based on fine-grained topological logic modeling is a 1×j dimensional vector Z. The Softmax function is used to process this vector Z to obtain the probability of a sample in each class, as shown in the following formula: ； in, It is the i-th component of Z.

3. A human behavior recognition device based on fine-grained topological logic modeling, characterized in that, include: The first processing module is used for pre-training a graph convolutional neural network based on fine-grained topological logic modeling; The second processing module is used to estimate the human joint position of the video frame by frame, and to construct a spatiotemporal graph within each frame based on the joint position data, according to the natural joint connectivity of the human body, by connecting the same joints between consecutive frames. The third processing module is used to normalize the positions of the joints in each frame of the spatiotemporal graph into two-dimensional coordinates to obtain the skeletal data of the graph structure. It is then fed into a pre-trained graph convolutional neural network based on fine-grained topological logic modeling; The fourth processing module is used to perform Softmax on the output of the graph convolutional neural network based on fine-grained topological logic modeling to obtain the probability that the input belongs to each class; The fifth processing module is used to select the class corresponding to the component with the largest value; The graph convolutional neural network based on fine-grained topological logic modeling consists of 10 GCN_TCN units and a global average pooling. Each GCN_TCN unit has a three-branch structure: the first branch serves as a skip link; the second branch first groups the skeletal data of the graph structure by channel, and then uses a fine-tuned attention mechanism and prior topological logic to compute the temporal fine-grained topological logic in each group, and applies this topological logic to perform graph convolution followed by temporal convolution; the last branch uses a high-order temporal data processing unit to dynamically infer and model high-order temporal data, and then sums the data from the three branches to obtain the output.

4. The human behavior recognition device based on fine-grained topological logic modeling as described in claim 3, characterized in that, If the total number of categories is j, the output of the convolutional neural network based on fine-grained topological logic modeling is a vector Z of shape 1×j. The Softmax function is used to process this vector Z to obtain the probability of a sample in each class, as shown in the following formula: ； in, It is the i-th component of Z.

Images