Human body behavior recognition method and device based on improved space-time diagram convolutional neural network

Abstract

The invention belongs to the field of human behavior recognition and computer vision, and provides a human behavior recognition method and device based on an improved space-time diagram convolutional neural network. The method relates to a graph neural network modeling technology and a spatial-temporal feature extraction technology. The method comprises the following steps: firstly, acquiring an original human body behavior video through a video acquisition system, and converting the original human body behavior video into a fixed-length image frame sequence; extracting human skeleton key point coordinates and confidence based on a YOLOv8-pose algorithm, and constructing a time sequence skeleton graph structure; screening skeleton targets with relatively high confidence coefficients to serve as skeleton feature data; the method comprises the following steps of: constructing a skeleton graph structure and a joint feature matrix on a space structure, automatically dividing the graph structure by utilizing a Laplacian feature mapping method, and extracting local and global features; in a time domain, a multi-scale cavity convolution method is adopted to extract human body action change trends with different time lengths; and finally, space and time dimension information is fused, and recognition and classification of behavior categories are completed. The human body behavior recognition method based on the improved space-time diagram convolutional neural network has the advantages of being clear in structure, easy to deploy, high in parameter automatic optimization capacity, suitable for various complex human body action scenes and the like, and meets the dual requirements for recognition precision and real-time performance in practical application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision recognition, in particular to a human behavior recognition method and device based on an improved space-time graph convolutional neural network. BACKGROUND

[0002] With the continuous development of computer vision technology, behavior recognition has become an important research direction in many fields such as intelligent monitoring, health care, human-computer interaction, etc. Behavior recognition refers to capturing and analyzing the actions of a person in an image or video to determine the specific behavior or activity they are performing. The main purpose of this technology is to automatically recognize the behavior performed by a person in a video or video sequence and implement a "W4 system" (i.e., determine the behavior subject, time of occurrence, location, and behavior content). Behavior recognition technology can be applied in various fields, such as video surveillance, human-computer interaction, motion analysis, health monitoring, etc.

[0003] In recent years, researchers have begun to focus on behavior recognition methods based on human skeleton data. Human skeleton point representation extracts the joint positions of a person as two-dimensional coordinate points, forming a stable and structured representation method. Compared with raw image data, it has less data redundancy, stronger motion structure expression ability, and higher robustness. Thanks to the development of pose estimation algorithms, the emergence of high-precision lightweight pose estimation networks such as OpenPose and YOLOv8-pose makes it possible to quickly and accurately extract human key points from videos, laying the foundation for skeleton point behavior recognition.

[0004] In the behavior recognition task, human skeletons can be naturally modeled as a graph structure, with nodes representing joints and edges representing the connection between joints. Therefore, graph convolutional networks (GCN) are widely used in skeleton behavior recognition. The core idea is to propagate features on the graph structure to extract semantic information in the spatial structure. The ST-GCN (Spatial Temporal Graph Convolutional Network) model is a typical graph structure-based spatio-temporal behavior recognition method that jointly models the spatial structure and temporal dynamics of skeleton data by constructing a space-time graph. However, the ST-GCN and its subsequent improved models still have some problems: first, the importance of each joint node in the original graph structure is not distinguished, and the local features of the key regions of the action cannot be effectively mined; second, the fixed convolution kernel is used for time convolution, and the receptive field is limited, making it difficult to adapt to behavior changes of different time scales.

[0005] In terms of spatial domain modeling, human joints have different importance in different actions, and local regions (such as hands, feet, and trunks) may have stronger discriminative ability for certain behaviors. Traditional GCN models use fixed graph topology, ignoring these differences. The method using Laplace eigenmap can cluster and reduce the dimension of the skeleton graph, divide the graph into multiple non-overlapping subgraphs, realize local modeling of key regions, and thus improve the ability to capture fine-grained action features.

[0006] In terms of time modeling, different human behaviors may have obvious rhythm changes in the time dimension, for example, waving hands, jumping, and slow running have different durations and dynamic ranges. Therefore, using fixed-scale convolution kernels to extract time features cannot meet the adaptation to different action patterns. To this end, dilated convolution is used to expand the time convolution receptive field, which obtains more rich temporal context information while only increasing a small amount of computation, and becomes an effective means to enhance the time modeling capability.

[0007] In summary, the current behavior recognition method still has limitations in spatial structure modeling and time feature extraction, especially in dealing with complex dynamic behaviors and multi-agent scenes. In order to solve the above problems, a new method combining graph structure division and multi-scale time modeling is needed to improve the accuracy and robustness of skeleton behavior recognition. SUMMARY

[0008] In view of the existing technical problems, the application provides a behavior recognition method based on an improved graph convolutional neural network, which uses a human skeleton blocking method to extract overall and partial spatial features, and uses different convolution kernels to extract features of different time scales.

[0009] In order to solve the above problems, the application adopts the following technical scheme: a human behavior recognition method based on an improved spatio-temporal graph convolutional neural network, which mainly consists of the following steps:

[0010] Step 1: Split the photographed human behavior video into picture frames, process the video into 200 frames, if insufficient, fill in the frames, if more, crop.

[0011] Step 2: Use YOLOv8-pose algorithm to extract human skeleton point coordinates and confidence.

[0012] Step 3: Select one or two skeletons with high average confidence in the video as the skeleton point dataset.

[0013] Step 4: Use graph structure and joint feature matrix to describe human skeleton features.

[0014] Step 5: Taking the human body graph structure and joint feature matrix as the input of the spatio-temporal graph convolution, in the spatial domain, the graph is divided into non-overlapping parts by using the Laplacian eigenmap dimension reduction method, and skeleton-level feature extraction is performed on the reduced graph, and part-level feature extraction is performed on the divided subgraph part, and then the features are aligned and fused. In the time domain, multi-scale feature extraction is performed in the time dimension by using a convolution kernel with different hole rates, thereby effectively expanding the time receptive field and enhancing the modeling capability of long and short time action patterns.

[0015] Step 6: The processed features are subjected to softmax classification to obtain the output result.

[0016] The indicators of the method mainly include two accuracy Top1-ACC and Top5-ACC, wherein Top1-ACC refers to the probability that the highest probability predicted category is the correct category; and Top5-ACC refers to the probability that the top five categories of the predicted probability contain the correct category. The formulas of the above indicators are as follows:

[0017]

[0018]

[0019] For the classification result, when the TOP1-accuracy and the TOP5-accuracy both reach the maximum value, the recognition effect of the model is optimal. In the present application, the data samples mainly come from the collected human behavior videos, and the key points are extracted by the YOLOv8-pose algorithm to form a skeleton sequence. These skeleton key points have a highly structured spatial topological relationship and obvious time dynamic characteristics, and have good distinguishability, which are suitable for recognition and classification between different human behaviors.

[0020] The behavior data collection process is as follows: first, after the original behavior video is subjected to frame decomposition and length uniformization processing, the coordinates and confidence information of the human skeleton points are extracted by the YOLOv8-pose pose estimation model; then, the person or the two persons with the highest confidence are selected as the analysis objects to construct high-quality skeleton graph sequence data. Next, the graph structure modeling and joint feature matrix are combined to form an information graph input containing spatial topology and node features. The skeleton graph is divided by using the Laplacian eigenmap to realize the structure modeling of the local subgraph; at the same time, a multi-scale hole convolution structure is introduced in the time domain to effectively expand the receptive field range and improve the modeling capability of the model to the time sequence dynamics. Finally, the behavior category output is completed by the softmax classifier.

[0021] Compared with the traditional ST-GCN or single structure model, the human behavior recognition method based on the improved space-time graph convolutional neural network has significant improvement in accuracy and stability, and also has good real-time performance and adaptability. In the presence of noise or multi-person behavior interference, the model can still realize effective key target focusing and behavior recognition through confidence filtering and structure division. At the same time, the method has high automation capability, and the whole processing process does not need human intervention, and can be directly embedded into the video analysis system in the actual application scene, meeting the technical needs of intelligent behavior recognition. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the overall implementation flowchart of the present application.

[0023] Figure 2 is the model structure diagram of the present application.

[0024] Figure 3 is the sample diagram of the dimension reduction algorithm of the present application.

[0025] Figure 4 is the human skeleton graph and division schematic diagram of the present application.

[0026] Figure 5 is the computer structure schematic diagram of the present application. DETAILED DESCRIPTION

[0027] The classification method of the present application will be described in detail below in combination with the drawings and specific examples:

[0028] Figure 1 The implementation process of the human behavior recognition method based on the improved space-time graph convolutional neural network is described in the figure, and the behavior recognition starts from the data set input, and a series of processing is performed until the clustering result is output. The implementation principle of the algorithm is as follows:

[0029] Data set input. The skeletal key point feature quantity in the human behavior video is extracted, as shown in Figure 1 The original behavior data is derived from the monitoring video or the public human behavior data set, and according to the system process shown in Figure 2 , each behavior video is uniformly processed into a fixed length frame sequence (the present application sets 150 frames), if the video frame number is insufficient, the frame is filled up, if it is exceeded, the core action part is randomly cropped. Subsequently, the YOLOv8-pose pose estimation algorithm is used to extract the human key points of each frame image, and the two-dimensional coordinates and confidence values of each joint are obtained.

[0030] To ensure the quality and consistency of the skeleton data, one or two people with higher average confidence in the video are selected as the analysis object to construct a stable and reliable skeleton point sequence. Then, according to the skeleton connection structure, the human body graph structure information is constructed, and the joint feature matrix is constructed to describe the human skeleton features.

[0031] Given a dataset with a total number of samples N, where each sample contains 18-dimensional skeletal joint feature quantities, the dataset is constructed as a 5D tensor:

[0032]

[0033] where N represents the total number of samples, C represents the number of channels, in the data set is the skeleton point coordinates X, Y coordinates and confidence Conf, T represents the time sequence, V represents the number of skeletal features, the data set is the 18 feature points extracted by YOLOv8pose, M is the effective number of people in this sample, usually 1 or 2.

[0034] The connection relationship between the joints is defined by the skeleton connection structure:

[0035]

[0036] Adjacency matrix is defined as:

[0037]

[0038] After constructing the graph and feature matrix through preprocessing, the preprocessed data is divided. First, construct the normalized Laplacian matrix:

[0039] L sym = I-D -1 / 2 AD -1 / 2 (6)

[0040] where is the degree matrix,

[0041] Solve the first K eigenvectors of L sym :

[0042] L sym u i = λ i u i (7)

[0043] Combine them into:

[0044]

[0045] Then do KMeans clustering for each node:

[0046] labeli = KMeans(U i ) (9)

[0047] Finally, the region number of each node is obtained.

[0048] After the completion of the division and clustering of the skeleton graph structure, for each subgraph region and its corresponding pooling graph representation, feature extraction in the spatial domain is also required. For the clustered subgraph structure, graph convolution operation is used to aggregate and learn the node information within the structure, extracting the geometric relationship of local structure and the action correlation between nodes. For the pooled global skeleton graph, skeleton-level graph convolution is used to extract the overall action form features, enhancing the modeling ability of global spatial dependency.

[0049] After the completion of the spatial convolution, the feature representations of each subgraph and the feature representations of the global pooling graph are stacked in time sequence to form a feature tensor with time sequence structure, denoted as:

[0050]

[0051] where T represents the time frame length, C represents the number of channels extracted per frame, and V represents the number of nodes. This tensor integrates multi-level feature information of subgraphs and skeleton graphs in the spatial domain.

[0052] In order to capture the changing trend of different action durations and rhythms, the invention introduces a multi-scale dilated temporal convolution module (Dilated Temporal Convolution, DTC) in the time dimension. By setting convolution kernels with different dilation rates, the time receptive field is expanded, and the model's adaptability to long and short actions is enhanced. Applying one-dimensional dilated convolution operation to the feature tensor F can be represented as:

[0053]

[0054] where L represents the size of the convolution kernel, w k (l) represents the lth weight of the Kth convolution kernel, d k represents the dilation rate. An output tensor

[0055] Then, all the scale outputs are fused:

[0056]

[0057] After parallel execution of multiple different convolution operations, splicing and fusion are performed to obtain enhanced time sequence features F'.

[0058] Finally, F' is sent to the fully connected layer and the Softmax classification layer to complete the behavior category prediction of the input skeleton sequence:

[0059]

[0060] Among them, W and b are the weight and bias of the classification layer respectively, To predict the probability distribution of the category. This structure fully combines the spatial local structure, global skeleton information and temporal dynamic features to achieve accurate recognition and classification of complex behaviors.

[0061] In this embodiment, a computing device and a camera device are provided. The computing device may be a server, a terminal or other computer system, such as Figure 5 As shown. The device generally includes but is not limited to: a processor, a storage module, an input / output interface and a communication module. The processor can be connected to components such as an input / output control system and a bus to implement the data processing function described in the present invention. The storage module may store a volatile or non-volatile storage medium, which stores an instruction program that can implement a specific function when executed by the processor. Through the above-mentioned hardware system, the computing device can perform steps such as data processing and model reasoning to achieve intelligent processing of target data. The input / output interface can also realize data exchange between the device and external devices to support various data interaction and control requirements. The camera device can be a surveillance camera, an industrial camera, a built-in camera module in a mobile terminal, or other devices with image acquisition capabilities, which are used to obtain continuous video data.

[0062] In another embodiment, a computer-readable storage medium is proposed, on which a computer program is stored. The program can be loaded and run by the above-mentioned computing device to execute a human behavior recognition method based on an improved spatiotemporal graph convolutional neural network to realize the application of behavior recognition.

[0063] It should be understood that the technical features described herein may be combined or separated in various ways according to actual needs. All functional implementations not explicitly defined are feasible within the scope of conventional knowledge of those skilled in the art and should be included within the scope of protection described in this specification. It should also be noted that the specific implementation examples provided herein are intended only to better understand the principles and application logic of the present invention and do not constitute a limitation on the scope of protection.

Claims

1. A human behavior recognition method and device based on an improved spatio-temporal graph convolutional neural network, characterized in that, The classification method comprises the following steps: 1st step: split the photographed human behavior video into picture frames, process all the videos into 200 frames, if insufficient, supplement frames, if more than that, randomly crop; 2nd step: use YOLOv8-pose algorithm to realize extraction of human skeleton point coordinates and confidence; 3rd step: select one or two skeletons with higher average confidence in the video as the skeleton point dataset; 4th step: use graph structure and joint feature matrix to describe human skeleton features; 5th step: take the human graph structure and joint feature matrix as the input of spatio-temporal graph convolution, in the spatial domain, use Laplace feature mapping dimension reduction method to divide the graph into non-overlapping parts, and perform skeleton-level feature extraction on the reduced graph, and then perform part-level feature extraction on the divided subgraph parts, and then align and fuse the features. In the time domain, use different scale time convolution kernels to expand the time scale feature receptive field. 6th step: perform softmax classification on the processed features to obtain the output result.

2. The human behavior recognition method of claim 1, wherein, The evaluation index of the classification method comprises two accuracy indexes, namely Top1-ACC and Top5-ACC, wherein Top1-ACC refers to the probability that the highest probability predicted category is the correct category; and Top5-ACC refers to the probability that the top five predicted categories contain the correct category. 3.The human skeleton behavior recognition method based on graph partitioning and dilated convolution of claim 1, wherein: The human skeleton data used is obtained through a video acquisition system, which comprises a video acquisition module; a skeleton extraction module based on YOLOv8-pose algorithm is used to extract human key point coordinates and confidence in each frame; a skeleton screening module selects a subjective observation target according to the confidence; a graph construction module converts the human skeleton structure into a graph structure and calculates the joint adjacency matrix; in the spatial domain, a Laplace feature mapping method is used to reduce the skeleton graph structure and automatically divide it into multiple structure-stable subgraph regions, and local and global spatial structure features are extracted; in the time domain, a multi-scale hollow convolution method is used to extract features of the time sequence, different hollow rates are used to expand the time receptive field, and the modeling ability of long-time dependence is enhanced; finally, the Softmax layer is used to complete the recognition and classification of the behavior category.

Images