Human behavior recognition method, device and storage medium based on skeleton points

Abstract

The present invention discloses a human behavior recognition method, device, and storage medium based on skeleton points, which are applied to the field of behavior recognition technology. The method comprises the following steps: S1. Data acquisition; S2. Construction of a human body detection and posture estimation network; S3. Training and testing of the human body detection and posture estimation network; S4. Performing human body detection and posture estimation; S5. Human multi-target tracking; S6. Repeating S4 and S5 until all video frames are processed, and finally obtaining a stacked array of human key point coordinates for all video frames; S7. Construction of a human behavior recognition network; S8. Training and testing of the human behavior recognition network; S9. Human behavior recognition. The present invention performs human behavior recognition based on skeleton points. By extracting the position transformation information of the skeleton points, the behavior of the target can be effectively understood, with high accuracy. At the same time, the amount of data processing is also reduced, and the method has high real-time performance.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior recognition, and more particularly to a human behavior recognition method, device and storage medium based on skeleton points. Background Art

[0002] With the continuous advancement of computer technology, the field of computer vision has achieved many outstanding research results, driving algorithms such as image processing and object detection to continuously advance towards meeting the needs of real-world scenarios. In recent decades, continuous breakthroughs in deep learning technology have led to a new wave of research in areas such as object detection and behavior recognition, gradually replacing traditional methods and becoming the prevalent processing framework for computer vision tasks.

[0003] Behavior recognition is one of the most challenging tasks in current computer vision and forms the foundation for subsequent tasks such as intention and trajectory prediction. Behavior recognition is defined as the classification of human actions detected in an input video sequence, enabling computers to interpret human behavior, such as whether the subject is walking, jumping, or running. This approach holds significant application potential in areas such as autonomous driving and intelligent surveillance. Currently, traditional methods or deep learning approaches are typically used to address behavior recognition.

[0004] Traditional behavior recognition methods rely on visual observation to manually design features that represent behavior. These features are then input into a designed classifier to produce a behavior classification result. However, these traditional methods are slow and complex. Furthermore, their heavy reliance on manual extraction results in poor model generalization and is susceptible to background effects such as lighting. Consequently, most traditional methods are unable to meet practical needs.

[0005] In recent years, with the development of deep learning technology, deep learning-based action recognition algorithms have gradually come into the attention of researchers due to their advantages such as being less susceptible to background influences and not requiring manual feature extraction. Deep learning-based action recognition methods can be roughly divided into skeleton-based methods and RGB video-based methods, depending on whether human key points are required as input. Human skeleton points provide a lot of important information for human action recognition. Even without detailed features such as the background, RGB color, and human appearance in the video, extracting the positional transformation information of skeleton points can effectively understand the target's behavior. These methods are robust to changes in lighting and scene. Therefore, skeleton-based action recognition methods have less computational complexity and are more adaptable to different scenarios than RGB video-based methods. They are more suitable for real-time action recognition and are therefore receiving increasing attention.

[0006] However, when applying behavior recognition methods in real-world scenarios, people often have high requirements for both accuracy and speed. This requires computing devices with strong computing power and large storage space, which incurs significant costs. Therefore, it is of great research value to develop a lightweight method for human behavior recognition based on skeleton points that can maintain both accuracy and real-time performance while being lightweight.

[0007] Therefore, it is an urgent problem for those skilled in the art to propose a human behavior recognition method, device and storage medium based on skeleton points to solve the difficulties existing in the prior art. Summary of the Invention

[0008] In view of this, the present invention provides a human behavior recognition method, device and storage medium based on skeleton points, which are used to solve the technical problems existing in the prior art.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A human behavior recognition method based on skeleton points comprises the following steps:

[0011] S1. Data collection: Use a camera to capture video data in real-world scenarios;

[0012] S2. Build a human body detection and pose estimation network: Based on the lightweight and improved YOLO-Pose algorithm, build a human body detection and pose estimation network consisting of three parts: ShuffleNetV2_k5 as the backbone network, BiFPN neck network, and Decoupled-Head decoupling detection head;

[0013] S3. Train and test the human detection and pose estimation network: Divide the COCO-Keypoints dataset into a training set, a test set, and a validation set. Use the training set to train the human detection and pose estimation network built in S2, use the validation set for validation, and finally use the test set to test the trained and validated human detection and pose estimation network.

[0014] S4. Perform human body detection and pose estimation: Use a lightweight and improved YOLO-Pose algorithm to process the captured video and obtain the human body detection frame and the coordinates of 17 human key points and their corresponding confidence levels;

[0015] S5. Human Multi-Target Tracking: The human detection frames obtained in S4, along with the coordinates of the 17 human key points and their corresponding confidence scores, are input into the BoT-SORT multi-target tracking algorithm. Each detected human is assigned a unique identity number, and the same human targets in the previous and current frames are matched. The coordinates of the human key points in the current frame are sorted by identity number.

[0016] S6. Repeat S4 and S5 until all video frames are processed, and finally obtain a stacked array of human key point coordinates of all video frames;

[0017] S7. Build a human action recognition network: Based on the PoseC3D algorithm, we improve it by replacing the original backbone network of the PoseC3D algorithm with the ShuffleNetV2_k5 network that incorporates the CBAM attention mechanism, thereby obtaining a human action recognition network.

[0018] S8. Train and test the human action recognition network: Divide the NTU-RGB+D 60 XSub dataset into a training set, a test set, and a validation set. Use the training set to train the human action recognition network from S7, use the validation set for validation, and finally use the test set to test the trained and validated human action recognition network.

[0019] S9. Human action recognition: Use the lightweight and improved PoseC3D algorithm to process the stacked array of human key point coordinates obtained in S6, and finally obtain the human action recognition result and confidence level.

[0020] Optionally, S2 is:

[0021] S21. The backbone network consists of a lightweight Focus downsampling module and a Shufflenet v2_k5 module. The backbone network outputs feature maps at four scales, downsampled by 4, 8, 16, and 32 times, which then enter the neck network for feature fusion.

[0022] S22. The neck network consists of a BiFPN network integrated with a CBAM attention mechanism module, where the 1×1 convolution is replaced by a GSConv module, and the C3 module is replaced by a lightweight C2f module.

[0023] S23. Connect the output feature map of the neck network to the improved lightweight decoupling head to obtain the prediction results and perform post-processing.

[0024] Optionally, the Focus downsampling module first slices the input into four parts and then stitches them together again, converting the high-resolution image into a low-resolution image, and finally passes the output through the CBR module. The Focus downsampling module reduces the loss caused by the downsampling process while reducing the number of floating-point operations.

[0025] Optionally, the CBR module consists of a two-dimensional convolutional layer, a BN layer, and a ReLU activation function.

[0026] Optionally, when training the human detection and pose estimation network in S3, the Mosaic and Mixup data augmentation methods are used, the SGD optimizer is used, the initial learning rate is 0.01, and the cosine annealing strategy is used to adjust the learning rate.

[0027] Optionally, S5 is:

[0028] S51 detects the human body position in the current frame and uses the Kalman filter algorithm to predict the human body position in the next frame;

[0029] S52. Perform data association and matching on the detected target and the predicted result;

[0030] S53. Update the successfully matched trajectory;

[0031] S54.BoT-SORT rematches the failed track with the detection box. If the track matches successfully, it is updated. If the detection box matches successfully, a new track is created. If the track matches unsuccessfully, another round of matching is performed. If it still fails, the track is deleted directly.

[0032] Optional, S7 specifically:

[0033] S71. Construct a ShuffleNet block module, which is composed of a stride 1 and a stride 2 building block. The building block replaces all 3×3 depthwise separable convolutions in ShuffleNet v2 with 5×5 depthwise separable convolution modules.

[0034] S72. By inserting the CBAM module into the ShuffleNet block of each stage, the two are integrated and used as the backbone network of PoseC3D; by adopting the improved ShuffleNetV2 network that integrates the CBAM attention mechanism as the backbone, the number of parameters and computational complexity are reduced.

[0035] Optional, S9 specifically:

[0036] S91. Process the stacked array of human key point coordinates obtained in S6, assuming that the number of video frames is F, the maximum identity number is N, and the number of skeleton points is K, to ultimately obtain an N×F×K×2 array storing the coordinates of the skeleton points and an N×F×K×1 array storing the confidence scores of the skeleton points;

[0037] S92. Take the coordinates of each skeleton point as the center and the confidence as the maximum value to obtain the binary Gaussian distribution heat map of the 2D key points. The calculation method is:

[0038] Among them, J is the key point heat map, i is the horizontal coordinate of the variable, x k is the horizontal coordinate of the key point, j is the vertical coordinate of the variable, y k is the vertical coordinate of the key point, σ is the variance, c k is the key point confidence, k is the key point label, and e is a natural constant;

[0039] S93. The 2D heat maps obtained in S92 are stacked to obtain a 3D heat map, which is used as input for subsequent behavior recognition. To reduce redundancy in the temporal dimension, a uniform sampling method is used to extract key frames. Specifically, assuming that N key frames are required, the video frame is first evenly divided into N segments, and then a frame is randomly selected from each segment to complete key frame extraction.

[0040] S94. Finally, the data is input into the PoseC3D network for inference, and the top n classifications and scores of the behaviors are obtained according to actual needs to complete the recognition.

[0041] A computer device as described in any of the above items, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the steps of the above-mentioned human behavior recognition method based on skeleton points.

[0042] A computer-readable storage medium as described in any one of the above items, wherein the computer-readable storage medium contains a program, and when the program is executed by a processor, it is used to implement the steps of the above-mentioned human behavior recognition method based on skeleton points.

[0043] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a method, device and storage medium for human behavior recognition based on skeleton points, which has the following beneficial effects:

[0044] 1) This invention combines target detection, posture estimation, multi-target tracking, and behavior recognition technologies to provide a lightweight human behavior recognition method with competitive recognition accuracy and speed while maintaining accuracy.

[0045] 2) Ordinary cameras instead of specialized depth cameras can meet the needs, reducing equipment costs;

[0046] 3) Using a lightweight and improved single-stage YOLO-Pose algorithm for joint human detection and pose estimation, only a single feature extraction step is required to obtain the human detection bounding box and key point coordinates. Compared with the two-stage method of pose estimation based on object detection, this simplifies the steps and improves the speed. At the same time, the lightweight improvement reduces the computational and storage costs while maintaining accuracy.

[0047] 4) Human behavior recognition based on skeleton points is not easily affected by lighting and background. It can effectively understand the target's behavior by simply extracting the position transformation information of the skeleton points, with high accuracy. It also reduces the amount of data processing and has high real-time performance.

[0048] 5) The lightweight and improved PoseC3D algorithm is used for human action recognition. It employs a three-dimensional convolutional neural network as its basic framework. Compared to the graph convolution method commonly used in current skeletal-based action recognition algorithms, it is more robust, compatible, and scalable, achieving better recognition results with less data. Furthermore, this lightweight improvement reduces computational and storage costs while maintaining accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 A flowchart of a human behavior recognition method based on skeleton points provided by the present invention;

[0051] Figure 2 A schematic flow chart of a human behavior recognition method based on skeleton points provided by the present invention;

[0052] Figure 3 A schematic diagram of the structure of the improved YOLO-Pose human body detection and pose estimation network provided by the present invention;

[0053] Figure 4 A schematic diagram of the structure of the backbone network and neck network in the improved YOLO-Pose human body detection and pose estimation network provided by the present invention;

[0054] Figure 5 A schematic diagram of the structure of the decoupling head in the improved YOLO-Pose human body detection and pose estimation network provided by the present invention;

[0055] Figure 6 This is a performance graph of the experimental results of the improved YOLO-Pose human body detection and pose estimation network provided by the present invention on the COCO-Keypoint dataset;

[0056] Figure 7 A schematic diagram of the structure of 17 key points of the human body detected by the improved YOLO-Pose human body detection and posture estimation network provided by the present invention;

[0057] Figure 8 This is a graph showing the top-1 and top-5 accuracy parameter curves of the improved PoseC3D action recognition algorithm provided by the present invention on the NTU-RGB+D 60XSub dataset. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] See also Figure 1 and Figure 2 As shown, the present invention discloses a human behavior recognition method based on skeleton points, comprising the following steps:

[0060] S1. Data collection: Use a camera to capture video data in real-world scenarios;

[0061] S2. Build a human body detection and pose estimation network: Based on the lightweight and improved YOLO-Pose algorithm, build a human body detection and pose estimation network consisting of three parts: ShuffleNetV2_k5 as the backbone network, BiFPN neck network, and Decoupled-Head decoupling detection head;

[0062] S3. Train and test the human detection and pose estimation network: Divide the COCO-Keypoints dataset into a training set, a test set, and a validation set. Use the training set to train the human detection and pose estimation network built in S2, use the validation set for validation, and finally use the test set to test the trained and validated human detection and pose estimation network.

[0063] S4. Perform human body detection and pose estimation: Use a lightweight and improved YOLO-Pose algorithm to process the captured video and obtain the human body detection frame and the coordinates of 17 human key points and their corresponding confidence levels;

[0064] S5. Human Multi-Target Tracking: The human detection frames obtained in S4, along with the coordinates of the 17 human key points and their corresponding confidence scores, are input into the BoT-SORT multi-target tracking algorithm. Each detected human is assigned a unique identity number, and the same human targets in the previous and current frames are matched. The coordinates of the human key points in the current frame are sorted by identity number.

[0065] S6. Repeat S4 and S5 until all video frames are processed, and finally obtain a stacked array of human key point coordinates of all video frames;

[0066] S7. Build a human action recognition network: Based on the PoseC3D algorithm, we improve it by replacing the original backbone network of the PoseC3D algorithm with the ShuffleNetV2_k5 network that incorporates the CBAM attention mechanism, thereby obtaining a human action recognition network.

[0067] S8. Train and test the human action recognition network: Divide the NTU-RGB+D 60 XSub dataset into a training set, a test set, and a validation set. Use the training set to train the human action recognition network from S7, use the validation set for validation, and finally use the test set to test the trained and validated human action recognition network.

[0068] S9. Human action recognition: Use the lightweight and improved PoseC3D algorithm to process the stacked array of human key point coordinates obtained in S6, and finally obtain the human action recognition result and confidence level.

[0069] Specifically, the specific structure of the human body detection and posture estimation network in S2 can be found in Figure 3 shown.

[0070] Specifically, the human body detection box in S4 is the largest circumscribed rectangle of the human body, which is used to identify the position of the human body. The confidence level refers to the probability calculated by the algorithm that the target contained in the detection box is a human body. The 17 key points of the human body obtained by posture estimation are nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee and left ankle, right ankle. These 17 key points of the human body can better represent the human body posture, such as Figure 7 shown.

[0071] Furthermore, S2 is specifically:

[0072] S21. The backbone network consists of a lightweight Focus downsampling module and a Shufflenet v2_k5 module. The backbone network outputs feature maps at four scales, downsampled by 4, 8, 16, and 32 times, which then enter the neck network for feature fusion.

[0073] S22. The neck network consists of a BiFPN network integrated with a CBAM attention mechanism module, where the 1×1 convolution is replaced by a GSConv module, and the C3 module is replaced by a lightweight C2f module.

[0074] S23. Connect the output feature map of the neck network to the improved lightweight decoupling head to obtain the prediction results and perform post-processing.

[0075] Specifically, CBAM is a lightweight attention mechanism module that can be plugged and played into any convolutional neural network. By combining channel-wise and spatial-level attention, it improves the channel-wise and spatial-level connections between features with minimal increase in computation and parameters. This improves the extraction of effective target features, thereby effectively enhancing model performance. The channel-wise attention module focuses on useful content in the image. Features from two channels undergo max-pooling and average-pooling operations in parallel before being fed into a shared neural network to generate two feature elements. Finally, these elements are multiplied and added together, followed by an activation layer, to produce the output, the channel-wise attention feature map. The spatial-level attention module focuses on the location of valuable information. The module's input first undergoes max-pooling and average-pooling to integrate information, then passes through a convolutional layer and activation to generate a spatial-level attention feature map. Finally, the module's output is weighted with the input features to obtain new, refined features with both channel-wise and spatial-level attention weights.

[0076] BiFPN was proposed by the Google team. The network introduces a learnable weight method to distinguish features of different scales and allows features to pass through the top-down and bottom-up structures repeatedly, so that multi-scale features can be better and more efficiently integrated. The position and structure of the BiFPN network in actual algorithm use are as follows Figure 4 shown.

[0077] BiFPN uses a fast normalized fusion method when performing weighted feature fusion operations. It uses a value normalized to the range of [0, 1] to weight the weights, which is guaranteed to be greater than 0 by the ReLU activation function. This is fast and stable. The calculation formula is

[0078] Among them, Output is the output; w is the weight; ε is the minimum value of 0.0001 to prevent numerical instability; Input is the input;

[0079] The C2f module consists of CBR and BottleNeck modules. The C2f module can obtain more gradient information while being lightweight.

[0080] Specifically, using different branches to perform calculations for classification and positioning tasks in target detection is beneficial to improving detection accuracy, which is also the basic idea of ​​the decoupled detection head. Improvements are made based on the EdgeYOLO detection head: the output feature map of the neck network is channel-compressed through a 1×1 convolution layer and then input into three parallel branches: positioning, classification, and key points. Each branch has a 3×3 convolution layer, which is then connected to a 1×1 convolution layer. Finally, the implicit representation layer is integrated into the convolution layer of each branch by drawing on the implicit knowledge idea, forming the following: Figure 5 The highly parallel, low-cost, and low-latency decoupled detection head shown.

[0081] All elements of the YOLO-Pose prediction output are

[0082] Among them, C x is the horizontal coordinate of the upper left corner of the bounding box, C y is the ordinate of the upper left corner of the bounding box; W is the width of the bounding box; H is the height of the bounding box; boxconf is the confidence of the bounding box; classconf is the confidence of the classification result; K x is the horizontal coordinate of the key point, K y is the vertical coordinate of the key point; Kconf is the confidence of the key point.

[0083] YOLO-Pose uses the CIoU loss with scale invariance and considering the aspect ratio of the center point of the bounding box as the loss function of the bounding box. The calculation formula is

[0084] Among them, Lbox is the bounding box loss, s is the scale, i and j are the positions, and k is the anchor box number; Boxgt is the true value of the k-th anchor box at position (i, j) with a scale of s; Boxpred is the predicted value of the k-th anchor box at position (i, j) with a scale of s.

[0085] For pose estimation, YOLO-Pose uses the IoU loss idea of ​​the bounding box analogy and uses OKS loss to supervise key points. OKS loss is scale-invariant and uses different weights to distinguish the importance of each key point. The calculation formula is

[0086] Among them, L kpts is the key point loss; d is the Euclidean distance between the true value and the predicted value of the key point; s is the target scale; k is the weight given to different key points; δ(v n ) is the flag indicating whether the key point is visible.

[0087] When the true value matches a preset anchor frame, YOLO-Pose predicts the key points of the human body through the center point of the anchor frame, calculates OKS for each key point and adds them together to obtain the final key point loss.

[0088] At the same time, YOLO-Pose uses BCE loss to learn key point confidence, which is used to determine whether a key point belongs to the person in the bounding box. The calculation formula is

[0089] Among them, L kpt_conf is the key point confidence loss; δ(v n) is the key point visibility flag, which is regarded as the true value here; pkpts is the key point prediction confidence.

[0090] Finally, the loss formula of YOLO-Pose is:

[0091]

[0092] Among them, λ cls is the hyperparameter of classification loss, assigned a value of 0.5, λ box is the hyperparameter of the bounding box loss, assigned a value of 0.05, λ kpts is the hyperparameter of key point loss, assigned a value of 0.1, λ kpts_conf is the hyperparameter of the key point confidence loss, assigned a value of 0.5; L total is the overall loss; L cls is the classification loss; L box is the bounding box loss; L kpts is the key point loss; L kpts_conf is the key point confidence loss.

[0093] Furthermore, the Focus downsampling module first slices the input into four parts and then stitches them together again, converting the high-resolution image into a low-resolution image, and finally passes the output through the CBR module. The Focus downsampling module reduces the loss caused by the downsampling process while reducing the number of floating-point operations.

[0094] Furthermore, the CBR module consists of a two-dimensional convolutional layer, a BN layer and a ReLU activation function.

[0095] Furthermore, when training the human detection and pose estimation network in S3, the Mosaic and Mixup data augmentation methods are adopted, the SGD optimizer is used, the initial learning rate is 0.01, and the cosine annealing strategy is used to adjust the learning rate.

[0096] Specifically, the weight decay is set to 0.0005, the batch size is set to 64, the target class is set to 1, the number of key points is set to 17, and the input image size is 640×640 for 300 epochs. The trained model weights are saved in each round.

[0097] The experimental results are as follows Figure 6As shown in the figure, "precision" is the accuracy, "recall" is the recall rate, "mAP" refers to the mean average precision, "mAP@0.5" is the mAP value when the IoU threshold is 0.5, and "mAP@0.5:0.95" is the mAP mean when the IoU thresholds are 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, and 0.95 respectively.

[0098] In addition, we selected several current mainstream lightweight object detection algorithms and conducted experiments on the COCO dataset with 640-pixel image resolution as input. We compared the model complexity and performance, as shown in Table 1. The comparison results show that the proposed algorithm is highly competitive in terms of average precision and parameter count.

[0099] Table 1: Performance comparison of the improved YOLO-Pose human detection and pose estimation network and other lightweight human detection networks

[0100]

[0101]

[0102] Furthermore, S5 is specifically:

[0103] S51 detects the human body position in the current frame and uses the Kalman filter algorithm to predict the human body position in the next frame;

[0104] S52. Perform data association and matching on the detected target and the predicted result;

[0105] S53. Update the successfully matched trajectory;

[0106] S54.BoT-SORT rematches the failed track with the detection box. If the track matches successfully, it is updated. If the detection box matches successfully, a new track is created. If the track matches unsuccessfully, another round of matching is performed. If it still fails, the track is deleted directly.

[0107] Specifically, the BoT-SORT multi-target tracking algorithm in S5 is as follows: first, target detection is performed on each frame image to obtain the bounding box coordinates of all targets, and finally the target similarity between adjacent frames is calculated for matching.

[0108] Improved Kalman filtering, camera motion compensation, and person re-identification technology were used to address issues such as excessive identity information changes, missed detections under occlusion, blur, and inconsistent trajectories.

[0109] BoT-SORT uses 8-tuples,

[0110] As the Kalman filter vector, we choose to directly predict the width and height instead of the aspect ratio, which can obtain a more accurate bounding box that can completely surround the human body.

[0111] Among them, x k is the state vector; x c is the horizontal coordinate of the center point of the bounding box; c is the vertical coordinate of the center point of the bounding box; w is the width of the bounding box; h is the height of the bounding box; is the first-order derivative of the corresponding variable.

[0112] Furthermore, S7 is specifically:

[0113] S71. Construct a ShuffleNet block module, which is composed of a stride 1 and a stride 2 building block. The building block replaces all 3×3 depthwise separable convolutions in ShuffleNet v2 with 5×5 depthwise separable convolution modules.

[0114] S72. By inserting the CBAM module into the ShuffleNet block of each stage, the two are integrated and used as the backbone network of PoseC3D; by adopting the improved ShuffleNetV2 network that integrates the CBAM attention mechanism as the backbone, the number of parameters and computational complexity are reduced.

[0115] Specifically, S8 uses the Adam optimizer for training, with an initial learning rate of 0.001, adjusted using a cosine annealing strategy, and a weight decay of 0.0001. The batch size is set to 64, and training is repeated for 200 epochs, with the training log and model saved for each epoch. All other hyperparameters not mentioned here are kept consistent with the default values ​​for PoseC3D training parameters.

[0116] The experimental results are as follows Figure 8 As shown in the figure, "top-1 acc" refers to the accuracy of the top-ranked classification that is consistent with the true classification when the prediction results are sorted from high to low probability; "top-5 acc" refers to the accuracy of the top five classifications that contain the true classification when the prediction results are sorted from high to low probability;

[0117] In addition, the comparison results of the improved algorithm with the popular skeleton-based behavior recognition algorithms STGCN and 2s-AGCN are shown in Table 2. Compared with the above two algorithms, the improved model of the present invention is lighter and more accurate, and has certain competitiveness.

[0118] Table 2: Performance comparison with other skeleton-based action recognition algorithms on the NTU-RGB+D 60 XSub dataset

[0119] method Input Modal Top-1 Accuracy Top-5 Accuracy Model size STGCN Skeleton Point 88.9％ 98.7％ 11.90MB 2s-AGCN Skeleton Point 88.6％ 98.5％ 13.40MB Improve PoseC3D Skeleton Point 90.3％ 98.5％ 7.09MB

[0120] The behavior categories that can be recognized by the trained behavior recognition algorithm are shown in Table 3.

[0121] Table 3: Recognizable human behavior categories

[0122]

[0123]

[0124] Furthermore, S9 is specifically:

[0125] S91. Process the stacked array of human key point coordinates obtained in S6, assuming that the number of video frames is F, the maximum identity number is N, and the number of skeleton points is K, to ultimately obtain an N×F×K×2 array storing the coordinates of the skeleton points and an N×F×K×1 array storing the confidence scores of the skeleton points;

[0126] S92. Take the coordinates of each skeleton point as the center and the confidence as the maximum value to obtain the binary Gaussian distribution heat map of the 2D key points. The calculation method is:

[0127] Among them, J is the key point heat map, i is the horizontal coordinate of the variable, x k is the horizontal coordinate of the key point, j is the vertical coordinate of the variable, y k is the vertical coordinate of the key point, σ is the variance, c k is the key point confidence, k is the key point label, and e is a natural constant;

[0128] S93. The 2D heat maps obtained in S92 are stacked to obtain a 3D heat map, which is used as input for subsequent behavior recognition. To reduce redundancy in the temporal dimension, a uniform sampling method is used to extract key frames. Specifically, assuming that N key frames are required, the video frame is first evenly divided into N segments, and then a frame is randomly selected from each segment to complete key frame extraction.

[0129] S94. Finally, the data is input into the PoseC3D network for inference, and the top n classifications and scores of the behaviors are obtained according to actual needs to complete the recognition.

[0130] A computer device as described above includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the steps of the above-mentioned human behavior recognition method based on skeleton points.

[0131] Specifically, a computer device is any device or apparatus capable of processing data.

[0132] A computer-readable storage medium as described above, wherein the computer-readable storage medium contains a program, and when the program is executed by a processor, it is used to implement the steps of the above-mentioned human behavior recognition method based on skeleton points.

[0133] Specifically, the computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart memory card (Smart Media Card, SMC), SD card, flash card (FlashCard), etc. equipped on the device.

[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0135] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A human behavior recognition method based on skeleton points, characterized in that: The following steps are involved: S1. Data collection: Use a camera to capture video data in real-world scenarios; S2. Build a human body detection and pose estimation network: Based on the lightweight and improved YOLO-Pose algorithm, build a human body detection and pose estimation network consisting of three parts: ShuffleNetV2_k5 as the backbone network, BiFPN neck network, and Decoupled-Head decoupling detection head; S3. Train and test the human detection and pose estimation network: Divide the COCO-Keypoints dataset into a training set, a test set, and a validation set. Use the training set to train the human detection and pose estimation network built in S2, use the validation set for validation, and finally use the test set to test the trained and validated human detection and pose estimation network. S4. Perform human body detection and pose estimation: Use a lightweight and improved YOLO-Pose algorithm to process the captured video and obtain the human body detection frame and the coordinates of 17 human key points and their corresponding confidence levels; S5. Human Multi-Target Tracking: The human detection frames obtained in S4, along with the coordinates of the 17 human key points and their corresponding confidence scores, are input into the BoT-SORT multi-target tracking algorithm. Each detected human is assigned a unique identity number, and the same human targets in the previous and current frames are matched. The coordinates of the human key points in the current frame are sorted by identity number. S6. Repeat S4 and S5 until all video frames are processed, and finally obtain a stacked array of human key point coordinates of all video frames; S7. Build a human action recognition network: Based on the PoseC3D algorithm, we improve it by replacing the original backbone network of the PoseC3D algorithm with the ShuffleNetV2_k5 network that incorporates the CBAM attention mechanism, thereby obtaining a human action recognition network. S8. Training and testing the human action recognition network: Divide the NTU-RGB+D 60XSub dataset into training, test, and validation sets. Use the training set to train the human action recognition network from S7, use the validation set for validation, and finally use the test set to test the trained and validated human action recognition network. S9. Human action recognition: Use the lightweight and improved PoseC3D algorithm to process the stacked array of human key point coordinates obtained in S6, and finally obtain the human action recognition result and confidence level.

2. The human behavior recognition method based on skeleton points according to claim 1, characterized in that: S2 is specifically: S21. The backbone network consists of a lightweight Focus downsampling module and a Shufflenet v2_k5 module. The backbone network outputs feature maps at four scales, downsampled by 4, 8, 16, and 32 times, which then enter the neck network for feature fusion. S22. The neck network consists of a BiFPN network integrated with a CBAM attention mechanism module, where the 1×1 convolution is replaced by a GSConv module, and the C3 module is replaced by a lightweight C2f module. S23. Connect the output feature map of the neck network to the improved lightweight decoupling head to obtain the prediction results and perform post-processing.

3. The human behavior recognition method based on skeleton points according to claim 2, characterized in that: The Focus downsampling module first slices the input into four parts and then stitches them together again, converting the high-resolution image into a low-resolution image. Finally, the output is obtained through the CBR module. The Focus downsampling module reduces the loss caused by the downsampling process while reducing the number of floating-point operations.

4. The human behavior recognition method based on skeleton points according to claim 3 is characterized in that: The CBR module consists of a two-dimensional convolutional layer, a BN layer and a ReLU activation function.

5. The human behavior recognition method based on skeleton points according to claim 1, characterized in that: When training the human body detection and pose estimation network in S3, the Mosaic and Mixup data enhancement methods are used, the SGD optimizer is used, the initial learning rate is 0.01, and the cosine annealing strategy is used to adjust the learning rate.

6. The human behavior recognition method based on skeleton points according to claim 1, characterized in that: S5 is specifically: S51 detects the human body position in the current frame and uses the Kalman filter algorithm to predict the human body position in the next frame; S52. Perform data association and matching on the detected target and the predicted result; S53. Update the successfully matched trajectory; S54.BoT-SORT rematches the failed track with the detection box. If the track matches successfully, it is updated. If the detection box matches successfully, a new track is created. If the track matches unsuccessfully, another round of matching is performed. If it still fails, the track is deleted directly.

7. The human behavior recognition method based on skeleton points according to claim 1, characterized in that: S7 specifically: S71. Construct a ShuffleNet block module, which is composed of a stride 1 and a stride 2 building block. The building block replaces all 3×3 depthwise separable convolutions in ShuffleNet v2 with 5×5 depthwise separable convolution modules. S72. By inserting the CBAM module into the ShuffleNet block of each stage, the two are integrated and used as the backbone network of PoseC3D; by adopting the improved ShuffleNetV2 network that integrates the CBAM attention mechanism as the backbone, the number of parameters and computational complexity are reduced.

8. The human behavior recognition method based on skeleton points according to claim 1, characterized in that: S9 specifically: S91. Process the stacked array of human key point coordinates obtained in S6, assuming that the number of video frames is F, the maximum identity number is N, and the number of skeleton points is K, to ultimately obtain an N×F×K×2 array storing the coordinates of the skeleton points and an N×F×K×1 array storing the confidence scores of the skeleton points; S92. Take the coordinates of each skeleton point as the center and the confidence as the maximum value to obtain the binary Gaussian distribution heat map of the 2D key points. The calculation method is: Among them, J is the key point heat map, i is the horizontal coordinate of the variable, x k is the horizontal coordinate of the key point, j is the vertical coordinate of the variable, y k is the vertical coordinate of the key point, σ is the variance, c k is the key point confidence, k is the key point label, and e is a natural constant; S93. The 2D heat maps obtained in S92 are stacked to obtain a 3D heat map, which is used as input for subsequent behavior recognition. To reduce redundancy in the temporal dimension, a uniform sampling method is used to extract key frames. Specifically, assuming that N key frames are required, the video frame is first evenly divided into N segments, and then a frame is randomly selected from each segment to complete key frame extraction. S94. Finally, the data is input into the PoseC3D network for inference, and the top n classifications and scores of the behaviors are obtained according to actual needs to complete the recognition.

9. A computer device, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the steps of the human behavior recognition method based on skeleton points described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The readable storage medium contains a program, which, when executed by a processor, is used to implement the steps of a human behavior recognition method based on skeleton points as described in any one of claims 1 to 8.

Images