Skeleton action recognition method and detection system thereof

Abstract

The invention discloses a skeleton action recognition method and a detection system thereof. The detection system comprises a power supply module, a skeleton signal acquisition module, an image preprocessing module, a receiving thread module, a processor module and an action recognition module. A multi-granularity skeleton feature network is adopted in the skeleton action recognition module, the network is divided into a plurality of stages, and skeleton features of different granularities are processed in each stage. Through multi-scale feature fusion operation on skeleton joint point information, mapping coarse-grained joint point features to a fine-grained space by using feature aggregation, and fusing the coarse-grained joint point features with fine-grained skeleton features; meanwhile, the fine-grained features are compressed to a coarse-grained space by using convolution operation and average operation, and are fused with the coarse-grained features. Through the cross-granularity feature fusion mode, the model can fuse more semantic information while reserving spatial information, so that the precision and speed of skeleton action recognition are improved, and the processing capacity of a system in a complex scene is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of action recognition, and in particular relates to a skeleton action recognition method and a detection system thereof. Background Art

[0002] Skeleton-based action recognition is an important task in the field of computer vision and has attracted more and more attention from researchers in recent years. This topic has extremely wide applications in video surveillance, human-computer interaction, and self-driving cars. In particular, skeleton data can be easily obtained from depth cameras (such as Microsoft Kinect V2) and currently mature 2D or 3D pose estimation algorithms. Compared with RGB video data, skeleton data has richer semantics and better robustness and flexibility for dynamic environments and complex backgrounds. With the continuous development of deep learning, skeleton-based action recognition mainly includes three mainstream methods, namely RNN-based, CNN-based, and GCN-based. GCN-based methods model the spatiotemporal dependencies in skeleton sequences by constructing spatial-temporal graphs. This type of method represents human skeleton data as a graph structure, where the nodes of the graph represent the joints of the human body and the edges represent the skeleton topological connection relationship or a custom dynamic association relationship. The GCN model extracts the human body structure features in the spatial dimension by performing graph convolution operations on the graph, and captures the dynamic features of the action by combining time series information (usually using one-dimensional convolution or sequence modeling methods). However, common skeleton action recognition methods have difficulty in effectively capturing the global dynamic pattern of the entire skeleton when performing skeleton action recognition, and fail to fully utilize the correlation information between local and overall features in the action, which limits the model's expressive ability. Summary of the invention

[0003] The purpose of the present invention is to provide a skeleton action recognition method and a detection system thereof in order to solve the above-mentioned problems.

[0004] The technical solution adopted by the present invention is as follows: a skeleton action recognition method and a detection system thereof, comprising a power supply module, a skeleton signal acquisition module, an image preprocessing module, a receiving thread module, a processor module, an action recognition module, an image filtering module, an image enhancement module, a human skeleton target detection module, a skeleton joint point extraction module and a human skeleton tracking module, wherein the output end of the power supply module is connected to the input end of the skeleton signal acquisition module, the output end of the skeleton signal acquisition module is connected to the input end of the image preprocessing module, the output end of the image preprocessing module is connected to the input end of the receiving thread module, the output end of the receiving thread module is connected to the input end of the processor module, and the output end of the processor module is connected to the input end of the action recognition module.

[0005] In a preferred embodiment, the image preprocessing module is internally provided with an image filtering module and an image enhancement module, and the overall output end of the image filtering module and the image enhancement module is connected to the input end of the image preprocessing module; the action recognition module is internally provided with a human skeleton target detection module, a skeleton joint point extraction module and a human skeleton tracking module, and the overall output end of the human skeleton target detection module, the skeleton joint point extraction module and the human skeleton tracking module is connected to the input end of the action recognition module.

[0006] In a preferred embodiment, the skeleton signal acquisition module uses the EZVIZ network camera of Hikvision as the image acquisition device terminal. The terminal side of the skeleton signal acquisition module uses a Python program to complete the acquisition of the image source and the request to the server. First, to obtain the image data of the network camera, the specific IP address and port number Port, user name user and password password of the network camera are required, and then the VideoCapture method in OpenCV is used to obtain the video stream image of the camera according to the Real-Time Streaming Protocol (RTSP).

[0007] In a preferred embodiment, the image filtering module uses a bilateral filtering method to perform image denoising; the bilateral filtering process for pixel information is nonlinear, and while taking the coordinates of the pixel point in the image into consideration, it also adds the difference influence of the pixel points in the surrounding area into the calculation range; this has an effect similar to Gaussian filtering for the processing of grayscale flat areas, while no processing is performed on abnormal areas, thereby maintaining edge information.

[0008] In a preferred embodiment, the image enhancement module converts the original grayscale range of the image into a new grayscale range by adjusting the scale factor for image enhancement. When the contrast of a certain area in the skeleton action image is low, a stretching function with a slope greater than 1 is used for image enhancement; when the regional contrast is high, a stretching function with a slope less than 1 is used for enhancement, thereby optimizing the visual effect of the image and improving the visibility of the low-contrast area. The region extraction module (12) performs Gaussian smoothing filtering on the image to remove noise. Then, the gradient amplitude and azimuth of the image are calculated, and non-maximum suppression (NMS) is applied to retain the points with the largest local changes. Next, a dual threshold method is used for edge detection, and fewer false edges are obtained through a high threshold, and disconnected edges are connected through a low threshold, ultimately achieving edge closure and accurate extraction.

[0009] In a preferred embodiment, the receiving thread module obtains real-time video data through the IP address and port number of the network camera, and the camera video source path is h264 / ch1 / main / av_stream. This path is used to obtain the video stream data captured by the camera. In order to perform image preprocessing, a Camloader class is constructed and an independent thread is started to process the image data. Considering that there may be differences in the processing capabilities of the client and the server, that is, there is a producer-consumer problem, a cache queue is used to coordinate the processing speed of upstream and downstream data, and the image data is placed in the cache queue frame by frame according to the timing.

[0010] In a preferred embodiment, the processor module deploys the processor on the cloud server through Nginx and uWSGI servers, and the entire deployment process is as follows: first, connect the network camera to the Internet, use the IP viewing tool to obtain the IP address of the camera, and use the Python program to obtain the video data of the camera through the RTSP protocol and test it; then, remotely connect to the cloud server through the SSH protocol (Xshell, Xftp and other tools can be used), and transfer the Flask project, algorithm model file and deep learning adaptation environment package in the local development environment to the cloud server; in the cloud server, use the pip command to install the dependency packages required by the Flask application, and create a virtual environment to ensure that the Flask built-in server can run normally; next, use the apt install command to install the Nginx server component, and use the pip command to install uWSGI and the process manager Supervisor; according to the Flask project configuration, set Nginx, uWSGI and Supervisor, and start the service to start the activity detection service; on the image acquisition device side, send the image data collected by the camera to the cloud server in real time through the Python program; the client accesses the URL address of the cloud server through the browser to obtain the activity detection result image processed by the server, and completes the cloud deployment of the entire activity detection algorithm.

[0011] In a preferred embodiment, the human skeleton target detection module adopts an instance segmentation-based target detection method, which is different from the traditional anchor-based method. The traditional method requires the design and adjustment of anchor frames for each data set, which is complex and time-consuming, while the instance segmentation-based method does not require anchor frame configuration, avoids parameter tuning problems, reduces a large amount of matching calculations, and reduces memory consumption. VFNet is the core model of this module. As a fully convolutional network, it can directly perform classification and regression prediction without relying on anchor frames, and can achieve detection effects comparable to those of traditional anchor frame methods. Due to its high efficiency and accuracy, VFNet performs well in processing multi-person human target detection. The accuracy of human skeleton target detection directly affects the performance of the subsequent skeleton key point extraction module (10). If the detection result is inaccurate, the extracted skeleton information will also be affected, thereby affecting the performance of the entire action recognition system. In order to ensure the accuracy of the model, the NTU-60 dataset was used for training. This dataset is designed specifically for action recognition and contains 56,000 video samples, covering 60 action categories, and annotated with 25 joint points. Each joint point corresponds to a specific part of the human body, such as the head, shoulders, and knees, and is suitable for skeleton posture estimation and action recognition tasks.

[0012] In a preferred embodiment, the skeleton joint point extraction module uses a high-resolution network (HRNet) as a feature extractor to extract the joint point coordinates and confidence information of the human skeleton from each detection frame. The high-resolution network can achieve high-precision extraction of joint point information in complex backgrounds by maintaining multi-scale feature expression capabilities, thereby providing higher quality input data for action recognition.

[0013] In a preferred embodiment, when processing a video sequence, the human skeleton tracking module detects the targets in consecutive frames frame by frame through a human target detector, and combines the multi-target tracking algorithm to associate the detection results of the same human body in different frames to generate a continuous and stable skeleton trajectory. Specifically, a matching strategy between the detection frame and the skeleton joint point features, such as a matching algorithm based on feature distance or a recursive optimization method, is used to ensure that each target remains consistent in the video. The module can also handle complex scenes such as target occlusion, appearance or disappearance to ensure the reliability of the tracking results.

[0014] In a preferred embodiment, the action recognition module uses an improved multi-granularity attention interactive skeleton action recognition model for skeleton action recognition. The model is optimized on the basis of CTR-GCN, and the skeleton features from different granularity levels are integrated through a multi-granularity attention mechanism to achieve accurate recognition of skeleton actions. Finally, the model is trained on the NTU-60 dataset, and the accuracy is improved. The module is applied to other skeleton action recognition models, and the recognition accuracy is also significantly improved.

[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. In the present invention, the HRNet algorithm module inside the action recognition module is divided into multiple stages, and branches of different resolutions in each stage run in parallel. A cross-resolution fusion operation is required between feature mapping branches of two different resolutions. Upsampling is used to convert low resolution into high resolution and fuse it with high resolution, while downsampling is used to convert high resolution into low resolution and fuse it with low resolution. This multi-scale feature fusion method of cross-resolution information interaction realizes semantic fusion, adds rich semantics on the basis of the original spatial information accuracy, thereby improving the speed of image processing and recognition, and bringing more convenience to people's use.

[0016] 2. In the present invention, the action recognition module adopts the CTR-GCN model to add a granularity division module, a cross-attention module and a feature fusion module, and improves the recognition performance through the interactive fusion of fine-grained and coarse-grained features. Fine-grained features retain the precise action details at the joint level, while coarse-grained features capture regional or overall motion patterns through node aggregation. Information is exchanged between features of different granularities through attention mechanisms and feature weighting methods, and fine-grained features are used to guide the generation of coarse-grained features, while coarse-grained features provide global information supplements in reverse. This fusion method of multi-granular features realizes the dynamic unification of details and the global, enhances the robustness and recognition accuracy of the model while retaining fine action information, thereby significantly improving the speed and effect of action recognition, and providing users with a more efficient user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a fine-grained skeleton node of the present invention Figure 2 Schematic diagram of the coarse-grained skeleton node of the present invention Figure 3 Schematic diagram of convolution operation of granularity division in the action recognition module of the present invention Figure 4 Schematic diagram of the cross-attention module of different granularity features in the action recognition module of the present invention Figure 5is the overall system block diagram of the present invention; Figure 6 This is a system block diagram of the image preprocessing module in the present invention; Figure 7 This is a system block diagram of the action recognition module in the present invention.

[0018] Markings in the figure: 1-power supply module, 2-skeleton signal acquisition module, 3-image preprocessing module, 4-receiving thread module, 5-processor module, 6-action recognition module, 7-image filtering module, 8-image enhancement module, 9-human skeleton target detection module, 10-skeleton joint extraction module, 11-human skeleton tracking module. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] Reference Figure 1-7 , Example

[0021] A skeleton action recognition method and a detection system thereof, comprising a power supply module 1, a skeleton signal acquisition module 2, an image preprocessing module 3, a receiving thread module 4, a processor module 5, an action recognition module 6, an image filtering module 7, an image enhancement module 8, a human skeleton target detection module 9, a skeleton joint point extraction module 10 and a human skeleton tracking module 11, wherein the output end of the power supply module 1 is connected to the input end of the skeleton signal acquisition module 2, the output end of the skeleton signal acquisition module 2 is connected to the input end of the image preprocessing module 3, the output end of the image preprocessing module 3 is connected to the input end of the receiving thread module 4, the output end of the receiving thread module 4 is connected to the input end of the processor module 5, and the output end of the processor module 5 is connected to the input end of the action recognition module 6.

[0022] The image preprocessing module 3 is internally provided with an image filtering module 7 and an image enhancement module 8, and the overall output end of the image filtering module 7 and the image enhancement module 8 is connected to the input end of the image preprocessing module 3; the action recognition module 6 is internally provided with a human skeleton target detection module 9, a skeleton joint point extraction module 10 and a human skeleton tracking module 11, and the overall output end of the human skeleton target detection module 9, the skeleton joint point extraction module 10 and the human skeleton tracking module 11 is connected to the input end of the action recognition module 6.

[0023] The skeleton signal acquisition module 2 uses the EZVIZ network camera of Hikvision as the image acquisition device terminal. The terminal side of the skeleton signal acquisition module 2 uses the Python program to complete the acquisition of the image source and the request to the server. First, to obtain the image data of the network camera, the specific IP address and port number Port, user name user and password password of the network camera are required, and then the VideoCapture method in OpenCV is used according to the Real-Time Streaming Protocol (RTSP) to obtain the video stream image of the camera.

[0024] The image filtering module 7 uses a bilateral filtering method to perform image denoising. Bilateral filtering processes pixel information nonlinearly. While taking the coordinates of the pixel points in the image into consideration, it also adds the difference influence of the pixels in the surrounding area into the calculation range. This has a similar effect to Gaussian filtering for the processing of the grayscale flat area, while no processing is performed in the abnormal area, thereby maintaining the edge information.

[0025] The image enhancement module 8 converts the original grayscale range of the image into a new grayscale range by adjusting the scale factor for image enhancement. When the contrast of a certain area in the skeleton action image is low, a stretching function with a slope greater than 1 is used for image enhancement; when the regional contrast is high, a stretching function with a slope less than 1 is used for enhancement, thereby optimizing the visual effect of the image and improving the visibility of the low-contrast area. The region extraction module (12) performs Gaussian smoothing filtering on the image to remove noise. Then, the gradient amplitude and azimuth of the image are calculated, and non-maximum suppression (NMS) is applied to retain the points with the largest local changes. Next, a dual threshold method is used for edge detection. A high threshold is used to obtain fewer false edges, and a low threshold is used to connect the disconnected edges, ultimately achieving edge closure and accurate extraction.

[0026] The receiving thread module 4 obtains real-time video data through the IP address and port number of the network camera. The camera video source path is h264 / ch1 / main / av_stream. This path is used to obtain the video stream data captured by the camera. In order to perform image preprocessing, a Camloader class is constructed and an independent thread is started to process the image data. Considering that there may be differences in the processing capabilities of the client and the server, that is, there is a producer-consumer problem, a cache queue is used to coordinate the processing speed of upstream and downstream data, and the image data is placed in the cache queue frame by frame according to the timing.

[0027] Processor module 5 deploys the processor to the cloud server through Nginx and uWSGI servers. The entire deployment process is as follows: First, connect the network camera to the Internet, use the IP viewing tool to obtain the IP address of the camera, and use the Python program to obtain the video data of the camera through the RTSP protocol and test it; then, remotely connect to the cloud server through the SSH protocol (Xshell, Xftp and other tools can be used), and transfer the Flask project, algorithm model file and deep learning adaptation environment package in the local development environment to the cloud server; in the cloud server, use the pip command to install the dependency packages required by the Flask application and create a virtual environment to ensure that the Flask built-in server can run normally; next, use the aptinstall command to install the Nginx server component, and use the pip command to install uWSGI and the process manager Supervisor; according to the Flask project configuration, set Nginx, uWSGI and Supervisor, and start the service to start the activity detection service; on the image acquisition device side, send the image data collected by the camera to the cloud server in real time through the Python program; the client accesses the URL address of the cloud server through the browser to obtain the activity detection result image processed by the server, and completes the cloud deployment of the entire activity detection algorithm.

[0028] The human skeleton target detection module 9 adopts an instance segmentation-based target detection method, which is different from the traditional anchor-based method. The traditional method requires the design and adjustment of anchor frames for each data set, which is complex and time-consuming. The instance segmentation-based method does not require anchor frame configuration, avoids parameter tuning problems, reduces a large number of matching calculations, and reduces memory consumption. VFNet is the core model of this module. As a fully convolutional network, it can directly perform classification and regression prediction without relying on anchor frames, and can achieve detection results comparable to traditional anchor frame methods. Due to its high efficiency and accuracy, VFNet performs well in processing multi-person human target detection. The accuracy of human skeleton target detection directly affects the performance of the subsequent skeleton key point extraction module (10). If the detection result is inaccurate, the extracted skeleton information will also be affected, which in turn affects the performance of the entire action recognition system. To ensure the accuracy of the model, the NTU-60 dataset was used for training. This dataset is designed specifically for action recognition and contains 56,000 video samples, covering 60 action categories, and annotated with 25 joint points. Each joint point corresponds to a specific part of the human body, such as the head, shoulders, and knees, and is suitable for skeleton posture estimation and action recognition tasks.

[0029] The skeleton joint extraction module 10 adopts a top-down multi-person posture estimation algorithm to extract the human skeleton, and uses a high-resolution network as a human joint extractor; the skeleton joint extraction module 10 uses a joint regression method of a regression heat map to locate the joint coordinates. This method uses a multi-stage cascade convolutional neural network to perform sufficient feature extraction and multi-scale fusion operations to extract features for each joint on the human body, and then predicts the position information of each joint in the form of several heat maps. The closer the position is to the center in the heat map, the greater the probability that it is the actual joint position. Finally, the specific coordinates of the joint are located from the predicted heat map according to the probability response. The skeleton joint extraction module 10 uses the HRNet network to extract the joint coordinate position by first extracting features through a basic convolutional network to form the most original high-resolution feature map, and the high-resolution features are maintained until the last layer of the network. Finally, the regression heat map method is used to estimate the joint position and obtain the joint coordinate information; the convolutional neural network part is the HRNet network used in this section. This method of first predicting a regression heat map for each joint point separately and then extracting the joint point position based on the probability response relies on the joint point feature extraction capability of the convolutional neural network. The predicted regression heat map incorporates rich multi-scale feature information, which enhances the robustness of the network model in joint point detection in the target two-dimensional posture estimation. This method of locating joint points has better performance.

[0030] When processing video sequences, the human skeleton tracking module 11 first uses a human target detector to detect targets in each frame, and then uses a multi-target tracking algorithm to associate the same target between different frames to achieve the purpose of tracking. The video sequence is used as the input of the human target detector, which continuously detects multiple target objects in each frame, and then uses different feature distance metrics to ensure that the same human body is matched and associated between adjacent frames, so that it forms a continuous and stable track Track between the appearance and disappearance of the video screen, and finally achieves the effect of multi-target tracking; The human skeleton tracking module 11 is divided into three parts during the recognition and tracking process: state estimation, matching problem, and trajectory management. For the state estimation part, the DeepSORT algorithm uses the Kalman filter algorithm, which is a filter algorithm with constant speed motion and linear observation model. The algorithm only needs to know the error covariance of the prediction result of the previous state and the current state observation value to complete the prediction of the next state. The Kalman filter algorithm predicts the trajectory position of the target on the image frame sequence. In any given video frame, the state of the target is modeled as an 8-dimensional space vector x: , where (u, v) corresponds to the center coordinates of the target detection bbox bounding box, r represents the aspect ratio of the bounding box, and ℎ represents the height of the bounding box. is the speed corresponding to the first four parameters in the image (i.e., the rate of change between frames). The Kalman filter algorithm uses x as the direct observation model of the target state and predicts the updated trajectory state as .

[0031] For the matching problem part of the human skeleton tracking module 11, it is necessary to match the target detection box Detectionbbox and the predicted target box Predictionbbox of the Kalman filter one by one. The DeepSORT algorithm uses the Hungarian algorithm to solve it. The detection object Detectionbbox of each frame is matched with the Kalman filter prediction value Predictionbbox of the confirmed state Track track in the tracking track set, and the object of the Hungarian algorithm used is the cost matrix, which is obtained by weighting the covariance distance Mahalanobis distance and cosine distance.

[0032] For the trajectory management part of the human skeleton tracking module 11, DeepSORT sets each tracking trajectory to two states: confirmed state and unconfirmed state. For targets that cannot be matched with any trajectory in the trajectory set, they are considered to be newly detected objects, and will be assigned to a trajectory Track and initialized to an unconfirmed state. Each trajectory contains a value age, which is used to record the frame difference between the current frame and the most recent successfully matched frame. The frame difference can be used to determine whether the trajectory life cycle has expired. Here, it is assumed that the trajectory life cycle is max_age. The value age is reset to zero when the Detection bbox and the Prediction bbox are successfully matched, and increases by 1 when the Kalman filter algorithm predicts. If the tracking trajectory Track successfully matches for 3 consecutive frames, its state will be replaced with a confirmed state. On the contrary, if any of the first 3 frames fails to match, the target Detection will be regarded as a misjudgment and deleted. If the trajectory is not successfully matched within the max_age frame, that is, age>max_age, or if the match fails in the unconfirmed state, the trajectory will be deleted from the track set Track Set.

[0033] In the present invention, the HRNet algorithm module inside the action recognition module 6 is divided into multiple stages, and branches of different resolutions in each stage run in parallel. A cross-resolution fusion operation is required between feature mapping branches of two different resolutions. Upsampling is used to convert low resolution into high resolution and merge it with high resolution, while downsampling is used to convert high resolution into low resolution and merge it with low resolution. This multi-scale feature fusion method of cross-resolution information interaction realizes semantic fusion, adds rich semantics on the basis of the original spatial information accuracy, thereby improving the speed of image processing and recognition, and bringing more convenience to people's use.

[0034] In the present invention, the action recognition module 6 adopts the CTR-GCN model to add a granularity division module, a cross-attention module and a feature fusion module, and improves the recognition performance through the interactive fusion of fine-grained and coarse-grained features. Fine-grained features retain the precise action details at the joint level, while coarse-grained features capture regional or overall motion patterns through node aggregation. Information is exchanged between features of different granularities through attention mechanisms and feature weighting methods, and fine-grained features are used to guide the generation of coarse-grained features, while coarse-grained features provide global information supplements in reverse. This fusion method of multi-granular features realizes the dynamic unification of details and the global, enhances the robustness and recognition accuracy of the model while retaining fine action information, thereby significantly improving the speed and effect of action recognition, and providing users with a more efficient user experience.

[0035] For the granularity division part, the action recognition module 6 combines the average division and the convolution division to aggregate the 25 fine-grained nodes in the original skeleton data into 10 coarse-grained nodes. The schematic diagram of the fine-grained skeleton joints is shown in Figure 1 As shown, the schematic diagram of the coarse-grained skeleton joints is as follows Figure 2 This process combines the averaging operation and the convolution operation, which not only retains the diversity of fine-grained information, but also captures more complex local dynamic relationships through the feature learning ability of convolution. For example, the basic node features are generated through the averaging operation, and then the deep features are further extracted by convolution, so that the coarse-grained nodes can represent more complex motion patterns.

[0036] The average operation is performed by dividing the 25 nodes into 10 sets, and averaging the m nodes in the set in the V dimension to obtain Vavg. The calculation formula is: .

[0037] The convolution operation is to operate on the fine-grained skeleton nodes, that is, the information containing 25 skeleton nodes, that is, to merge multiple nodes in a set in the node dimension, and the merged new tensor has a node dimension of m, and then perform a convolution operation on the node dimension, so that its node dimension changes from m to 1, thereby completing the merging of the node, and similarly perform convolution operations on the nodes in other sets, and finally generate coarse-grained skeleton node information containing 10 skeleton nodes. The specific steps are as follows: Figure 3 shown.

[0038] For the cross-attention part, the action recognition module 6 receives feature representations from different granularities, namely the coarse-grained feature vector X and the fine-grained feature vector Y, and generates a fused feature X_att by calculating the attention weight. The module adopts a multi-head attention mechanism, and the input features are calculated and weighted by several attention heads respectively. The fused features obtained by summarizing can more comprehensively represent the original information. The schematic diagram of the cross-attention module is shown in Figure 4shown.

[0039] The attention mechanism realizes the dynamic fusion of multi-granularity information by explicitly establishing the correlation between features of different granularity levels. This mechanism can not only effectively capture the skeleton features of different granularity levels, but also further optimize the feature fusion process through interactive weight adjustment.

[0040] To verify the effectiveness of the action recognition module 6 of the present invention, we conducted experiments on the NTU RGB+D dataset. This dataset contains 56,880 video clips, covering 60 types of daily and interactive actions, and is a commonly used benchmark dataset in the field of action recognition. The Top-1 accuracy is used as the evaluation indicator, and compared with mainstream algorithms such as ST-GCN and 2s-AGCN. The experimental results are shown in Tables 1 and 2:

[0041] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0042] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A skeleton action recognition method and detection system thereof, comprising a power supply module (1), a skeleton signal acquisition module (2), an image preprocessing module (3), a receiving thread module (4), a processor module (5), an action recognition module (6), an image filtering module (7), an image enhancement module (8), a human skeleton target detection module (9), and a skeleton joint point extraction module (10), characterized in that: The output end of the power supply module (1) is connected to the input end of the skeleton signal acquisition module (2), the output end of the skeleton signal acquisition module (2) is connected to the input end of the image preprocessing module (3), the output end of the image preprocessing module (3) is connected to the input end of the receiving thread module (4), the output end of the receiving thread module (4) is connected to the input end of the processor module (5), and the output end of the processor module (5) is connected to the input end of the action recognition module (6).

2. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: The image preprocessing module (3) is provided with an image filtering module (7) and an image enhancement module (8) inside, and the overall output end of the image filtering module (7) and the image enhancement module (8) is connected to the input end of the image preprocessing module (3); the action recognition module (6) is provided with a human skeleton target detection module (9), a skeleton joint point extraction module (10) and a human skeleton tracking module (11) inside, and the overall output end of the human skeleton target detection module (9), the skeleton joint point extraction module (10) and the human skeleton tracking module (11) is connected to the input end of the action recognition module (6).

3. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: The skeleton signal acquisition module (2) uses the EZVIZ network camera of Hikvision as an image acquisition device terminal. The terminal side of the skeleton signal acquisition module (2) uses a Python program to complete the acquisition of image sources and the request to the server. First, to obtain the image data of the network camera, the specific IP address and port number Port, user name user and password password of the network camera are required. Then, the VideoCapture method in OpenCV is used to obtain the video stream image of the camera according to the Real Time Streaming Protocol (RTSP).

4. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: The image filtering module (7) uses a bilateral filtering method to perform image denoising. The processing of pixel information by bilateral filtering is nonlinear. While taking the coordinates of the pixel point in the image into consideration, the difference influence of the pixel points in the surrounding area is also added to the calculation range. In this way, the processing of the grayscale flat area has an effect similar to that of Gaussian filtering, while no processing is performed on the abnormal area, thereby maintaining the edge information.

5. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: The image enhancement module (8) converts the original grayscale range of the image into a new grayscale range by adjusting the scale factor to perform image enhancement. When the contrast of a certain area in the skeleton action image is low, a stretching function with a slope greater than 1 is used for image enhancement; when the regional contrast is high, a stretching function with a slope less than 1 is used for enhancement, thereby optimizing the visual effect of the image and improving the visibility of the low-contrast area. The region extraction module (12) performs Gaussian smoothing filtering on the image to remove noise. Then, the gradient amplitude and azimuth of the image are calculated, and non-maximum suppression (NMS) is applied to retain the point with the largest local change. Next, a dual threshold method is used for edge detection, and fewer false edges are obtained by a high threshold, and disconnected edges are connected by a low threshold, ultimately achieving edge closure and accurate extraction.

6. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: The receiving thread module (4) obtains real-time video data through the IP address and port number of the network camera, and the camera video source path is h264 / ch1 / main / av_stream. This path is used to obtain the video stream data captured by the camera. In order to perform image preprocessing, a Camloader class is constructed and an independent thread is started to process the image data. Considering that there may be differences in the processing capabilities of the client and the server, that is, there is a producer-consumer problem, a cache queue is used to coordinate the processing speed of upstream and downstream data, and the image data is placed in the cache queue frame by frame according to the timing.

7. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: The processor module (5) deploys the processor to the cloud server through Nginx and uWSGI servers. The entire deployment process is as follows: first, connect the network camera to the Internet, use the IP viewing tool to obtain the IP address of the camera, and use the Python program to obtain the video data of the camera through the RTSP protocol and perform a test; then, remotely connect to the cloud server through the SSH protocol (Xshell, Xftp and other tools can be used), and transfer the Flask project, algorithm model file and deep learning adaptation environment package in the local development environment to the cloud server; In the cloud server, use the pip command to install the dependency packages required by the Flask application and create a virtual environment to ensure that the Flask built-in server can run normally. Next, use the apt install command to install the Nginx server component, and use the pip command to install uWSGI and the process manager Supervisor. According to the Flask project configuration, set up Nginx, uWSGI, and Supervisor, and start the service, and start the activity detection service. On the image acquisition device side, use the Python program to send the image data collected by the camera to the cloud server in real time. The client accesses the URL address of the cloud server through a browser, obtains the activity detection result image processed by the server, and completes the cloud deployment of the entire activity detection algorithm.

8. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: The human skeleton target detection module (9) adopts an instance segmentation-based target detection method, which is different from the traditional anchor-based method. The traditional method requires the design and adjustment of anchor frames for each data set, which is complex and time-consuming. The instance segmentation-based method does not require anchor frame configuration, avoids parameter tuning problems, reduces a large amount of matching calculations, and reduces memory consumption. VFNet is the core model of this module. As a fully convolutional network, it can directly perform classification and regression prediction without relying on anchor frames, and can achieve detection results comparable to those of traditional anchor frame methods. Due to its high efficiency and accuracy, VFNet performs well in processing multi-person human target detection. The accuracy of human skeleton target detection directly affects the performance of the subsequent skeleton key point extraction module (10). If the detection result is inaccurate, the extracted skeleton information will also be affected, thereby affecting the performance of the entire action recognition system. In order to ensure the accuracy of the model, the NTU-60 dataset was used for training. This dataset is designed specifically for action recognition and contains 56,000 video samples, covering 60 action categories, and annotated with 25 joint points. Each joint point corresponds to a specific part of the human body, such as the head, shoulders, and knees, and is suitable for skeleton posture estimation and action recognition tasks.

9. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: The skeleton joint point extraction module (10) uses a high-resolution network (HRNet) as a feature extractor to extract the joint point coordinates and confidence information of the human skeleton from each detection frame. The high-resolution network can achieve high-precision extraction of joint point information under complex backgrounds by maintaining multi-scale feature expression capabilities, thereby providing higher quality input data for action recognition.

10. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: When processing a video sequence, the human skeleton tracking module (11) detects the targets in consecutive frames frame by frame through a human target detector, and combines a multi-target tracking algorithm to associate the detection results of the same human body in different frames to generate a continuous and stable skeleton trajectory. Specifically, a matching strategy between the detection frame and the skeleton joint point features is used, such as a matching algorithm based on feature distance or a recursive optimization method, to ensure that each target in the video remains consistent. The module can also handle complex scenes such as target occlusion, appearance or disappearance, to ensure the reliability of the tracking results.

11. A skeleton action recognition method and detection system as claimed in claim 1, characterized in that: The action recognition module (6) adopts an improved multi-granularity attention interactive skeleton action recognition model for skeleton action recognition. The model is optimized based on CTR-GCN (graph convolutional network) and integrates skeleton features from different granularity levels through a multi-granularity attention mechanism to achieve accurate recognition of skeleton actions.

Images