Motion recognition method and device based on human skeleton conversion point cloud, terminal and medium

Abstract

The present application relates to the technical field of human behavior recognition, and particularly relates to a motion recognition method and device based on human skeleton conversion point cloud, a terminal and a medium.The motion recognition method based on human skeleton conversion point cloud comprises: performing time embedding and space embedding on each joint node in a human skeleton sequence to obtain time sequence position encoding and joint topology encoding of each joint node; fusing joint nodes of different frames in the human skeleton sequence which have been time embedded and space embedded according to a time axis to form an ordered skeleton point cloud; converting the skeleton point cloud into a high-dimensional skeleton point cloud based on a linear embedding mode; and inputting the high-dimensional skeleton point cloud into a trained motion classification model to obtain a motion recognition result.The present application converts a skeleton sequence into a point cloud, constructs a skeleton point cloud mode with time sequence dynamics and human topology characteristics, effectively retains the integrity of the skeleton structure and the continuity of the motion, and improves the motion recognition accuracy and efficiency in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of human behavior recognition technology, and in particular to a method, device, terminal and medium for action recognition based on human skeleton conversion point cloud. Background Technology

[0002] Human behavior recognition refers to the technology of automatically analyzing and understanding human actions and behaviors based on video or other sensor data. It is one of the important research directions in the fields of computer vision and artificial intelligence. This technology can be applied to applications such as public safety monitoring, autonomous driving and robot navigation, video content retrieval, human-computer interaction, and intelligent entertainment.

[0003] To improve the performance of human behavior recognition, existing research has explored various data modalities, including RGB images, depth maps, human skeleton sequences, and 3D point cloud data. Among these, human skeletons and point clouds, as typical 3D structural representations, have attracted widespread attention due to their ability to directly model spatial information.

[0004] Skeleton data typically consists of the three-dimensional coordinates of key human joints and their connections. It has advantages such as small data volume, clear structure, and insensitivity to changes in background and lighting, making it suitable for describing human movement patterns. However, existing skeleton-based behavior recognition methods mostly rely on predefined human topology and focus on modeling the connections between joints. They often fail to fully depict the complex spatial geometric relationships in human movements, especially the interaction features between non-topologically related joints, thus limiting their ability to perform fine-grained motion recognition and model complex poses.

[0005] Point cloud data, composed of a large number of discrete three-dimensional points, can relatively completely reflect the overall shape and local geometric structure of the human body in space, making it suitable for global and local spatial modeling. While point cloud-based behavior recognition methods have advantages in geometric representation, point clouds themselves are characterized by disorder, weak semantic information, and high computational cost, making it difficult to directly depict the functional relationships and temporal constraints between human joints, resulting in high modeling complexity.

[0006] Therefore, existing technologies have shortcomings and need to be improved and developed. Summary of the Invention

[0007] This application provides a method, device, terminal, and medium for action recognition based on human skeleton conversion point cloud, in order to solve the technical problem in related technologies that it is difficult to model complex postures and the modeling complexity is high, which leads to low accuracy and efficiency of action recognition results in complex scenes.

[0008] To achieve the above objectives, this application adopts the following technical solution: An action recognition method based on human skeleton converted point cloud, wherein the method includes: Obtain the human skeleton sequence to be identified, and perform temporal embedding and spatial embedding on each joint point in the human skeleton sequence to obtain the temporal position code and joint topology code corresponding to each joint point; The joints of different frames in the human skeleton sequence, which are embedded temporally and spatially, are fused along the time axis to form an ordered skeleton point cloud. Each joint in the skeleton point cloud includes three-dimensional spatial coordinates, temporal position encoding, and joint topology encoding. The skeleton point cloud is transformed into a high-dimensional skeleton point cloud based on linear embedding. The high-dimensional skeleton point cloud is input into the trained action classification model to obtain the action recognition result.

[0009] In one embodiment of this application, a human skeleton sequence to be identified is obtained, and temporal and spatial embedding are performed on each joint point in the human skeleton sequence to obtain the temporal position code and joint topology code corresponding to each joint point, including: A human skeleton sequence to be identified is obtained. A normalized time sequence number is added to each joint point in the human skeleton sequence as a time channel, and a normalized spatial sequence number is added as a spatial channel to obtain the temporal position code and joint topology code corresponding to each joint point.

[0010] In one embodiment of this application, converting the skeleton point cloud into a high-dimensional skeleton point cloud based on linear embedding includes: The original feature dimension of the skeleton point cloud is expanded to a predetermined dimension by a learnable linear transformation layer to obtain a high-dimensional skeleton point cloud.

[0011] In one embodiment of this application, the high-dimensional skeleton point cloud is input into a trained action classification model to obtain action recognition results, including: Obtain a trained action classification model, which includes a point cloud backbone network. The point cloud backbone network adopts a hierarchical structure and is composed of multiple layers stacked sequentially. The high-dimensional skeleton point cloud is input into the action classification model. Based on the hierarchical progressive structure of the point cloud backbone network, layer-by-layer downsampling and point cloud fully connected feature aggregation are performed to obtain deep aggregation features. Based on the deep aggregation features, the action recognition result is obtained.

[0012] In one embodiment of this application, layer-by-layer downsampling and point cloud fully connected feature aggregation are performed based on the hierarchical structure of the point cloud backbone network to obtain deep aggregated features, including: In each level of the hierarchical structure, the skeleton point cloud input at each level is downsampled based on the topological prior of the human skeleton to obtain downsampled points. Perform a K-nearest neighbor operation on each downsampled point to extract the features of the neighboring points of the downsampled point; Based on the point cloud fully connected network, the features of the neighboring points are aggregated to obtain deep aggregated features at each level.

[0013] In one embodiment of this application, feature aggregation is performed on the features of the neighboring points based on a fully connected point cloud network to obtain deep aggregated features at each level, including: After the neighbor point features are input into the first fully connected residual module for processing, the processed neighbor point features are obtained. The max pooling function is used to aggregate the features of the processed neighboring points to obtain aggregated features; Deep aggregation features are extracted from the aggregation features based on the second fully connected residual module; The first fully connected residual module and the second fully connected residual module each include several residual point blocks. Each residual point block includes two repeating units. Each unit includes a fully connected layer, a batch normalization layer, and a ReLU activation layer. Each residual point block adopts residual skip connections.

[0014] This application also provides an action recognition device based on human skeleton converted point cloud, wherein the device includes: The embedding module is used to acquire the human skeleton sequence to be identified, and to perform temporal embedding and spatial embedding on each joint point in the human skeleton sequence to obtain the temporal position code and joint topology code corresponding to each joint point. The fusion module is used to fuse the joints of different frames in the human skeleton sequence that have been embedded temporally and spatially according to the time axis to form an ordered skeleton point cloud. Each joint in the skeleton point cloud includes three-dimensional spatial coordinates, temporal position encoding and joint topology encoding. The conversion module is used to convert the skeleton point cloud into a high-dimensional skeleton point cloud based on linear embedding. The recognition module is used to input the high-dimensional skeleton point cloud into the trained action classification model to obtain the action recognition result.

[0015] This application also provides a terminal, which includes: a memory, a processor, and a motion recognition program based on human skeleton conversion point cloud stored in the memory and executable on the processor. When the motion recognition program based on human skeleton conversion point cloud is executed by the processor, it implements the steps of the motion recognition method based on human skeleton conversion point cloud as described above.

[0016] This application also provides a computer-readable storage medium storing a computer program that can be executed to implement the steps of the action recognition method based on human skeleton conversion point cloud as described above.

[0017] The beneficial effects of this invention are as follows: The method of this embodiment acquires a human skeleton sequence to be identified, performs temporal and spatial embedding on each joint point in the human skeleton sequence to obtain the temporal position code and joint topology code corresponding to each joint point; the joint points of different frames in the human skeleton sequence that have undergone temporal and spatial embedding are fused along the time axis to form an ordered skeleton point cloud, where each joint point in the skeleton point cloud includes three-dimensional spatial coordinates, temporal position code, and joint topology code; the skeleton point cloud is converted into a high-dimensional skeleton point cloud based on linear embedding; the high-dimensional skeleton point cloud is input into a trained action classification model to obtain action recognition results. This application, by converting the skeleton sequence into a point cloud, constructs a skeleton point cloud modality that combines temporal dynamics and human topological characteristics, effectively preserving the integrity of the skeleton structure and the continuity of actions, and improving the accuracy and efficiency of action recognition results in complex scenes. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the action recognition method based on human skeleton conversion point cloud in this invention.

[0019] Figure 2 This is a schematic diagram of the skeleton sequence being merged into a pseudo-point cloud frame by frame in the action recognition method based on human skeleton conversion point cloud in this invention.

[0020] Figure 3 This is a schematic diagram illustrating the logical principle of the skeleton-point cloud capture and switching module in the action recognition method based on human skeleton-point cloud conversion in this invention.

[0021] Figure 4 This is a framework diagram of the fully connected network of the point cloud in the action recognition method based on human skeleton conversion point cloud in this invention.

[0022] Figure 5 This is a framework diagram of the fully connected network of the point cloud in the action recognition method based on human skeleton conversion point cloud in this invention.

[0023] Figure 6 This is a functional principle block diagram of a preferred embodiment of the motion recognition device based on human skeleton conversion point cloud in this invention.

[0024] Figure 7 This is a functional principle block diagram of a preferred embodiment of the terminal in this invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0026] In existing technologies, skeleton modeling and point cloud modeling are generally considered two relatively independent research directions. Some multimodal methods attempt to simultaneously introduce skeletons and point clouds as inputs, achieving information complementarity through feature-level or decision-level fusion. However, such methods often require the construction of complex multi-branch network structures and rely on the synchronous acquisition and alignment of multimodal data, resulting in high system complexity and strict requirements for data quality and acquisition conditions, which is not conducive to practical deployment and widespread application.

[0027] Therefore, the problems with existing technologies include: First, the skeleton method has limited geometric modeling capabilities and is difficult to fully represent the spatial relationships in human movements; second, the point cloud method lacks clear semantic and topological constraints, resulting in low modeling efficiency; third, existing multimodal fusion schemes are complex in structure and costly, making it difficult to achieve a unified and efficient behavior recognition framework.

[0028] The overall process of this application embodiment includes: skeleton sequence → skeleton point cloud conversion (forming skeleton point cloud) → point cloud network. While maintaining the efficiency of skeleton data, it introduces the advantages of point cloud geometric modeling to achieve human behavior recognition with unified modal modeling, thereby overcoming the shortcomings of the above-mentioned prior art.

[0029] The following describes, with reference to the accompanying drawings, an action recognition method, apparatus, terminal, and medium based on human skeleton-converted point cloud embodiments of this application. Addressing the problem mentioned in the background art that the related technologies suffer from difficulty in modeling complex postures and high modeling complexity, leading to low accuracy and efficiency in action recognition in complex scenes, this application provides an action recognition method based on human skeleton-converted point cloud. In this method, a human skeleton sequence to be recognized is acquired; each joint point in the human skeleton sequence is temporally and spatially embedded to obtain a temporal position code and a joint topology code corresponding to each joint point; the joint points of different frames in the human skeleton sequence, after temporal and spatial embedding, are fused along the time axis to form an ordered skeleton point cloud. Each joint point in the skeleton point cloud includes three-dimensional spatial coordinates, a temporal position code, and a joint topology code; the skeleton point cloud is converted into a high-dimensional skeleton point cloud using linear embedding; and the high-dimensional skeleton point cloud is input into a trained action classification model to obtain the action recognition result. This application constructs a skeleton point cloud modality that combines temporal dynamics and human topological characteristics by converting skeleton sequences into point clouds. This effectively preserves the integrity of the skeleton structure and the continuity of actions, improving the accuracy and efficiency of action recognition results in complex scenes.

[0030] Please see Figure 1 The action recognition method based on human skeleton conversion point cloud described in this embodiment of the invention includes the following steps: Step S100: Obtain the human skeleton sequence to be identified, and perform temporal embedding and spatial embedding on each joint point in the human skeleton sequence to obtain the temporal position code and joint topology code corresponding to each joint point. The human skeleton sequence uses joint coordinates to represent human posture, and then connects the postures frame by frame to form time-series data. Each frame in the human skeleton sequence is a static skeleton diagram. Arranging multiple consecutive frames in time forms the human skeleton sequence, such as... Figure 2 As shown.

[0031] Step S200: Merge the joints of different frames in the human skeleton sequence that have been embedded temporally and spatially according to the time axis to form an ordered skeleton point cloud. Each joint in the skeleton point cloud includes three-dimensional spatial coordinates, temporal position encoding and joint topology encoding.

[0032] This application embodiment constructs a data-driven cross-modal unified framework Pose2Point (P2P), which realizes end-to-end conversion from skeleton sequence to skeleton point cloud. It uses a pure point cloud method to model features and constructs a skeleton point cloud modality that combines temporal dynamics and human topological characteristics, effectively preserving the integrity of the skeleton structure and the continuity of movement.

[0033] To accurately convert human skeleton sequences into skeleton point clouds with minimal loss of key information, this application provides a skeleton-to-point cloud conversion module that fully considers the characteristics of both skeletons and point clouds. Point clouds, as a dense modality, typically achieve interaction between local and global features through downsampling and clustering. However, the number of joints in a single-frame skeleton is limited, and direct downsampling leads to excessively sparse points, resulting in the loss of significant information. Therefore, embodiments of this application fuse the joint data of the skeleton sequence along the time axis to form a denser skeleton point cloud, enabling comprehensive processing of the entire skeleton sequence's information.

[0034] For example, the skeleton sequence can be represented as ,in, Let T represent the set of real numbers, T be the number of frames in the skeleton diagram in the skeleton sequence, V be the number of joints in each frame of the skeleton diagram, and C (usually = 3, i.e., X, Y, Z coordinates) be the coordinate channels, such as... Figure 3 As shown.

[0035] Skeleton point clouds can be represented as ,in, This is the number of points before conversion (which can be downsampled later). This represents the feature dimension after embedding.

[0036] This application embodiment converts human skeleton sequences into skeleton point clouds to facilitate action recognition within a point cloud modeling framework. This allows the originally structured skeleton data to participate in strong geometric modeling in a point cloud modality, while preserving the temporal and topological semantic information of the skeleton, thus balancing efficiency and expressive power.

[0037] In one embodiment of this application, step S100 specifically involves: obtaining a human skeleton sequence to be identified, adding a normalized time sequence number as a time channel and a normalized spatial sequence number as a spatial channel to each joint point in the human skeleton sequence, so as to obtain the temporal position code and joint topology code corresponding to each joint point.

[0038] For example, the skeleton point cloud formed by directly stacking skeleton sequences is disordered and cannot maintain the temporal order and motion continuity between frames. To address this problem, before merging joints from different frames, this invention first processes the skeleton sequence... Each joint in the skeleton sequence is embedded temporally and spatially. For the t-th frame and the i-th joint... By adding normalized time serial numbers As a time channel and normalized spatial index As a space passage To achieve enhanced joints : .

[0039] in, This normalizes the time sequence number to [-0.5, 0.5]. This normalizes the spatial index to [-0.5, 0.5].

[0040] In addition to normalized timing indices, trigonometric function coding, such as sine-cosine timing coding and frequency scaling trigonometric function coding, can also be used, such as timing embedding layers and spatiotemporal joint learnable coding.

[0041] In this way, the temporal channel preserves frame position information, which helps to restore the action sequence; the spatial channel encodes the joint topological position, which helps to distinguish different joint roles and effectively enriches the features of the skeleton point cloud, so that it not only contains the three-dimensional position information of the joints, but also temporal and topological information, thereby helping the subsequent model to better capture the coherence and trend of the action.

[0042] like Figure 1 As shown, the action recognition method based on human skeleton conversion point cloud further includes the following steps: Step S300: Convert the skeleton point cloud into a high-dimensional skeleton point cloud based on linear embedding.

[0043] For example, the skeleton point cloud is an ordered skeleton point cloud formed by fusing all joints after temporal and spatial embedding processing along the time axis, represented as: .

[0044] That is, the skeleton point cloud contains Each of the three-dimensional points contains, in addition to its three-dimensional spatial coordinates, normalized temporal position codes and joint topological codes, forming... 3D feature vectors.

[0045] In this embodiment of the application, step S300 specifically involves: expanding the original feature dimension of the skeleton point cloud to a predetermined dimension through a learnable linear transformation layer to obtain a high-dimensional skeleton point cloud.

[0046] For example, the present invention targets skeleton point clouds. A linear embedding enhancement module was designed to expand the original feature dimension to a learnable linear transformation layer. Dimension, represented as: ; in, The function represents the linear layer embedding function. The linear embedding enhancement module effectively improves feature representation through nonlinear mapping in high-dimensional space, providing richer feature inputs for subsequent networks and facilitating global and local geometric modeling.

[0047] This application embodiment, through a point cloud construction strategy that integrates spatiotemporal coding, not only fully preserves the dynamic evolution characteristics of the skeleton sequence, but also fully leverages the advantages of point cloud networks in modeling complex geometric relationships, thereby realizing feature mapping from raw skeleton data to high-dimensional skeleton point clouds.

[0048] like Figure 1 As shown, the action recognition method based on human skeleton conversion point cloud further includes the following steps: Step S400: Input the high-dimensional skeleton point cloud into the trained action classification model to obtain the action recognition result.

[0049] In this embodiment of the application, step S400 specifically includes: Step S410: Obtain the trained action classification model, which includes a point cloud backbone network. The point cloud backbone network adopts a hierarchical structure and is composed of multiple layers stacked sequentially. Step S420: Input the high-dimensional skeleton point cloud into the action classification model, perform layer-by-layer downsampling and point cloud fully connected feature aggregation based on the hierarchical progressive structure of the point cloud backbone network to obtain deep aggregated features, and obtain the action recognition result based on the deep aggregated features.

[0050] For example, a high-dimensional skeleton point cloud is input into a point cloud backbone network for further modeling, and the modeled features are used for action classification. The point cloud backbone network adopts a hierarchical structure, consisting of n P2P modules, with each P2P module representing a level. Each P2P module contains two core components: a downsampling module and a point cloud fully connected (MLP) feature extraction module.

[0051] This application's embodiments, through layer-by-layer downsampling and feature abstraction, achieve a gradual refinement from local geometry to global action semantics. This results in a progressively expanding receptive field, high computational efficiency, strong multi-scale feature representation, and good robustness, enabling efficient and accurate completion of point cloud action recognition tasks, such as... Figure 4 As shown.

[0052] In one embodiment of this application, the step "performing layer-by-layer downsampling and point cloud fully connected feature aggregation based on the hierarchical structure of the point cloud backbone network to obtain deep aggregated features" specifically includes: In each level of the hierarchical structure, the skeleton point cloud input at each level is downsampled based on the topological prior of the human skeleton to obtain downsampled points. Perform a K-nearest neighbor operation on each downsampled point to extract the features of the neighboring points of the downsampled point; Based on the point cloud fully connected network, the features of the neighboring points are aggregated to obtain deep aggregated features at each level.

[0053] Among them, the topological prior of the human skeleton is based on a fixed hierarchical graph structure prior of human anatomy and kinematics, which can be pre-stored and dynamically constructed in some scenarios.

[0054] Specifically, in the first In the processing of the layer P2P module, let the input skeleton point cloud be... The first level takes a high-dimensional skeleton point cloud as input, the second level takes a downsampled point cloud as input from the first level, the third level takes a point cloud as input from the second level, and so on, until the top level is reached. Downsampling is performed using the topological prior of the human skeleton to obtain the downsampled point cloud representation. Next, a K-Nearest Neighbor (KNN) operation is performed on each sampling point to obtain the features of neighboring points. Let i represent the i-th joint and j represent the j-th neighboring point. The features of the neighboring points are then processed using a fully connected point cloud network (PointMLP, framework diagram as shown below). Figure 2The modeling module shown performs feature aggregation modeling and aggregates through max pooling to capture local structure.

[0055] In this embodiment of the application, the step "aggregating the features of the neighboring points based on the fully connected point cloud network to obtain deep aggregated features at each level" specifically includes: After the neighbor point features are input into the first fully connected residual module for processing, the processed neighbor point features are obtained. The max pooling function is used to aggregate the features of the processed neighboring points to obtain aggregated features; Deep aggregation features are extracted from the aggregation features based on the second fully connected residual module; The first fully connected residual module and the second fully connected residual module each include several residual point blocks. Each residual point block includes two repeating units. Each unit includes a fully connected layer, a batch normalization layer, and a ReLU activation layer. Each residual point block adopts residual skip connections.

[0056] Specifically, such as Figure 5 As shown, in the fully connected point cloud network, geometric affine transformation is first performed on the features of neighboring points. The geometric affine transformation module is an existing technology; its function is to normalize and perform affine transformation on the features of neighboring points to alleviate the modeling difficulties caused by the sparse and irregular local geometric structures. Inputting the 3D point cloud (x, y, z) from the neighboring point features, the geometric affine transformation module first groups its local neighboring points based on a certain center point, then calculates the offset of the neighboring points relative to the center point, and calculates the normalization coefficient σ based on the feature differences of all local neighbors. Subsequently, the centered features are divided by σ, and then scaled and translated channel-by-channel using learnable parameters α and β to obtain the transformed local features. This process improves the stability and robustness of feature representation while maintaining the original geometric relationships. Then, the geometrically affine-transformed neighboring point features are input into the first fully connected residual module (i.e., the previous fully connected residual module). Learn local features, then use the max pooling function. Feature aggregation is performed, and finally the result is passed through the second fully connected residual module (i.e., the later fully connected residual module). Extracting deep aggregation features Both the first and second fully connected residual modules include N residual blocks. Each residual block consists of two repeating units. Each unit contains a fully connected layer (MLP), a batch normalization (BN) layer, and a ReLU activation layer. Each residual block employs residual skip connections to enhance feature representation while preserving shallow features. The modeling process for deep aggregated features is represented as follows: .

[0057] Where K represents the K nearest neighbors of joint i, Indicates the first The neighboring features of the j-th neighboring point of joint i in the hierarchy.

[0058] Geometric affine transformations are linear geometric transformations that preserve the straight-line and parallel relationships, and include operations such as rotation, scaling, translation, and shearing. In point cloud action recognition, geometric affine transformations are used for point cloud pose normalization and data augmentation, improving the model's robustness to geometric transformations.

[0059] The two fully connected residual modules in this embodiment are connected across layers, allowing the gradient to be directly backpropagated to the shallow layer, thus solving the gradient vanishing problem in deep fully connected networks. Stacking two fully connected residual modules can learn more complex feature transformations. Max pooling selects the maximum value from the K features in the KNN neighborhood, which can highlight the most representative features in the neighborhood. This allows this application to ensure the depth of feature representation while taking into account computational efficiency and noise resistance.

[0060] Extensive experiments were conducted on benchmark datasets such as NTU RGB+D 60 and NTU RGB+D 120, consistently validating the effectiveness of the proposed method. Compared to existing skeleton-based modal methods, this application achieves superior human behavior recognition performance, providing a simpler and more efficient solution for skeleton-point cloud cross-modal research, and enhancing the generalization ability and robustness of human behavior modeling in complex scenarios.

[0061] Specifically, the NTU-RGB+D 120 dataset was expanded from the NTU-RGB+D 60 dataset by adding 57,367 additional skeleton sequences (covering 60 additional action categories). A total of 106 volunteers performed 1,139,945 samples across 120 categories, captured from three camera views. The dataset comprises 32 settings, each representing a specific location and background. The benchmarks for training and testing included: (1) Cross-subject (X-Sub): Data from 53 participants was used for training, and data from the remaining 53 participants was used for testing. (2) Cross-setting (X-Set): Training samples were taken from even-numbered settings, while odd-numbered settings were reserved for testing.

[0062] All experiments were conducted on four NVIDIA GeForce RTX 3070 Ti GPUs, and the skeleton-to-point cloud conversion module of this application was tested on a PointMLP (Point Cloud Fully Connected Network). The results show that removing the skeleton-to-point cloud conversion module reduces the accuracy of NTU-RGB+D 120 X-Sub by approximately 7.97%. These results fully demonstrate the importance of the skeleton-to-point cloud conversion module to the overall performance of the model, significantly improving recognition accuracy.

[0063] In one embodiment, such as Figure 6 As shown, based on the above-described action recognition method based on human skeleton converted point cloud, the present invention also provides an action recognition device based on human skeleton converted point cloud, comprising: The embedding module 100 is used to acquire the human skeleton sequence to be identified, and to perform temporal embedding and spatial embedding on each joint point in the human skeleton sequence to obtain the temporal position code and joint topology code corresponding to each joint point. The fusion module 200 is used to fuse the joints of different frames in the human skeleton sequence that have been embedded temporally and spatially according to the time axis to form an ordered skeleton point cloud. Each joint in the skeleton point cloud includes three-dimensional spatial coordinates, temporal position encoding and joint topology encoding. The conversion module 300 is used to convert the skeleton point cloud into a high-dimensional skeleton point cloud based on linear embedding. The recognition module 400 is used to input the high-dimensional skeleton point cloud into a trained action classification model to obtain action recognition results.

[0064] In one embodiment, the embedding module 100 is further configured to acquire the human skeleton sequence to be identified, add a normalized time sequence number as a time channel to each joint point in the human skeleton sequence, and add a normalized spatial sequence number as a spatial channel to obtain the temporal position code and joint topology code corresponding to each joint point.

[0065] In one embodiment, the transformation module 300 is further configured to extend the original feature dimension of the skeleton point cloud to a predetermined dimension through a learnable linear transformation layer to obtain a high-dimensional skeleton point cloud.

[0066] In one embodiment, the identification module 400 includes: The acquisition unit is used to acquire a trained action classification model, which includes a point cloud backbone network. The point cloud backbone network adopts a hierarchical structure and is composed of multiple layers stacked sequentially. The recognition unit is used to input the high-dimensional skeleton point cloud into the action classification model, perform layer-by-layer downsampling and point cloud fully connected feature aggregation based on the hierarchical progressive structure of the point cloud backbone network to obtain deep aggregated features, and obtain action recognition results based on the deep aggregated features.

[0067] In one embodiment, the recognition unit is further configured to downsample the skeleton point cloud input at each level of the hierarchical structure based on the topological prior of the human skeleton to obtain downsampled points; perform a K-nearest neighbor operation on each downsampled point to extract the neighboring point features of the downsampled point; and perform feature aggregation on the neighboring point features based on the fully connected point cloud network to obtain the deep aggregated features of each level.

[0068] In one embodiment, the identification unit is further configured to input the neighbor point features into a first fully connected residual module for processing to obtain processed neighbor point features; perform feature aggregation on the processed neighbor point features using a max pooling function to obtain aggregated features; and extract deep aggregated features from the aggregated features based on a second fully connected residual module; wherein, both the first fully connected residual module and the second fully connected residual module include several residual point blocks, each residual point block includes two repeating units, each unit includes a fully connected layer, a batch normalization layer, and a ReLU activation layer; and each residual point block employs residual skip connections.

[0069] It should be noted that the foregoing explanation of the action recognition method based on human skeleton conversion point cloud also applies to the action recognition device based on human skeleton conversion point cloud in this embodiment, and will not be repeated here.

[0070] This invention discloses an action recognition device based on human skeleton converted into point cloud. It acquires a human skeleton sequence to be recognized, performs temporal and spatial embedding on each joint point in the sequence to obtain a temporal position code and a joint topology code for each joint point. The joint points from different frames of the human skeleton sequence, after temporal and spatial embedding, are fused along the time axis to form an ordered skeleton point cloud. Each joint point in the skeleton point cloud includes three-dimensional spatial coordinates, a temporal position code, and a joint topology code. The skeleton point cloud is then converted into a high-dimensional skeleton point cloud using linear embedding. This high-dimensional skeleton point cloud is input into a trained action classification model to obtain the action recognition result. This application, by converting the skeleton sequence into a point cloud, constructs a skeleton point cloud modality that combines temporal dynamics and human topological characteristics, effectively preserving the integrity of the skeleton structure and the continuity of actions, thus improving the accuracy and efficiency of action recognition results in complex scenes.

[0071] Figure 7 A schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0072] When the processor 502 executes the program, it implements the action recognition method based on human skeleton conversion point cloud provided in the above embodiments.

[0073] Furthermore, the terminal also includes: Communication interface 503 is used for communication between memory 501 and processor 502.

[0074] The memory 501 is used to store computer programs that can run on the processor 502.

[0075] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0076] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0077] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0078] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0079] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described motion recognition method based on human skeleton conversion point cloud.

[0080] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0081] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0082] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0083] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can read and execute instructions from or in conjunction with such an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, then editing, interpreting or otherwise processing them as necessary, and then storing them in computer memory.

[0084] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0085] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0087] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for action recognition based on human skeleton converted point cloud, characterized in that, The method includes: Obtain the human skeleton sequence to be identified, and perform temporal embedding and spatial embedding on each joint point in the human skeleton sequence to obtain the temporal position code and joint topology code corresponding to each joint point; The joints of different frames in the human skeleton sequence, which are embedded temporally and spatially, are fused along the time axis to form an ordered skeleton point cloud. Each joint in the skeleton point cloud includes three-dimensional spatial coordinates, temporal position encoding, and joint topology encoding. The skeleton point cloud is transformed into a high-dimensional skeleton point cloud based on linear embedding. The high-dimensional skeleton point cloud is input into the trained action classification model to obtain the action recognition result; The high-dimensional skeleton point cloud is input into a trained action classification model to obtain action recognition results, including: Obtain a trained action classification model, which includes a point cloud backbone network. The point cloud backbone network adopts a hierarchical structure and is composed of multiple layers stacked sequentially. The high-dimensional skeleton point cloud is input into the action classification model. Based on the hierarchical progressive structure of the point cloud backbone network, layer-by-layer downsampling and point cloud fully connected feature aggregation are performed to obtain deep aggregation features. Based on the deep aggregation features, the action recognition result is obtained. Based on the hierarchical structure of the point cloud backbone network, layer-by-layer downsampling and fully connected feature aggregation of the point cloud are performed to obtain deep aggregated features, including: In each level of the hierarchical structure, the skeleton point cloud input at each level is downsampled based on the topological prior of the human skeleton to obtain downsampled points. Perform a K-nearest neighbor operation on each downsampled point to extract the features of the neighboring points of the downsampled point; Based on the point cloud fully connected network, the features of the neighboring points are aggregated to obtain deep aggregated features at each level. Based on a fully connected point cloud network, feature aggregation is performed on the features of neighboring points to obtain deep aggregated features at each level, including: After the neighbor point features are input into the first fully connected residual module for processing, the processed neighbor point features are obtained. The max pooling function is used to aggregate the features of the processed neighboring points to obtain aggregated features; Deep aggregation features are extracted from the aggregation features based on the second fully connected residual module; The first fully connected residual module and the second fully connected residual module each include several residual point blocks. Each residual point block includes two repeating units. Each unit includes a fully connected layer, a batch normalization layer, and a ReLU activation layer. Each residual point block adopts residual skip connections.

2. The action recognition method based on human skeleton converted point cloud according to claim 1, characterized in that, Obtain the human skeleton sequence to be identified, and perform temporal and spatial embedding on each joint point in the human skeleton sequence to obtain the temporal position code and joint topology code corresponding to each joint point, including: A human skeleton sequence to be identified is obtained. A normalized time sequence number is added to each joint point in the human skeleton sequence as a time channel, and a normalized spatial sequence number is added as a spatial channel to obtain the temporal position code and joint topology code corresponding to each joint point.

3. The action recognition method based on human skeleton converted point cloud according to claim 1, characterized in that, The skeleton point cloud is transformed into a high-dimensional skeleton point cloud based on linear embedding, including: The original feature dimension of the skeleton point cloud is expanded to a predetermined dimension by a learnable linear transformation layer to obtain a high-dimensional skeleton point cloud.

4. A motion recognition device based on human skeleton converted point cloud, characterized in that, The device includes: The embedding module is used to acquire the human skeleton sequence to be identified, and to perform temporal embedding and spatial embedding on each joint point in the human skeleton sequence to obtain the temporal position code and joint topology code corresponding to each joint point. The fusion module is used to fuse the joints of different frames in the human skeleton sequence that have been embedded temporally and spatially according to the time axis to form an ordered skeleton point cloud. Each joint in the skeleton point cloud includes three-dimensional spatial coordinates, temporal position encoding and joint topology encoding. The conversion module is used to convert the skeleton point cloud into a high-dimensional skeleton point cloud based on linear embedding. The recognition module is used to input the high-dimensional skeleton point cloud into the trained action classification model to obtain action recognition results; The high-dimensional skeleton point cloud is input into a trained action classification model to obtain action recognition results, including: Obtain a trained action classification model, which includes a point cloud backbone network. The point cloud backbone network adopts a hierarchical structure and is composed of multiple layers stacked sequentially. The high-dimensional skeleton point cloud is input into the action classification model. Based on the hierarchical progressive structure of the point cloud backbone network, layer-by-layer downsampling and point cloud fully connected feature aggregation are performed to obtain deep aggregation features. Based on the deep aggregation features, the action recognition result is obtained. Based on the hierarchical structure of the point cloud backbone network, layer-by-layer downsampling and fully connected feature aggregation of the point cloud are performed to obtain deep aggregated features, including: In each level of the hierarchical structure, the skeleton point cloud input at each level is downsampled based on the topological prior of the human skeleton to obtain downsampled points. Perform a K-nearest neighbor operation on each downsampled point to extract the features of the neighboring points of the downsampled point; Based on the point cloud fully connected network, the features of the neighboring points are aggregated to obtain deep aggregated features at each level. Based on a fully connected point cloud network, feature aggregation is performed on the features of neighboring points to obtain deep aggregated features at each level, including: After the neighbor point features are input into the first fully connected residual module for processing, the processed neighbor point features are obtained. The max pooling function is used to aggregate the features of the processed neighboring points to obtain aggregated features; Deep aggregation features are extracted from the aggregation features based on the second fully connected residual module; The first fully connected residual module and the second fully connected residual module each include several residual point blocks. Each residual point block includes two repeating units. Each unit includes a fully connected layer, a batch normalization layer, and a ReLU activation layer. Each residual point block adopts residual skip connections.

5. A terminal, characterized in that, include: The device includes a memory, a processor, and a motion recognition program based on human skeleton conversion point cloud stored in the memory and executable on the processor. When the motion recognition program based on human skeleton conversion point cloud is executed by the processor, it implements the steps of the motion recognition method based on human skeleton conversion point cloud as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed to implement the steps of the action recognition method based on human skeleton conversion point cloud as described in any one of claims 1 to 3.

Images