A motion recognition method, device and medium based on human bone three-dimensional data

Abstract

The application relates to a motion recognition method and device based on human body skeleton three-dimensional data, and a medium, the method comprising the following steps: acquiring real-time human body skeleton three-dimensional posture data, inputting the data into a pre-trained motion recognition model after conversion into a preset format, and acquiring a human body motion recognition result, wherein the motion recognition model comprises: a multi-feature input module for acquiring comprehensive features including Cartesian motion information, global joint size information and global joint angle information based on the human body skeleton three-dimensional posture data converted into the preset format; and a feature fusion and output module for acquiring the human body motion recognition result based on the comprehensive features. Compared with the prior art, the application has the advantages of a lighter network structure, fast operation speed, high recognition accuracy and the like.

Description

Technical Field

[0001] This invention relates to the field of human motion recognition technology, and in particular to a motion recognition method, device, and medium based on three-dimensional human skeletal data. Background Technology

[0002] In recent years, artificial intelligence, metaverse, augmented reality, and human-computer interaction technologies have developed rapidly, and human motion recognition has received increasing attention. Currently, there are many motion recognition methods, such as RGB-based methods, but the accuracy of direct recognition using RGB images is low and greatly affected by environmental interference; methods based on wearable devices, but these are expensive and have poor operability; and optical flow-based methods, but these require high ambient light intensity, making them unsuitable for practical deployment.

[0003] Existing action recognition methods have the following drawbacks:

[0004] (1) Some current skeleton-based motion recognition methods have problems such as large model size, large number of parameters, low computing speed, and inability to achieve real-time performance.

[0005] (2) Some skeleton-based motion recognition methods have fewer parameters but lower accuracy.

[0006] (3) Existing motion recognition methods cannot solve the problems of motion speed difference, viewpoint position change and object distance change in motion recognition.

[0007] In summary, there is currently a lack of action recognition methods that can address the issues of low accuracy caused by variations in motion speed, changes in viewpoint position, and changes in object distance in existing methods. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a motion recognition method, device, and medium based on three-dimensional human skeletal data. This invention addresses the speed difference problem by obtaining Cartesian motion information based on three-dimensional human skeletal posture data; addresses the viewpoint position change problem by obtaining global joint dimension information based on three-dimensional human skeletal posture data; and addresses the object distance change problem by obtaining global joint angle information based on three-dimensional human skeletal posture data. By utilizing the aforementioned Cartesian motion information, global joint dimension information, and global joint angle information for feature fusion, and obtaining the recognition result, this invention solves or partially solves the problem of low accuracy in recognition results caused by motion speed differences, viewpoint position changes, and object distance changes in existing methods.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] One aspect of the present invention provides a motion recognition method based on three-dimensional human skeletal data, comprising the following steps:

[0011] Real-time 3D pose data of the human skeleton is acquired, converted into a preset format, and then input into a pre-trained motion recognition model to obtain human motion recognition results.

[0012] The action recognition model includes:

[0013] The multi-feature input module is used to obtain comprehensive features, including Cartesian motion information, global joint size information, and global joint angle information, based on the human skeleton 3D pose data converted to a preset format.

[0014] The feature fusion and output module is used to obtain the human action recognition result based on the comprehensive features.

[0015] As a preferred technical solution, the acquisition of Cartesian motion information includes the following steps:

[0016] Based on the human skeleton 3D pose data converted to a preset format, obtain the skeleton feature sequence;

[0017] Based on the skeletal feature sequence, slow motion features are obtained;

[0018] The skeletal feature sequence is augmented by downsampling, and rapid motion features are obtained based on the augmented sequence.

[0019] Based on the slow motion characteristics and the fast motion characteristics, the Cartesian motion information is obtained.

[0020] As a preferred technical solution, the slow motion characteristics are obtained using the following formula:

[0021]

[0022] In the formula, v s (k) represents the speed of the slow motion at time k. This represents the derivative or difference of the slow action at time k with respect to time.

[0023] The rapid motion characteristics are obtained using the following formula:

[0024]

[0025] In the formula, v f (k) represents the velocity of the rapid action at time k, p f (k) represents the position at time k. This represents the derivative or difference of the fast action at time k with respect to time, and 2k±1 indicates taking an odd number of frames.

[0026] As a preferred technical solution, the acquisition of the global joint dimension information includes the following steps:

[0027] Based on the human skeleton 3D pose data converted to a preset format, global skeleton point information is obtained;

[0028] The global joint size information is obtained based on the Euclidean distance between each global skeleton point.

[0029] As a preferred technical solution, the Euclidean distance between each global skeleton point is obtained using the following formula:

[0030]

[0031]

[0032] Among them, v ij and d ij Let i and j represent the vector formed by nodes i and j, respectively, and the Euclidean distance between them.

[0033] As a preferred technical solution, the acquisition of the global joint angle information includes the following steps:

[0034] Based on the human skeleton 3D pose data converted to a preset format, global skeleton point information is obtained;

[0035] The global joint angle information is obtained based on the angle between each global skeleton point and any two other points.

[0036] As a preferred technical solution, the global joint angle information is obtained using the following formula:

[0037]

[0038] In the formula, v ij Let θ represent the vector formed by nodes i and j. ijk This represents the angle formed by the three nodes i, j, and k.

[0039] As a preferred technical solution, the acquisition of the three-dimensional posture data of the human skeleton includes the following steps:

[0040] Real-time RGB-D data of the human body is acquired, and the three-dimensional pose data of the human skeleton is obtained based on the MediaPipe algorithm.

[0041] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the above-described motion recognition method based on human skeletal three-dimensional data.

[0042] In another aspect, the present invention provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the above-described motion recognition method based on three-dimensional human skeletal data.

[0043] Compared with the prior art, the present invention has the following advantages.

[0044] (1) High recognition accuracy: In response to the problems of speed difference, viewpoint position change and object distance change, the present invention acquires comprehensive features based on the three-dimensional posture data of human skeleton, including Cartesian motion information, global joint size information and global joint angle information. The comprehensive features are used to perform feature fusion and obtain recognition results, which solves or partially solves the problem that the existing methods have low recognition accuracy due to the problems of motion speed difference, viewpoint position change and object distance change.

[0045] (2) The system proposed in this invention not only has a lighter network structure, but also can achieve high-accuracy action recognition, and can be deployed in practical applications. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the action recognition network model in Example 1;

[0047] Figure 2 This is a schematic diagram of a single-frame global joint angle model;

[0048] Figure 3 This is a schematic diagram of the workflow of the action recognition system in Example 2. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0050] Example 1

[0051] This embodiment provides a motion recognition method based on three-dimensional human skeletal data, including the following steps:

[0052] Real-time human skeleton 3D pose data is acquired, converted into a preset format, and then input into a pre-trained motion recognition network model to obtain human motion recognition results.

[0053] like Figure 1As shown, the action recognition network model comprises four cascaded modules. Cascaded module 1 is a multi-feature input module, containing four features: Cartesian fast motion, Cartesian slow motion, global joint size, and global joint angle. These four features are calculated from the human 3D pose data output by the pose estimator. Cascaded module 2 extracts useful information from each feature and consists of one-dimensional convolutional layers and pooling layers. Cascaded module 3 fuses the refined information from each feature to further recognize human actions and consists of one-dimensional convolutional layers, pooling layers, and fully connected layers. Cascaded module 4 outputs the human action recognition result.

[0054] The methods for obtaining the features of Cartesian fast and slow motion are as follows:

[0055] In some global motions, during global trajectory translation and global trajectory rotation, the geometric features of the skeleton do not change, only the Cartesian coordinate information changes, so it is necessary to introduce Cartesian motion features.

[0056] For the same motion, the speeds of the motions may not be exactly the same; some may be faster, and some may be slower, resulting in inconsistent rates of change in Cartesian motion characteristics. To learn a robust global motion characteristic, both fast and slow motions should be considered simultaneously, making the velocity scale more generalizable.

[0057] To improve generalization in terms of speed, Cartesian action features were expanded on the skeletal feature sequence by downsampling, and these were called fast features.

[0058]

[0059] Among them, v f (k) is defined as the velocity of the fast motion at time k / frame, p f (k) represents the position at time k. This represents the derivative / difference of the fast action at time k with respect to time, and 2k±1 indicates taking an odd number of frames.

[0060] The Cartesian motion characteristics before expansion (Formula 2) can be considered as slow motion, while those after expansion (Formula 1) can be considered as fast motion.

[0061]

[0062] Among them, v s (k) represents the speed of the slow motion at time k / frame. This represents the derivative / difference of the slow action at time k with respect to time.

[0063] To make the model more lightweight, we flattened the Cartesian motion features of each frame's action, creating a model with a dimension of [missing information]. The one-dimensional vectors are used to form a two-dimensional input matrix for the Cartesian motion features of each video, as shown in Equation (3).

[0064]

[0065] in, n represents the size of the Cartesian motion feature in each frame. joint Indicates the number of skeletal joints. The dimension of the Cartesian coordinates of the skeletal joints.

[0066] Therefore, for slow motion, the magnitude of the Cartesian motion feature for each video is... For slow movements, the magnitude of the Cartesian motion feature for each movement is: To match a uniform video frame rate, the horizontal dimension is expanded (resized) using linear interpolation. n frame Indicates the number of video frames.

[0067] The method for obtaining the global joint size feature is as follows:

[0068] Cartesian motion features can represent the macroscopic features of the observed object, but the Cartesian coordinate information changes with different viewpoints. Even without actual motion, the Cartesian coordinate information changes depending on the viewpoint. This is undesirable, as the Cartesian motion features change with different viewpoints, necessitating the introduction of geometric features that can display detailed information. To address this issue, we introduce Global Joint Distances (GJD) features, considering global skeleton points and calculating the Euclidean distance between each pair of points, as shown in Equation (4).

[0069]

[0070] Where, ν ij and d ij Let i and j represent the vector formed by nodes i and j, respectively, and the Euclidean distance between them.

[0071] i and j are the indices of the skeletal points. The Euclidean distance between each pair of points is calculated, forming a symmetric matrix with a diagonal value of 0. To reduce redundancy, the global joint distance feature only retains the upper triangular matrix that does not contain the diagonal, as shown in Equation (5).

[0072]

[0073] The size of the GJD feature for each frame is given by formula (6), where n joint This represents the number of skeletal points; therefore, the size of the GJD feature for each video is... Right now

[0074]

[0075] The method for obtaining global joint angular features is as follows:

[0076] Human motion posture data is affected by object distance (i.e., the distance from the camera to the observed object). For the same action, when the object distance is inconsistent, both the Cartesian motion features and the global joint size features will change, which is undesirable. To address this, we propose a global joint angle feature.

[0077] Each point can calculate the angle between itself and the other two points. Considering the global skeleton points, we calculate the inverse cosine function between the vector formed by all skeleton points and all global points, as shown in Equation (7).

[0078]

[0079] Where, ν ij ν represents the vector formed by nodes i and j; similarly, ν ik Let θ represent the vector formed by nodes i and k. ijk This represents the angle formed by the three nodes i, k, and y.

[0080] like Figure 2 As shown, the magnitudes of the three directions ijk are the number of bone points, and the value of each point is obtained by the angle relationship between the vectors formed by the three points. That is, the global angular feature formed by each frame is a three-dimensional matrix, as shown in Equation (8):

[0081]

[0082] Figure 2 In the GJA model, the three diagonal planes α, β, and γ contain at least two points with the same index, so the values ​​on all three planes are always zero. To reduce redundancy, the eigenvalues ​​of the diagonal planes are discarded, and only the effective matrix portion without diagonal planes is used as the GJA feature. Technically, we flatten the 3D GJA features of each frame, and their size is shown in formula (9). Thus, the size of the GJA feature for each video is... Right now

[0083]

[0084] This embodiment employs a one-dimensional convolutional design to create a lightweight multi-feature fusion network structure with only 2M parameters, achieving real-time performance for action recognition. The multi-input module includes Cartesian motion features, Global Joint Distances (GJD) features, and Global Joint Angles (GJA) features, which can respectively address issues related to motion speed differences, viewpoint position changes, and object distance changes. This network not only boasts a lighter structure but also achieves high-accuracy action recognition, making it suitable for practical applications.

[0085] Example 2

[0086] like Figure 3 This embodiment provides an action recognition system for implementing the action recognition method based on human skeletal 3D data as described in Embodiment 1. The system includes a receiving module, a pose estimator, an action recognition network model, and an output module. The receiving module receives real-time RGB-D data of the human body and can employ a depth camera; in this embodiment, a Realsense D435i depth camera is used to acquire real-time RGB-D data of the human body. The pose estimator acquires human 3D pose data; this invention uses the MediaPipe algorithm as the pose estimator for acquiring human skeletal 3D data. The action recognition network model identifies human actions. The output module displays or outputs the human action recognition results to the controlled object in real time.

[0087] Example 3

[0088] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the motion recognition method based on human skeletal three-dimensional data as described in Embodiment 1.

[0089] Example 4

[0090] This embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the motion recognition method based on human skeletal three-dimensional data as described in Embodiment 1.

[0091] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A motion recognition method based on three-dimensional human skeletal data, characterized in that, Includes the following steps: Real-time 3D pose data of the human skeleton is acquired, converted into a preset format, and then input into a pre-trained motion recognition model to obtain human motion recognition results. The action recognition model includes: The multi-feature input module is used to obtain comprehensive features, including Cartesian motion information, global joint size information, and global joint angle information, based on the human skeleton 3D pose data converted to a preset format. The feature fusion and output module is used to obtain the human action recognition result based on the comprehensive features. The acquisition of the Cartesian motion information includes the following steps: Based on the human skeleton 3D pose data converted to a preset format, obtain the skeleton feature sequence; Based on the skeletal feature sequence, slow motion features are obtained; The skeletal feature sequence is augmented by downsampling, and rapid motion features are obtained based on the augmented sequence. Based on the slow motion characteristics and the fast motion characteristics, the Cartesian motion information is obtained. The slow motion characteristics are obtained using the following formula: In the formula, Indicates the slow motion in the first... The speed of time, Indicates the first The derivative or difference of slow motion with respect to time at any given moment. The rapid motion characteristics are obtained using the following formula: In the formula, For rapid action in the first The speed of time, Indicates the first Location at any given moment Indicates the first The derivative or difference of a fast motion with respect to time. This indicates that an odd number of frames are taken. The acquisition of the global joint dimension information includes the following steps: Based on the human skeleton 3D pose data converted to a preset format, global skeleton point information is obtained; The global joint size information is obtained based on the Euclidean distance between each global skeleton point. The Euclidean distance between each global skeleton point is obtained using the following formula: in, and Representing nodes respectively and nodes The vector formed and the Euclidean distance ， Indicates the number of skeletal points. The acquisition of the global joint angle information includes the following steps: Based on the human skeleton 3D pose data converted to a preset format, global skeleton point information is obtained; The global joint angle information is obtained based on the angle between each global skeleton point and any two other points. The global joint angle information is obtained using the following formula: In the formula, Represents a node and nodes The vector formed Indicates that by node ,node and nodes The included angle formed by the three nodes.

2. The motion recognition method based on three-dimensional human skeletal data according to claim 1, characterized in that, The acquisition of the three-dimensional pose data of the human skeleton includes the following steps: Real-time RGB-D data of the human body is acquired, and the three-dimensional pose data of the human skeleton is obtained based on the MediaPipe algorithm.

3. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the motion recognition method based on human skeletal three-dimensional data as described in any one of claims 1-2.

4. A computer-readable storage medium, characterized in that, It includes one or more programs that are executed by one or more processors of an electronic device, the one or more programs including instructions for performing the motion recognition method based on human skeletal three-dimensional data as described in any one of claims 1-2.

Images