A method for representing three-dimensional human motion features for intelligent behavior recognition

Abstract

This invention discloses a three-dimensional human motion feature representation method for intelligent behavior recognition, solving the problems of dimensional redundancy or underexpression of human motion features in existing machine learning models in the field of computer vision, which are based on three-dimensional Cartesian coordinate system data. The proposed three-dimensional human motion features have a smaller data dimensionality. Starting from concepts in the mechanical field such as joint constraints and degrees of freedom, the human body is regarded as a mechanical motion mechanism with complex degrees of freedom. The intelligent recognition features of human behavior are constructed using the degrees of freedom of the mechanism as a framework to optimize the description of human motion posture. When used in human behavior recognition engineering in computer vision, the proposed three-dimensional human motion features can accurately and comprehensively express human posture while ensuring minimal data dimensionality, thereby significantly improving the training speed of deep learning models and supporting high-speed and accurate human behavior recognition and its digital twin modeling.

Description

Technical Field

[0001] This application relates to the field of rapid recognition of human behavior in computer vision, and more particularly to a method for representing three-dimensional human motion features for intelligent behavior recognition. Background Technology

[0002] In recent years, the deep integration of artificial intelligence and manufacturing technologies has profoundly transformed the production model of the manufacturing industry, and intelligent manufacturing is currently undergoing a significant strategic transformation. Manufacturing is the process by which humans use tools to transform raw materials into products and services that meet the needs of production and daily life. The development of intelligent manufacturing has effectively enhanced the efficiency and quality of production activities. Even today, humans remain the most autonomous, creative, and perceptible factor in production activities and will play an even greater role in the autonomous decision-making process of the next generation of intelligent manufacturing. How to accurately acquire the movement trajectories or behavioral postures of technicians within the factory, thereby realizing the construction of a human-centered intelligent manufacturing system, is currently a major research goal and development direction in the field of intelligent manufacturing.

[0003] To achieve the above goals, the core issue is how to enable computer systems to accurately understand human behavior. Currently, the mainstream approach is human behavior recognition technology. In this research field, some scholars have achieved good recognition results using traditional manual feature extraction or deep learning methods under certain constraints, but the accuracy needs improvement. Most methods input structured data consisting of the three-dimensional coordinates of human joints. By converting this coordinate information into vectors or matrices and inputting them into a neural network for learning and iteration, the parameter values ​​of the neural network model are finally obtained. Relying on the powerful learning and feature extraction capabilities of neural networks, this operating mode can indeed achieve certain results on behavioral action datasets with relatively low complexity. However, it is undeniable that this approach does not fully explore the intrinsic connections and degrees of freedom features of joints, and the neural network model is prone to overfitting when the behavior becomes more complex. Summary of the Invention

[0004] This application provides a three-dimensional human motion feature representation method for intelligent behavior recognition. Its technical objective is to construct a refined joint feature based on the principle of minimum degrees of freedom of the skeletal structure. While ensuring the comprehensiveness of feature construction, it eliminates the redundancy of human joint features, and achieves the goal of minimizing the complexity of joint features while being able to completely distinguish all human behaviors, so as to support the rapid construction of a deep learning model for human behavior recognition based on computer vision.

[0005] The above-mentioned technical objective of this application is achieved through the following technical solution:

[0006] A method for representing three-dimensional human motion features for intelligent behavior recognition, comprising:

[0007] The coordinate data of the human skeleton model is obtained through a depth vision sensor;

[0008] The human skeleton model is divided into sections, and the mathematical representation of each joint and skeleton in each section is defined. Then, the degree of freedom of each joint is analyzed.

[0009] A coordinate system is established for each joint, and the displacement characteristics, attitude angle characteristics, spin angle characteristics, and included angle characteristics of each joint are calculated based on the results of the degree of freedom analysis.

[0010] The displacement features, attitude angle features, spin angle features, and included angle features are summarized to express the three-dimensional human motion features.

[0011] The beneficial effects of this application are as follows:

[0012] (1) This application proposes a three-dimensional human motion feature representation method for intelligent behavior recognition. Compared with the 75-dimensional data of traditional coordinate features, this three-dimensional human motion feature only requires 36-dimensional data, but can still describe the human motion posture in detail.

[0013] (2) The three-dimensional human motion features proposed in this application have a more refined data structure and faster computing speed when used in human behavior recognition engineering of computer vision, providing a corresponding foundation for realizing high-speed and accurate digital twin technology.

[0014] (3) The method of this application starts from the concepts of mechanical fields such as joint constraints and degrees of freedom, regards the human body as a high-precision instrument with complex degrees of freedom, and focuses on high-level features such as angles that cannot be extracted by deep networks to construct feature models, providing a theoretical basis for the non-destructive description of human digital twin models and the rapid conversion of digital-physical models. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the calculation of three-dimensional human motion features in the method described in this application.

[0016] Figure 2 This is a schematic diagram of the human skeleton model in this application;

[0017] Figure 3 This is a schematic diagram illustrating the calculation of the attitude angle of joint point J5 in a specific embodiment of this application;

[0018] Figure 4 This is a schematic diagram illustrating the calculation of the spin angle of joint point J5 in a specific embodiment of this application;

[0019] Figure 5This is a schematic diagram illustrating the calculation of the included angle of joint point J6 in a specific embodiment of this application. Detailed Implementation

[0020] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0021] like Figure 1 As shown, the three-dimensional human motion feature representation method for intelligent behavior recognition described in this application includes:

[0022] S1: Obtain coordinate data of the human skeleton model through a depth vision sensor.

[0023] Specifically, the coordinate data of the human skeleton model includes the three-dimensional spatial coordinates of 25 joints, namely: pelvis J1, spine (hip) J2, neck J3, head J4, left shoulder J5, left elbow J6, left wrist J7, left hand J8, right shoulder J9, and right elbow J1. 10 Right wrist J 11 Right hand J 12 Left thigh J 13 Left knee J 14 Left ankle J 15 Left foot J 16 Right thigh J 17 Right knee J 18 Right ankle J 19 Right foot J 20 Spine (shoulder) J 21 Left finger J 22 Left thumb J 23 Right finger J 24 Right thumb J 25 As shown in Table 1. The human skeleton model is as follows. Figure 2 As shown.

[0024] S2: Divide the human skeleton model into sections, define the mathematical representation of each joint and skeleton in each section, and then perform a degree of freedom analysis on each joint.

[0025] Specifically, the human skeleton model is divided into seven sections: head, left upper limb, right upper limb, left lower limb, right lower limb, upper trunk, and lower trunk. The mathematical representation of each joint is J. k The mathematical representation of each skeleton is J. k J l .

[0026] Analyze the degrees of freedom of each joint. The descriptive features of a joint vary depending on its degrees of freedom. For example, a joint with three rotational degrees of freedom requires two attitude angles and one spin angle to describe it; a joint with three translational degrees of freedom requires three displacement features to describe it; and a joint with one rotational degree of freedom is described using one included angle.

[0027] The translational and rotational degrees of freedom are calculated for each joint point; the translational degrees of freedom are characterized by absolute coordinates (i.e., displacement characteristics) in a Cartesian coordinate system, and the rotational degrees of freedom are characterized by the attitude angle and spin angle characteristics of the rigid body; the attitude angle characteristics are two-dimensional angle characteristics, expressed as follows: Each joint has a maximum of three translational and rotational degrees of freedom.

[0028] Results of the degrees of freedom analysis

[0029] S3: Establish a coordinate system for each joint point, and calculate the displacement characteristics, attitude angle characteristics, spin angle characteristics, and included angle characteristics of each joint point based on the results of the degree of freedom analysis.

[0030] The coordinate system for each joint is established, for example, with joint J. k Establish an original coordinate system CS as the reference, with the positive X-axis of CS represented as... The positive Y-axis direction of CS is represented as The positive Z-axis direction of CS is determined by the right-hand rule.

[0031] Table 1. Human Joint Point Correspondence Table

[0032] Serial Number Joint name Serial Number Joint name <![CDATA[J1]]> pelvis <![CDATA[J 14 ]]> Left knee <![CDATA[J2]]> Spine (hips) <![CDATA[J 15 ]]> left ankle <![CDATA[J3]]> neck <![CDATA[J 16 ]]> left foot <![CDATA[J4]]> head <![CDATA[J 17 ]]> Right thigh <![CDATA[J5]]> left shoulder <![CDATA[J 18 ]]> Right knee <![CDATA[J6]]> left elbow <![CDATA[J 19 ]]> Right ankle <![CDATA[J7]]> left wrist <![CDATA[J 20 ]]> Right foot <![CDATA[J8]]> left hand <![CDATA[J 21 ]]> Spine (shoulder) <![CDATA[J9]]> right shoulder <![CDATA[J 22 ]]> Left fingers <![CDATA[J 10 ]]> right elbow <![CDATA[J 23 ]]> left thumb <![CDATA[J 11 ]]> right wrist <![CDATA[J 24 ]]> Right finger <![CDATA[J 12 ]]> right hand <![CDATA[J 25 ]]> Right thumb <![CDATA[J 13 ]]> Left thigh

[0033] The displacement feature represents the joint point J. k The displacement increment in space is represented as (Δx1, Δy1, Δz1).

[0034] The attitude angle features The calculations include:

[0035] (1) Calculate the skeleton vector attitude angle Represented as:

[0036]

[0037] in, This represents a unit vector in the same direction as the Z-axis of CS, where CS represents the vector with respect to the joint J. k Using the original coordinate system as the reference, the positive X-axis direction of CS is represented as The positive Y-axis direction of CS is represented as The positive Z-axis direction of CS is determined by the right-hand rule; k, k', k”, and l all represent the joint number, and there are a total of 25 joint numbers.

[0038] (2) Calculate the skeleton vector The attitude angle θ is expressed as:

[0039]

[0040] in, Represents a unit vector in the same direction as the X-axis of CS; Represents a unit vector in the same direction as the Y-axis of CS; express The projection on the X-axis; express The projection on the Y-axis; express Projection onto the OXY plane.

[0041] Taking the left shoulder joint J5 as a specific example, the posture angle is calculated as follows: Figure 3 As shown, first calculate the skeleton of the left upper arm. attitude angle Represented as:

[0042]

[0043] in, Let represent a unit vector in the same direction as the Z-axis of CS, where CS represents the original coordinate system based on joint J5, and the positive X-axis of CS is represented as . The positive Y-axis direction of CS is represented as The positive Z-axis direction of CS is determined by the right-hand rule.

[0044] Left upper arm skeleton The attitude angle θ is expressed as:

[0045]

[0046] in, express The projection on the X-axis; express The projection on the Y-axis; express Projection onto the OXY plane.

[0047] The key points of the attitude angles of all skeleton vectors are summarized in Table 2. In the table, RHR represents the right-hand rule, which means that the direction of the axis is determined by the right-hand rule.

[0048] Table 2 summarizes the key points containing attitude angles.

[0049]

[0050] The calculation of the spin angle characteristic includes:

[0051] Construct skeleton vectors Given a base coordinate system CS0, the positive X-axis direction of CS0 is represented as... The positive Y-axis direction of CS0 is represented as The positive Z-axis direction of CS0 is determined according to the right-hand rule;

[0052] Construct skeleton vectors Given a rotating coordinate system CS1, the positive X-axis direction of CS1 is represented as... The positive Z-axis direction of CS1 is represented as The positive Y-axis direction of CS1 is determined according to the right-hand rule; l' represents the joint number;

[0053] Using the X-axis of CS1 as the key axis, calculate the cross product of the X-axis of CS1 and the X-axis of CS0. The included angle α is expressed as:

[0054]

[0055] in, This indicates the positive X-axis direction of CS0; Indicates the positive X-axis direction of CS1;

[0056] outer product Substituting the included angle α into Rodriguez's formula, we obtain the rotation matrix R, and then calculate the rotated vector. Represented as:

[0057]

[0058] in, Represents a unit vector in the same direction as the Y-axis of CS1;

[0059] Calculate skeleton vector The spin angle ψ is expressed as:

[0060]

[0061] in, This represents a unit vector in the same direction as the Y-axis of CS0.

[0062] Taking the left shoulder joint J5 as a specific example, the spin angle ψ is calculated as follows: Figure 4 As shown, establish a rotating coordinate system CS1. The positive X-axis and Z-axis directions of CS1 are respectively used as the positive directions. The positive Y-axis direction is obtained according to the right-hand rule. The specific calculation process will not be described in detail. Table 3 is a summary of joints including spin angles.

[0063] Table 3 Summary of key points including spin angle

[0064]

[0065] The calculation of the included angle feature includes:

[0066] Constructing with key J k Using the coordinate system CS′ as the reference, then This indicates the positive X-axis direction of CS′. Let Z represent the positive direction of CS′. Using the right-hand rule, determine the positive direction of CS′'s Y-axis. Then, the included angle β is expressed as:

[0067]

[0068] Taking the left elbow joint point J6 as a specific example, as follows: Figure 5 As shown, establish coordinate system CS′. Let X and Z be the positive directions of CS′ respectively, and determine the positive direction of Y according to the right-hand rule.

[0069] Calculate skeleton vector and The inner product of the product, divide the result by and After obtaining the modulus, the included angle β is calculated using the inverse cosine formula, and is expressed as:

[0070]

[0071] Table 4 Summary of joints containing included angles

[0072]

[0073] S4: Summarize the displacement features, attitude angle features, spin angle features and included angle features to realize the expression of three-dimensional human motion features.

[0074] Specifically, the three-dimensional human motion features are represented as follows:

[0075]

[0076] in, Represents skeleton vector The attitude angle features, Represents skeleton vector attitude angle features, Represents skeleton vector The attitude angle features, Represents skeleton vector The attitude angle features, Represents skeleton vector The attitude angle features, Represents skeleton vector The attitude angle features, Represents skeleton vector The attitude angle features; β0 represents the skeleton vector. and The included angle, β1 represents the skeleton vector and The included angle, β2 represents the skeleton vector and The included angle, β3 represents the skeleton vector. and The included angle, β4 represents the skeleton vector. and The included angle, β5 represents the skeleton vector. and The included angle, β6 represents the skeleton vector. and The included angle, β7 represents the skeleton vector. and The included angle, β8 represents the skeleton vector. and The included angle, β9 represents the skeleton vector. and The included angle; ψ0 represents the skeleton vector. The spin angle characteristic, ψ1 represents the skeleton vector The spin angle characteristic, ψ2 represents the skeleton vector. The spin angle characteristic, ψ3 represents the skeleton vector. The spin angle characteristic, ψ4 represents the skeleton vector. The spin angle characteristic, ψ5 represents the skeleton vector. The spin angle characteristic, ψ6 represents the skeleton vector. The spin angle characteristic, ψ7 represents the skeleton vector. The spin angle characteristic, ψ8 represents the skeleton vector. The spin angle characteristic.

[0077] The above are exemplary embodiments of this application, and the scope of protection of this application is defined by the claims and their equivalents.

Claims

1. A method for representing three-dimensional human motion features for intelligent behavior recognition, characterized in that, include: The coordinate data of the human skeleton model is obtained through a depth vision sensor; The human skeleton model is divided into sections, and the mathematical representation of each joint and skeleton in each section is defined. Then, the degree of freedom of each joint is analyzed. A coordinate system is established for each joint, and the displacement characteristics, attitude angle characteristics, spin angle characteristics, and included angle characteristics of each joint are calculated based on the results of the degree of freedom analysis. The displacement features, attitude angle features, spin angle features, and included angle features are summarized to achieve the expression of three-dimensional human motion features; Among them, the attitude angle features ) The calculations include: Calculate skeleton vector attitude angle , represented as: ; in, Indicates and The unit vectors in the same direction as the Z-axis, Indicated by key points The original coordinate system as the reference, The positive direction of the X-axis is represented as , The positive direction of the Y-axis is represented as , The positive direction of the Z-axis is determined by the right-hand rule; , '、 ''、 Each number represents a key point number; there are a total of 25 key point numbers. Calculate skeleton vector attitude angle , represented as: ; in, Indicates and Unit vectors in the same direction as the X-axis; Indicates and The unit vectors in the same direction as the Y-axis; express The projection on the X-axis; express The projection on the Y-axis; express Projection onto the OXY plane; The calculation of the spin angle characteristic includes: Construct skeleton vectors basic coordinate system ,but The positive direction of the X-axis is represented as , The positive direction of the Y-axis is represented as , The positive direction of the Z-axis is determined according to the right-hand rule; Construct skeleton vectors Rotating coordinate system ,but The positive direction of the X-axis is represented as , The positive direction of the Z-axis is represented as , The positive direction of the Y-axis is determined according to the right-hand rule; Indicates the number of the joint point; Will Using the X-axis as the key axis, calculate X-axis and of outer product of axis and included angle , represented as: ； in, express The positive direction of the X-axis; express The positive direction of the X-axis; outer product and included angle Substituting into the Rodriguez formula, the rotation matrix is ​​calculated. Then calculate the rotated vector. , represented as: ； in, Indicates and of Unit vectors in the same direction as the axis; Calculate skeleton vector spin angle , represented as: ; in, Indicates and of Unit vectors in the same direction as the axis; The calculation of the included angle feature includes: Building with key points coordinate system based on ,but express The positive direction of the X-axis, express The positive direction of the Z-axis is determined by the right-hand rule. If the positive direction of the Y-axis is used, then the included angle is... Represented as: 。 2. The method as described in claim 1, characterized in that, The coordinate data of the human skeleton model includes the three-dimensional spatial coordinates of 25 joints, including the pelvis. Spine and hip ,neck ,head left shoulder , left elbow , left wrist left hand right shoulder , right elbow , right wrist right hand left thigh left knee left ankle left foot Right thigh Right knee Right ankle Right foot Spine and shoulder left fingers left thumb Right fingers Right thumb .

3. The method as described in claim 2, characterized in that, The degree-of-freedom analysis for each joint includes: The degrees of freedom (DOF) of each joint are analyzed, and the translational and rotational DDFs of each joint are calculated. The translational DDFs are characterized using absolute coordinates in a Cartesian coordinate system, and the rotational DDFs are characterized using the rigid body's attitude angle and spin angle features. The attitude angle features are two-dimensional angle features, expressed as follows: ) Each joint has at most three translational and rotational degrees of freedom.

4. The method as described in claim 3, characterized in that, The displacement feature represents the joint point. The displacement increment in space is expressed as .

5. The method as described in claim 4, characterized in that, The displacement features, attitude angle features, spin angle features, and included angle features are summarized to express the three-dimensional human motion features, which are represented as follows: ； in, ( Represents skeleton vector The attitude angle features, ( Represents skeleton vector The attitude angle features, Represents skeleton vector The attitude angle features, Represents skeleton vector The attitude angle features, Represents skeleton vector The attitude angle features, Represents skeleton vector The attitude angle features, Represents skeleton vector Attitude angle characteristics; Represents skeleton vector and The included angle, Represents skeleton vector and The included angle, Represents skeleton vector and The included angle, Represents skeleton vector and The included angle, Represents skeleton vector and The included angle, Represents skeleton vector and The included angle, Represents skeleton vector and The included angle, Represents skeleton vector and The included angle, Represents skeleton vector and The included angle, Represents skeleton vector and The included angle; Represents skeleton vector Spin angle characteristics, Represents skeleton vector Spin angle characteristics, Represents skeleton vector Spin angle characteristics, Represents skeleton vector Spin angle characteristics, Represents skeleton vector Spin angle characteristics, Represents skeleton vector Spin angle characteristics, Represents skeleton vector Spin angle characteristics, Represents skeleton vector Spin angle characteristics, Represents skeleton vector The spin angle characteristic.

Images