Human motion recognition method and system

Abstract

The present invention provides a human motion recognition method and system, the method comprising: real-time monitoring of motion parameters of human joints in a human coordinate system, respectively constructing a next moment state variable prediction matrix calculation model, a next moment state vector error prediction value calculation model and a fusion threshold calculation model; according to the constructed next moment state variable prediction model matrix and state vector error prediction value calculation model, gain correction is performed on the next moment state vector error prediction value, the next moment human state variable matrix and the next moment accurate state vector prediction value calculation model are constructed, and whether the deviation of the state variable prediction values ​​at two different moments meets the fusion threshold condition is judged; if it meets, the fused human motion recognition result is output, and if it does not meet, the above steps are continued to be repeated. The present invention can ensure that the human motion parameters at each moment are smoothly fused, and the state parameter results finally output are smoothly and non-intermittently fused to form accurate human motion state data within the monitoring time range.

Description

Technical Field

[0001] The present invention belongs to the technical field of modern sports service support, and in particular relates to a human motion recognition method and system. Background Art

[0002] Human motion status monitoring or identification technologies and devices are widely used in kinematic analysis, medical diagnosis, personnel care, virtual reality and other fields.

[0003] In terms of sports training for athlete cultivation, most wearable devices on the market, such as bracelets, only contain data recording and analysis functions such as step counting, mileage, and calorie consumption, and do not have motion recognition and counting functions for certain types of sports. In order to facilitate coaches to better analyze the performance of athletes in training and competitions, people hope to be able to record the number of technical movements of athletes on the field. In the past, human motion recognition and prediction methods such as those provided by Chinese patents 2020800063201 and 202010491566.4 were used to analyze athlete data through real-time monitoring of video or image, which is not only time-consuming but also laborious. Therefore, it is hoped that motion recognition and counting can be achieved by using lighter wearable devices or avoiding video recording of human motion to perform motion recognition for each frame. For example, in terms of the technical requirements for human motion recognition methods for athlete training, the most widely used are in the fields of professional table tennis and badminton.

[0004] At present, scholars at home and abroad have started research on human motion recognition algorithms. For example, US Patent US4570217A discloses a method for human motion recognition using a distributed sensor network, which proposes a linear settlement method to process sensor network data, but there are many sensors, the amount of data is large, and the real-time performance is not high. Li Feng et al. (Human motion recognition based on three-axis acceleration sensor [J]. Computer Research and Development, 2016, 53 (3): 621-631.) used the accelerometer and gyroscope sensor of the mobile phone to collect human motion data, and analyzed the time domain and frequency domain of the original data to extract relevant eigenvalues. Then the J48 decision maker combined with the Markov model was used to obtain the recognition result, which can recognize the state of people going upstairs, going downstairs, running, walking, and standing still. However, the algorithm can only recognize a certain state of a person within a period of time, and cannot count the actions and recognize the next action of the human body, and predict the human motion trend after the smooth fusion of the human motion data of athletes at different times. Zhou Lin et al. (Zhou Lin, Lei Liping, Yang Longpin. Human behavior recognition system based on multi-sensor [J]. Sensors and Microsystems, 2016, 35(3):89-91, 95.) also used decision trees and windowing processing to realize the recognition of human squatting, lying and other actions. However, due to the use of windowing processing, it is still impossible to achieve the prediction of human movement trends after smooth fusion of human movement data at different times.

[0005] In terms of daily physical exercise, in today's society, exercise and fitness are increasingly needed by the public, especially for office white-collar workers who always stay in air-conditioned rooms, and their sub-health condition is getting worse and worse. Exercise close to nature and stepping on the soil is a scientific and effective aerobic fitness exercise. However, due to factors such as climate and environment, people's enthusiasm for exercise is reduced. Treadmill exercise is an indoor exercise that is not affected or restricted by external environments such as weather, and is especially popular among office white-collar workers. However, more and more people are injured due to running.

[0006] In today's world, information technology is developing rapidly, and the social, economic and user demands for fitness equipment are undergoing tremendous changes. Most of the existing human motion state monitoring or identification methods and system devices can only classify and judge specific motion states, and their functions are relatively simple. The development and design technology of methods, software and system devices that can automatically complete the recognition of multiple human motion states under daily conditions is still imperfect. Traditional human motion recognition methods can no longer meet the needs of human motion state recognition, targeted exercise results, humanized exercise, and fall prevention during rehabilitation in the fields of kinematic analysis, medical diagnosis, personnel care, virtual reality, etc. Summary of the invention

[0007] In view of the above-mentioned defects, the present invention provides a method and system for human motion recognition. Based on the Internet of Things and wearable sensors, the present invention monitors the three-axis motion speed, angular velocity and acceleration of a moving human body in real time, respectively constructs a next moment state variable prediction matrix calculation model, a next moment state vector error prediction value calculation model, a next moment human state variable matrix, and a next moment accurate vector prediction value calculation fusion threshold calculation model, and judges whether the deviation of the human motion variable prediction value at different moments meets the fusion threshold condition based on the constructed fusion threshold model, and finally outputs smooth fusion accurate human motion state data, which can effectively improve the accuracy of human motion recognition and provide coherent fusion and high-accuracy human motion recognition continuous data.

[0008] The present invention provides the following technical solution: a human motion recognition method, wherein the method converts a human motion recognition result into feature information, wherein the feature information is 3D key point information, and comprises the following steps:

[0009] S1: Real-time monitoring of the three-axis motion speed, three-axis motion angular velocity and three-axis acceleration of human joints in the human coordinate system, and building a prediction matrix calculation model for the state variables at the next moment;

[0010] S2: Construct a calculation model for the state vector error prediction value at the next moment;

[0011] S3: Real-time monitoring of the three-dimensional motion coordinate values ​​of human joints in the human coordinate system, and building a fusion threshold calculation model;

[0012] S4: According to the next moment state variable prediction model matrix constructed in step S1 and the state vector error prediction value calculation model constructed in step S2, the state vector error prediction value at the next moment is gain corrected, and the human body state variable matrix at the next moment and the accurate state vector prediction value calculation model at the next moment are constructed;

[0013] S5: Determine whether the deviation of the predicted values ​​of the state variables at the i-th moment and the j-th moment meets the fusion threshold condition calculated in step S3, where j>i; if so, output the fused human motion recognition result; if not, repeat steps S1-S4.

[0014] Furthermore, the human body joints include the center point of the back of the head, the cervical vertebrae, the left shoulder, the left elbow, the left hand, the right shoulder, the right elbow, the right hand, the center point of the torso, the left hip, the left knee, the left foot, the right hip, the right knee, and the right foot.

[0015] Furthermore, the method for constructing the next moment state variable prediction matrix calculation model in step S1 includes the following steps:

[0016] S11: Based on the real-time monitoring of the three-axis motion speed and three-axis acceleration of the human joints in the human coordinate system, construct the human joint velocity vector matrix V in the human coordinate system k and the human joint acceleration vector matrix A k Then according to the human joint velocity vector matrix V k and the human joint acceleration vector matrix A k Construct the human body state variable matrix X at time k k :

[0017] in

[0018]

[0019]

[0020] Where, q∈Q, Q=1,2…,15, Q is the total number of human joints monitored in real time, is the x-axis velocity of the qth joint at time k in the human body coordinate system, is the y-axis velocity of the q-th joint at time k in the human body coordinate system, is the z-axis velocity of the qth joint at time k in the human body coordinate system; is the x-axis acceleration of the qth joint in the human coordinate system at time k, is the y-axis acceleration of the qth joint in the human body coordinate system at time k, is the z-axis acceleration of the q-th joint in the human body coordinate system at time k; k∈{i, j}, k+1∈{i, j}, {i, j} is the set of all time points from the i-th time to the j-th time;

[0021] S12: Based on the three-axis angular velocity obtained by real-time monitoring, construct the human motion transformation matrix M at time k in the human coordinate system k :

[0022] Among them, B is the state coefficient matrix, is the angular velocity matrix in the human body coordinate system at time k The skew-symmetric matrix of is the skew-symmetric matrix of angular velocity;

[0023] in, I3 is a 3×3 identity matrix, 03 is a 3×3 zero matrix with all elements being 0, and η is the acceleration steady state calculation coefficient with Gauss-Markov white noise added;

[0024] Angular velocity matrix in, is the x-axis angular velocity matrix of the human body coordinate system at time k monitored in real time, is the y-axis angular velocity matrix of the human body coordinate system at time k monitored in real time, is the z-axis angular velocity matrix in the human body coordinate system at time k monitored in real time; is the x-axis angular velocity of the qth joint at time k in the human body coordinate system, is the y-axis angular velocity of the q-th joint at time k in the human body coordinate system, is the z-axis angular velocity of the qth joint at time k in the human body coordinate system; the angular velocity skew-symmetric matrix

[0025] S13: Construct the next moment state variable prediction matrix calculation model X k+1|k :X k+1|k =M k X k , where the next moment state variable prediction matrix X k+1|k is the human body state variable matrix X based on time k k Calculated.

[0026] Furthermore, the method for constructing a state vector error prediction value calculation model at the next moment in step S2 includes the following steps:

[0027] S21: According to the three-axis motion speed of the human joints in the human coordinate system obtained by real-time monitoring in step S1, a three-axis motion speed matrix in the human coordinate system at time k is constructed The skew-symmetric matrix Among them, the three-axis motion velocity matrix in the human body coordinate system at time k is is the x-axis motion velocity matrix of the human body coordinate system at time k monitored in real time, is the y-axis motion velocity matrix of the human body coordinate system at time k monitored in real time, is the z-axis motion velocity matrix in the human body coordinate system at time k monitored in real time;

[0028] S22: The three-axis motion velocity matrix in the human body coordinate system at time k constructed according to step S21 The skew-symmetric matrix Construct the human body state variable matrix X at time k k and the human body state variable matrix X at the next moment k+1 The covariance matrix C between k :

[0029]

[0030] in, is the covariance of the three-axis motion velocity of Q joints in the human body coordinate system at the kth moment and the k+1th moment, is the covariance of the acceleration of the three-axis motion of Q joints in the human body coordinate system at the kth moment and the k+1th moment;

[0031] S23: The covariance matrix C constructed according to step S21 k And the human body motion transformation matrix M at time k in the human body coordinate system constructed in step S12 k , construct the state vector error prediction value calculation model P k+1|k :P k+1|k =M k P k (M k ) T +C k , where the state vector error prediction value P k+1|k is the predicted value P of the state vector error based on time k k Calculated.

[0032] Furthermore, the step S4 performs gain correction on the state vector error prediction value at the next moment, constructs the human body state variable matrix at the next moment and the accurate state vector prediction value calculation model at the next moment, including the following steps:

[0033] S41: Calculating the state vector error prediction value model P constructed according to step S2 k+1|k , construct the gain correction matrix G at time k+1 k+1 : G k+1 =P k+1|k (E k ) T [E k P k+1|k (E k ) T ] -1 , where E k is the human body state variable matrix X at time k k Add measurement noise γ k The measured human body state variable matrix Y at time k+1 is obtained k+1 The coefficient matrix, Y k+1 =E k X k +γ k ;

[0034] S42: The next moment state variable prediction matrix X constructed according to the S1 step k+1|k , the gain correction matrix G at time k+1 constructed in step S41 k+1 And the measured human body state variable matrix Y at time k+1 is calculated k+1 , calculate the human body state variable matrix X at time k+1 k+1 :

[0035] X k+1 =X k+1|k +G k+1 [Y k+1 -E k X k+1|k ]

[0036] S43: The gain correction matrix G at time k+1 constructed according to step S41 k+1 , coefficient matrix E k The state vector error prediction value calculation model P of the next moment constructed in step S2 k+1|k , construct the next moment accurate state vector prediction value calculation model P k+1 :P k+1 =(1-G k+1 E k ) k+1|k .

[0037] Furthermore, the coefficient matrix E in step S41 is k for:

[0038] Among them, I3 is a 3×3 identity matrix, and 03 is a 3×3 zero matrix in which all elements are 0.

[0039] Furthermore, the fusion threshold condition is 0.01<D(X i , X j )<0.3, if the fusion threshold D(X i , X j )≥0.3, repeating steps S1-S4 to ensure that the human motion parameters at each moment are smoothly fused, so that the human motion state parameter result at the i-th moment and the human motion state parameter result at the j-th moment are smoothly and continuously fused to form accurate human motion state data from the i-th moment to the j-th moment;

[0040] The fusion threshold D(X) constructed in step S3 i , X j ) is calculated as follows:

[0041]

[0042] Among them, τ q is the i-th state variable matrix X i To the jth state variable matrix X j The weight coefficient of the qth joint in all state variable matrices of |X i | is the state variable matrix X at the i-th moment i The rank of |X j | is the state variable matrix X at the jth moment j rank.

[0043] Furthermore, the weight coefficient τ of the qth joint in the state variable matrix q The calculation formula is:

[0044]

[0045] Among them, τ intra is the internal variability coefficient of Q joints at the kth moment, τ inter is the internal variability coefficient of Q joints at the kth moment and other moments in the time interval between the ith moment and the jth moment;

[0046]

[0047]

[0048] Among them, H q (i, j) is the fusion adjustment calculation model of the q-th joint state variable matrix at the i-th moment and the j-th moment, Trace(H q(ij)) is H q (i, j) is solved using the Trace(·) function.

[0049] Furthermore, the qth joint state variable matrix fusion adjustment calculation model H at the i-th moment and the j-th moment q (i, j) is as follows:

[0050]

[0051] Among them, mean(·) is the average function, is the boundary position matrix of the qth joint at the kth moment, is the rank of each boundary position matrix from the i-th moment to the j-th moment, is the x-axis coordinate value of the qth joint in the human body coordinate system at the kth moment obtained by real-time monitoring in step S2, is the y-axis coordinate value of the q-th joint in the human body coordinate system at the k-th moment obtained by real-time monitoring in step S2, The z-axis coordinate value of the qth joint in the human body coordinate system at the kth moment obtained by real-time monitoring in step S2.

[0052] The present invention also provides a human motion recognition system, which includes a three-axis gyroscope, a three-dimensional positioning module, a data analysis and calculation module, and a central control module; the three-axis gyroscope is used to monitor the three-axis motion speed, three-axis motion angular velocity, and three-axis acceleration of human joints in a human coordinate system in real time;

[0053] The three-dimensional positioning module is used to monitor the three-dimensional motion coordinate values ​​of human joints in the human body coordinate system in real time;

[0054] The data analysis and calculation module is used to calculate the state variable prediction matrix at the next moment and construct a state vector error prediction value calculation model at the next moment according to the real-time monitoring parameters of the three-axis gyroscope, and to construct a fusion threshold calculation model;

[0055] The central control module is used to calculate the state variable prediction matrix at the next moment, perform gain correction on the state vector error prediction value at the next moment, construct an accurate state vector prediction value calculation model for the next moment, determine whether the deviation of the state variable prediction values ​​at the i-th moment and the j-th moment meets the fusion threshold condition, and finally output the human body motion state parameter result at the i-th moment that is fused without interruption and smoothed with the human body motion state parameter result at the j-th moment to form accurate human body motion state data from the i-th moment to the j-th moment.

[0056] The beneficial effects of the present invention are:

[0057] 1. The human motion recognition method provided by the present invention monitors the three-axis motion speed, three-axis motion angular velocity and three-axis acceleration of 15 joints of the human body, including the center point of the back of the head, cervical spine, left shoulder, left elbow, left hand, right shoulder, right elbow, right hand, center point of the torso, left hip, left knee, left foot, right hip, right knee and right foot in real time in the human body coordinate system, and constructs a human joint velocity vector matrix V based on the time (time k) according to the three-axis speed, three-axis angular velocity and three-axis acceleration of the 15 joints in the human body coordinate system. k and the human joint acceleration vector matrix A k The human body state variable matrix X k , then based on the human body state variable matrix X k By constructing the human motion transformation matrix M k Predict the state variable prediction matrix calculation model X at the next moment (k+1 moment) k+1|k , including the state coefficient matrix B and the angular velocity matrix in the human body coordinate system at time k The skew-symmetric matrix The human motion transformation matrix M k , can effectively improve the human body state variable matrix X based on k moments k The calculated state variable prediction matrix calculation model X at the next moment (k+1 moment) k+1|k accuracy.

[0058] 2. The present invention obtains the state variable prediction matrix calculation result X at the next moment (k+1 moment) k+1|k In the case of k, the three-axis motion speed matrix of the human body coordinate system at time k is constructed by monitoring the three-axis motion speed of each joint in the human body coordinate system in real time based on the three-axis gyroscope. The skew-symmetric matrix Based on the next moment state variable prediction matrix calculation result X k+1|k , construct the state vector error prediction value calculation model P at the next moment k+1|k , and calculate the model P based on the state vector error prediction value k+1|k , construct the gain correction matrix G at time k+1 k+1 , use the gain correction matrix to perform gain correction on the state vector error prediction value at the next moment, and obtain the human body state variable matrix X at the next moment k+1 And the next moment accurate state vector prediction value calculation model P k+1After continuous iterative optimization, the constructed fusion threshold calculation model is used to determine whether the deviation of the predicted value of the state variable at the i-th moment and the j-th moment meets the fusion threshold condition, ensuring that the human motion parameters at each moment are smoothly fused, so that the human motion state parameter results at the i-th moment and the human motion state parameter results at the j-th moment are smoothly and uninterruptedly fused to form accurate human motion state data from the i-th moment to the j-th moment. Then, the constructed reinforcement training model is applied to the parameters obtained by real-time monitoring of the three-axis gyroscope and the three-dimensional positioning module. After data analysis and calculation, the self-constructed fusion threshold calculation model is applied to determine whether the human motion state parameters at each moment are smooth, ensuring that the human motion parameters at each moment are smoothly fused, so that the human motion state parameter results at the i-th moment and the human motion state parameter results at the j-th moment are smoothly and uninterruptedly fused to form accurate human motion state data from the i-th moment to the j-th moment.

[0059] 3. The fusion threshold calculation model constructed by the present invention is based on the rank of the human body state variable matrix at the i-th moment |X i | and the rank of the human body state variable matrix at the jth moment |X j |After subtraction, assign joint weight coefficient τ q , which can effectively provide a method to determine whether the three-axis velocity, three-axis angular velocity, and three-axis acceleration of each joint at the i-th moment are smoothly fused with the three-axis velocity, three-axis angular velocity, and three-axis acceleration of each joint at the j-th moment, and use the mean(·) function to calculate the fusion adjustment result H of the q-th joint state variable matrix at each moment q (i, j), and based on this, the internal variability coefficient τ of the Q joints at the kth moment and other moments in the time interval between the ith moment and the jth moment is calculated intra , in calculating the qth joint state variable matrix at each moment, the fusion adjustment result H q (i, j) is calculated using the Trace(·) function, which minimizes the intra-class distance while outputting all strings through the Trace(·) function, thereby achieving the purpose of maximizing the inter-class distance and preventing the occurrence of inaccurate judgment criteria caused by high data noise.

[0060] 4. The human motion recognition method provided by the present invention can be used for safety assurance of sports equipment, physical education, rehabilitation exercises and emergency rescue. Through the state parameters provided by the present invention that can accurately judge the changes in human motion, it can sensitively, quickly and accurately identify human falls or predict falls, or predict and / or identify the occurrence of inadequate physical exercise movements. It can also be applied to the front-end application of sports and virtual game product development, effectively improving the effect of virtual game products on VR and AR virtual audio-visuals. By using the human motion state parameters at different times in the early stage and inputting them into the database, different motion posture display models for virtual games are constructed, which can improve the authenticity and atmosphere of virtual audio-visuals.

[0061] 5. The human motion recognition method provided by the present invention avoids the image recognition of human motion by real-time monitoring sensors of video data output parameters such as cameras, and avoids the defect of large amount of calculation for recognizing human motion frame by frame in the data processing process. At the same time, compared with the image human motion recognition method, it can also accurately predict the sudden change or change of human motion, expand the application field of the human motion recognition method, and increase its practicality.

[0062] 6. The present invention includes the human joint velocity vector matrix V k and the human joint acceleration vector matrix A k The human body state variable matrix X k It is constructed based on the real-time monitoring data of the three-axis gyroscope, which avoids the occurrence of mismeasurement or missed measurement caused by linear real-time monitoring data, and can effectively improve the comprehensiveness and accuracy of human body state parameter data, thereby further improving the accuracy of judging whether the fusion of different human body motion state parameters is smooth. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The present invention will be described in more detail below based on embodiments and with reference to the accompanying drawings, wherein:

[0064] Figure 1 A schematic diagram of the flow chart of the human action recognition method provided by the present invention;

[0065] Figure 2 A schematic diagram of the distribution of 15 joint points for human motion recognition provided by the present invention;

[0066] Figure 3 The present invention provides a next-moment state variable prediction matrix calculation model X k+1|k Schematic diagram of the method flow chart;

[0067] Figure 4 The state vector error prediction value calculation model P for constructing the next moment provided by the present invention k+1|kSchematic diagram of the process;

[0068] Figure 5 The present invention provides a gain correction for the state vector error prediction value at the next moment to construct the human body state variable matrix X at the next moment. k+1 And the next moment accurate state vector prediction value calculation model P k+1 Schematic diagram of the process;

[0069] Figure 6 This is a schematic diagram of the structure of the human motion recognition system provided by the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Example 1

[0072] like Figure 1 As shown, the human motion recognition method provided by this embodiment converts the human motion recognition result into feature information, and the feature information is 3D key point information, including the following steps:

[0073] S1: Real-time monitoring of the three-axis motion speed, three-axis motion angular velocity and three-axis acceleration of human joints in the human coordinate system, and building the next moment state variable prediction matrix calculation model X k+1|k ;

[0074] S2: Construct the state vector error prediction value calculation model P at the next moment k+1|k ;

[0075] S3: Real-time monitoring of the three-dimensional motion coordinate values ​​of human joints in the human coordinate system, and building a fusion threshold calculation model;

[0076] S4: The next moment state variable prediction model matrix X constructed according to step S1 k+1|k The state vector error prediction value calculation model P constructed in step S2 k+1|k , perform gain correction on the state vector error prediction value at the next moment, and construct the human body state variable matrix X at the next moment k+1 And the next moment accurate state vector prediction value calculation model P k+1 ;

[0077] S5: Determine whether the deviation of the predicted values ​​of the state variables at the i-th moment and the j-th moment meets the fusion threshold condition calculated in step S3, where j>i; if so, output the fused human motion recognition result; if not, repeat steps S1-S4.

[0078] Example 2

[0079] like Figure 1 As shown, the human motion recognition method provided by this embodiment converts the human motion recognition result into feature information, and the feature information is 3D key point information, including the following steps:

[0080] S1: Real-time monitoring of the three-axis motion speed, three-axis motion angular velocity and three-axis acceleration of human joints in the human coordinate system, and building the next moment state variable prediction matrix calculation model X k+1|k ;

[0081] S2: Construct the state vector error prediction value calculation model P at the next moment k+1|k ;

[0082] S3: Real-time monitoring of the three-dimensional motion coordinate values ​​of human joints in the human coordinate system, and building a fusion threshold calculation model;

[0083] S4: The next moment state variable prediction model matrix X constructed according to step S1 k+1|k The state vector error prediction value calculation model P constructed in step S2 k+1|k , perform gain correction on the state vector error prediction value at the next moment, and construct the human body state variable matrix X at the next moment k+1 And the next moment accurate state vector prediction value calculation model P k+1 ;

[0084] S5: Determine whether the deviation of the predicted values ​​of the state variables at the i-th moment and the j-th moment meets the fusion threshold condition calculated in step S3, where j>i; if so, output the fused human motion recognition result; if not, repeat steps S1-S4.

[0085] like Figure 2 As shown, as a preferred embodiment of the present invention, the human joints focused on by the human motion recognition method provided by the present invention include the center point of the back of the head, cervical vertebrae, left shoulder, left elbow, left hand, right shoulder, right elbow, right hand, center point of the torso, left hip, left knee, left foot, right hip, right knee, and right foot, a total of 15 joint points.

[0086] As another preferred embodiment of the present invention, Figure 3 As shown, in step S1, the next moment state variable prediction matrix calculation model X is constructed k+1|k The method comprises the following steps:

[0087] S11: Based on the real-time monitoring of the three-axis motion speed and three-axis acceleration of the human joints in the human coordinate system, construct the human joint velocity vector matrix V in the human coordinate system k and the human joint acceleration vector matrix A k Then according to the human joint velocity vector matrix V k and the human joint acceleration vector matrix A k Construct the human body state variable matrix X at time k k :

[0088] in

[0089]

[0090]

[0091] Where, q∈Q, Q=1,2…,15, Q is the total number of human joints monitored in real time, is the x-axis velocity of the qth joint at time k in the human body coordinate system, is the y-axis velocity of the q-th joint at time k in the human body coordinate system, is the z-axis velocity of the qth joint at time k in the human body coordinate system; is the x-axis acceleration of the qth joint in the human coordinate system at time k, is the y-axis acceleration of the qth joint in the human body coordinate system at time k, is the z-axis acceleration of the q-th joint in the human body coordinate system at time k; k∈{i, j}, k+1∈{i, j}, {i, j} is the set of all time points from the i-th time to the j-th time;

[0092] S12: Based on the three-axis angular velocity obtained by real-time monitoring, construct the human motion transformation matrix M at time k in the human coordinate system k :

[0093] Among them, B is the state coefficient matrix, is the angular velocity matrix in the human body coordinate system at time k The skew-symmetric matrix of is the skew-symmetric matrix of angular velocity;

[0094] in, I3 is a 3×3 unit matrix, 03 is a 3×3 zero matrix with all elements being 0, η is the acceleration steady state calculation coefficient with Gauss-Markov white noise added,

[0095]

[0096] ε1 is the acceleration vector A towards the human joints k The Gauss-Markov white noise is added, and the human joint acceleration vector A is converted by adding Gauss-Markov white noise ε1 and calculating the acceleration constant state coefficient η of the Gauss-Markov white noise. k Transformed into modeling the dynamic behavior of motion acceleration through stochastic modeling methods

[0097] Angular velocity matrix in, is the x-axis angular velocity matrix in the human body coordinate system at time k monitored in real time by the three-axis gyroscope, is the y-axis angular velocity matrix in the human body coordinate system at time k monitored in real time by the three-axis gyroscope, is the z-axis angular velocity matrix in the human body coordinate system at time k monitored in real time by the three-axis gyroscope; is the x-axis angular velocity of the qth joint at time k in the human body coordinate system, is the y-axis angular velocity of the q-th joint at time k in the human body coordinate system, is the z-axis angular velocity of the qth joint at time k in the human body coordinate system;

[0098] Angular velocity skew-symmetric matrix

[0099] S13: Construct the next moment state variable prediction matrix calculation model X k+1|k :X k+1|k =M k X k , where the next moment state variable prediction matrix X k+1|k is the human body state variable matrix X based on time k k Calculated.

[0100] As another preferred embodiment of the present invention, in order to evaluate the state vector value X based on time k k To calculate the error of the state vector prediction matrix at the next moment, it is necessary to construct the state vector error prediction value calculation model P at the next moment k+1|k , the method is as follows Figure 4 As shown, in the method for human motion recognition provided by the present invention, a state vector error prediction value calculation model P for the next moment is constructed in S2 k+1|k The method comprises the following steps:

[0101] S21: According to the three-axis motion speed of the human joints in the human coordinate system obtained in real time according to step S1, the three-axis motion speed matrix in the human coordinate system at time k is constructed The skew-symmetric matrix Among them, the three-axis motion velocity matrix in the human body coordinate system at time k is is the x-axis motion velocity matrix in the human body coordinate system at time k monitored in real time by the three-axis gyroscope, is the y-axis motion velocity matrix in the human body coordinate system at time k monitored in real time by the three-axis gyroscope, is the z-axis motion velocity matrix in the human body coordinate system at time k monitored in real time by the three-axis gyroscope;

[0102]

[0103] S22: The three-axis motion velocity matrix in the human body coordinate system at time k constructed according to step S21 The skew-symmetric matrix Construct the human body state variable matrix X at time k k and the human body state variable matrix X at the next moment (i.e., moment k+1) k+1 The covariance matrix C between k :

[0104]

[0105] in, is the covariance of the three-axis motion velocity of Q joints in the human body coordinate system at the kth moment and the k+1th moment, is the covariance of the acceleration of the three-axis motion of Q joints in the human body coordinate system at the kth moment and the k+1th moment;

[0106] S23: Covariance matrix C constructed according to step S21 k And the human motion transformation matrix M at time k in the human coordinate system constructed in step S12 k , construct the state vector error prediction value calculation model P k+1|k :P k+1|k =M k P k (M k ) T +C k , where the state vector error prediction value P k+1|k is the predicted value P of the state vector error based on time k k Calculated.

[0107] As another preferred embodiment of the present invention, Figure 5 As shown, in step S4, the state vector error prediction value at the next moment is corrected by gain, and the human body state variable matrix X at the next moment is constructed. k+1 And the next moment accurate state vector prediction value calculation model P k+1, including the following steps:

[0108] S41: Calculation model P of the state vector error prediction value constructed according to step S2 k+1|k , construct the gain correction matrix G at time k+1 k+1 : G k+1 =P k+1|k (E k ) T [E k P k+1|k (E k ) T +L k ] -1 , where E k is the human body state variable matrix X at time k k Add measurement noise γ k The measured human body state variable matrix Y at time k+1 is obtained k+1 The coefficient matrix, Y k+1 =E k X k +γ k ;

[0109] S42: The next moment state variable prediction matrix X constructed according to step S1 k+1|k , the gain correction matrix G at time k+1 constructed in step S41 k+1 And the measured human body state variable matrix Y at time k+1 is calculated k+1 , calculate the human body state variable matrix X at time k+1 k+1 :

[0110] X k+1 =X k+1|k +G k+1 [Y k+1 -E k X k+1|k ];

[0111] S43: The gain correction matrix G at time k+1 constructed according to step S41 k+1 , coefficient matrix E k The state vector error prediction value calculation model P for the next moment constructed by step S2 k+1|k , construct the next moment accurate state vector prediction value calculation model P k+1 :P k+1 =(1-G k+1 E k ) k+1|k .

[0112] Among them, the coefficient matrix E in step S41 k for:

[0113] Among them, I3 is a 3×3 identity matrix, and 03 is a 3×3 zero matrix in which all elements are 0.

[0114] Example 3

[0115] Based on Example 2, as a preferred embodiment of the present invention, the fusion threshold condition in step S4 is 0.01<D(X i , X j )<0.3, if the fusion threshold D(X i , X j )≥0.3, repeat steps S1-S4 to ensure that the human motion parameters at each moment are smoothly integrated, so that the human motion state parameter result at the i-th moment and the human motion state parameter result at the j-th moment are smoothly integrated without interruption, forming accurate human motion state data from the i-th moment to the j-th moment;

[0116] The fusion threshold D(X) constructed in step S2 i , X j ) is calculated as follows:

[0117]

[0118] Among them, τ q is the i-th state variable matrix X i To the jth state variable matrix X j The weight coefficient of the qth joint in all state variable matrices of |X i | is the state variable matrix X at the i-th moment i The rank of |X j | is the state variable matrix X at the jth moment j rank.

[0119] As a preferred embodiment of the present invention, the weight coefficient τ of the qth joint in the state variable matrix is q The calculation formula is:

[0120]

[0121] Among them, τ intra is the internal variability coefficient of Q joints at the kth moment, τ inter is the internal variability coefficient of Q joints at the kth moment and other moments in the time interval between the ith moment and the jth moment;

[0122]

[0123]

[0124] Among them, H q(i, j) is the fusion adjustment calculation model of the q-th joint state variable matrix at the i-th moment and the j-th moment, Trace(H q (i, j)) is H q (i, j) is solved using the Trace(·) function, which is a MATLAB function commonly used in the art to find the sum of the elements on the diagonal of a square matrix.

[0125] As another preferred embodiment of the present invention, the qth joint state variable matrix fusion adjustment calculation model H at the i-th moment and the j-th moment q (i, j) is as follows:

[0126]

[0127] Wherein, mean(·) is the average function of Matlab commonly used in this field to calculate the average value, is the boundary position matrix of the qth joint at the kth moment, is the rank of each boundary position matrix from the i-th moment to the j-th moment, is the x-axis coordinate value of the qth joint in the human body coordinate system at the kth moment obtained by real-time monitoring in step S2, is the y-axis coordinate value of the qth joint in the human body coordinate system at the kth moment obtained by real-time monitoring in step S2, It is the z-axis coordinate value of the qth joint in the human body coordinate system at the kth moment obtained by real-time monitoring in step S2.

[0128] Example 4

[0129] like Figure 6 As shown, the human motion recognition system provided by the present invention includes a three-axis gyroscope, a three-dimensional positioning module, a data analysis and calculation module, and a central control module; the three-axis gyroscope is used to monitor the three-axis motion speed, three-axis motion angular velocity, and three-axis acceleration of human joints in the human coordinate system in real time;

[0130] A three-dimensional positioning module is used to monitor the three-dimensional motion coordinate values ​​of human joints in a human coordinate system in real time; the three-dimensional positioning module can be a GPRS positioning module, a GPS positioning module, a GNSS positioning module, and a Beidou navigation positioning module commonly used in the prior art;

[0131] The data analysis and calculation module is used to calculate the state variable prediction matrix at the next moment and construct the state vector error prediction value calculation model at the next moment according to the real-time monitoring parameters of the three-axis gyroscope, and to construct the fusion threshold calculation model;

[0132] The central control module is used to calculate the state variable prediction matrix at the next moment, perform gain correction on the state vector error prediction value at the next moment, and build the accurate state vector prediction value calculation model P at the next moment. k+1 , use the calculated accurate vector prediction value calculation results of the next moment to perform optimization iterations at different moments, judge whether the deviation of the state variable prediction values ​​at the i-th moment and the j-th moment meets the fusion threshold condition, and finally output the human body motion state parameter results of the i-th moment that are fused without interruption and smoothed with the human body motion state parameter results of the j-th moment to form accurate human body motion state data from the i-th moment to the j-th moment.

[0133] The artificial motion recognition system provided by the present invention may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0134] The artificial action recognition system provided by the present invention includes at least one type of scale storage medium, and the computer readable medium can be a computer readable signal medium or a computer readable storage medium or any combination of the above two. The computer readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer readable program code is carried. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0135] In some embodiments, the central processing module can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, which is used to run the program code stored in the central processing module or process data to complete steps S1-S4 of the human motion recognition method provided by the present invention.

[0136] It should be noted that the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments, and the technical terms "including", "comprising" or any other variants in the technical solution of the present invention are intended to cover non-exclusive inclusion, so that the process, device, article or method including a series of elements not only includes the technical feature elements after the above technical terms, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, device, article or method including the element.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0138] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A human motion recognition method, wherein the human motion recognition result is converted into feature information, wherein the feature information is 3D key point information, and wherein: The following steps are involved: S1: Real-time monitoring of the three-axis motion speed, three-axis motion angular velocity and three-axis acceleration of human joints in the human coordinate system, and building a prediction matrix calculation model for the state variables at the next moment; S2: Construct a calculation model for the state vector error prediction value at the next moment; S3: Real-time monitoring of the three-dimensional motion coordinate values ​​of human joints in the human coordinate system, and building a fusion threshold calculation model; S4: According to the next moment state variable prediction model matrix constructed in step S1 and the state vector error prediction value calculation model constructed in step S2, the state vector error prediction value at the next moment is gain corrected, and the human body state variable matrix at the next moment and the accurate state vector prediction value calculation model at the next moment are constructed; S5: Determine whether the deviation of the predicted value of the state variable at the i-th moment and the j-th moment meets the fusion threshold condition calculated in step S3, wherein j>i; if so, output the fused human action recognition result; if not, repeat steps S1-S4; The method for constructing the next moment state variable prediction matrix calculation model in step S1 comprises the following steps: S11: Based on the real-time monitoring of the three-axis motion speed and three-axis acceleration of the human joints in the human coordinate system, construct the human joint velocity vector matrix V in the human coordinate system k and the human joint acceleration vector matrix A k Then according to the human joint velocity vector matrix V k and the human joint acceleration vector matrix A k Construct the human body state variable matrix X at time k k : in Where, q∈Q, Q=1, 2…, 15, Q is the total number of human joints monitored in real time, is the x-axis velocity of the qth joint at time k in the human body coordinate system, is the y-axis velocity of the q-th joint at time k in the human body coordinate system, is the z-axis velocity of the qth joint at time k in the human body coordinate system; is the x-axis acceleration of the qth joint in the human coordinate system at time k, is the y-axis acceleration of the qth joint in the human body coordinate system at time k, is the z-axis acceleration of the q-th joint in the human body coordinate system at time k; k∈{i, j}, k+1∈{i, j}, {i, j} is the set of all time points from the i-th time to the j-th time; S12: Based on the three-axis angular velocity obtained by real-time monitoring, construct the human motion transformation matrix M at time k in the human coordinate system k : Among them, B is the state coefficient matrix, is the angular velocity matrix in the human body coordinate system at time k The skew-symmetric matrix of is the skew-symmetric matrix of angular velocity; in, I3 is a 3×3 identity matrix, 03 is a 3×3 zero matrix with all elements being 0, and η is the acceleration steady state calculation coefficient with Gauss-Markov white noise added; Angular velocity matrix in, is the x-axis angular velocity matrix of the human body coordinate system at time k monitored in real time, is the y-axis angular velocity matrix of the human body coordinate system at time k monitored in real time, is the z-axis angular velocity matrix in the human body coordinate system at time k monitored in real time; is the x-axis angular velocity of the qth joint at time k in the human body coordinate system, is the y-axis angular velocity of the q-th joint at time k in the human body coordinate system, is the z-axis angular velocity of the qth joint at time k in the human body coordinate system; the angular velocity skew-symmetric matrix S13: Construct the next moment state variable prediction matrix calculation model X k+1|k :X k+1|k =M k X k , where the next moment state variable prediction matrix X k+1|k is the human body state variable matrix X based on time k k Calculated; The method for constructing the state vector error prediction value calculation model at the next moment in step S2 comprises the following steps: S21: constructing a three-axis motion speed matrix of the human body coordinate system at time k according to the three-axis motion speed of the human body joints in the human body coordinate system obtained by real-time monitoring in step S1 The skew-symmetric matrix Among them, the three-axis motion velocity matrix in the human body coordinate system at time k is is the x-axis motion velocity matrix of the human body coordinate system at time k monitored in real time, is the y-axis motion velocity matrix of the human body coordinate system at time k monitored in real time, is the z-axis motion velocity matrix in the human body coordinate system at time k monitored in real time; S22: The three-axis motion velocity matrix in the human body coordinate system at time k constructed according to step S21 The skew-symmetric matrix Construct the human body state variable matrix X at time k k and the human body state variable matrix X at the next moment k+1 The covariance matrix C between k : in, is the covariance of the three-axis motion velocity of Q joints in the human body coordinate system at the kth moment and the k+1th moment, is the covariance of the acceleration of the three-axis motion of Q joints in the human body coordinate system at the kth moment and the k+1th moment; S23: The covariance matrix C constructed according to step S21 k And the human body motion transformation matrix M at time k in the human body coordinate system constructed in step S12 k , construct the state vector error prediction value calculation model P k+1|k :P k+1|k =M k P k (M k ) T +C k , where the state vector error prediction value P k+1|k is the predicted value P of the state vector error based on time k k Calculated; The step S4 performs gain correction on the state vector error prediction value at the next moment, constructs the human body state variable matrix at the next moment and the accurate state vector prediction value calculation model at the next moment, including the following steps: S41: Calculate the state vector error prediction value model P constructed according to step S2 k+1|k , construct the gain correction matrix G at time k+1 k+1 : G k+1 =P k+1|k (E k ) T [E k P k+1|k (E k ) T ] -1 , where E k is the human body state variable matrix X at time k k Add measurement noise γ k The measured human body state variable matrix Y at time k+1 is obtained k+1 The coefficient matrix, Y k+1 =E k X k +γ k ; S42: The next moment state variable prediction matrix X constructed according to the S1 step k+1|k , the gain correction matrix G at time k+1 constructed in step S41 k+1 And the measured human body state variable matrix Y at time k+1 is calculated k+1 , calculate the human body state variable matrix X at time k+1 k+1 : X k+1 =X k+1|k +G k+1 [Y k+1 -E k X k+1|k ]; S43: The gain correction matrix G at time k+1 constructed according to step S41 k+1 , coefficient matrix E k The state vector error prediction value calculation model P of the next moment constructed in step S2 k+1|k , construct the next moment accurate state vector prediction value calculation model P k+1 :P k+1 =(1-G k+1 E k ) k+1|k .

2. The human motion recognition method according to claim 1, characterized in that: The human body joints include the center point of the back of the head, the cervical vertebrae, the left shoulder, the left elbow, the left hand, the right shoulder, the right elbow, the right hand, the center point of the torso, the left hip, the left knee, the left foot, the right hip, the right knee, and the right foot.

3. The human motion recognition method according to claim 1, characterized in that: The coefficient matrix E in step S41 k for: Among them, I3 is a 3×3 identity matrix, and 03 is a 3×3 zero matrix in which all elements are 0.

4. The human motion recognition method according to claim 1, characterized in that: The fusion threshold condition is 0.01<D(X i , X j )<0.3, if the fusion threshold D(X i , X j )≥0.3, repeating steps S1-S4 to ensure that the human motion parameters at each moment are smoothly fused, so that the human motion state parameter result at the i-th moment and the human motion state parameter result at the j-th moment are smoothly and continuously fused to form accurate human motion state data from the i-th moment to the j-th moment; The fusion threshold D(X) constructed in step S3 i , X j ) is calculated as follows: Among them, τ q is the i-th state variable matrix X i To the jth state variable matrix X j The weight coefficient of the qth joint in all state variable matrices of |X i | is the state variable matrix X at the i-th moment i The rank of |X j | is the state variable matrix X at the jth moment j rank.

5. The human motion recognition method according to claim 4, characterized in that: The weight coefficient τ of the qth joint in the state variable matrix q The calculation formula is: Among them, τ intra is the internal variability coefficient of Q joints at the kth moment, τ inter is the internal variability coefficient of Q joints at the kth moment and other moments in the time interval between the ith moment and the jth moment; Among them, H q (i, j) is the fusion adjustment calculation model of the q-th joint state variable matrix at the i-th moment and the j-th moment, Trace(H q (i, j)) is H q (i, j) is solved using the Trace(·) function.

6. The human motion recognition method according to claim 4, characterized in that: The qth joint state variable matrix fusion adjustment calculation model H at the i-th moment and the j-th moment q (i, j) is as follows: Among them, mean(·) is the average function, is the boundary position matrix of the qth joint at the kth moment, is the rank of each boundary position matrix from the i-th moment to the j-th moment, is the x-axis coordinate value of the qth joint in the human body coordinate system at the kth moment obtained by real-time monitoring in step S2, is the y-axis coordinate value of the q-th joint in the human body coordinate system at the k-th moment obtained by real-time monitoring in step S2, The z-axis coordinate value of the qth joint in the human body coordinate system at the kth moment obtained by real-time monitoring in step S2.

7. A human motion recognition system, used to implement the human motion recognition method according to claim 1, characterized in that: The system includes a three-axis gyroscope, a three-dimensional positioning module, a data analysis and calculation module, and a central control module; the three-axis gyroscope is used to monitor the three-axis motion speed, three-axis motion angular velocity, and three-axis acceleration of human joints in the human coordinate system in real time; The three-dimensional positioning module is used to monitor the three-dimensional motion coordinate values ​​of human joints in the human body coordinate system in real time; The data analysis and calculation module is used to calculate the state variable prediction matrix at the next moment and construct a state vector error prediction value calculation model at the next moment according to the real-time monitoring parameters of the three-axis gyroscope, and to construct a fusion threshold calculation model; The central control module is used to calculate the state variable prediction matrix at the next moment, perform gain correction on the state vector error prediction value at the next moment, construct an accurate state vector prediction value calculation model for the next moment, determine whether the deviation of the state variable prediction values ​​at the i-th moment and the j-th moment meets the fusion threshold condition, and finally output the human body motion state parameter result at the i-th moment that is fused without interruption and smoothed with the human body motion state parameter result at the j-th moment to form accurate human body motion state data from the i-th moment to the j-th moment.

Images