Human action recognition and prediction method based on motion timing feature coding

Abstract

The application is suitable for the technical field of action recognition and prediction, and provides a human action recognition and prediction method based on motion time sequence feature coding, which comprises the following steps: establishing a human skeleton model, constructing an action sequence based on the human skeleton model, segmenting the action sequence according to an automatic segmentation model to obtain a sub-action sequence, extracting key frames from the sub-action sequence according to a clustering algorithm, encoding and assigning the key frames to obtain an encoding table of the key frame corresponding digital sequence, re-distributing the weight proportion of ordinary frames and key frames based on an attention mechanism, constructing a human action recognition model according to a long short-term memory recurrent neural network and the attention mechanism, and obtaining a test action sequence and an action recognition result according to the human action recognition model. The application can effectively improve the recognition and prediction efficiency of the model by using key frames to describe the motion sequence for the behavior action sequence of the human body.

Description

Technical Field

[0001] This application belongs to the field of action recognition and prediction technology, and in particular relates to a method for human action recognition and prediction based on motion temporal feature encoding. Background Technology

[0002] Seamless interaction between robotic or artificial intelligence systems and human-dominated urban environments requires a certain level of behavior recognition and prediction capabilities. Efficient action recognition and prediction technologies can help intelligent systems better understand human intentions and behaviors.

[0003] Meng et al. proposed a human motion recognition system suitable for embedded computer vision applications in security systems, human-computer interaction, and intelligent environments. They utilized hierarchical motion history histogram features to represent motion information, extracted a low-dimensional feature vector, and applied it to a human motion recognition system based on a support vector machine (SVM) classifier. A. Aravind Kumar designed a human activity recognition system based on video analysis, primarily employing a method that tracks the trajectories of pixels. This method uses signal processing techniques to identify their attributes, extracts features, trains the data using an SVM, and ultimately classifies new videos.

[0004] However, motion generation models based on 3D human pose are usually limited to a few activities and are difficult to generalize to new sports or applications. Typically, histograms are acquired for each frame and then calculated, resulting in poor real-time performance. Furthermore, during tracking, points or regions of interest need to be manually selected for matching, and some points may be lost when the region rotates or changes its shape, significantly impacting feature extraction. Summary of the Invention

[0005] The purpose of this application is to provide a human motion recognition method based on motion temporal feature encoding, which aims to solve the problem that motion generation models based on 3D human posture are usually limited to a few activities and are difficult to extend to new sports or applications.

[0006] The embodiments of this application are implemented as follows: a human action recognition method based on motion temporal feature encoding, the method comprising:

[0007] Establish a human skeleton model, and construct an action sequence based on the human skeleton model;

[0008] The action sequence is segmented according to the automatic segmentation model to obtain sub-action sequences;

[0009] Keyframes are extracted from the sub-action sequence using a clustering algorithm;

[0010] The keyframes are encoded and assigned values ​​to obtain an encoding table of the digital sequence corresponding to the keyframes;

[0011] Based on an attention mechanism, the weight ratio of ordinary frames and keyframes is reallocated; the attention mechanism is a weighted summation and weight reallocation.

[0012] A human motion recognition model is constructed based on the long short-term memory recurrent neural network and the attention mechanism.

[0013] Another objective of this application is a human motion prediction method based on motion temporal feature encoding, the method comprising:

[0014] Obtain the action sequence;

[0015] The action sequence is segmented according to the automatic segmentation model to obtain sub-action sequences;

[0016] Keyframes are extracted from the sub-action sequence using a clustering algorithm;

[0017] The keyframes are encoded and assigned values ​​to obtain an encoding table of the digital sequence corresponding to the keyframes;

[0018] Based on the Hidden Markov Model, the Hidden Markov Model parameters are trained and optimized according to the encoding table.

[0019] Action sequences are predicted using a keyframe-based NG-HMM model.

[0020] The human motion recognition method based on motion temporal feature encoding provided in this application establishes a human skeleton model and constructs a motion sequence based on the model. During human limb movement, angle and spatial position information of the limb bones can be obtained according to different semantics and postures of the movement. The motion sequence is segmented using an automatic segmentation model to obtain sub-motion sequences, combining the prediction and fitting characteristics of time series with the regularity of human motion over time series, which is more consistent with the motion laws and overcomes the limitation of traditional models that can only predict short-term motion sequences, allowing the model to segment long-term motion sequences. Keyframe extraction is performed on the sub-motion sequences using a clustering algorithm, which can easily achieve keyframe extraction and has good clustering effect, effectively recognizing motion. Similar frames in the sequence are categorized to achieve keyframe extraction. By encoding and assigning values ​​to these keyframes, an encoding table of corresponding digital sequences is obtained, transforming the original action sequence representing posture into a digital sequence, reducing computational complexity and improving the recognition rate of human actions. Based on an attention mechanism, the weight ratio of ordinary frames and keyframes is reallocated, aggregating motion feature representations with different weights to reduce information fragmentation. Attention is introduced into the human action sequence, giving keyframes more attention, enabling more effective human action recognition. A human action recognition model is constructed based on a long short-term memory recurrent neural network and the aforementioned attention mechanism. Frames with larger weight values ​​receive a larger proportion of network training, and frames carrying more useful information are more likely to be output as recognition results. This significantly improves data utilization and the performance of the recognition model. Attached Figure Description

[0021] Figure 1 A flowchart of a human motion recognition method based on motion temporal feature encoding provided in an embodiment of this application;

[0022] Figure 2 An angular feature map of human limb bone segments in the walking motion state of the human motion recognition method based on motion temporal feature encoding provided in the embodiments of this application;

[0023] Figure 3 A sequence of walking motions following attention keyframes in the human motion recognition method based on motion temporal feature encoding provided in this application embodiment;

[0024] Figure 4 The human skeleton and Euler angles of the human action recognition method based on motion temporal feature encoding provided in the embodiments of this application;

[0025] Figure 5 An LSTM mesh architecture diagram of the human action recognition method based on motion temporal feature encoding provided in the embodiments of this application;

[0026] Figure 6 A flowchart of a human motion prediction method based on motion temporal feature encoding provided in an embodiment of this application;

[0027] Figure 7 The NG-HMM model structure diagram for the human motion prediction method based on motion temporal feature encoding provided in the embodiments of this application;

[0028] Figure 8 The image shows motion data acquisition for the human motion recognition and prediction method based on motion temporal feature encoding provided in this application embodiment.

[0029] Figure 9 This is a block diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this application, a first script may be referred to as a second script, and similarly, a second script may be referred to as a first script.

[0032] like Figure 1 As shown, in one embodiment, a human action recognition method based on motion temporal feature encoding is proposed, which may specifically include the following steps:

[0033] Step S102: Establish a human skeleton model and construct an action sequence based on the human skeleton model;

[0034] In this embodiment, the method for recognizing human movements first involves collecting and processing motion data from different actions. During human limb movement, angle and spatial position information of the limb bones can be obtained based on the different semantics and postures of the movement. The human skeleton model in this application is a tree-like hierarchical structure model, consisting of a root node and multiple subtrees. The entire skeleton model can be roughly divided into 18 bone segments, each with a parent bone segment and several child bone segments, connected by joints.

[0035] Step S104: Segment the action sequence according to the automatic segmentation model to obtain sub-action sequences;

[0036] This application proposes an unsupervised segmentation algorithm based on the structural representation of the angles between limb bone segments and fitting an autoregressive moving average (ARMA) model. It combines the predictive and fitting characteristics of the ARMA model over time with the regularity of human movement over time. The algorithm calculates the time inflection points in the human motion sequence, identifies and extracts these inflection points using a fitness algorithm, and achieves motion sequence segmentation. This method overcomes the limitation of the ARMA model, which is only suitable for predicting short-term motion sequences, and allows the ARMA model to segment long-term motion sequences.

[0037] Step S106: Extract keyframes from the sub-action sequence using a clustering algorithm;

[0038] In this embodiment, after automatically segmenting a complex action sequence containing multiple actions into multiple sub-action sequences using an automatic segmentation model, the K-Means clustering algorithm is used to extract keyframes from the sub-action sequences. The K-Means algorithm is an unsupervised learning algorithm characterized by its relatively simple implementation and good clustering effect. This algorithm has a wide range of applications. Furthermore, the K-means algorithm can effectively classify similar frames in an action sequence to achieve the purpose of keyframe extraction.

[0039] Step S108: Encode and assign values ​​to the key frames to obtain the encoding table of the digital sequence corresponding to the key frames;

[0040] In this embodiment, a label encoding method is used to reduce the dimensionality of the feature matrix of the action sequence. The extracted human action keyframes are encoded and assigned values, transforming the original action sequence into a numerical sequence, reducing computational complexity and improving the recognition rate of human actions. During the construction of the encoding table, since action data possesses temporal characteristics, it is necessary to analyze these temporal characteristics for better encoding processing. Taking walking as an example... Figure 2 The images show the temporal angular characteristics of the human limbs during walking. It can be observed that the bone regions of the same limb exhibit periodicity and causality in time. This can serve as a basis for determining the keyframe type.

[0041] Step S110: Based on the attention mechanism, the weight ratio of ordinary frames and keyframes is reallocated; the attention mechanism is a weighted summation and weight reallocation.

[0042] In this embodiment, many motion frames in a large amount of motion data provide the same useful information, while some impressive motion frames may contain the most discriminative information that records the main movements. Therefore, a keyframe-based attention mechanism is used for attention allocation, and motion feature representations with different weights are aggregated to reduce information fragmentation. Introducing attention into the human motion model, giving keyframes more attention, can more effectively identify human motion movements. Figure 3 This is for extracting the walking motion sequence after keyframes.

[0043] Step S112: Construct a human action recognition model based on the long short-term memory recurrent neural network and the attention mechanism;

[0044] In this embodiment, after assigning different weight ratios to action frames in the action sequence, the action sequence with attention weights and the corresponding digit sequence are input into a Long Short-Term Memory (LSTM) recurrent neural network for training. Action frames with larger weights will receive a larger proportion of network training, and action frames carrying more useful information are more likely to be output as recognition results. This greatly improves data utilization and the performance of the recognition model.

[0045] In one embodiment, such as Figure 4 As shown, a human skeleton model is established, and an action sequence is constructed based on the human skeleton model. Specifically, this may include the following steps:

[0046] Step S202: Select the hip as the root node of the human skeleton model;

[0047] Step S204: Based on the root node, represent the positions of other nodes using spatial coordinates;

[0048] Step S206: Use feature vectors to describe the movement of the human skeleton model;

[0049] Step S208: Multiple actions form an action sequence.

[0050] In this embodiment, when the human body moves, the movement of each limb can be described as the movement of that limb segment relative to its parent segment joint. The human limbs periodically switch between flexion and extension postures, exhibiting periodic changes, and these changes between limbs become correlated. Therefore, limb segmentation angles can be used to improve the semantic description of the motion sequence. In the establishment of the human skeleton model, the hip node is typically selected as the root node of the tree-like human skeleton model, constraining its child nodes. The skeleton model is represented by the spatial coordinates of each joint point; therefore, it is necessary to convert the rotation angle data of each joint point into joint coordinates.

[0051]

[0052] P = P root +O r-1 ...+O2+O1+O0

[0053] Where Mr is the rotation matrix of the joint, Proot is the position of the root node, and Or is the position of the child node relative to the parent node. Figure 4 (b) shows the rotation order of Euler angles in the Cartesian coordinate system ZXY, where roll is represented by r, y is represented by y, and pitch is represented by p.

[0054] In this embodiment, a feature vector based on manual feature fusion is selected. This feature vector contains three parts: the angle between adjacent bone segments, the angle between limb bone segments and the central bone segment (the bone segment between the hip joint and the vertebral joint), and the spatial distance from each bone segment to the central joint (hip joint). By combining the angles between limb bone segments with the spatial distances between joints, the semantics of the human body's overall movement posture can be effectively described. The movement angle features of different bone segments are determined by the variation in the angles between each bone segment. The calculation of the angles between limb bone segments and the central bone segment, and the angles between adjacent limb bone segments is as follows:

[0055]

[0056]

[0057] Where, θ∈[0, 180°], θ A θ is the direction vector on the central spinal segment partition. B ={θ a θ b , ..., θ h} represents the direction vectors on the various limb regions of the human body.

[0058] The three-dimensional spatial characteristics of movement in different bone segments are determined by the variation in the spatial distance between each limb bone segment and the central node. The calculation of the spatial distance between nodes is as follows:

[0059]

[0060] Among them, X A =(x a y a , z a ) and X B =(x b y b , z b These are the three-dimensional spatial coordinates of the central node (hip node) of the human skeleton and the three-dimensional spatial coordinates of each limb bone segment in the Cartesian coordinate system.

[0061] In one embodiment, the action sequence is segmented according to an automatic segmentation model to obtain sub-action sequences, which may specifically include the following steps:

[0062] Step S302: Fit the motion sequence to the automatic segmentation model;

[0063] Step S304: Select the segmentation window, calculate the segmentation points according to the automatic segmentation model, and obtain the sub-action sequence.

[0064] In this embodiment, the ARMA model is an important model for studying time series data. It consists of an autoregressive (AR) model and a moving average (MA) model. In the ARMA model, one variable Y... t Data at any time t is represented as its observation sequence Y. t-1 Y t-2 ..., Y t-p and historical random disturbance sequence ε t-1 ε t-2 、…,ε t-q A linear combination of p and q. ARMA(p, q) is shown below:

[0065] Y t =AR + MA, AR = c + β1Y t-1 +β1Y t-2 +...+β p Y t-p’

[0066] MA=λ1ε t +λ2ε t-2 +...+λ q ε t-q +c

[0067] Where p and q are the orders of AR and MA, respectively. β p and λ q ...

[0068] The ARMA model incorporates the angle features between each limb bone segment and the central spinal segment in a human limb movement sequence. The ARMA model of the angle sequence is expressed by the formula:

[0069]

[0070] Where θ i The fitted data are for the angles between limb bone segments. These are linear approximation coefficients. It represents the residual.

[0071] After fitting the motion sequence to the ARMA model, a suitable segmentation window is selected, and segmentation points for the limb bone angle feature sequence are calculated based on the ARMA model. The limb bone angle information sequence of the human skeletal posture is extracted, and median filtering is used to obtain the final set of segmentation points. The formula can then be derived as follows:

[0072]

[0073] s = median(S) i )

[0074] In one embodiment, extracting keyframes from the sub-action sequence using a clustering algorithm may specifically include the following steps:

[0075] Step S402: Divide the sub-action sequence into multiple clusters, calculate the mean vector of the clusters, wherein the clusters have the same dimension and the sum of squared errors within the clusters is minimized;

[0076] Step S404: Based on the Euclidean distance formula, recalculate the mean vector of the cluster to obtain a definite mean vector;

[0077] Step S406: Determine the mean vector as the keyframe of the sub-action sequence and extract it.

[0078] In this embodiment, the K-Means algorithm divides the dataset into k clusters, where the action frames in each cluster are closest to the cluster center. For the motion sequence U = [u1, u2, ..., u...], ... n Assuming each sample has the same dimension, and a cluster set C = {C1, C2, ..., C} is used as the input to the model. k The K-Means algorithm can divide these n samples into k clusters, where 1 < k < n, and minimize the sum of squared errors within each cluster, E.

[0079]

[0080] Where μ i It is a cluster C i The mean vector, i.e. the centroid of the cluster.

[0081]

[0082] During algorithm execution, k points are randomly selected as initial cluster centers. Then, for each point in the dataset, its nearest cluster center is calculated. In this study, Euclidean distance is one of the most commonly used spatial distance metrics. This method is universal and applicable in three-dimensional space, as shown in the formula:

[0083]

[0084] Where μ i and μ j They are clusters C i and C j The mean vector. Recalculate the new centroid μ for all sample points in C. i This process continues until all centroid vectors no longer change, ultimately outputting the re-partitioned clusters. The k determined centroids are used as keyframes for the motion sequence and extracted.

[0085] In one embodiment, encoding the keyframes to obtain an encoding table of the corresponding digital sequences of the keyframes may specifically include the following steps:

[0086] Step S502: Determine the action type and key pose of the key frame;

[0087] Step S504: Based on the code table, assign a digital sequence corresponding to the key frame. The code table contains digital sequences corresponding to all action types and key poses.

[0088] In this embodiment of the application, for the establishment of the encoding table, a feature vector of the action pose of a keyframe is first defined as F. ak Where 'a' represents the 'a'th action type; and 'k' represents the 'k'th key pose. The code table contains the feature vectors of all action types and key poses, so the code table is also defined as: CB = {F...} ik}, i = 1, ..., I. The feature vectors of action types and key poses in the code table are arranged according to the time order of the training sample data, and include a total of k feature vectors. These action types and key poses are assigned values ​​1, 2, ..., K according to their order in the code table. The key poses of different action types are converted into numerical sequences {c1, c2, ..., c...} r This achieves the purpose of encoding, enabling human motion analysis methods to be better applied to various behavioral tasks.

[0089] In one embodiment, based on an attention mechanism, the weight ratio between ordinary frames and keyframes is reallocated, which may specifically include the following steps:

[0090] Step S602: Determine the correlation between ordinary frames and keyframes in the sub-action sequence;

[0091] Step S604: Calculate the weight values ​​of ordinary frames and keyframes in the sub-action sequence based on their relevance.

[0092] In this embodiment, an attention mechanism is applied to assign weights to keyframes generated under different motion types. The attention mechanism is a weighted summation and weight allocation mechanism that filters and extracts frames in the sequence that are highly similar to the keyframes, and then redistributes the weight ratios of these frames according to the attention mechanism-based weight values. Specifically, the similarity between different frames and the keyframes in the feature sequence is used to determine their proportion in the redistributed weight ratio.

[0093] In this embodiment of the application, it is assumed that the total number of frames of the feature action sequence U is n, according to the formula The transformed feature sequence U′ is obtained as U′=[α1U1,α2U2,…,α i U i ,…α n U n ], where U i The feature matrix before processing, α i This refers to the weights for each action frame. The key to the method is calculating a suitable α. i .

[0094] Qi represents the correlation between each frame in the feature sequence and the keyframe. A higher correlation results in a higher weight assigned to the corresponding frame; conversely, a lower correlation results in a lower weight assigned to the corresponding frame. The correlation calculation formula is as follows:

[0095]

[0096] Where cov(·,·) is the covariance between ordinary frames and keyframes in the feature sequence, and σ U σ represents the standard deviation of ordinary frames in the feature sequence. L Let be the standard deviation of the keyframes. The correlation Q between each frame in the feature sequence and the keyframes is obtained. i Then, the weight values ​​assigned to each frame can be obtained.

[0097] α i =f(Q) i U i )

[0098] Where f(·) represents the correlation Q between frames. i The weighting function is defined by α, which represents the output probability of the current sequence. α represents the final state of the action in the motion sequence and serves as the attention weight value for each action frame.

[0099] In one embodiment, such as Figure 5 As shown, a human action recognition model is constructed based on a long short-term memory recurrent neural network and the attention mechanism, which may specifically include the following steps:

[0100] Step S702: Input the sample feature action sequence and label number sequence into the Bi-LSTM layer;

[0101] Step S704: Output the trained state matrix using the fully connected layer;

[0102] Step S706: The network output is converted into probability vectors for various action states through a softmax layer;

[0103] Step S708: Obtain relevant parameters for different action types through the classification layer.

[0104] In the embodiments of this application, such as Figure 5 The LSTM network structure shown takes a sequence of sample action features of length N and dimension 20, and a sequence of labeled numbers of length N and dimension 1, containing 20 action states, as input to a Bi-LSTM layer. This yields a feature sequence with 128 hidden states. A fully connected layer (FC) then outputs a trained state matrix of length N and dimension 20. A softmax layer converts the network output into probability vectors for each action state. Finally, a classification layer obtains the relevant parameters for different action types.

[0105] In one embodiment, such as Figure 6 As shown, a method for predicting human actions based on motion temporal feature encoding includes:

[0106] Step S802, obtain the action sequence;

[0107] Step S804: Segment the action sequence according to the automatic segmentation model to obtain sub-action sequences;

[0108] Step S806: Extract keyframes from the sub-action sequence using a clustering algorithm;

[0109] Step S808: Encode and assign values ​​to the key frame to obtain the encoding table of the digital sequence corresponding to the key frame;

[0110] Step S810: Based on the Hidden Markov Model, train and optimize the Hidden Markov Model parameters according to the encoding table;

[0111] Step S812: Establish a keyframe-based NG-HMM model to predict action sequences.

[0112] In this embodiment, a portion of the prediction method is based on related steps in the recognition method. Human behavior can be viewed as an observable sequence, and the sequence of human motion postures is dynamic in time; that is, the state to which the observed behavioral skeleton image belongs changes over time. For example, a behavioral cycle begins by approaching one state, and after the posture changes over time, it approaches another state, then a third, a fourth, and so on. For a certain action, its characteristics are relatively stable within a sufficiently small time period, and the overall process can be viewed as a transition from one characteristic to another. Therefore, a Markov chain can be used to describe a behavior, connecting many linear models.

[0113] During human movement, the tension and relaxation of skeletal muscles follow a specific sequence. Therefore, we can view a behavior as a series of consecutive postures. In this process, the current posture depends only on the previous posture, which perfectly conforms to the Markov property. From a statistical perspective, a person's movement cycle can be viewed as a probabilistic model, where there is a probabilistic dependency between the state at a certain moment (such as standing, walking, running, etc.) and the possible states at the next moment. This dependency can be represented by a probability transition matrix, whose elements represent the probability of transitioning from one state to another. During the behavioral cycle, a person may remain in a specific state for a period of time, and this dwell time can be characterized by a probability distribution. Therefore, statistical methods can be used to model and analyze a person's movement cycle. A Markov matrix can describe the transition relationships between these states.

[0114] The matrix encodes dynamic information, including the duration density of motion states at a given time point and the transition probability from one state to another. This encoding method helps us analyze the duration and transition patterns of human motion cycles when modeling and analyzing them. However, for the observer, the dwell time and transition of a certain state in human motion is a hidden process that is not easily observed directly. Therefore, it is necessary to predict and estimate the change process of motion states from the observed motion data using probabilistic inference techniques. By observing the skeleton image sequence of the action posture, the state and its transition can be indirectly obtained, making Hidden Markov Model (HMM)-based methods suitable for modeling and describing it.

[0115] In one embodiment, training and optimizing the parameters of a Hidden Markov Model (HMM) based on the encoding table may specifically include the following steps:

[0116] Step S902: Add the digital sequence corresponding to the keyframe to the corresponding HMM model parameters;

[0117] Step S904: Train the HMM model on the digit sequence and optimize the model parameters using the Baum-Welch algorithm;

[0118] Step S906: Output the trained probability distribution matrix using the Viterbi algorithm and infer the motion intention.

[0119] In this embodiment of the application, for a finite number of different pose state sequences Q = {q1, q2, ... q...} N}, where N is the number of states in the model, and the state at time t can only be {q1, q2, ... q}. N One of the elements in}. For a random vector O = {o1, o2, ..., o} T Each observation vector has a corresponding output probability for different states. Here, T represents the length of the time series. Each action can be well modeled by a set of Hidden Markov Model parameters as λ = (A, B, π). Using Bayes' rule P(O... i |λ) is used to calculate the probability of the action generated by the model. Here, parameter A is the matrix representing the state transition probability, parameter π is the initial state probability distribution, and parameter B represents the output probability of all states.

[0120] For the hidden states in the Hidden Matrix Model (HMM), we replace the real frames in the original action sequence with 20 motion keyframes (including five action categories) obtained after clustering, serving as the hidden states in the HMM model. The feature matrix of each frame in the test sequence is used as the observation. A motion database is built using the sample motion data, and the initial parameters of the recognition model are obtained through the temporal features of the action data in the database.

[0121] Assume the sample starts from the hidden state q i Transfer to q j The frequency count is A ij The state transition matrix is ​​as follows:

[0122] A = [α] ij ]

[0123]

[0124] Assume the hidden state of the sample is q j And the observation state is O k The frequency count is B jk We will observe the state O = [O1, O2...O2]. T Each action frame in the image is compared with the 20 previously extracted action keyframes using a formula to obtain the output probability for all states:

[0125]

[0126] Where m and n represent the row and column of each action frame, and These are the average values ​​of the state matrix and the observation matrix, respectively. Assume the initial hidden state for all samples is q. i Let the frequency count be C(i), then the initial probability distribution is as follows:

[0127]

[0128] To perform the prediction task, the parameter λ needs to be trained on the sample data given a sequence of motion samples. The initial parameter model λ is obtained from the motion database. I The parameter λ is refined iteratively using the Baurn-Welch algorithm. State transition probabilities. and initial probability distribution It is the result of re-estimation, and the HMM model parameters are obtained after training with the Baum-Welch algorithm.

[0129] In this embodiment, during the testing phase, test data is fed into the trained HMM model, and the resulting Bayesian probability distribution matrix represents its likelihood. For the probability distribution matrix obtained from the test, we use the Viterbi algorithm to estimate the probability of ending all possible paths and calculate the optimal hidden state path, i.e., solving the problem of maximizing P(I|O, λ), defining δ... t (i) is along a state path q1, q2, ..., q t And q t =θ i In this case, the observation sequence O = {o1, o2, ... o} is obtained. t The maximum probability value of}:

[0130]

[0131] Calculate the optimal path sequence Q * The relevant calculation formula is:

[0132] Initialization: δ1(i) = π i b i (o1), ψ1(i)=0, 1≤i≤N

[0133] recursion:

[0134] termination:

[0135]

[0136] Optimal state path sequence:

[0137]

[0138] Where N is the number of states in the model, δ t (i) is the cumulative output probability of hidden state i in time period t, i.e., output O = {o1, o2...o...} t The state transition path with the highest probability q1, q2, ..., q t The probability value in ψ t (i) is a two-dimensional array, if at time t it is in state q i Then the state at time t-1 is ψ t (i), argmax represents making δ t (i) The value of i when it is at its maximum. Let P be the state of the optimal state path sequence at time t. * This is the final probability output value. The optimal hidden state sequence can be obtained using this algorithm.

[0139] In one embodiment, an action sequence is predicted using a keyframe-based NG-HMM model. The probability of the optimal discrete hidden state sequence is predicted based on the HMM model, and the NG-HMM model is established.

[0140] In this embodiment, the prediction method represents a series of human actions as a discrete time series based on keyframes. This discrete time series can be viewed as a text sequence, where each key pose can be considered a basic unit (word), and the probability of the next movement is predicted based on previous movements (context); for example, the position and direction of the next foot can be inferred based on a person's gait and walking habits. Furthermore, human movement is a temporal and causal process, and its output variables are not independent but interdependent. Therefore, it can be constructed as a structured prediction problem. The N-gram features of the N-gram model describe the contextual information of continuous moments (positions), largely reflecting the concept of "structure" in structured prediction tasks. Therefore, an NG-HMM prediction model is constructed by improving the prediction method based on the HMM model and combining the N-gram language model with a human motion database based on motion keyframes.

[0141] In the embodiments of this application, such as Figure 7 The NG-HMM model structure diagram shown illustrates how, based on the Viterbi algorithm predicting the optimal hidden state path, an N-Gram model is derived that can extract the transition structure of motion states based on keyframes. The model uses a standard database to determine the position of the local state action sequence in the codebook space, as well as the subsequent motion states, achieving medium- to long-term sequence prediction of motion poses.

[0142] Continuous motion data can be represented as a series of segmented motion pattern data O = {o1, o2, ..., o...} t ..., o T Using the previous HMM model, the optimal discrete hidden state path S = {S1, ..., S2} is predicted. t-1 S t}, where t represents a continuous time period, S t This refers to the motion state within time interval t. The characteristics of this motion state are relatively stable within time interval t, and it can be represented using a corresponding motion keyframe. Assume the hidden state search space of path S is Q = {q1, q2, ... q...} N}, then the predicted state s for time period T+1 T+1 The probability of occurrence can be used to build an N-Gram model based on the probability of the previous optimal hidden state path. Using the state transition matrix in the HMM model, the formula can be derived.

[0143]

[0144] Here, N is the number of states in the model, and C(·) is the frequency of the state-action sequence appearing in the prediction results of the training actions. The set of transition probabilities and output probabilities forms a directed graph. Furthermore, effectively utilizing the repetitiveness of motion in long-term history can achieve medium- to long-term motion prediction.

[0145] In this embodiment, the network was built and the algorithm was designed using a Matlab-based data analysis programming language, and the BVH data files were exported using Axis Neuron Pro motion capture software. Simultaneously, the Perception Neuron Pro inertial motion capture device manufactured by Noitom was used for data acquisition.

[0146] Perception Neuron is an inertial sensor-based motion capture system consisting of sensors, signal collection and transmission equipment, and a signal processing system. The inertial sensors are responsible for acquiring data, the signal collection and transmission equipment collects and transmits the signals acquired by the sensors, and finally, the signal processing system uses the PerceptionNeuron software on a PC to process the signals acquired by the sensors, calculate the relative offset positions of the joints in the human body, and thus obtain and reconstruct three-dimensional motion data. The Perception Neuron system combines navigation and orientation systems and has advantages such as simple operation and high data acquisition accuracy.

[0147] The Perception Neuron motion capture system utilizes wearable IMU sensors (including accelerometers, gyroscopes, and magnetometers) to measure data such as body acceleration, angular velocity, and magnetic field strength. The system can simultaneously acquire data from multiple sensors to obtain more accurate human motion posture data. During data acquisition, sensors are placed on key parts of the body, such as the head, arms, waist, and feet. As the body moves, the sensors measure a range of motion-related information, such as acceleration, angular velocity, and the strength of the Earth's magnetic field. This data is then wirelessly transmitted to a computer for processing and storage. The Perception Neuron system uses high-precision data processing algorithms to filter out noise and calibrate the acquired data, ensuring its accuracy and reliability. The acquired data can be stored in its raw format or a standard motion capture format for subsequent signal processing and 3D reconstruction applications.

[0148] In this embodiment of the application, in order to design and evaluate the proposed motion recognition system, motion data of four subjects were measured, including three men and one woman. Figure 8 It shows different types of movement postures during the measurement, including walking, running, raising hands, squatting, and raising legs.

[0149] It should be understood that although the steps in the flowcharts of the various embodiments of this application are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0150] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A human motion recognition method based on motion temporal feature encoding, characterized in that, The method includes: Establish a human skeleton model, and construct an action sequence based on the human skeleton model; The action sequence is segmented using an automatic segmentation model to obtain sub-action sequences; keyframes are extracted from the sub-action sequences using a clustering algorithm. The keyframes are encoded and assigned values ​​to obtain an encoding table for the corresponding digital sequence of the keyframes; Based on an attention mechanism, the weight ratio of ordinary frames and keyframes is reallocated; the attention mechanism is a weighted summation and weight reallocation. Based on the long short-term memory recurrent neural network and the attention mechanism, a human action recognition model is constructed; The step of segmenting the action sequence according to the automatic segmentation model to obtain sub-action sequences includes the following steps: Fit the action sequence to the automatic segmentation model; Select a segmentation window, calculate the segmentation points according to the automatic segmentation model, and obtain the sub-action sequence; the automatic segmentation model is an autoregressive moving average model. The step of encoding and assigning values ​​to the keyframes to obtain the encoding table of the corresponding digital sequences of the keyframes includes: Determine the action type and key pose of the keyframe; Based on a code table, a digital sequence is assigned to each key frame. The code table contains digital sequences corresponding to all action types and key poses.

2. The human motion recognition method based on motion temporal feature encoding according to claim 1, characterized in that, The process of establishing a human skeleton model and constructing an action sequence based on the human skeleton model includes the following steps: Choose the hip as the root node of the human skeleton model; Based on the root node, the positions of other nodes are represented using spatial coordinates; The movement of the human skeleton model is described using feature vectors; Multiple actions form an action sequence.

3. The human motion recognition method based on motion temporal feature encoding according to claim 1, characterized in that, The keyframe extraction of the sub-action sequence based on the clustering algorithm includes: The sub-action sequences are divided into multiple clusters, and the mean vector of each cluster is calculated. The clusters have the same dimension and the sum of squared errors within the cluster is minimized. Based on the Euclidean distance formula, the mean vector of the cluster is recalculated to obtain a definite mean vector; The mean vector is determined as the keyframe of the sub-action sequence and extracted.

4. The human motion recognition method based on motion temporal feature encoding according to claim 1, characterized in that, The attention-based mechanism for reallocating the weight ratio between ordinary frames and keyframes includes: Determine the correlation between ordinary frames and keyframes in the sub-action sequence; The weights of ordinary frames and keyframes in the sub-action sequence are calculated based on their relevance.

5. The human motion recognition method based on motion temporal feature encoding according to claim 1, characterized in that, The construction of a human action recognition model based on a long short-term memory recurrent neural network and the attention mechanism includes: The sample feature action sequence and the label number sequence are input into the Bi-LSTM layer; The trained state matrix is ​​output using the fully connected layer; The softmax layer converts the network output into probability vectors for various action states. The relevant parameters for different action types are obtained through the classification layer.

6. A method for predicting human motion based on motion temporal feature encoding, characterized in that, The method includes: Obtain the action sequence; The action sequence is segmented according to the automatic segmentation model to obtain sub-action sequences; Keyframes are extracted from the sub-action sequence using a clustering algorithm; The keyframes are encoded and assigned values ​​to obtain an encoding table of the digital sequence corresponding to the keyframes; Based on the Hidden Markov Model, the Hidden Markov Model parameters are trained and optimized according to the encoding table. Establish a keyframe-based NG-HMM model to predict action sequences; The step of segmenting the action sequence according to the automatic segmentation model to obtain sub-action sequences includes the following steps: Fit the action sequence to the automatic segmentation model; Select a segmentation window, calculate the segmentation points according to the automatic segmentation model, and obtain the sub-action sequence; the automatic segmentation model is an autoregressive moving average model. The step of encoding and assigning values ​​to the keyframes to obtain the encoding table of the corresponding digital sequences of the keyframes includes: Determine the action type and key pose of the keyframe; Based on a code table, a digital sequence is assigned to each key frame. The code table contains digital sequences corresponding to all action types and key poses.

7. The human motion prediction method based on motion temporal feature encoding according to claim 6, characterized in that, The training and optimization of Hidden Markov Model (HMM) parameters based on the encoding table includes the following steps: Add the numerical sequence corresponding to the keyframe to the corresponding HMM model parameters; An HMM model was trained on the digit sequence, and the model parameters were optimized using the Baum-Welch algorithm. The Viterbi algorithm is used to output the probability distribution matrix after training and infer the motion intention.

8. The human motion prediction method based on motion temporal feature encoding according to claim 6, characterized in that, The action sequence is predicted using a keyframe-based NG-HMM model. The probability of the optimal discrete hidden state sequence is obtained based on the HMM model prediction, and the NG-HMM model is established.

Images