Motion recognition method and device, storage medium and electronic device

Abstract

The present invention discloses an action recognition method and device, a storage medium and an electronic device. The method comprises: extracting an action feature sequence of a target action currently performed by the target object from a target video clip containing a target object; inputting the action feature sequence into multiple context recognition models constructed based on an attention mechanism to obtain multiple emission matrix parameters; obtaining state transfer matrix parameters obtained by training a hidden Markov model constructed based on a Gaussian mixture distribution mechanism to a target state; integrating and calculating each emission matrix parameter and state transfer matrix parameter to obtain an action recognition coefficient corresponding to each action category; and identifying the target action category corresponding to the maximum value in the action recognition coefficient as the action category matching the target action. The present invention solves the technical problem of low recognition accuracy of existing action recognition technologies.

Description

Technical Field

[0001] The present invention relates to the field of motion recognition technology, and in particular to a motion recognition method and device, a storage medium and an electronic device. Background Art

[0002] Motion recognition is widely used in human-computer interaction, virtual reality, video surveillance, public safety and other fields, and has gradually become one of the most active research fields in computer vision. In terms of human-computer interaction, motion recognition enables machines to understand human behavior through human body movements. For example, somatosensory game consoles capture the player's movements in three-dimensional space and process the captured information data to make the characters in the game perform the same movements. This human-computer interactive entertainment method can achieve a better experience effect. For example, in smart homes, we can control various home appliances through different movements, making our lives more convenient. In the field of video surveillance, manual video analysis is not only labor-intensive and resource-intensive, but also not real-time. However, after the camera is equipped with a motion recognition algorithm, it can detect abnormal behavior of pedestrians in a timely manner and issue an alarm.

[0003] Existing motion recognition technology mainly uses relevant algorithms to establish relevant models for motion data. The established model learns the motion data, that is, updates the model parameters, so that it has human-like cognitive experience and can determine the category to which new motion data belongs. However, the existing motion recognition algorithm has the technical problem of low motion recognition accuracy. Summary of the invention

[0004] The embodiments of the present invention provide a motion recognition method and device, a storage medium and an electronic device, so as to at least solve the technical problem of low motion recognition accuracy existing in the existing motion recognition algorithms.

[0005] According to one aspect of an embodiment of the present invention, there is provided an action recognition method, comprising: extracting an action feature sequence of a target action currently performed by the target object from a target video clip containing the target object, wherein the action feature sequence is used to indicate the spatiotemporal variation characteristics of the target object when performing the target action; inputting the action feature sequence into multiple context recognition models constructed based on an attention mechanism to obtain multiple emission matrix parameters, wherein each of the multiple context recognition models constructed based on the attention mechanism is a neural network model for identifying an action category obtained by training based on multiple sample video clips; obtaining state transfer matrix parameters obtained by training a hidden Markov model constructed based on a Gaussian mixture distribution mechanism to a target state, wherein the hidden Markov is a neural network model for determining a hidden state category obtained by training based on multiple sample video clips; integrating and calculating each emission matrix parameter and state transfer matrix parameter to obtain an action recognition coefficient corresponding to each action category; and identifying the target action category corresponding to the maximum value in the action recognition coefficient as the action category matching the target action.

[0006] According to another aspect of an embodiment of the present invention, there is also provided an action recognition device, comprising: an extraction unit, for extracting an action feature sequence of a target action currently performed by a target object from a target video clip containing the target object, wherein the action feature sequence is used to indicate the spatiotemporal variation characteristics of the target object when performing the target action; an input unit, for inputting the action feature sequence into a plurality of context recognition models constructed based on an attention mechanism to obtain a plurality of emission matrix parameters, wherein each of the plurality of context recognition models constructed based on an attention mechanism is a neural network model for identifying an action category obtained by training based on a plurality of sample video clips; an acquisition unit, for obtaining state transfer matrix parameters obtained by a hidden Markov model constructed based on a Gaussian mixture distribution mechanism when trained to a target state, wherein the hidden Markov is a neural network model for determining a hidden state category obtained by training based on a plurality of sample video clips; a calculation unit, for integrating and calculating each emission matrix parameter and a state transfer matrix parameter respectively to obtain an action recognition coefficient corresponding to each action category; and an identification unit, for identifying the target action category corresponding to the maximum value in the action recognition coefficient as the action category matching the target action.

[0007] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned action recognition method when running.

[0008] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the motion recognition method through the computer program.

[0009] In an embodiment of the present invention, an action feature sequence of a target action currently performed by a target object is extracted from a target video clip containing a target object; the action feature sequence is input into multiple context recognition models constructed based on an attention mechanism to obtain multiple emission matrix parameters; the state transfer matrix parameters obtained by a hidden Markov model constructed based on a Gaussian mixture distribution mechanism when trained to a target state are obtained; the emission matrix parameters and the state transfer matrix parameters are integrated and calculated respectively to obtain action recognition coefficients corresponding to each action category; the target action category corresponding to the maximum value in the action recognition coefficient is identified as the action category matching the target action. At the same time, the correlation and importance of the sequence features are examined, and the connection between the various sub-actions contained in an action is fully considered, thereby achieving the technical effect of improving the action recognition rate and solving the technical problem of low action recognition accuracy in existing action recognition algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0011] Figure 1 is a schematic diagram of a network environment of an optional motion recognition method according to an embodiment of the present invention;

[0012] Figure 2 is a schematic diagram of an optional action recognition method according to an embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of another optional action recognition method according to an embodiment of the present invention;

[0014] Figure 4 is a schematic diagram of another optional action recognition method according to an embodiment of the present invention;

[0015] Figure 5 is a schematic structural diagram of an optional motion recognition device according to an embodiment of the present invention;

[0016] Figure 6 is a schematic diagram of an optional action recognition model according to an embodiment of the present invention; DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0019] According to one aspect of an embodiment of the present invention, a method for action recognition is provided. Optionally, as an optional implementation, the method for action recognition can be applied to, but is not limited to, Figure 1In the motion recognition system in the network environment shown, the motion recognition system includes a device 102, a network 110 and a server 112. The device 102 includes an extraction unit 103, an input unit 104, an acquisition unit 105, a calculation unit 106 and a recognition unit 107. Among them, the extraction unit 103 is used to extract the action feature sequence of the target action currently performed by the target object from the target video clip containing the target object, wherein the action feature sequence is used to indicate the spatiotemporal change characteristics when the target object performs the target action; the input unit 104 is used to input the action feature sequence into multiple context recognition models constructed based on the attention mechanism to obtain multiple emission matrix parameters, wherein each of the multiple context recognition models constructed based on the attention mechanism is a neural network model for identifying an action category obtained by training based on multiple sample video clips; the acquisition unit 105 is used to obtain the state transfer matrix parameters obtained by the hidden Markov model constructed based on the Gaussian mixture distribution mechanism when trained to the target state, wherein the hidden Markov is a neural network model for determining the hidden state category obtained by training based on multiple sample video clips; the calculation unit 106 is used to integrate and calculate each emission matrix parameter and state transfer matrix parameter respectively, and obtain the action recognition coefficient corresponding to each action category; the identification unit 107 is used to identify the target action category corresponding to the maximum value in the action recognition coefficient as the action category matched by the target action. Specifically, please refer to steps S101 to S110.

[0020] S101, the server 112 obtains a video clip and sends the obtained video clip to the network 110;

[0021] S102, the network 110 sends the video clip to the device 102;

[0022] S103, the device 102 extracts an action feature sequence of a target action currently performed by the target object from the target video segment containing the target object;

[0023] S104, the device 102 inputs the action feature sequence into multiple context recognition models built based on the attention mechanism to obtain multiple emission matrix parameters;

[0024] S105, the device 102 obtains state transfer matrix parameters obtained by training a hidden Markov model constructed based on a Gaussian mixture distribution mechanism to a target state;

[0025] S106, the device 102 integrates and calculates each emission matrix parameter and state transfer matrix parameter to obtain an action recognition coefficient corresponding to each action category;

[0026] S107, the device 102 identifies the target action category corresponding to the maximum value of the action recognition coefficient as the action category matching the target action;

[0027] S108, device 102 sends the recognition result to network 110

[0028] S109, the network 110 sends the recognition result to the server 112;

[0029] S110, the server 112 receives the recognition result and sends an indication message according to the recognition result.

[0030] In an embodiment of the present invention, an action feature sequence of a target action currently performed by a target object is extracted from a target video clip containing a target object; the action feature sequence is input into multiple context recognition models constructed based on an attention mechanism to obtain multiple emission matrix parameters; the state transfer matrix parameters obtained by a hidden Markov model constructed based on a Gaussian mixture distribution mechanism when trained to a target state are obtained; the emission matrix parameters and the state transfer matrix parameters are integrated and calculated respectively to obtain action recognition coefficients corresponding to each action category; the target action category corresponding to the maximum value in the action recognition coefficient is identified as the action category matching the target action. At the same time, the correlation and importance of the sequence features are examined, and the connection between the various sub-actions contained in an action is fully considered, thereby achieving the technical effect of improving the action recognition rate and solving the technical problem of low action recognition accuracy in existing action recognition algorithms.

[0031] Optionally, in this embodiment, the user device may be, but is not limited to, a computer device that supports running an application client, such as a mobile phone, a tablet computer, a laptop computer, a PC, etc. The server and the user device may, but is not limited to, implement data exchange through a network, and the network may include, but is not limited to, a wireless network or a wired network. The wireless network includes: Bluetooth, WIFI, and other networks that implement wireless communication. The wired network may include, but is not limited to: a wide area network, a metropolitan area network, and a local area network. The above is only an example, and this embodiment does not make any limitation to this.

[0032] See also Figure 2 According to one aspect of an embodiment of the present invention, there is provided a method for action recognition, comprising:

[0033] S202, extracting an action feature sequence of a target action currently performed by the target object from a target video clip containing the target object, wherein the action feature sequence is used to indicate spatiotemporal variation characteristics when the target object performs the target action;

[0034] S204, inputting the action feature sequence into a plurality of context recognition models constructed based on the attention mechanism to obtain a plurality of emission matrix parameters, wherein each of the plurality of context recognition models constructed based on the attention mechanism is a neural network model for identifying an action category obtained by training based on a plurality of sample video clips;

[0035] S206, obtaining state transfer matrix parameters obtained by training a hidden Markov model constructed based on a Gaussian mixture distribution mechanism to a target state, wherein the hidden Markov is a neural network model for determining hidden state categories obtained by training based on multiple sample video clips;

[0036] S208, integrating and calculating each emission matrix parameter and state transfer matrix parameter to obtain an action recognition coefficient corresponding to each action category;

[0037] S210, identifying the target action category corresponding to the maximum value of the action recognition coefficient as the action category matched by the target action.

[0038] In an embodiment of the present invention, an action feature sequence of a target action currently performed by a target object is extracted from a target video clip containing a target object; the action feature sequence is input into multiple context recognition models constructed based on an attention mechanism to obtain multiple emission matrix parameters; the state transfer matrix parameters obtained by a hidden Markov model constructed based on a Gaussian mixture distribution mechanism when trained to a target state are obtained; the emission matrix parameters and the state transfer matrix parameters are integrated and calculated respectively to obtain action recognition coefficients corresponding to each action category; the target action category corresponding to the maximum value in the action recognition coefficient is identified as the action category matching the target action. At the same time, the correlation and importance of the sequence features are examined, and the connection between the various sub-actions contained in an action is fully considered, thereby achieving the technical effect of improving the action recognition rate and solving the technical problem of low action recognition accuracy in existing action recognition algorithms.

[0039] In an optional embodiment of the present invention, in the above step S204, the action feature sequence is input into multiple context recognition models constructed based on the attention mechanism to obtain multiple emission matrix parameters, wherein each of the multiple context recognition models constructed based on the attention mechanism is a neural network model for identifying an action category obtained by training based on multiple sample video clips, and the following steps may be included:

[0040] S1, in each context recognition model, performs the following operations respectively:

[0041] S2, through the bidirectional long short-term memory module in the context recognition model, the position encoding of the currently input action feature sequence is performed to obtain the encoding result;

[0042] S3, through the multi-layer self-learning attention module connected to the bidirectional long short-term memory module, assigns the corresponding attention weight of each layer to the encoding result to obtain the emission matrix parameters corresponding to the context recognition model.

[0043] Optionally, the above context recognition model can adopt a BiLSTM-Self-Attention model structure: the main body of the model is composed of 6 layers of Multi-Self-Attention modules, and a BiLSTM module is used for position encoding. Therefore, the model fully considers the relationship between feature time series and gives different degrees of attention to each part of the feature. Through learning, the model pays more attention to the features that contribute greatly to action recognition.

[0044] Optionally, the BiLSTM-Self-Attention model structure can also be as follows Figure 6 The network structure shown includes:

[0045] Input layer: used to input sequence features of dimension (T, N) and feed them into the BiLSTM-Self-Attention model, where T indicates the sequence length and N indicates the number of features.

[0046] Embedding layer: maps the feature quantity of each dimension into a vector of fixed length;

[0047] LSTM layer: uses bidirectional LSTM to calculate the embedding vector;

[0048] Attention layer: Use Attention weighting on the results of bidirectional LSTM;

[0049] Output layer: Output layer, outputs specific results.

[0050] Through the method recorded in the above embodiment, a context recognition model built based on the attention mechanism is used to obtain multiple emission matrices, and the position encoding of the currently input action feature sequence is performed, which fully considers the relationship between the feature time series and gives different degrees of attention to each part of the feature. It is achieved that through learning, the model pays more attention to the features that contribute greatly to action recognition, thereby improving the accuracy of action recognition.

[0051] In another optional embodiment of the present invention, before the above step S202, extracting the action feature sequence of the target action currently performed by the target object from the target video clip containing the target object, may further include the following steps:

[0052] S1, obtain multiple sample video clips;

[0053] S2, training the initialized hidden Markov model using a plurality of sample video clips until a first training convergence condition is reached;

[0054] S3, determining the hidden Markov model that has reached the first training convergence condition as reaching the target state, and saving the state transfer matrix parameters in the hidden Markov model at the target state;

[0055] S4, using multiple sample video clips and labels corresponding to each hidden state category in the hidden Markov model under the target state, training the initialization context recognition model corresponding to each action category until the second training convergence condition is reached.

[0056] Through the method recorded in the above embodiment, by training the hidden Markov model before extracting action features and retaining the state transfer matrix parameters of the hidden Markov model obtained through training, the state transfer matrix parameters obtained through deep learning are realized, so that the recognition model can decompose the action into a combination of sequence sub-actions, fully consider the conversion relationship between sub-actions, and improve the accuracy of action recognition.

[0057] In another optional embodiment of the present invention, the aforementioned training of the initialization hidden Markov model using a plurality of sample video clips may further include:

[0058] S1, use the maximum expectation algorithm to train the initialized hidden Markov model to obtain the state transfer matrix parameters;

[0059] S2, obtaining the hidden state category corresponding to the sample feature sequence extracted from the sample video clip through the Vibit algorithm, and using the hidden state category as a reference label for training the context recognition model.

[0060] Optionally, the above hidden Markov model can be a GMM-HMM model. The number of hidden variables contained in each action is set to 5. The hidden variables in action recognition are the action fragments that each action may contain, such as: push-ups contain prone and push-ups, and jumping jacks contain opening and closing of both hands. The input feature dimension of the GMM-HMM model is (T, N), where T represents the sequence length and N represents the number of features.

[0061] Optionally, the EM algorithm may be used to train the GMM-HMM model and retain its state transfer matrix parameters.

[0062] Optionally, the hidden state category corresponding to the sequence is obtained by decoding with a Vibit algorithm, and the hidden state category is used as a label for subsequent deep learning model training.

[0063] Through the method recorded in the above embodiment, by training the hidden Markov model before extracting action features and retaining the state transfer matrix parameters of the hidden Markov model obtained through training, the state transfer matrix parameters and hidden state categories obtained through deep learning are realized, so that the recognition model can use the hidden state categories obtained through training as labels to train the context recognition model, thereby fully considering the conversion relationship between sub-actions and improving the accuracy of action recognition.

[0064] In an optional embodiment of the present invention, the above step S202, extracting the action feature sequence of the target action currently performed by the target object from the target video clip containing the target object, may also include:

[0065] S1, extracting the skeletal 3D key point features corresponding to the target object appearing in the target video clip;

[0066] S2, based on the positional relationship between the skeleton key points indicated by the skeleton three-dimensional key point features, calculate the action feature sequence when the target object performs the target action.

[0067] Optionally, several types of action video clips may be collected and divided into a training set and a test set, and then a Kinect 2.0 camera may be used to obtain 3D key point features of the human skeleton.

[0068] Through the method described in the above embodiment, a feature sequence is extracted according to the target video clip, and feature points with low sensitivity to limb length are selectively selected as the extraction objects of action features, thereby reducing the influence of the selection of action feature extraction position on the recognition result and improving the accuracy of action recognition.

[0069] In an optional embodiment of the present invention, the feature extraction step calculates the action feature sequence when the target object performs the target action based on the positional relationship between the skeleton key points indicated by the skeleton three-dimensional key point features, and may also include:

[0070] S1, determining the coordinate position relationship corresponding to each of a plurality of key point combinations based on the skeletal three-dimensional key point features, wherein each key point combination includes two skeletal three-dimensional key points;

[0071] S2, determining the angle change vector set of key point combination matching according to the coordinate position relationship;

[0072] S3, the angle change vector set of each key point combination matching is determined as the action feature sequence of the target action.

[0073] Optionally, in order to obtain more effective features for action classification, deep feature extraction is performed on 3D key points. Since different people have different limb lengths, angles with low sensitivity to limb length are used as action features.

[0074] Optionally, the key points of left shoulder-left elbow, right shoulder-right elbow, left elbow-left wrist, right elbow-right wrist, left shoulder-left hip, right shoulder-right hip, left hip-left knee, right hip-right knee, left knee-left ankle, and right knee-right ankle are combined into 10 limb combinations of interest, and the point closest to the center of the body is used as the origin (such as the left shoulder as the origin in the left shoulder-left elbow combination, and the left elbow as the origin in the left elbow-left wrist combination), the three-dimensional coordinates of another key point coordinate from the origin are calculated, and then the θ, ψ, and inter-frame angle change parameters in the spherical coordinate system are calculated, thereby obtaining 10*3, i.e. 30, deep motion features.

[0075] Through the method described in the above embodiment, a feature sequence is extracted according to the target video clip, and feature points with low sensitivity to limb length are selectively selected as the extraction objects of action features, thereby reducing the influence of the selection of action feature extraction position on the recognition result and improving the accuracy of action recognition.

[0076] Combine the following Figure 3 to Figure 4 , a specific embodiment of the present invention is illustrated.

[0077] Alternatively, if Figure 3 FIG. 2 shows the training process of the neural network model for identifying an action category according to the present invention.

[0078] S301, Kinect collects 3D key points: collects 3D key point information in video data through a Kinect 2.0 camera, wherein the video data includes an image of a specific human action, and the 3D key point information indicates a combination of key points on the human body related to the action;

[0079] S302, extracting spatial and temporal features: extracting spatial and temporal feature information sequences based on the collected three-dimensional key point information;

[0080] S303, retaining the state transfer matrix of the trained GMM-HMM: inputting the collected spatial and temporal feature information sequence as input into the trained GMM-HMM model, outputting the state transfer matrix parameters and hidden state category parameters, and retaining the state transfer matrix parameters of the GMM-HMM model;

[0081] S304, combining the hidden state category training of the GMM-HMM with the recognition model BiLSTM-Self-Attention: using the hidden state category parameters of the trained GMM-HMM model as labels for training a BiLSTM-Self-Attention model corresponding to a specific human body action, and obtaining a BiLSTM-Self-Attention-HMM model corresponding to a specific human body action.

[0082] Optionally, the input of the BiLSTM-Self-Attention-HMM model is the same as that of the GMM-HMM model. The BiLSTM-Self-Attention-HMM model is trained using the hidden state categories obtained by decoding the GMM-HMM model as labels, and is trained by gradient descent using SoftMax as the loss function. A BiLSTM-Self-Attention-HMM model needs to be trained for each action.

[0083] Optionally, the above steps S301-S304 may be performed multiple times for multiple video data of the same action, and a BiLSTM-Self-Attention-HMM model corresponding to a specific human action may be trained. At the same time, the above training steps may be performed for multiple actions, and BiLSTM-Self-Attention-HMM models corresponding to multiple actions may be trained.

[0084] Alternatively, if Figure 4 As shown, a specific action recognition method of the present invention is shown.

[0085] S401, Kinect collects 3D key points: collects 3D key point information in video data through a Kinect 2.0 camera, wherein the video data includes an image of a specific human action, and the 3D key point information indicates a combination of key points on the human body related to the action;

[0086] S402, extracting spatial and temporal features: extracting spatial and temporal feature information sequences based on the collected three-dimensional key point information;

[0087] S403, outputting emission matrix parameters: inputting the collected spatial and temporal feature information sequence as an input into a trained BiLSTM-Self-Attention-HMM model corresponding to a human action, and outputting emission matrix parameters corresponding to a specific action;

[0088] S404, calculating the action score by combining the state transfer matrix of the GMM-HMM obtained through the training: calculating the score corresponding to the action by combining the output emission matrix parameters and the state transfer matrix parameters of the GMM-HMM model retained corresponding to the specific action.

[0089] Optionally, the process of identifying an action by the BiLSTM-Self-Attention-HMM model can be as follows: first, the sequence features of dimension (T, N) are sent to the trained BiLSTM-Self-Attention-HMM model, where T indicates the sequence length and N indicates the number of features, so as to obtain the probability P(y|x) of the feature relative to the latent variable. P(y|x) converted into the generation probability is used as the emission matrix parameter of the HMM model, combined with the state transfer matrix parameters obtained by the GMM-HMM algorithm, and the action score under this sequence feature can be obtained by using the forward-backward algorithm. The action sequence is sent to N BiLSTM-Self-Attention-HMM models to obtain N action scores, and the action category corresponding to the model with the largest action score is taken as the recognition result.

[0090] In an embodiment of the present invention, an action feature sequence of a target action currently performed by a target object is extracted from a target video clip containing a target object; the action feature sequence is input into multiple context recognition models constructed based on an attention mechanism to obtain multiple emission matrix parameters; the state transfer matrix parameters obtained by a hidden Markov model constructed based on a Gaussian mixture distribution mechanism when trained to a target state are obtained; the emission matrix parameters and the state transfer matrix parameters are integrated and calculated respectively to obtain action recognition coefficients corresponding to each action category; the target action category corresponding to the maximum value in the action recognition coefficient is identified as the action category matching the target action. At the same time, the correlation and importance of the sequence features are examined, and the connection between the various sub-actions contained in an action is fully considered, thereby achieving the technical effect of improving the action recognition rate and solving the technical problem of low action recognition accuracy in existing action recognition algorithms.

[0091] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0092] According to another aspect of the embodiments of the present invention, a control device for implementing the above-mentioned motion recognition method is also provided. Figure 5As shown, the device comprises:

[0093] An extraction unit 501 is used to extract an action feature sequence of a target action currently performed by the target object from a target video clip containing the target object, wherein the action feature sequence is used to indicate the spatiotemporal variation characteristics of the target object when performing the target action;

[0094] An input unit 502 is used to input the action feature sequence into multiple context recognition models constructed based on the attention mechanism to obtain multiple emission matrix parameters, wherein each of the multiple context recognition models constructed based on the attention mechanism is a neural network model for identifying an action category obtained by training based on multiple sample video clips;

[0095] An acquisition unit 503 is used to acquire state transfer matrix parameters obtained by training a hidden Markov model constructed based on a Gaussian mixture distribution mechanism to a target state, wherein the hidden Markov is a neural network model for determining hidden state categories obtained by training based on multiple sample video clips;

[0096] A calculation unit 504 is used to integrate and calculate the parameters of each emission matrix and the parameters of the state transfer matrix to obtain the action recognition coefficient corresponding to each action category;

[0097] The identification unit 505 is used to identify the target action category corresponding to the maximum value of the action identification coefficient as the action category matched by the target action.

[0098] In an embodiment of the present invention, an action feature sequence of a target action currently performed by a target object is extracted from a target video clip containing a target object; the action feature sequence is input into multiple context recognition models constructed based on an attention mechanism to obtain multiple emission matrix parameters; the state transfer matrix parameters obtained by a hidden Markov model constructed based on a Gaussian mixture distribution mechanism when trained to a target state are obtained; the emission matrix parameters and the state transfer matrix parameters are integrated and calculated respectively to obtain action recognition coefficients corresponding to each action category; the target action category corresponding to the maximum value in the action recognition coefficient is identified as the action category matching the target action. At the same time, the correlation and importance of the sequence features are examined, and the connection between the various sub-actions contained in an action is fully considered, thereby achieving the technical effect of improving the action recognition rate and solving the technical problem of low action recognition accuracy in existing action recognition algorithms.

[0099] Optionally, according to another aspect of the embodiment of the present invention, the input unit 502 may further include an input subunit, and the input subunit is used to perform the following operations in each context recognition model:

[0100] Through the bidirectional long short-term memory module in the context recognition model, the position encoding of the currently input action feature sequence is performed to obtain the encoding result;

[0101] Through a multi-layer self-learning attention module connected to a bidirectional long short-term memory module, the encoding result is assigned the corresponding attention weight of each layer to obtain the emission matrix parameters corresponding to the context recognition model.

[0102] Optionally, according to another aspect of the embodiment of the present invention, the extraction unit may also be used for:

[0103] Obtain multiple sample video clips;

[0104] Using a plurality of sample video clips to train the initialized hidden Markov model until a first training convergence condition is reached;

[0105] Determine the hidden Markov model that has reached the first training convergence condition as reaching the target state, and save the state transfer matrix parameters in the hidden Markov model at the target state;

[0106] Using multiple sample video clips and labels corresponding to each hidden state category in the hidden Markov model under the target state, the initialization context recognition model corresponding to each action category is trained until the second training convergence condition is reached.

[0107] Optionally, according to another aspect of the embodiment of the present invention, when the extraction unit is used to train the initialized hidden Markov model using a plurality of sample video clips, it further includes:

[0108] The maximum expectation algorithm is used to train the initialized hidden Markov model to obtain the state transfer matrix parameters;

[0109] The latent state category corresponding to the sample feature sequence extracted from the sample video clip is obtained through the Vibit algorithm, and the latent state category is used as a reference label for training the context recognition model.

[0110] Optionally, according to another aspect of the embodiment of the present invention, the extraction unit 501 may further include:

[0111] A first extraction subunit is used to extract the skeletal three-dimensional key point features corresponding to the target object appearing in the target video clip;

[0112] The second extraction subunit is used to calculate the action feature sequence when the target object performs the target action based on the position relationship between the skeleton key points indicated by the skeleton three-dimensional key point features.

[0113] Optionally, according to another aspect of the embodiment of the present invention, the second extraction subunit may also be used for:

[0114] Determine the coordinate position relationship corresponding to each of the plurality of key point combinations based on the skeletal three-dimensional key point features, wherein each key point combination includes two skeletal three-dimensional key points;

[0115] Determine the angle change vector set of key point combination matching according to the coordinate position relationship;

[0116] The angle change vector set of each key point combination is determined as the action feature sequence during the target action.

[0117] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0118] According to another aspect of an embodiment of the present invention, an electronic device for implementing the above-mentioned action recognition method is also provided. The electronic device includes a memory and a processor. The memory stores a computer program. The processor is configured to execute the steps in any one of the above-mentioned method embodiments through the computer program.

[0119] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0120] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0121] S1, extracting an action feature sequence of a target action currently performed by the target object from a target video clip containing the target object, wherein the action feature sequence is used to indicate the spatiotemporal variation characteristics of the target object when performing the target action;

[0122] S2, inputting the action feature sequence into multiple context recognition models constructed based on the attention mechanism to obtain multiple emission matrix parameters, wherein each of the multiple context recognition models constructed based on the attention mechanism is a neural network model for identifying an action category obtained by training based on multiple sample video clips;

[0123] S3, obtaining state transfer matrix parameters of a hidden Markov model constructed based on a Gaussian mixture distribution mechanism when trained to a target state, wherein the hidden Markov is a neural network model for determining hidden state categories obtained by training based on multiple sample video clips;

[0124] S4, integrating and calculating each emission matrix parameter and state transfer matrix parameter respectively to obtain the action recognition coefficient corresponding to each action category;

[0125] S5, identifying the target action category corresponding to the maximum value of the action recognition coefficient as the action category matched by the target action.

[0126] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0127] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0128] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0129] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0133] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for motion recognition, characterized in that: include: Obtain multiple sample video clips; The initialization hidden Markov model is trained using the multiple sample video clips until a first training convergence condition is reached, including: using a maximum expectation algorithm to train the initialization hidden Markov model to obtain state transfer matrix parameters; obtaining a hidden state category corresponding to a sample feature sequence extracted from the sample video clip by a Vibit algorithm, and using the hidden state category as a reference label for training a context recognition model; Determine the hidden Markov model that reaches the first training convergence condition as reaching a target state, and save the state transfer matrix parameters in the hidden Markov model at the target state; Using the multiple sample video clips and the labels corresponding to the hidden state categories in the hidden Markov model under the target state, training the initialization context recognition model corresponding to each action category until a second training convergence condition is reached; Extracting an action feature sequence of a target action currently performed by the target object from a target video clip containing the target object, and selecting feature points on the target object whose sensitivity to limb length is lower than a certain threshold as extraction objects in the action feature sequence, wherein the action feature sequence is used to indicate the spatiotemporal variation characteristics of the target object when performing the target action; Inputting the action feature sequence into a plurality of context recognition models constructed based on the attention mechanism to obtain a plurality of emission matrix parameters, wherein each of the plurality of context recognition models constructed based on the attention mechanism is a neural network model for identifying an action category obtained by training based on a plurality of sample video clips; Obtaining state transfer matrix parameters of a hidden Markov model constructed based on a Gaussian mixture distribution mechanism when trained to a target state, wherein the hidden Markov is a neural network model for determining hidden state categories obtained by training based on the multiple sample video clips; Integrate and calculate the emission matrix parameters and the state transfer matrix parameters respectively to obtain the action recognition coefficient corresponding to each action category; The target action category corresponding to the maximum value of the action recognition coefficients is identified as the action category matching the target action.

2. The method according to claim 1, characterized in that: The action feature sequence is input into multiple context recognition models based on the attention mechanism to obtain multiple emission matrix parameters including: The following operations are performed in each of the context recognition models: Performing position encoding on the currently input action feature sequence through the bidirectional long short-term memory module in the context recognition model to obtain an encoding result; Through a multi-layer self-learning attention module connected to the bidirectional long short-term memory module, the encoding result is respectively assigned an attention weight corresponding to each layer to obtain the emission matrix parameters corresponding to the context recognition model.

3. The method according to claim 1, characterized in that Extracting an action feature sequence of a target action currently performed by the target object from a target video clip containing the target object comprises: Extracting skeletal three-dimensional key point features corresponding to the target object appearing in the target video clip; Based on the positional relationship between the skeleton key points indicated by the skeleton three-dimensional key point features, the action feature sequence when the target object performs the target action is calculated.

4. The method according to claim 3, characterized in that Based on the positional relationship between the skeleton key points indicated by the skeleton three-dimensional key point features, calculating the action feature sequence when the target object performs the target action includes: Determine the coordinate position relationship corresponding to each of a plurality of key point combinations based on the three-dimensional key point features of the skeleton, wherein each of the key point combinations includes two three-dimensional key points of the skeleton; Determine the angle change vector set of the key point combination matching according to the coordinate position relationship; The angle change vector set of each key point combination matching is determined as the action feature sequence during the target action.

5. A motion recognition device, characterized in that: include: An extraction unit, used for obtaining a plurality of sample video clips; The initialized hidden Markov model is trained using the multiple sample video clips until a first training convergence condition is reached, including: using a maximum expectation algorithm to train the initialized hidden Markov model to obtain state transfer matrix parameters; obtaining hidden state categories corresponding to sample feature sequences extracted from the sample video clips by a Vibit algorithm, and using the hidden state categories as reference labels for training a context recognition model; determining the hidden Markov model that has reached the first training convergence condition as reaching a target state, and saving the state transfer matrix parameters in the hidden Markov model under the target state; using the multiple sample video clips and the labels corresponding to each hidden state category in the hidden Markov model under the target state, training the initialized context recognition model corresponding to each action category until a second training convergence condition is reached; extracting an action feature sequence of a target action currently performed by the target object from a target video clip containing a target object, and selecting feature points on the target object whose sensitivity to limb length is lower than a certain threshold as extraction objects in the action feature sequence, wherein the action feature sequence is used to indicate the spatiotemporal variation characteristics of the target object when performing the target action; An input unit, used for inputting the action feature sequence into a plurality of context recognition models constructed based on an attention mechanism to obtain a plurality of emission matrix parameters, wherein each of the plurality of context recognition models constructed based on an attention mechanism is a neural network model for identifying an action category obtained by training based on a plurality of sample video clips; An acquisition unit, used to acquire state transfer matrix parameters obtained by training a hidden Markov model constructed based on a Gaussian mixture distribution mechanism to a target state, wherein the hidden Markov is a neural network model for determining hidden state categories obtained by training based on the multiple sample video clips; A calculation unit, used for integrating and calculating each of the emission matrix parameters and the state transfer matrix parameters to obtain an action recognition coefficient corresponding to each of the action categories; The recognition unit is used to recognize the target action category corresponding to the maximum value of the action recognition coefficient as the action category matched by the target action.

6. The device according to claim 5, characterized in that The input unit includes an input subunit, and the input subunit is used to perform the following operations in each of the context recognition models: Performing position encoding on the currently input action feature sequence through the bidirectional long short-term memory module in the context recognition model to obtain an encoding result; Through a multi-layer self-learning attention module connected to the bidirectional long short-term memory module, the encoding result is respectively assigned an attention weight corresponding to each layer to obtain the emission matrix parameters corresponding to the context recognition model.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the action recognition method described in any one of claims 1 to 4 when running.

8. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to execute the action recognition method according to any one of claims 1 to 4 through the computer program.

Images