Action recognition method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an action recognition method and device, computer equipment, a readable storage medium and a program product. The gesture sequence of the target object is obtained from the to-be-recognized video, action recognition is performed on the gesture sequence through the action recognition network model, and the action recognition result of the to-be-recognized video is obtained. According to the method, whether actions are compliant or not is judged in a manner of mapping a sample posture sequence of a training data set to a hidden layer space of Gaussian distribution for clustering, so that a model does not learn specific pixel points, but focuses on learning and understanding information about human body postures and actions contained in the sample posture sequence; the learning method is helpful for improving the generalization ability and robustness of the model, so that the recognition accuracy is improved, the cost of data labeling is reduced, and the consumption of computing resources is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an action recognition method, apparatus, computer equipment, readable storage medium and program product. Background Art

[0002] In the field of computer vision technology, gesture recognition and motion capture technology have been widely used. Through computer vision technology, people's gestures and movements can be monitored and analyzed in real time to determine whether they meet the standards. Common methods include convolutional neural networks (CNNs) based on deep learning.

[0003] However, the characteristics of traditional convolutional neural networks require a large amount of labeled data for training, which is not only time-consuming and labor-intensive, but also the detection performance is easily affected by the quality and quantity of data. Summary of the invention

[0004] Based on this, it is necessary to provide a motion recognition method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides an action recognition method, comprising:

[0006] Acquire a posture sequence of a target object from a video to be identified, wherein the posture sequence includes joint point information of the target object at different times;

[0007] The action recognition is performed on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized, wherein the action recognition result includes a result that the action of the target object in the video to be recognized is compliant or compliant. The action recognition network model is obtained by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution.

[0008] In one embodiment, the step of obtaining a posture sequence of a target object from a video to be recognized includes:

[0009] Sampling the video to be identified based on a preset sampling rate to obtain multiple video frames;

[0010] Performing target detection on each of the video frames to determine the position of each object in each of the video frames;

[0011] Extracting joint point information of each object in each video frame based on posture estimation according to the position of each object in each video frame;

[0012] Determine the joint point information of the target object in the multiple video frames by target tracking, wherein the target object is an object continuously tracked in the multiple video frames;

[0013] A posture sequence of the target object is generated according to the joint point information of the target object in the multiple video frames.

[0014] In one embodiment, performing action recognition on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized includes:

[0015] Mapping the posture sequence of the target object through the action recognition network model to obtain an intermediate distribution and a first activation value of the posture sequence;

[0016] In the hidden layer space, determining a feature representation of the posture sequence of the target object, performing an inverse operation on the feature representation to obtain a second activation value, wherein the second activation value is used to characterize the similarity between the distribution after the inverse operation and the intermediate distribution;

[0017] Determine an action recognition result of the video to be recognized according to the first activation value and the second activation value.

[0018] In one embodiment, performing action recognition on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized includes:

[0019] Inputting the posture sequence into the action recognition network model for action recognition, and if it is determined that the posture sequence is out-of-distribution data, obtaining the distribution point positions of the maximum deviation values ​​in the posture sequence;

[0020] Performing weighted summation on the distribution points of the maximum deviation value to obtain fused distribution points;

[0021] Obtaining the offset between the fused distribution point and the distribution center of the target set;

[0022] The action recognition result of the video to be recognized is determined based on the offset.

[0023] In one embodiment, the action recognition network model is obtained by mapping a sample posture sequence of a training data set to a hidden space of a Gaussian distribution, including:

[0024] Acquire a training data set, the training data set comprising sample posture sequences of a plurality of sample videos and corresponding classification labels, the classification labels being used to characterize classifications of compliance of actions in the sample videos;

[0025] Mapping the sample posture sequences of the plurality of sample videos to a hidden layer space to obtain a high-dimensional feature representation of the sample posture sequences;

[0026] According to the classification labels of the plurality of sample videos, the high-dimensional feature representation is modeled based on Gaussian distribution to obtain the action recognition network model.

[0027] In one embodiment, mapping the sample posture sequences of the plurality of sample videos to a hidden layer space to obtain a high-dimensional feature representation of the sample posture sequences includes:

[0028] Normalizing the sample pose sequence of the sample video through a normalization layer to obtain a normalized sample pose sequence;

[0029] By using a sorting layer, the sorting of input channels is permuted to convert the normalized sample posture sequence into a multi-dimensional matrix;

[0030] An affine transformation is performed on half of the multidimensional matrix through an affine layer, and a feature representation of the sample posture sequence is determined based on a first tensor after the affine transformation and an untransformed second tensor in the multidimensional matrix.

[0031] In a second aspect, the present application further provides a motion recognition device, the device comprising:

[0032] A posture sequence acquisition module is configured to acquire a posture sequence of a target object from a video to be recognized, wherein the posture sequence includes joint point information of the target object at different times;

[0033] The action recognition module is configured to perform action recognition on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized, wherein the action recognition result includes a result of whether the action of the target object in the video to be recognized is compliant or compliant. The action recognition network model is obtained by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution.

[0034] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect above.

[0036] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0037] The above-mentioned action recognition method, device, computer equipment, computer-readable storage medium and computer program product obtain the action recognition result of the video to be recognized by obtaining the posture sequence of the target object from the video to be recognized and performing action recognition on the posture sequence through the action recognition network model. It determines whether the action is compliant by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution for clustering, so that the model does not learn specific pixels, but focuses on learning and understanding the information about human posture and action contained in the sample posture sequence. This learning method helps to improve the generalization ability and robustness of the model, thereby improving the recognition accuracy; in addition, it only needs to mark a compliant label for an input video, without accurately marking each frame in the video, therefore, it also reduces the cost of data annotation, thereby reducing the consumption of computing resources and protecting data privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 is a flow chart of an action recognition method in one embodiment;

[0040] Figure 2 A schematic diagram of a flow chart of steps for obtaining a posture sequence in one embodiment;

[0041] Figure 3 A schematic diagram of a flow chart of a method for acquiring a motion recognition network model in one embodiment;

[0042] Figure 4 is a flow chart of an action recognition method in another embodiment;

[0043] Figure 5 is a structural block diagram of a motion recognition device in one embodiment;

[0044] Figure 6 is an internal structure diagram of a computer device in one embodiment,

[0045] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] In some scenarios, people's posture standards (such as standing and sitting postures, etc.) and operation standards (referring to specific operation actions, such as playing with mobile phones, taking pictures of screens, operating cash, etc.) must meet the requirements. For this scenario, traditional technology uses the following methods to determine whether the action is compliant, including:

[0048] The human posture-based method uses an autoencoder to reconstruct or predict future postures. It uses the reconstruction error as a proxy for the normality score. However, autoencoders often have strong generalization capabilities, that is, abnormal samples can be reconstructed like normal samples. As a result, the accuracy of detection is highly dependent on the generalization ability and performance of the autoencoder. Although the use of memory mechanisms or multimodal data can suppress this generalization to a certain extent, this will bring additional memory consumption and computational overhead.

[0049] The method of using anomaly detection algorithm based on RGB (color mode, i.e. red, green and blue) scheme. However, this method is easily affected by interference factors of background and foreground, resulting in poor scene generalization ability. Abnormal action detection is a long-tail distribution problem. Usually, abnormal actions account for a small proportion. Therefore, changes in the shape of people's clothes caused by different seasons and work uniforms, differences in different regions, differences in viewing angles, and differences in light in normal actions will further lead to the regression of the algorithm's generalization ability.

[0050] The convolutional neural network based on deep learning is adopted. However, the characteristics of convolutional neural networks determine that a large amount of labeled data is needed to support stable output results and better detection performance, which is a major disadvantage for many tasks where it is difficult to obtain samples. It is also very difficult to obtain abnormal samples in abnormal action detection. In addition, processing large amounts of data requires higher computing resources, and the increasing computing cost is also one of the bottlenecks in the development of anomaly detection algorithms.

[0051] Based on this, an embodiment of the present application provides an action recognition method, which determines whether an action is compliant by projecting action posture information into a high-dimensional space for clustering, thereby greatly reducing the demand for training data.

[0052] In one embodiment, Figure 1As shown, a motion recognition method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0053] Step 102: Acquire a posture sequence of the target object from the video to be recognized.

[0054] The video to be identified may be a video that requires action recognition, and may be shot based on a specific usage scenario, for example, a bank, a supermarket checkout counter, or other related scenarios.

[0055] The posture sequence is the joint point information of the target object at different times extracted from the video to be identified. Specifically, the joint point information can be the two-dimensional coordinate information of the predefined key points. Among them, the key points can be pre-set based on the human posture model, or can also be pre-defined based on other methods, which is not limited in the present embodiment. The target object is the object continuously tracked in the video to be identified. For the target object being a human object, the joint points can be the key points of the human posture framework, including but not limited to the head key points, shoulder key points, arm key points, wrist key points, finger key points, leg key points, etc. It can be understood that in specific applications, appropriate key points can be selected based on the needs of the actual scene.

[0056] In this embodiment, the terminal may obtain a posture sequence of the target object from the video to be recognized, and perform action recognition based on subsequent steps.

[0057] Step 104, performing action recognition on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized.

[0058] Among them, the action recognition network model is obtained by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution. Specifically, the sample posture sequence of the training data set has a corresponding classification label, and the classification label is used to characterize the classification of compliance or non-compliance of the action corresponding to the sample posture sequence of the training data set. The action recognition result includes the results of compliance or non-compliance of the action of the target object in the video to be identified. By mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution for clustering, the model does not learn specific pixels, but focuses on learning and understanding the information about human posture and action contained in the sample posture sequence. This learning method helps to improve the generalization ability and robustness of the model, so that it can adapt to human posture estimation tasks in different scenarios and conditions.

[0059] Specifically, the posture sequence of the target object obtained from the video to be recognized can be directly input into the action recognition network model obtained by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution, and the action recognition result of the video to be recognized is obtained as the output of the model.

[0060] In the above-mentioned action recognition method, the terminal obtains the posture sequence of the target object from the video to be recognized, and performs action recognition on the posture sequence through the action recognition network model to obtain the action recognition result of the video to be recognized. It determines whether the action is compliant by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution for clustering, so that the model does not learn specific pixels, but focuses on learning and understanding the information about human posture and action contained in the sample posture sequence. This learning method helps to improve the generalization ability and robustness of the model, thereby improving the recognition accuracy, while reducing the cost of data annotation, thereby reducing the consumption of computing resources.

[0061] In an exemplary embodiment, Figure 2 As shown, in step 102, obtaining a posture sequence of a target object from a video to be identified may specifically include:

[0062] Step 202: Sample the video to be identified based on a preset sampling rate to obtain multiple video frames.

[0063] The sampling rate may be a preset sampling speed or sampling frequency for extracting discrete video frames from the video to be identified, which defines the number of samples of discrete video frames extracted from the continuous video to be identified per second, for example, 4 or 5 frames per second.

[0064] In this embodiment, the terminal may sample the video to be identified based on a preset sampling rate, thereby obtaining a plurality of sampled video frames, wherein the plurality of video frames may be sorted based on their positions in the video to be identified, thereby ensuring the timing of the video frames.

[0065] Step 204: perform target detection on each video frame to determine the position of each object in each video frame.

[0066] Among them, target detection is to find all the targets of interest in the image and determine their positions. The target detection of this embodiment can be to find all the human frames (i.e., objects) and their positions in the video frame. Specifically, YOLOX (You Only Look Once, a target detection model) can be used to perform target detection on each video frame, so as to determine the position of each human frame, i.e., object, in each video frame.

[0067] Step 206 , extracting joint point information of each object in each video frame based on posture estimation according to the position of each object in each video frame.

[0068] Among them, the joint point information can be the two-dimensional coordinate information of predefined key points. Posture estimation can be to accurately detect and locate the positions of various joints of the human body in an image or video through computer vision technology. In this embodiment, based on the position of each object in each video frame and the predefined key points, the joint point information of each object in each video frame can be extracted based on posture estimation. Specifically, posture estimation can be implemented based on RTMPose (a deep learning posture estimation method). RTMPose adopts a top-down paradigm, obtains a bounding box by using a detector, and then estimates the posture of each object separately.

[0069] For example, if the predefined key points include a head key point, two shoulder key points, and two leg key points, then according to the position of each object in each video frame, the pose estimation based on RTMPose can be performed to extract the joint point information of each object in each video frame. The joint point information of each object can be expressed as [[255, 340], [155, 450], [355, 450], [200, 900], [260, 900]].

[0070] Step 208: determining the joint point information of the target object in multiple video frames through target tracking.

[0071] Among them, target tracking is to track these targets in subsequent frames of the video sequence after the targets are detected to associate the targets between different frames. The target object is the object that is continuously tracked in multiple video frames of the video sequence.

[0072] Specifically, target tracking can determine the target objects that are continuously tracked in the video frames of the video sequence, and determine the joint point information of each target object in multiple video frames of the video sequence. Since the video frames are time-sequential, the joint point information of the target object in each video frame of the video sequence represents the posture characteristics of the target object at the corresponding time point.

[0073] Step 210: Generate a posture sequence of the target object according to the joint point information of the target object in multiple video frames.

[0074] Among them, the posture sequence is obtained by merging the joint point information of the same target object in multiple video frames, that is, at different time points. Specifically, the joint point information of the same target object at different time points can be merged into a two-dimensional matrix to obtain the posture sequence of the target object. Among them, each row represents a time point, and each column represents the coordinate vector of a certain joint point of the target object at that time point. It can be understood that when there are multiple target objects in the video to be identified, the posture sequence can also include the joint point information of multiple target objects at different time points.

[0075] Based on the above embodiment, the continuous posture tracking of each target object in the video to be recognized can be realized. The whole process includes two main parts: posture estimation and posture tracking. The posture estimation is used to extract the joint points in each frame, and the target tracking is used to associate these joint points to form a continuous posture sequence, so that the posture sequence can be more effectively managed and analyzed for action recognition. Since the present embodiment is not limited to a specific posture estimator, the posture estimation can be highly decoupled, so that better accuracy and operation optimization can be achieved.

[0076] In an exemplary embodiment, Figure 3 As shown, the action recognition network model is obtained by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution, which may specifically include:

[0077] Step 302: Obtain a training data set.

[0078] Among them, the training data set is used for model training. The training data set may include sample posture sequences of multiple sample videos and corresponding classification labels, and the classification labels are used to characterize the classification of compliant or non-compliant actions in the sample videos. Specifically, in the unsupervised mode (i.e., training mode), negative sample annotation is not required, and all positive samples are used for learning, and non-positive judgments are negative by default. In the supervised mode (i.e., inference mode), the trained mixed Gaussian distribution can be used as prior knowledge to force the model to assign low probabilities to abnormal samples (for example, samples that have not been seen). The sample posture sequence can be obtained by standardizing the posture sequence of the sample video in the training data set to ensure that different features have similar scales. For example, the sequence length of the posture sequence can be standardized to a fixed length.

[0079] The training dataset can be collected from the Internet or based on specific usage scenarios. For example, it can be the ShanghaiTech dataset (an abnormal action detection dataset), Avenue dataset (a dataset for video anomaly detection), or Ped dataset (a pedestrian anomaly detection dataset) in the Internet.

[0080] The ShanghaiTech dataset focuses on abnormal human behavior, such as pulling and fighting, riding a bicycle on campus, etc. This dataset contains 437 high-resolution campus surveillance videos, covering 13 scenes, and 130 abnormal events are annotated in the dataset. Its background is complex and diverse, and the lighting conditions and camera angles vary greatly, making it suitable for video anomaly detection research under weak supervision settings. The dataset also provides pixel-level abnormal event annotations, which helps the model learn to locate key frames of abnormal behavior in video streams. Due to the diversity and complexity of its scenes, the dataset can also be used to evaluate the generalization ability of the model in different environments.

[0081] Avenue dataset contains 16 training videos and 21 test videos captured on CUHK campus avenue, with a total of 30,652 frames (15,328 frames for training and 15,324 frames for testing). These videos are used to study and evaluate video anomaly detection algorithms, including capture of normal situations as well as test videos containing abnormal events, such as throwing objects, wandering, and running.

[0082] The Ped dataset is acquired by a camera fixed at a high altitude, overlooking the pedestrian walkway. The dataset contains two subsets, Peds1 and Peds2, each corresponding to a different scene. Peds1 contains 34 training video samples and 36 test video samples, mainly containing people walking towards and away from the camera, with a certain degree of perspective distortion. Peds2 contains 16 training video samples and 12 test video samples, in which pedestrians move parallel to the camera plane. Abnormal events in the dataset include the circulation of non-pedestrian entities in the walkway, such as cyclists, skaters, and carts, as well as abnormal movement patterns of pedestrians, such as crossing the walkway or the surrounding grass. These anomalies are all naturally occurring and are not deliberately arranged for the assembly of the dataset.

[0083] The training data set can also be based on surveillance videos collected in specific security scenarios, such as bank counters, supermarket cashier counters, hospitals, schools, etc. Specifically, in the bank scenario, illegal actions can be simulated based on the security characteristics of the bank scenario to obtain abnormal samples for training.

[0084] Step 304 , mapping the sample posture sequences of the plurality of sample videos to the hidden layer space to obtain a high-dimensional feature representation of the sample posture sequences.

[0085] The hidden space is a high-dimensional, abstract space that is used to capture and represent complex structures and patterns in data (such as sample pose sequences). In this space, sample pose sequences are represented as high-dimensional feature representations that can reveal the intrinsic relationships and similarities between data.

[0086] In order to map the sample pose sequence to the hidden space, a mapping function is usually required. This mapping function can convert the original sample pose sequence into a high-dimensional feature representation in the hidden space. In this embodiment, the mapping function can be implemented by neural network parameterization, and the specific mapping relationship can be expressed as:

[0087]

[0088] in, is the mapping function implemented by the neural network, are the parameters of the neural network, yes The reversible function process is designed to ensure the reversibility and efficient computation of the Jacobian matrix determinant. For input data such as sample pose sequences, For High-dimensional feature representation after mapping.

[0089] In this process, the network learns how to capture key features in the pose sequence and represent them as points in a high-dimensional space.

[0090] Step 306 , according to the classification labels of the multiple sample videos, high-dimensional feature representation is modeled based on Gaussian distribution to obtain an action recognition network model.

[0091] Usually after mapping, There is a tractable density, such as a spherical multivariate Gaussian distribution.

[0092] Specifically, the transformed variables The probability density of can be obtained by the following formula:

[0093]

[0094] for The probability density of for The probability density of Represents the mapping function About input The determinant of the derivative of .

[0095] Jacobian capture and The volume change between , can be explained by applying logarithms and the chain rule:

[0096]

[0097] The basic idea is to choose transformations where the Jacobian matrix is ​​triangular. For these transformations, the logarithmic determinant is trivial to compute, so it is straightforward to estimate In contrast, other generative approaches consider the probability distribution of data is intractable, so other methods use suboptimal scoring functions such as reconstruction or prediction.

[0098] Therefore, the main variation patterns of posture data can be captured in the latent space and represented in the form of a probability distribution.

[0099] Gaussian distribution is also a probability distribution, which describes the situation where data fluctuates around a certain central value. In posture estimation, assuming that posture sequences obey Gaussian distribution means that these sequences vary randomly around a certain "average" posture. This embodiment models high-dimensional feature representations based on Gaussian distribution according to the classification labels of multiple sample videos, thereby obtaining an action recognition network model. In the hidden space, high-dimensional feature representations of similar sample posture sequences are clustered together to form different clusters.

[0100] Specifically, assuming that the feature distribution in the hidden space obeys a Gaussian distribution, this distribution can be fitted and described by a mixed Gaussian model. This means that each point in the hidden space can be described by a mean vector and a covariance matrix, in this way, a complex posture sequence can be mapped to a relatively simple probability distribution.

[0101] Since Gaussian distribution has good mathematical properties and is easy to process and analyze, using Gaussian distribution to model posture sequences can help the model better handle the uncertainty and diversity of posture data. It can also flexibly adapt to different data characteristics and needs by adjusting the parameters of the mixed Gaussian distribution (such as mean and variance). It can also provide a smooth and continuous way to describe posture changes, so that the model can respond sensitively to small posture changes. In short, mapping the sample posture sequence of training samples to a hidden space based on Gaussian distribution is an effective method that can help the model better understand and utilize the intrinsic structure of posture data.

[0102] Since the present embodiment uses the posture sequence as the clustering source, this design greatly reduces the pressure of data annotation, from the past of giving dozens of labels to the video (each frame in the video needs to be accurately labeled), to only giving each sample video a binary classification label of yes or no, which greatly improves the speed of data processing. And the amount of training data required is greatly reduced, only 300 to 1000 4-second to 6-second videos are needed to obtain excellent recognition results. Because it is very difficult to obtain samples of abnormal actions in high-security scenarios such as banks, and traditional recognition models usually require thousands of training data, the solution of this application can greatly reduce the demand for training data.

[0103] In one scenario, in order to train the model, a loss function can also be defined, which is used to measure the difference between the model output and the real data. The loss function is minimized based on an optimization algorithm, such as stochastic gradient descent. This can help adjust the model parameters to better fit the training data set. The model is trained by using the training data set and the model parameters are updated iteratively for multiple times until the loss function converges or the preset number of iterations is reached. The performance of the model can also be evaluated by the validation set and the test set to ensure that it has good generalization ability. Various indicators can be used to measure the accuracy and stability of the model, such as Mean Squared Error (MSE), Mean Absolute Error (MAE), etc. Once the model is fully trained and its performance is verified, it can be applied to actual action recognition tasks. For example, a new posture sequence can be input into the model, and the model will project it into the hidden space and judge whether the action represented by the posture information is compliant based on its similarity with each cluster. Specifically, if the projected point falls within a known cluster, the action is considered compliant; otherwise, the action is considered to be abnormal or unknown.

[0104] In an exemplary embodiment, mapping sample pose sequences of multiple sample videos to a hidden layer space to obtain a high-dimensional feature representation of the sample pose sequence may include: normalizing the sample pose sequence of the sample video through a normalization layer to obtain a normalized sample pose sequence. Performing a graph convolution operation and permuting the order of input channels through a sorting layer to convert the normalized sample pose sequence into a multidimensional matrix. Performing an affine transformation on half of the multidimensional matrix through an affine layer, and determining the feature representation of the sample pose sequence based on the first tensor after the affine transformation and the untransformed second tensor in the multidimensional matrix.

[0105] Specifically, a graph convolution network (GCN) can be used to process sample pose sequences. The architecture can include multiple recurrent steps, each of which consists of three layers: a normalization layer, a sorting layer, and an affine layer. This architecture ensures reversibility because each component in the network is bijective. The normalization layer is an activation normalization layer that uses data-dependent initialization to apply scale and bias, similar to batch normalization. The sorting layer arranges the order of the input channels, and can use the learned 1×1 convolution to achieve reversible soft permutation through linear transformation and initialized to a random rotation matrix. Finally, an affine transformation is performed through the affine layer.

[0106] In some embodiments, the sample pose sequence of the sample video may be based on Figure 2 The two-dimensional matrix is ​​obtained by processing in a way, where each row represents a time point and each column represents the coordinate vector of the joint point of the sample object at that time point. By using a weighted adjacency matrix to describe the coordinates of the joint points in these human postures, a weighted graph relationship is formed. Considering the temporal dimension, the continuous human skeleton joint graph is finally represented as a time graph, that is, the joint point graph of each frame is spliced ​​again along the temporal dimension. Then the graph convolutional network GCN is used to abstract the feature vector again.

[0107] The specific processing process may include: inputting the two-dimensional matrix into the normalization layer, and then normalizing all the values ​​therein. In this embodiment, the maximum activation value is limited, and the output of this process is still a two-dimensional matrix. Among them, the normalization layer can use an "activation normalization layer", which uses randomly initialized initialization parameters to normalize the data.

[0108] The order of the input channels is then permuted through the sorting layer. Specifically, the learned 1×1 convolution kernel can be used to perform matrix multiplication on all graph data. This layer implements reversible soft permutation through linear transformation, which is initialized to a random rotation matrix. The output of this step is a multidimensional matrix with dimensions B*C*T*N, where B is the amount of data in a batch, which is 1 if there is only one data, C is the number of channels of the new matrix after convolution, which can be set to 512 in this embodiment, T is the sequence length, which is 5 if there are five frames, and N is the number of nodes in the graph network, which is 17 if the number of predefined joint points is 17.

[0109] The affine layer determines its high-dimensional feature representation by keeping half of the input multidimensional matrix unchanged and performing an affine transformation on the other half based on the first half, where the first tensor is obtained after the transformation and the untransformed part is the second tensor. This method can simplify the calculation of the inverse transformation and Jacobian matrix.

[0110] In an exemplary embodiment, performing action recognition on a posture sequence through an action recognition network model to obtain an action recognition result of a video to be recognized may include: first using a human posture estimation network to obtain a continuous human posture sequence, and then mapping the posture sequence of the target object through the action recognition network model to obtain an intermediate distribution and a first activation value of the posture sequence; in a hidden layer space, determining a feature representation of the posture sequence of the target object, performing an inverse operation on the feature representation, and obtaining a second activation value, wherein the second activation value is used to characterize the similarity between the distribution after the inverse operation and the intermediate distribution; and determining the action recognition result of the video to be recognized based on the first activation value and the second activation value.

[0111] In some embodiments, an evaluation function can be used as a reference for determining abnormality, i.e., non-compliance. The evaluation function in this embodiment is as follows:

[0112]

[0113] in, It represents a segment. If the input video is divided into 10 segments, That is 10. represents the mapping function, It is the data that needs to be mapped, such as posture sequence, yes The reversible transformation function of are weights, determined by the model.

[0114] Specifically, the input video clip, i.e. the video to be recognized, can first be abstracted into a posture sequence, which is usually represented as a two-dimensional matrix, which contains the coordinates of the human joints of the target object in each frame of the video. Then, this posture sequence is input into the action recognition network model that has been trained in advance. The model will process this input and give an intermediate distribution d1 and a first activation value s1. Here, d1 represents an internal representation or probability distribution of the network for the input data, while s1 is a value generated by the network during the processing process, which can be obtained by the model parameters based on the input data, and may reflect a certain similarity or abnormality between the input data and the network expectations. In the hidden space of the network, the feature representation of the posture sequence is inversely operated by using Normalization Flow (a generative network based on reversible mapping and probability density estimation). The purpose of this inverse operation is to convert the feature representation of the posture sequence back to the original data distribution space, thereby obtaining another score s2, i.e. the second activation value. This s2 represents the similarity between the distribution of d1 and the original distribution. Performing matrix multiplication on s1 and s2, the score of the video clip is obtained. This score is compared with a preset threshold. For example, if the score exceeds the threshold, an alarm mechanism is triggered, indicating that the video clip may contain non-compliant gestures or require further review.

[0115] In some embodiments, in order to improve the accuracy of recognition, a long complete video is usually divided into multiple video clips, and then the scores of the multiple video clips are averaged to obtain the final score of the complete video. The final score can then be compared with a preset threshold. For example, if the score exceeds the threshold, an alarm mechanism is triggered, indicating that the video may contain non-compliant gestures or require further review.

[0116] The above process combines the processing power of neural networks and the distribution conversion characteristics of Normalization Flow. It simplifies anomaly detection into human activities. Since human posture sequences are highly semantic, intuitive, and low-dimensional signals, they contain information about the normality of the video. The human posture estimation scheme also provides explainability for abnormalities in the scene, because each person's posture sequence is assigned a normality score. And only using human posture sequence representation can reduce privacy violations and biases associated with appearance-based methods. By learning data distribution and scoring samples based on their distribution probability instead of relying on reconstruction errors, interfering parameters in the video are excluded, such as background clutter, lighting changes, and other factors unrelated to the normality of human activities. It has a good effect on personal privacy protection and can accurately judge abnormal situations in the video to be identified.

[0117] In an exemplary embodiment, performing action recognition on a posture sequence through an action recognition network model to obtain an action recognition result of a video to be recognized may also include:

[0118] The posture sequence is input into the action recognition network model for action recognition. If the posture sequence is determined to be out-of-distribution data (OOD), the distribution points with maximum deviation values ​​in the posture sequence are obtained; the distribution points with maximum deviation values ​​are weighted summed to obtain fused distribution points; the offset between the fused distribution points and the distribution center of the target set is obtained; and the action recognition result of the video to be recognized is determined based on the offset.

[0119] In some scenarios, there is ambiguity in the definition of the distinction between non-compliant actions, i.e. abnormal actions, and compliant actions. For example, the distinction between pulling and fighting and playing and frolicking mainly focuses on the amplitude and duration of the action. Therefore, we can select the maximum deviation value in the out-of-distribution data in the input posture sequence, perform weighted summation on these representative distribution points, and then perform offset calculation with the smoothed distribution center in the normal category set, and use the offset value as a reference for judging the abnormality.

[0120] Specifically, by inputting the posture sequence into the action recognition network model for action recognition, if the posture sequence is determined to be OOD (i.e., it has significant differences from the normal distribution in the model, and usually exhibits characteristics or patterns different from normal data), the maximum deviation value in the posture sequence is found, and its distribution point in the feature space is calculated. Specifically, the k-means clustering algorithm can be used to find the center point of each cluster, or the feature mean can be directly calculated. And these representative distribution points are summed to obtain a comprehensive representative distribution point, i.e., the fused distribution point, which is then compared with the normal category set, i.e., the smoothed distribution center in the target set, to calculate the difference (offset) between the two. This offset can be used to measure the degree of dissimilarity between the OOD sequence and the normal category. According to the size of the offset, a threshold is set to determine whether the OOD sequence is abnormal. For example, if the offset exceeds the preset threshold, the OOD sequence is considered abnormal, i.e., non-compliant; otherwise, it is considered normal, i.e., compliant.

[0121] Through the above method, anomalies or outliers in the data can be effectively detected and identified, thereby improving the accuracy and robustness of the human posture estimation system. And by combining the idea of ​​machine learning, the posture information is projected in a high-dimensional space for clustering to determine whether the action is compliant. After the action features are extracted by the deep learning network, all actions are clustered into two categories: compliant and non-compliant. This can efficiently handle the clustering problem of feature vectors and is very lightweight. Therefore, it can give real-time feedback to the front-end device and prevent the occurrence of irregular behaviors in time.

[0122] In an exemplary embodiment, Figure 4 As shown, the above-mentioned action recognition method is further described below, which may specifically include:

[0123] Step 401: posture estimation and tracking.

[0124] By performing posture estimation and tracking on the video to be identified, the specific implementation of posture estimation and tracking can refer to the above Figure 2 The embodiment shown is not described in detail in this embodiment.

[0125] Step 402: gesture sequence input.

[0126] in, , and They are posture sequences of different target objects tracked in the video to be identified. Each posture sequence is a two-dimensional matrix, each row represents a time point, and each column represents the coordinate vector of the joint point of the target object corresponding to the time point.

[0127] Network adopts Figure 3 The action recognition network model trained by the method shown in the figure is then , and The data is input into the action recognition network model and then processed by the model.

[0128] Step 403: gesture sequence mapping.

[0129] Specifically, the action recognition network model transforms the posture sequence of each target object into , and are mapped to the hidden space to determine their feature representation in the hidden space. For example, the posture sequence of target object No. 1 Mapped to the hidden space, the corresponding feature representation is obtained , the posture sequence of target object No. 2 Mapped to the hidden space, the corresponding feature representation is obtained , the posture sequence of target object No. 3 Mapped to the hidden space, the corresponding feature representation is obtained .

[0130] Step 404 , evaluating the posture sequence probability.

[0131] Specifically, since each feature representation has a probability density in the hidden space, for the posture sequence Feature representation after mapping , and its corresponding probability density is , for the posture sequence Feature representation after mapping , and its corresponding probability density is , for the posture sequence Feature representation after mapping , and its corresponding probability density is .

[0132] Since the hidden space in this embodiment is reversible, the probability of the posture sequence before mapping can be calculated based on the probability density of the feature representation after mapping. Specifically, the following formula can be used for calculation:

[0133]

[0134] in, Indicates The probability density of the feature representation after the posture sequence mapping of the target object is Indicates The probability of the posture sequence of each target object can then be compared with a preset threshold. For example, if the probability exceeds the threshold, an alarm mechanism is triggered, indicating that the video may contain non-compliant posture actions for further review.

[0135] Since this embodiment performs anomaly detection by using the distribution classification of human posture in high-dimensional space and judges the anomaly by judging the distribution classification of posture, compared with the traditional method of using posture reconstruction and judging the anomaly according to the reconstruction score, it can reduce the consumption of computing resources and improve the recognition accuracy.

[0136] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0137] Based on the same inventive concept, the embodiment of the present application also provides a motion recognition device for implementing the motion recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the motion recognition device provided below can refer to the limitations on the motion recognition method above, and will not be repeated here.

[0138] In an exemplary embodiment, Figure 5 As shown, a motion recognition device is provided, including: an acquisition module 502 and a motion recognition module 504, wherein:

[0139] An acquisition module 502 is configured to acquire a posture sequence of a target object from a video to be recognized, wherein the posture sequence includes joint point information of the target object at different times;

[0140] The action recognition module 504 is configured to perform action recognition on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized, wherein the action recognition result includes a result of whether the action of the target object in the video to be recognized is compliant or compliant. The action recognition network model is obtained by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution.

[0141] In an exemplary embodiment, the acquisition module is also used to: sample the video to be identified based on a preset sampling rate to obtain multiple video frames; perform target detection on each of the video frames to determine the position of each object in each of the video frames; extract joint point information of each object in each of the video frames based on posture estimation according to the position of each object in each of the video frames; determine the joint point information of the target object in the multiple video frames through target tracking, the target object being an object continuously tracked in the multiple video frames; and generate a posture sequence of the target object based on all continuous joint point information of the target object in the multiple video frames.

[0142] In an exemplary embodiment, the action recognition module is also used to: map the posture sequence of the target object through the action recognition network model to obtain the intermediate distribution and the first activation value of the posture sequence; determine the feature representation of the posture sequence of the target object in the hidden layer space, perform an inverse operation on the feature representation, and obtain a second activation value, wherein the second activation value is used to characterize the similarity between the distribution after the inverse operation and the intermediate distribution; and determine the action recognition result of the video to be recognized based on the first activation value and the second activation value.

[0143] In an exemplary embodiment, the action recognition module is also used to: input the posture sequence into the action recognition network model for action recognition, if the posture sequence is determined to be out-of-distribution data, obtain the distribution points of the maximum deviation value in the posture sequence; perform weighted summation on the distribution points of the maximum deviation value to obtain fused distribution points; obtain the offset between the fused distribution points and the distribution center of the target set; and determine the action recognition result of the video to be identified based on the offset.

[0144] In an exemplary embodiment, the action recognition network model is obtained by mapping the sample posture sequence of a training data set to a hidden space of a Gaussian distribution, including: obtaining a training data set, the training data set including sample posture sequences of multiple sample videos and corresponding classification labels, the classification labels being used to characterize the classification of action compliance in the sample videos; mapping the sample posture sequences of the multiple sample videos to a hidden space to obtain a high-dimensional feature representation of the sample posture sequence; and modeling the high-dimensional feature representation based on a Gaussian distribution according to the classification labels of the multiple sample videos to obtain the action recognition network model.

[0145] In an exemplary embodiment, the sample posture sequences of the multiple sample videos are mapped to the hidden layer space to obtain a high-dimensional feature representation of the sample posture sequence, including: normalizing the sample posture sequence of the sample video through a normalization layer to obtain a normalized sample posture sequence; permuting the order of the input channels through a sorting layer to convert the normalized sample posture sequence into a multi-dimensional matrix; performing an affine transformation on half of the multi-dimensional matrix through an affine layer, and determining the feature representation of the sample posture sequence based on the first tensor after the affine transformation and the untransformed second tensor in the multi-dimensional matrix.

[0146] Each module in the above-mentioned action recognition device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0147] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store model data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an action recognition method is implemented.

[0148] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an action recognition method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0149] Those skilled in the art will understand that Figure 6 or Figure 7The structure shown in the figure is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the computer device to which the present application scheme is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0150] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0152] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0154] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0155] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0156] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for motion recognition, characterized in that: The method comprises: Acquire a posture sequence of a target object from a video to be identified, wherein the posture sequence includes joint point information of the target object at different times; The action recognition is performed on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized, wherein the action recognition result includes a result that the action of the target object in the video to be recognized is compliant or compliant. The action recognition network model is obtained by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution.

2. The method according to claim 1, characterized in that The step of obtaining a posture sequence of a target object from a video to be identified includes: Sampling the video to be identified based on a preset sampling rate to obtain multiple video frames; Performing target detection on each of the video frames to determine the position of each object in each of the video frames; Extracting joint point information of each object in each video frame based on posture estimation according to the position of each object in each video frame; Determine the joint point information of the target object in the multiple video frames by target tracking, wherein the target object is an object continuously tracked in the multiple video frames; A posture sequence of the target object is generated according to the joint point information of the target object in the multiple video frames.

3. The method according to claim 1, characterized in that The step of performing action recognition on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized includes: Mapping the posture sequence of the target object through the action recognition network model to obtain an intermediate distribution and a first activation value of the posture sequence; In the hidden layer space, determining a feature representation of the posture sequence of the target object, performing an inverse operation on the feature representation to obtain a second activation value, wherein the second activation value is used to characterize the similarity between the distribution after the inverse operation and the intermediate distribution; Determine an action recognition result of the video to be recognized according to the first activation value and the second activation value.

4. The method according to claim 1, characterized in that: The step of performing action recognition on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized includes: Inputting the posture sequence into the action recognition network model for action recognition, and if it is determined that the posture sequence is out-of-distribution data, obtaining the distribution point positions of the maximum deviation values ​​in the posture sequence; Performing weighted summation on the distribution points of the maximum deviation value to obtain fused distribution points; Obtaining the offset between the fused distribution point and the distribution center of the target set; The action recognition result of the video to be recognized is determined based on the offset.

5. The method according to any one of claims 1 to 4, characterized in that: The action recognition network model is obtained by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution, including: Acquire a training data set, the training data set comprising sample posture sequences of a plurality of sample videos and corresponding classification labels, the classification labels being used to characterize classifications of compliance of actions in the sample videos; Mapping the sample posture sequences of the plurality of sample videos to a hidden layer space to obtain a high-dimensional feature representation of the sample posture sequences; According to the classification labels of the plurality of sample videos, the high-dimensional feature representation is modeled based on Gaussian distribution to obtain the action recognition network model.

6. The method according to claim 5, characterized in that Mapping the sample posture sequences of the plurality of sample videos to a hidden layer space to obtain a high-dimensional feature representation of the sample posture sequences includes: Normalizing the sample pose sequence of the sample video through a normalization layer to obtain a normalized sample pose sequence; By using a sorting layer, the sorting of input channels is permuted to convert the normalized sample posture sequence into a multi-dimensional matrix; An affine transformation is performed on half of the multidimensional matrix through an affine layer, and a feature representation of the sample posture sequence is determined based on the first tensor after the affine transformation and the untransformed second tensor in the multidimensional matrix.

7. A motion recognition device, characterized in that: The device comprises: A posture sequence acquisition module is configured to acquire a posture sequence of a target object from a video to be recognized, wherein the posture sequence includes joint point information of the target object at different times; The action recognition module is configured to perform action recognition on the posture sequence through an action recognition network model to obtain an action recognition result of the video to be recognized, wherein the action recognition result includes a result of whether the action of the target object in the video to be recognized is compliant or compliant. The action recognition network model is obtained by mapping the sample posture sequence of the training data set to the hidden space of the Gaussian distribution.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images