3D human action recognition method and device based on unsupervised mask algorithm

Abstract

The present invention relates to the field of human action recognition, and provides a 3D human action recognition method and device based on an unsupervised mask algorithm. The method comprises: acquiring two-dimensional image information and depth information of a human action to be tested; converting the two-dimensional image information and the depth information to obtain human skeleton data to be recognized; preprocessing said human skeleton data to obtain temporal skeleton data to be recognized and spatial skeleton data to be recognized; and sequentially inputting said temporal skeleton data and said spatial skeleton data into a feature extractor and a linear classifier which are pre-trained and fine-tuned, to obtain an action recognition result. A large amount of unsupervised data can be used for algorithm training and optimization, avoiding additional costs incurred by labeling unsupervised data as supervised data, and also achieving the accuracy comparable to that of a model trained using the supervised data, thereby effectively improving the accuracy of recognizing human action sequences using an unsupervised algorithm.

Description

A 3D human action recognition method and device based on an unsupervised mask algorithm TECHNICAL FIELD

[0001] The present application relates to the field of human action recognition, and in particular to a 3D human action recognition method and device based on an unsupervised mask algorithm. BACKGROUND

[0002] In existing human action recognition technology, a basic supervised action recognition framework is provided, which can realize the recognition of single-person or multi-person skeleton point positions and the pose estimation of human actions.

[0003] However, supervised training requires a large amount of labeled data, which requires a large cost, and most of the current skeleton data is still unsupervised data without labels. The accuracy of the existing unsupervised action recognition framework still has a large room for improvement compared with the supervised action recognition framework. Therefore, there is an urgent need for a system that can use unsupervised data for training and has a high motion classification accuracy to perform human action recognition. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a 3D human action recognition method and device based on an unsupervised mask algorithm, which realizes accurate human action recognition by using skeleton data.

[0005] In a first aspect, a 3D human action recognition method based on an unsupervised mask algorithm is provided, which comprises:

[0006] obtaining two-dimensional picture information and depth information of a human action to be measured;

[0007] obtaining human skeleton data to be recognized by converting the two-dimensional picture information and the depth information;

[0008] preprocessing the human skeleton data to be recognized to obtain time skeleton data to be recognized and space skeleton data to be recognized;

[0009] inputting the time skeleton data to be recognized and the space skeleton data to be recognized into a pre-trained and fine-tuned feature extractor and a linear classifier in sequence to obtain an action recognition result.

[0010] According to an implementable manner in the embodiments of the present application, before obtaining the two-dimensional picture information and the depth information of the human action to be measured, the method further comprises:

[0011] directly obtaining skeleton data in a preset data set for a pre-training process and a fine-tuning process of a model;

[0012] preprocessing the skeleton data in the preset data set to obtain time skeleton data and space skeleton data;

[0013] The skeleton features of the skeleton data are mined by using an unsupervised mask algorithm, and the time skeleton features of the time skeleton data are mined by using a time-based motion-dependent representation mask learning module and the space skeleton features of the space skeleton data are mined by using a space-based motion-dependent representation mask learning module, to complete a pre-training process;

[0014] The feature extractor is formed by using the time-based motion-dependent representation learning module encoder and the space-based motion-dependent representation learning module encoder, and a linear classifier is added at the bottom of the feature extractor to complete a fine-tuning process.

[0015] According to an implementable manner in the embodiments of the present application, the skeleton data in the preset data set is preprocessed, including:

[0016] The skeleton data in the preset data set is filled by using a bilinear interpolation method, and the skeleton data in the preset data set is adjusted to a fixed time length;

[0017] The skeleton data in the preset data set is subjected to dimension transformation to obtain time skeleton data and space skeleton data ,

[0018] wherein R represents the skeleton data in the preset data set, T represents the fixed time length frame number after the bilinear interpolation, V represents the number of skeleton points of the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, represents the time skeleton data, represents the space skeleton data.

[0019] The time skeleton data is subjected to a difference operation based on a motion skeleton formula, to obtain motion skeleton data, wherein the motion skeleton formula is:

[0020] ;

[0021] wherein M represents the motion information of the skeleton data corresponding to the frame number, T represents the fixed time length frame number after the bilinear interpolation, represents the skeleton data of the i-th frame, and i represents the time frame.

[0022] According to an implementable manner in the embodiments of the present application, the time skeleton data features are mined by using a time-based motion-dependent representation mask learning module, including:

[0023] The motion skeleton data is used as a guide to embed the semantic richness prior of feature mask, the motion intensity algorithm is used to calculate the motion intensity of each time frame, the motion intensity is converted into a probability distribution to obtain a first mask probability, the first mask probability is arranged in descending order to obtain the position index of the first 20 first mask probabilities as a first mask index. The motion intensity algorithm is as follows:

[0024]

[0025] wherein, represents the motion skeleton data corresponding to the frame number of the skeleton data, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, I represents the motion intensity, i represents the skeleton point number, and j represents the corresponding three-dimensional coordinate channel;

[0026] The time skeleton data is masked according to the first mask index, and the corresponding time skeleton data not masked by the mask is selected according to the first mask index;

[0027] The unmasked time skeleton data is extracted by the time encoder to obtain pre-encoding time feature information;

[0028] The pre-encoding time feature information is spliced with the learnable time mask token according to the first mask index to obtain time encoding feature information;

[0029] The time encoding feature information is decoded by the time decoder to obtain time decoding information.

[0030] According to an implementable manner in the embodiment of the application, a space-based motion-dependent representation mask learning module is used to mine the features of the space skeleton data, including:

[0031] According to the space skeleton data, the entropy of each skeleton point in the time sequence is calculated by a skeleton point entropy algorithm, and the entropy is converted into a probability distribution to obtain a second mask probability. The second mask probability is arranged in descending order to obtain the position index of the first 20 second mask probabilities as a second mask index.

[0032] wherein, the skeleton point entropy algorithm is as follows:

[0033]

[0034]

[0035] wherein, V represents the number of skeleton points, P(j) represents the probability frequency distribution of the skeleton point in the time sequence, G(j) represents the entropy in a certain spatial dimension, and G represents the sum of the entropies of the three dimensions, represents the sum operation on G(j);

[0036] masking the spatial skeleton data by the second mask index, and selecting corresponding spatial skeleton data not shielded by the second mask index according to the second mask index;

[0037] extracting features of the spatial skeleton data not shielded by the mask through a spatial encoder to obtain pre-encoding spatial feature information;

[0038] splicing the pre-encoding spatial feature information and a learnable spatial mask token according to the second mask index to obtain spatial encoding feature information;

[0039] decoding the spatial encoding feature information according to a time decoder to obtain spatial decoding information.

[0040] According to an implementable manner in the embodiments of the present application, the pre-training process specifically includes:

[0041] defining a loss function as a mean square error function of the time decoding information and the spatial decoding information on the motion data, and completing the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module.

[0042] According to an implementable manner in the embodiments of the present application, the fine-tuning process specifically includes:

[0043] using the time encoder in the time-based motion-dependent representation mask learning module and the spatial encoder in the space-based motion-dependent representation mask learning module in the pre-training model to jointly constitute a feature extractor.

[0044] fine-tuning the human body recognition module using a small amount of supervised skeleton data, wherein the parameters of the time encoder and the spatial encoder are frozen in the fine-tuning process, and the parameters of the bottom linear classifier are fine-tuned.

[0045] In a second aspect, a 3D human body action recognition device based on an unsupervised mask algorithm is provided, which includes:

[0046] An acquisition unit is configured to acquire two-dimensional picture information and depth information of a human body action to be measured.

[0047] A conversion unit is configured to convert the two-dimensional picture information and the depth information to obtain human body skeleton data to be recognized.

[0048] A preprocessing unit is configured to preprocess the human body skeleton data to be recognized to obtain time skeleton data to be recognized and spatial skeleton data to be recognized.

[0049] A recognition unit is configured to input the time skeleton data to be recognized and the spatial skeleton data to be recognized into a feature extractor and a linear classifier that have been pre-trained and fine-tuned in sequence to obtain an action recognition result.

[0050] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory storing computer program instructions, the processor reading and executing the computer program instructions to perform the steps of the method for 3D human action recognition based on an unsupervised mask algorithm in the first aspect or any of the implementation forms of the first aspect.

[0051] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing computer program instructions, the computer program instructions being executed by a processor to implement the steps of the method for 3D human action recognition based on an unsupervised mask algorithm in the first aspect or any of the implementation forms of the first aspect.

[0052] According to the technical content provided by the embodiment of the present application, the two-dimensional picture information and the depth information of the human action to be measured are obtained, the two-dimensional picture information and the depth information are converted to obtain human skeleton data to be recognized, the human skeleton data to be recognized is preprocessed to obtain time skeleton data to be recognized and space skeleton data to be recognized, the time skeleton data to be recognized and the space skeleton data to be recognized are sequentially input into the pre-trained and fine-tuned feature extractor and linear classifier to obtain the action recognition result, a large amount of unsupervised data can be used for algorithm training and optimization, the additional cost generated by labeling the unsupervised data as supervised data is avoided, and the accuracy of the model trained using the supervised data can be achieved, the time and space dimension feature information in the skeleton data is mined by using the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module, and the accuracy of the unsupervised algorithm in recognizing the human action sequence is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0053] FIG. 1 is a flowchart of a method for 3D human action recognition based on an unsupervised mask algorithm according to an embodiment of the present application;

[0054] FIG. 2 is a schematic diagram of a time-based motion-dependent representation mask learning module and a space-based motion-dependent representation mask learning module according to an embodiment of the present application;

[0055] FIG. 3 is a structural schematic diagram of a device for 3D human action recognition based on an unsupervised mask algorithm according to an embodiment of the present application;

[0056] FIG. 4 is a hardware structural schematic diagram of a computer device according to an embodiment of the present application.

[0057] DETAILED DESCRIPTION

[0058] The features and exemplary embodiments of each aspect of the present application will be described in detail below with reference to the accompanying drawings and specific embodiments, in order to make the purpose, technical scheme and advantages of the present application more clear and apparent,

[0059] The application will be described in further detail. It should be understood that the specific embodiments described herein are merely configured to explain the application and are not configured to limit the application. The application can be implemented without some of the specific details for those skilled in the art. The following description of the embodiments is merely to provide a better understanding of the application by showing examples of the application.

[0060] FIG. 1 shows a 3D human action recognition method based on an unsupervised mask algorithm according to an example of the application, which comprises:

[0061] S110, obtaining two-dimensional picture information and depth information of a human action to be measured.

[0062] The two-dimensional picture information and the depth information can be obtained by using a binocular camera device, a laser radar monocular camera, or other camera devices. The two-dimensional picture information refers to various information carried by an image represented on a two-dimensional plane, including image pixel values, resolution information, and the like. The depth information refers to the distance and depth values of each pixel in an image or a scene, including two-dimensional image information and the distance information between objects in the image.

[0063] S120, obtaining human skeleton data to be recognized by converting the two-dimensional picture information and the depth information.

[0064] Using the two-dimensional picture information and the depth information obtained by the camera device, a human body detection and pose estimation algorithm is used to detect and extract key points of the human body, such as the head, shoulders, elbows, wrists, hips, knees, and ankles, from the two-dimensional image.

[0065] The depth image is then converted into the same coordinate system as the two-dimensional image to ensure that each pixel in the depth image corresponds to a pixel in the two-dimensional image.

[0066] The human body key points detected in the two-dimensional image are associated with the corresponding pixels in the depth image. This can be achieved by mapping the key point coordinates in the two-dimensional image to the depth image, and then obtaining the associated depth values from the depth image.

[0067] Using the associated depth information, a three-dimensional skeleton of the human body is reconstructed according to the positions and depth values of the human body key points, and finally the human skeleton data to be recognized is obtained.

[0068] S130, pre-processing the skeleton data to be recognized to obtain time skeleton data to be recognized and space skeleton data to be recognized.

[0069] For the skeleton data to be recognized, a bilinear interpolation algorithm is used to adjust the skeleton data to be recognized to a uniform length.

[0070] The time skeleton data to be recognized is obtained by performing dimension transformation on the skeleton data to be recognized 。

[0071] The space skeleton data to be recognized is obtained by performing dimension transformation on the skeleton data to be recognized 。

[0072] wherein T represents the frame number of the fixed time length after bilinear interpolation, V represents the number of skeleton points in the source data set, and C represents x, y, and z coordinates.

[0073] S140, sequentially inputting the time skeleton data to be recognized and the space skeleton data to be recognized into the pre-trained and fine-tuned feature extractor and linear classifier to obtain the action recognition result.

[0074] The time skeleton data to be recognized and the space skeleton data to be recognized are respectively input into the pre-trained feature extractor to obtain time features and space features, the time features and the space features are spliced and input into the fine-tuned linear classifier to obtain the action recognition result.

[0075] It can be seen that, by obtaining the two-dimensional picture information and the depth information of the human body action to be measured, the two-dimensional picture information and the depth information are converted to obtain the human skeleton data to be recognized, the human skeleton data to be recognized is preprocessed to obtain the time skeleton data to be recognized and the space skeleton data to be recognized, the time skeleton data to be recognized and the space skeleton data to be recognized are sequentially input into the pre-trained and fine-tuned feature extractor and linear classifier to obtain the action recognition result, that is, the human body action recognition method with high accuracy is realized by using a small amount of supervised data.

[0076] As an implementable manner, before obtaining the two-dimensional picture information and the depth information of the human body action to be measured, the method further includes as shown in FIG. 2:

[0077] The skeleton data in the preset data set is directly obtained for model pre-training and fine-tuning process;

[0078] The skeleton data in the preset data set is preprocessed to obtain time skeleton data and space skeleton data;

[0079] The skeleton features in the skeleton data are mined by using an unsupervised mask algorithm, and the time skeleton data features are mined by using a time-based motion-dependent representation mask learning module, and the space skeleton data features are mined by using a space-based motion-dependent representation mask learning module, to complete the pre-training process;

[0080] The feature extractor is composed of the time-based motion-dependent representation learning module encoder part and the space-based motion-dependent representation learning module encoder part, and a linear classifier is added at the bottom of the feature extractor to complete the fine-tuning process.

[0081] The preset data set can be an NTU-RGB+D 60 data set, and the method for mining the skeleton features in the skeleton data by using the unsupervised mask algorithm includes: a pre-training and fine-tuning process of a feature extractor and a linear classifier; specifically, the skeleton data in the NTU-RGB+D 60 data set is directly obtained for the model pre-training and fine-tuning process.

[0082] The skeleton data in the data set is preprocessed to obtain time skeleton data and space skeleton data.

[0083] According to the time skeleton data and the space skeleton data, the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module are used for feature mining, respectively, to complete the model pre-training process.

[0084] The feature extractor is composed of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module, and a linear classifier is added at the bottom of the feature extractor to complete the fine-tuning part.

[0085] As an implementable way, the time-based motion-dependent representation mask learning module is used to mine the time skeleton data features, which includes:

[0086] The motion skeleton data is used as a semantic richness prior for embedding feature masking, the motion intensity of each time frame is calculated based on a motion intensity algorithm, and the motion intensity is converted into a probability distribution to obtain a first mask probability. The first mask probability is arranged in descending order to obtain the position index of the first 20 first mask probabilities as a first mask index. The motion intensity algorithm is:

[0087] ;

[0088] wherein, The motion skeleton data corresponding to the frame number of the skeleton data is represented by V, the number of skeleton points is represented by C, the three-dimensional coordinate channel number is represented by I, the skeleton point sequence number is represented by i, and the corresponding three-dimensional coordinate channel is represented by j.

[0089] The time skeleton data is masked according to the first mask index, and the corresponding time skeleton data not masked by the mask is selected according to the first mask index.

[0090] The time encoder is used for feature extraction on the unmasked time skeleton data, and pre-encoding time feature information is obtained;

[0091] The pre-encoding time feature information is spliced with a learnable time mask token according to a first mask index, and time encoding feature information is obtained.

[0092] The time decoder is used for decoding on the time encoding feature information, and time decoding information is obtained.

[0093] The first mask probability is a mask probability obtained based on motion intensity, and the first mask index is a mask index obtained based on the first mask probability.

[0094] As an implementable manner, a spatial-based motion-dependent representation mask learning module is used to mine spatial skeleton data features, including:

[0095] According to the spatial skeleton data, the entropy of each skeleton point in the time sequence is calculated by a skeleton point entropy algorithm, and the entropy is converted into a probability distribution to obtain a second mask probability. The second mask probability is arranged in descending order, and the position index of the top 20 second mask probabilities is taken as a second mask index.

[0096] The skeleton point entropy algorithm is:

[0097] ;

[0098] ;

[0099] V represents the number of skeleton points, P(j) represents the probability frequency distribution of the skeleton point in the time sequence, G(j) represents the entropy in a certain spatial dimension, and G represents the sum of the three-dimensional spatial entropy, G(j) is summed up.

[0100] The spatial skeleton data is masked by the second mask index, and the corresponding spatial skeleton data not masked by the mask is selected according to the second mask index.

[0101] The spatial encoder is used for feature extraction on the spatial skeleton data not masked by the mask, and pre-encoding spatial feature information is obtained.

[0102] The pre-encoding spatial feature information is spliced with a learnable spatial mask token according to the second mask index, and spatial encoding feature information is obtained.

[0103] The spatial decoder is used for decoding on the spatial encoding feature information, and spatial decoding information is obtained.

[0104] Wherein, the second mask probability is a mask probability obtained based on the entropy of the time sequence, and the second mask index is a mask index obtained based on the second mask probability.

[0105] As an implementable manner, the pre-training process specifically comprises:

[0106] The loss function is defined as the mean square error function of the time decoding information and the space decoding information on the motion data, and the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module is completed.

[0107] As an implementable manner, the fine-tuning process specifically comprises: using the time encoder in the time-based motion-dependent representation mask learning module and the space encoder in the space-based motion-dependent representation mask learning module in the pre-trained model to jointly constitute a feature extractor.

[0108] The human body recognition module is fine-tuned using a small amount of supervised skeleton data, wherein the parameters of the time encoder and the space encoder are frozen in the fine-tuning process, and the parameters of the bottom linear classifier are fine-tuned.

[0109] The embodiments of the present application are further described below in combination with Fig. 2:

[0110] I. Preprocessing stage:

[0111] (1) Directly obtain the skeleton data in the NTU-RGB+D 60 dataset for model pre-training and fine-tuning process;

[0112] (2) For the skeleton data, the bilinear interpolation algorithm is used to adjust the skeleton data to a uniform length.

[0113] (3) For the time skeleton data, the time skeleton data is obtained by performing dimension transformation on the skeleton data .

[0114] (4) For the space skeleton data, the space skeleton data is obtained by performing dimension transformation on the skeleton data .

[0115] Wherein, T represents the frame number of the fixed time length after bilinear interpolation, V represents the number of skeleton points in the source dataset, and C represents the x, y, z coordinates in the source dataset.

[0116] (5) Based on the motion skeleton formula, the time skeleton data is obtained, and the motion skeleton data is obtained by performing difference operation on the time skeleton data, wherein the motion skeleton formula is:

[0117] ;

[0118] wherein M represents motion information of the skeleton data corresponding to the frame number, T represents the frame number of the fixed time length after the bilinear interpolation, represents the skeleton data of the i-th frame, and i represents the time frame.

[0119] II. Time-based motion-dependent mask learning:

[0120] The motion intensity of each time frame is calculated using the motion skeleton data as a guide to embed the semantic richness prior of the feature mask, and the motion intensity is converted into a probability distribution to obtain a mask probability. Then, the mask probability is arranged in descending order to obtain the position index of the top 20 mask probabilities as the mask index. The motion intensity calculation formula is:

[0121] ;

[0122] wherein, represents the motion information of the skeleton data corresponding to the frame number, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, I represents the motion intensity, i represents the skeleton point number, and j represents the corresponding three-dimensional coordinate channel.

[0123] The formula for converting the motion intensity into the mask index is:

[0124] ,

[0125] ,

[0126] ,

[0127] wherein II represents the calculation of the motion intensity using the Softmax function, represents a regularization term, U represents a random distribution of 0 to 1, Index-of-Top-K represents the index of the largest value, represents the mask index.

[0128] The time skeleton data is masked according to the obtained mask index, that is, the corresponding time skeleton data that is not masked by the mask is selected according to the mask index. In the corresponding unmasked time skeleton data part in the figure, the unmasked data is represented by a solid box, and the masked data is represented by a dashed box.

[0129] The unmasked time skeleton data is feature-extracted according to the time encoder to obtain pre-encoding time feature information.

[0130] The time encoder adopts a convolutional neural network combined with an eight-layer self-attention-based feature extraction block network structure, wherein the self-attention-based feature extraction block is composed of a multi-head attention layer, a normalization layer and a forward propagation layer, and a higher level of features is obtained through the extraction operation of the stacked self-attention-based feature extraction block network structure. The time encoder is used to mine unmasked time skeleton data features.

[0131] According to the pre-encoding time feature information extracted by the time encoder, the pre-encoding time feature information is combined with a mask index and a learnable time mask token, the learnable time mask token corresponds to the gray part in the time encoding feature information module in the figure, and the gray box and the white box jointly constitute the time encoding feature information.

[0132] The time decoding information is obtained by decoding the time encoding feature information according to the time decoder.

[0133] The time decoder adopts the same feature extraction block as the time encoder, and adopts a three-layer self-attention-based feature extraction block network structure to decode the input time encoding feature information.

[0134] III. Spatial-based motion-dependent mask learning:

[0135] According to the spatial skeleton data, the entropy of each skeleton point in the time sequence is calculated, and the entropy is converted into a probability distribution to obtain a mask probability. Then, the mask probability is arranged in descending order to obtain the position index of the top 20 maximum probabilities as the mask index. The calculation formula of the skeleton point entropy is:

[0136] ;

[0137] ;

[0138] Wherein, V represents the number of skeleton points, P(j) represents the probability frequency distribution of the skeleton point in the time sequence, G(j) represents the entropy in a certain spatial dimension, and G represents the sum of the three-dimensional spatial entropy, Indicates the summation operation on G(j);

[0139] The formula for converting the entropy intensity into the mask index is:

[0140] ,

[0141] ,

[0142] ,

[0143] Wherein, Π represents the calculation of the entropy by using a Softmax function, represents a regularization term, U represents a random distribution of 0 to 1, and Index-of-Top-K represents an index of a larger value, represents a mask index.

[0144] According to the obtained mask index, the spatial skeleton data is masked, that is, the corresponding spatial skeleton data not masked by the mask index is selected according to the mask index, and the unmasked data in the unmasked spatial skeleton data part in the figure is represented by a solid box, and the masked data is represented by a dashed box.

[0145] According to the spatial encoder, the unmasked spatial skeleton data is feature extracted to obtain pre-encoding spatial feature information.

[0146] Among them, the spatial encoder adopts a convolutional neural network combined with an eight-layer self-attention-based feature extraction block network structure, wherein the self-attention-based feature extraction block is composed of a multi-head attention layer, a normalization layer, and a forward propagation layer. Through the extraction operation of the stacked self-attention-based feature extraction block network structure, higher-level features are obtained. And the spatial encoder excavates the features of the unmasked spatial skeleton data.

[0147] According to the pre-encoding spatial feature information extracted by the spatial encoder, the pre-encoding spatial feature information is spliced according to the mask index and the learnable spatial mask token. The learnable spatial mask token corresponds to the gray part in the spatial encoding feature information module in the figure, and the gray box and the white box together constitute the spatial encoding feature information.

[0148] According to the spatial decoder, the spatial encoding feature information is decoded to obtain spatial decoding information.

[0149] Among them, the spatial decoder adopts the same feature extraction block as the spatial encoder, and adopts a three-layer self-attention-based feature extraction block network structure to decode the input spatial encoding feature information.

[0150] Four, pre-training and fine-tuning stage

[0151] (1) Pre-training process:

[0152] First, define the loss function as the mean square error function of the time decoding information and the spatial decoding information on the motion data, complete the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module.

[0153] Specifically, the setting of the loss function will focus on the masked position, and only the position specified by the mask index in the decoding information and the motion information will be calculated according to the mask index, so as to force the model to better learn the significant features of time and space.

[0154] (2) Fine-tuning process:

[0155] Obtaining a small amount of supervised skeleton data, and using the same data preprocessing on the supervised data to obtain time skeleton data and spatial skeleton data.

[0156] The time encoder part in the time-based motion-dependent representation mask learning module and the space encoder part in the space-based motion-dependent representation mask learning module in the pre-training model are used to jointly constitute a feature extractor.

[0157] According to the feature extractor extracted in the pre-training process, a linear classifier is added to the bottom layer to jointly constitute a human action recognition module.

[0158] The human recognition module is fine-tuned using a small amount of supervised skeleton data, wherein the parameters of the feature extractor are frozen during the fine-tuning process, that is, the parameters of the time encoder and the space encoder are frozen, and the parameters of the bottom layer linear classifier are fine-tuned.

[0159] The time skeleton data and the spatial skeleton data are used as inputs of the human action recognition module, a loss function is defined as a cross-entropy function of the model output and the real category, and the fine-tuning process is completed.

[0160] FIG. 3 is a structure diagram of a 3D human action recognition device 300 based on an unsupervised algorithm according to an embodiment of the present application. As shown in FIG. 3, the device can include an acquisition unit 301, a conversion unit 302, a preprocessing unit 303, and a recognition unit 304, and can further include a pre-training fine-tuning unit. The main functions of each component module are as follows:

[0161] The acquisition unit 301 is configured to acquire two-dimensional picture information and depth information of a human action to be measured.

[0162] The conversion unit 302 is configured to obtain human skeleton data to be recognized by converting the two-dimensional picture information and the depth information.

[0163] The preprocessing unit 303 is configured to preprocess the human skeleton data to be recognized to obtain time skeleton data to be recognized and spatial skeleton data to be recognized.

[0164] The recognition unit 304 is configured to input the time skeleton data to be recognized and the spatial skeleton data to be recognized into the feature extractor and the linear classifier that have been pre-trained and fine-tuned in sequence to obtain an action recognition result.

[0165] As an implementable manner, the device includes a pre-training fine-tuning unit, which is configured to:

[0166] Directly obtain skeleton data in a preset data set for pre-training and fine-tuning of the model.

[0167] The skeleton data in the preset data set is preprocessed to obtain time skeleton data and space skeleton data;

[0168] The skeleton features of the skeleton data in the preset data set are mined by using an unsupervised mask algorithm, the time skeleton features of the time skeleton data are mined by using a time-based motion-dependent representation mask learning module, and the space skeleton features of the space skeleton data are mined by using a space-based motion-dependent representation mask learning module, to complete a pre-training process;

[0169] The feature extractor is formed by using the time-based motion-dependent representation learning module encoder and the space-based motion-dependent representation learning module encoder, and a linear classifier is added at the bottom of the feature extractor to complete a fine-tuning process.

[0170] As an implementable manner, the preprocessing unit 303 is further configured to:

[0171] The skeleton data in the preset data set is filled by using a bilinear interpolation method, and the skeleton data in the preset data set is adjusted to a fixed time length;

[0172] The skeleton data in the preset data set is subjected to dimension transformation to obtain time skeleton data and space skeleton data ,

[0173] wherein R represents the skeleton data in the preset data set, T represents the fixed time length frame number after bilinear interpolation, V represents the number of skeleton points of the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, X t represents the time skeleton data, and X j represents the space skeleton data;

[0174] The time skeleton data is subjected to a difference operation based on a motion skeleton formula to obtain motion skeleton data, wherein the motion skeleton formula is:

[0175] ;

[0176] wherein M represents the motion information of the skeleton data corresponding to the frame number, T represents the fixed time length frame number after bilinear interpolation, X i represents the skeleton data of the i th frame, and i represents the time frame.

[0177] As an implementable manner, the pre-training fine-tuning unit is specifically configured to:

[0178] The motion skeleton data is used as a guide to embed the semantic richness prior of feature mask, the motion intensity algorithm is used to calculate the motion intensity of each time frame, the motion intensity is converted into a probability distribution to obtain a first mask probability, the first mask probability is arranged in descending order to obtain the position index of the first 20 first mask probabilities as a first mask index. The motion intensity algorithm is as follows:

[0179] ;

[0180] wherein, represents the motion skeleton data corresponding to the frame number of the skeleton data, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, I represents the motion intensity, i represents the skeleton point number, and j represents the corresponding three-dimensional coordinate channel;

[0181] The time skeleton data is masked according to the first mask index, and the corresponding time skeleton data not masked by the mask is selected according to the first mask index;

[0182] The unmasked time skeleton data is extracted by the time encoder to obtain pre-encoding time feature information;

[0183] The pre-encoding time feature information is spliced with the learnable time mask token according to the first mask index to obtain time encoding feature information;

[0184] The time encoding feature information is decoded by the time decoder to obtain time decoding information.

[0185] As an implementable manner, the pre-training fine-tuning unit is specifically used for:

[0186] According to the spatial skeleton data, the skeleton point entropy algorithm is used to calculate the entropy of each skeleton point in the time sequence, and the entropy is converted into a probability distribution to obtain a second mask probability, the second mask probability is arranged in descending order to obtain the position index of the first 20 second mask probabilities as a second mask index.

[0187] wherein, the skeleton point entropy algorithm is as follows:

[0188] ;

[0189] ;

[0190] wherein, V represents the number of skeleton points, P(j) represents the probability frequency distribution of the skeleton point in the time sequence, G(j) represents the entropy in a certain spatial dimension, and G represents the sum of the entropies of three dimensions, represents the sum operation on C(j);

[0191] The spatial skeleton data is masked by a second mask index, and the spatial skeleton data corresponding to the spatial skeleton data not masked by the second mask index is selected;

[0192] The spatial skeleton data not masked by the mask is subjected to feature extraction by a spatial encoder to obtain pre-encoding spatial feature information;

[0193] The pre-encoding spatial feature information is spliced with a learnable spatial mask token according to the second mask index to obtain spatial encoding feature information.

[0194] The spatial encoding feature information is decoded according to a time decoder to obtain spatial decoding information.

[0195] As an implementable manner, the pre-training fine-tuning unit is specifically configured to: define a loss function as a mean square error function of the time decoding information and the spatial decoding information on the motion data, and complete the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module.

[0196] As an implementable manner, the pre-training fine-tuning unit is specifically configured to: utilize the time encoder in the time-based motion-dependent representation mask learning module and the spatial encoder in the space-based motion-dependent representation mask learning module in the pre-training model to jointly constitute a feature extractor.

[0197] The human body recognition module is fine-tuned by using a small amount of supervised skeleton data, wherein the parameters of the time encoder and the spatial encoder are frozen in the fine-tuning process, and the parameters of the bottom linear classifier are fine-tuned.

[0198] FIG. 4 shows a hardware structure schematic diagram of a computer device provided by an embodiment of the present application.

[0199] The computer device can include a processor 401 and a memory 402 having computer program instructions stored therein.

[0200] Specifically, the processor 401 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured as one or more integrated circuits implementing an embodiment of the present application.

[0201] The memory 402 can include mass storage for data or instructions. By way of example, and not limitation, the memory 402 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a tape drive, a USB drive, or a combination of two or more of these. In one example, the memory 402 can include removable or non-removable (or fixed) media, where the memory 402 is a nonvolatile storage. The memory 402 can be internal or external to the integrated gateway disaster recovery device.

[0202] In one example, the memory 402 can be a read-only memory (ROM). In one example, the ROM can be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0203] The processor 401 implements the methods / step S110-S140 in the embodiments shown in FIG. 1 by reading and executing computer program instructions stored in the memory 402, and achieves the corresponding technical effects achieved by the embodiments shown in FIG. 1 when executing the methods / step. For brevity, the description is not repeated here.

[0204] In one example, the computer device can further include a communication interface 403 and a bus 410. As shown in FIG. 4, the processor 401, the memory 402, and the communication interface 403 are connected through the bus 410 and complete communication among each other.

[0205] The communication interface 403 is mainly used to realize the communication among the modules, devices, units, and / or equipment in the embodiments of the present application.

[0206] Bus 410 includes a hardware, software, or both that couples components of the online data traffic metering device to each other. As an example without limitation, bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infmiband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Where suitable, bus 410 can include one or more buses. Although specific busses are described and shown, the present application contemplates any suitable bus or interconnect.

[0207] The computer device can be used to execute the 3D human action recognition method based on unsupervised mask algorithm in the embodiments of the present application, so as to realize the 3D human action recognition method and device based on unsupervised mask algorithm described in combination with Fig. 1 and Fig. 3.

[0208] In addition, in combination with the 3D human action recognition method based on unsupervised mask algorithm in the above embodiments, the embodiments of the present application can provide a computer storage medium to realize. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to realize any one of the 3D human action recognition methods based on unsupervised mask algorithm in the above embodiments.

[0209] It needs to be clear that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of simplicity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present application.

[0210] The functional blocks shown in the above structural block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a carrier wave in a transmission medium or communication link over a carrier wave. The "machine-readable medium" can include any medium that can store or transfer information. Examples of the machine-readable medium include an electronic circuit, a semiconductor memory device, a ROM, a flash memory, an erasable ROM (EROM), a floppy diskette, a CD-ROM, an optical disk, a hard disk, a fiber optic medium, a radio frequency (RF) link, and the like. The code segments can be downloaded via the computer network, such as the Internet, an intranet, and the like.

[0211] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0212] The above description is merely a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the above-described system, modules and units for the convenience and brevity of description, which can refer to the corresponding processes in the foregoing method embodiments, which will not be described here. It should be understood that the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A 3D human action recognition method based on an unsupervised mask algorithm, characterized in that: The method comprises: Obtain two-dimensional image information and depth information of the human body motion to be measured; Obtaining human skeleton data to be identified by converting the two-dimensional image information and the depth information; Preprocessing the human skeleton data to be identified to obtain time skeleton data to be identified and space skeleton data to be identified; The temporal skeleton data to be identified and the spatial skeleton data to be identified are sequentially input into a pre-trained and fine-tuned feature extractor and a linear classifier to obtain an action recognition result.

2. The 3D human motion recognition method based on an unsupervised mask algorithm according to claim 1, characterized in that: Before obtaining the two-dimensional image information and depth information of the human body motion to be measured, the method further includes: Directly obtain skeleton data from the preset dataset for model pre-training and fine-tuning; Preprocessing the skeleton data in the preset data set to obtain temporal skeleton data and spatial skeleton data; An unsupervised mask algorithm is used to mine skeleton features of the skeleton data in the preset data set, and a time-based motion-dependent representation mask learning module is used to mine temporal skeleton features of the temporal skeleton data and a space-based motion-dependent representation mask learning module is used to mine spatial skeleton features of the spatial skeleton data to complete the pre-training process; A feature extractor is constructed using a time-based motion dependency representation learning module encoder and a space-based motion dependency representation learning module encoder, and a linear classifier is added to the bottom layer of the feature extractor to complete the fine-tuning process.

3. The 3D human motion recognition method based on an unsupervised mask algorithm according to claim 2, characterized in that: The preprocessing of the skeleton data in the preset data set includes: Filling the skeleton data in the preset data set using a bilinear interpolation method, and adjusting the skeleton data in the preset data set to a fixed time length; The skeleton data in the preset data set is dimensionally transformed to obtain the time skeleton data and the space skeleton data ， Wherein R represents the skeleton data in the preset data set, T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points in the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, Xt represents the temporal skeleton data, and Xj represents the spatial skeleton data; Perform differential operation on the time skeleton data based on the motion skeleton formula to obtain the motion skeleton data, where the motion skeleton formula is: ； Where M represents the motion information of the corresponding frame number of the skeleton data, and T represents the fixed time length frame number after bilinear interpolation. Represents the skeleton data of the i-th frame, where i represents the time frame.

4. The 3D human motion recognition method based on an unsupervised mask algorithm according to claim 3, characterized in that: The method of mining temporal skeleton data features using a time-based motion-dependent representation mask learning module includes: The motion skeleton data is used as a semantic richness prior to guide feature masking embedding. The motion intensity of each time frame is calculated based on the motion intensity algorithm, and the motion intensity is converted into a probability distribution to obtain a first mask probability. The first mask probabilities are arranged in descending order, and the position indexes of the first 20 first mask probabilities are obtained as the first mask index. The motion intensity algorithm is as follows: ； in, Represents the motion skeleton data corresponding to the frame number of the skeleton data, V represents the number of skeleton points, C represents the number of 3D coordinate channels, I represents the motion intensity, i represents the sequence number of the skeleton point, and j represents the corresponding 3D coordinate channel; Masking the temporal skeleton data according to the first mask index, and selecting corresponding unmasked temporal skeleton data according to the first mask index; performing feature extraction on the unmasked temporal skeleton data by a temporal encoder to obtain pre-encoded temporal feature information; splicing the pre-coded time feature information with the learnable time mask token according to the first mask index to obtain time coding feature information; The time coding feature information is decoded by a time decoder to obtain time decoding information.

5. The 3D human motion recognition method based on unsupervised mask algorithm according to claim 4, characterized in that: The method of mining spatial skeleton data features using a space-based motion-dependent representation mask learning module includes: Based on the spatial skeleton data, the entropy of each skeleton point in the time series is calculated using the skeleton point entropy algorithm, and the entropy is converted into a probability distribution to obtain the second mask probability. The second mask probabilities are arranged in descending order, and the position indexes of the first 20 second mask probabilities are obtained as the second mask indexes. The skeleton point entropy algorithm is: ； ； Among them, V represents the number of skeleton points, P(j) represents the probability frequency distribution of skeleton points in the time series, G(j) represents the entropy in a certain spatial dimension, and G represents the sum of the entropy of the three-dimensional space. Indicates the summation operation on G(j); Masking the spatial skeleton data using the second mask index, and selecting corresponding unmasked spatial skeleton data according to the second mask index; Performing feature extraction on the spatial skeleton data not shielded by the mask by a spatial encoder to obtain pre-coded spatial feature information; concatenating the pre-coded spatial feature information with the learnable spatial mask token according to the second mask index to obtain spatial coding feature information; The spatial coding feature information is decoded according to a temporal decoder to obtain spatial decoding information.

6. The 3D human motion recognition method based on unsupervised mask algorithm according to claim 5, characterized in that: The pre-training process specifically includes: The loss function is defined as the mean square error function of the temporal decoding information and the spatial decoding information to the motion data, and the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module is completed.

7. The 3D human motion recognition method based on unsupervised mask algorithm according to claim 5, characterized in that: The fine-tuning process specifically includes: The feature extractor is composed of the temporal encoder in the time-based motion dependency representation mask learning module and the spatial encoder in the space-based motion dependency representation mask learning module in the pre-trained model. A small amount of supervised skeleton data is used to fine-tune the human recognition module, wherein the parameters of the temporal encoder and the spatial encoder are frozen during the fine-tuning process, and the parameters of the underlying linear classifier are fine-tuned.

8. A 3D human motion recognition device based on an unsupervised algorithm, characterized in that: The device comprises: An acquisition unit, used to acquire two-dimensional image information and depth information of the human body motion to be measured; a conversion unit, configured to obtain human skeleton data to be identified by converting the two-dimensional image information and the depth information; a preprocessing unit, configured to preprocess the human skeleton data to be identified to obtain time skeleton data to be identified and space skeleton data to be identified; The recognition unit is used to input the time skeleton data to be recognized and the space skeleton data to be recognized into a pre-trained and fine-tuned feature extractor and a linear classifier in sequence to obtain an action recognition result.

9. A computer device, characterized in that: The device includes: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the 3D human action recognition method based on the unsupervised mask algorithm according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that The computer storage medium stores computer program instructions, which, when executed by a processor, implement the 3D human motion recognition method based on an unsupervised mask algorithm according to any one of claims 1 to 7.

Images