A 3D human action recognition method and device based on unsupervised mask algorithm

Abstract

The present invention relates to the field of human motion recognition, and provides a 3D human motion recognition method and device based on an unsupervised mask algorithm. Two-dimensional image information and depth information of a human motion to be detected are obtained, and human skeleton data to be recognized are obtained by converting the two-dimensional image information and the depth information. The human skeleton data to be recognized are preprocessed to obtain time skeleton data to be recognized and space skeleton data to be recognized. The time skeleton data to be recognized and the space skeleton data to be recognized are sequentially input into a pre-trained and fine-tuned feature extractor and a linear classifier to obtain a motion recognition result. A large amount of unsupervised data can be used for algorithm training and optimization, avoiding the extra cost of marking unsupervised data as supervised data, and having an accuracy comparable to that of a model trained using supervised data, effectively improving the accuracy of unsupervised algorithms in recognizing human motion sequences.

Description

Technical Field

[0001] The present invention relates to the field of human motion recognition, and in particular to a 3D human motion recognition method and device based on an unsupervised mask algorithm. Background Art

[0002] The existing human action recognition technology provides a basic supervised action recognition framework, which can realize the recognition of the skeleton point positions of a single person or multiple people and realize the posture estimation of human actions.

[0003] However, supervised training requires a large amount of labeled data, which is very costly. Currently, most skeleton data is still unlabeled unsupervised data. The accuracy of existing unsupervised action recognition frameworks is still far from being improved compared to supervised action recognition frameworks. Therefore, there is an urgent need for a system that can use unsupervised data for training and has a high accuracy rate in motion classification to perform human action recognition. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a 3D human motion recognition method and device based on an unsupervised mask algorithm, which realizes accurate human motion recognition by utilizing skeleton data.

[0005] In the first aspect, a 3D human action recognition method based on an unsupervised mask algorithm is proposed, the method comprising:

[0006] Obtain two-dimensional image information and depth information of the human body motion to be measured;

[0007] By converting the two-dimensional image information and the depth information, the human skeleton data to be identified is obtained;

[0008] Preprocessing the human skeleton data to be identified to obtain the time skeleton data to be identified and the space skeleton data to be identified;

[0009] The temporal skeleton data to be recognized and the spatial skeleton data to be recognized are sequentially input into the pre-trained and fine-tuned feature extractor and linear classifier to obtain the action recognition result.

[0010] According to an achievable manner in an embodiment of the present application, before obtaining the two-dimensional image information and depth information of the human body motion to be measured, the method further includes:

[0011] Directly obtain the skeleton data in the preset data set for the model pre-training and fine-tuning process;

[0012] Preprocessing the skeleton data in the preset data set to obtain time skeleton data and space skeleton data;

[0013] The skeleton features of the skeleton data are mined using an unsupervised mask algorithm, and the temporal skeleton features of the temporal skeleton data are mined using a time-based motion-dependent representation mask learning module, and the spatial skeleton features of the spatial skeleton data are mined using a space-based motion-dependent representation mask learning module to complete the pre-training process;

[0014] The feature extractor is composed of a time-based motion dependency representation learning module encoder and a space-based motion dependency representation learning module encoder, and a linear classifier is added to the bottom layer of the feature extractor to complete the fine-tuning process.

[0015] According to an achievable method in an embodiment of the present application, preprocessing the skeleton data in the preset data set includes:

[0016] The skeleton data is filled using a bilinear interpolation method, and the skeleton data in the preset data set is adjusted to a fixed time length;

[0017] The skeleton data in the preset data set is transformed into time skeleton data X t ∈R T×V×C and space skeleton data X j ∈R V×T×C ,

[0018] Where R represents the skeleton data in the preset data set, T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points in the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, X t represents the time skeleton data, X j Represents spatial skeleton data;

[0019] The time skeleton data is differentially operated based on the motion skeleton formula to obtain the motion skeleton data, where the motion skeleton formula is:

[0020]

[0021] Where M represents the motion information of the skeleton data corresponding to the frame number, T represents the fixed time length frame number after bilinear interpolation, and X i,:,: Represents the skeleton data of the i-th frame, where i represents the time frame.

[0022] According to an achievable method in an embodiment of the present application, a time-based motion-dependent representation mask learning module is used to mine time skeleton data features, including:

[0023] Using motion skeleton data as a semantic richness prior to guide feature masking embedding, the motion intensity of each time frame is calculated based on the motion intensity algorithm, and the motion intensity is converted into a probability distribution to obtain the first mask probability. The first mask probabilities are arranged in descending order, and the position indexes of the first 20 first mask probabilities are obtained as the first mask index. Among them, the motion intensity algorithm is:

[0024]

[0025] Among them, M :,i,j Represents the motion skeleton data corresponding to the frame number of the skeleton data, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, I represents the motion intensity, i represents the sequence number of the skeleton point, and j represents the corresponding three-dimensional coordinate channel;

[0026] Masking the temporal skeleton data according to the first mask index, and selecting the corresponding temporal skeleton data not masked according to the first mask index;

[0027] Extracting features from the unmasked temporal skeleton data through a temporal encoder to obtain pre-encoded temporal feature information;

[0028] concatenating the pre-coded time feature information with the learnable time mask token according to the first mask index to obtain the time coding feature information;

[0029] The time coding feature information is decoded by a time decoder to obtain time decoding information.

[0030] According to an achievable method in an embodiment of the present application, a space-based motion-dependent representation mask learning module is used to mine spatial skeleton data features, including:

[0031] According to the spatial skeleton data, the entropy of each skeleton point in the time series is calculated by the skeleton point entropy algorithm, and the entropy is converted into a probability distribution to obtain the second mask probability. The second mask probabilities are arranged in descending order, and the position indexes of the first 20 second mask probabilities are obtained as the second mask index. Among them, the skeleton point entropy algorithm is:

[0032]

[0033] G=sum(G(j)) j∈X,Y,Z

[0034] Where V represents the number of skeleton points, P(j) represents the probability frequency distribution of skeleton points in the time series, G(j) represents the entropy in a certain spatial dimension, G represents the sum of the entropies of the three-dimensional space, and sum(G(j)) represents the summation operation of G(j);

[0035] Masking the spatial skeleton data by using the second mask index, and selecting the corresponding spatial skeleton data not shielded by the mask according to the second mask index;

[0036] The spatial skeleton data which is not shielded by the mask is subjected to feature extraction by a spatial encoder to obtain pre-coded spatial feature information;

[0037] concatenating the pre-coded spatial feature information with the learnable spatial mask token according to the second mask index to obtain spatial coded feature information;

[0038] The spatially encoded feature information is decoded according to the temporal decoder to obtain spatially decoded information.

[0039] According to an achievable method in the embodiment of the present application, the pre-training process specifically includes:

[0040] The loss function is defined as the mean square error function of the temporal decoding information and the spatial decoding information to the motion data, and the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module is completed.

[0041] According to an achievable method in an embodiment of the present application, the fine-tuning process specifically includes:

[0042] The feature extractor is composed of the temporal encoder in the time-based motion dependency representation mask learning module and the spatial encoder in the space-based motion dependency representation mask learning module in the pre-trained model.

[0043] A small amount of supervised skeleton data is used to fine-tune the human recognition module. During the fine-tuning process, the parameters of the temporal encoder and the spatial encoder are frozen, and the parameters of the underlying linear classifier are fine-tuned.

[0044] In the second aspect, a 3D human action recognition device based on an unsupervised mask algorithm is proposed, the device comprising:

[0045] An acquisition unit, used to acquire two-dimensional image information and depth information of the human body motion to be measured;

[0046] A conversion unit, used to obtain human skeleton data to be identified by converting the two-dimensional image information and the depth information;

[0047] A preprocessing unit, used for preprocessing the human skeleton data to be identified, to obtain the time skeleton data to be identified and the space skeleton data to be identified;

[0048] The recognition unit is used to input the time skeleton data to be recognized and the space skeleton data to be recognized into the pre-trained and fine-tuned feature extractor and linear classifier in sequence to obtain the action recognition result.

[0049] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a processor, and a memory storing computer program instructions, wherein the processor reads and executes the computer program instructions to execute the steps of a 3D human motion recognition method based on an unsupervised mask algorithm in the first aspect or any embodiment of the first aspect.

[0050] In a fourth aspect, an embodiment of the present invention provides a computer storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of a 3D human motion recognition method based on an unsupervised mask algorithm in the first aspect or any embodiment of the first aspect.

[0051] According to the technical content provided in the embodiments of the present application, two-dimensional image information and depth information of the human body action to be measured are obtained, and human skeleton data to be identified are obtained by converting the two-dimensional image information and the depth information. The human skeleton data to be identified are preprocessed to obtain time skeleton data to be identified and space skeleton data to be identified. The time skeleton data to be identified and the space skeleton data to be identified are input into a pre-trained and fine-tuned feature extractor and a linear classifier in sequence to obtain action recognition results. A large amount of unsupervised data can be used for algorithm training and optimization, avoiding the additional cost of marking unsupervised data as supervised data, and having an accuracy comparable to that of a model trained with supervised data. The feature information of time and space dimensions in the skeleton data is mined using a time-based motion-dependent representation mask learning module and a space-based motion-dependent representation mask learning module, thereby effectively improving the accuracy of the unsupervised algorithm in identifying human action sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flowchart of a 3D human action recognition method based on an unsupervised mask algorithm provided by an example of the present invention

[0053] Figure 2 Schematic diagram of a time-based motion-dependent representation mask learning module and a space-based motion-dependent representation mask learning module provided by an example of the present invention

[0054] Figure 3 Schematic diagram of a 3D human motion recognition device based on an unsupervised mask algorithm provided by an example of the present invention

[0055] Figure 4 is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention; DETAILED DESCRIPTION

[0056] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention more clear, the following is combined with the accompanying drawings and specific embodiments.

[0057] The present invention is further described in detail. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0058] Figure 1 A 3D human action recognition method based on an unsupervised mask algorithm provided by an example of the present invention is shown, comprising:

[0059] S110, obtaining two-dimensional image information and depth information of the human body motion to be measured.

[0060] Two-dimensional image information and depth information can be obtained by using binocular camera equipment, laser radar monocular camera and other camera equipment. Among them, two-dimensional image information refers to various information carried by the image represented on the two-dimensional plane, including image pixel value, resolution and other information. Depth information refers to the distance and depth value of each pixel in the image or scene, including two-dimensional image information and information about the distance between objects in the image.

[0061] S120, obtaining human skeleton data to be identified by converting the two-dimensional image information and the depth information.

[0062] Using the two-dimensional image information and depth information obtained by the camera device, using the human body detection and posture estimation algorithm, the key points of the human body, such as the head, shoulders, elbows, wrists, hips, knees and ankles, are detected and extracted from the two-dimensional image.

[0063] The depth image is then converted into the same coordinate system as the two-dimensional image to ensure that each pixel in the depth image corresponds to a pixel in the two-dimensional image.

[0064] The human key points detected in the 2D image are associated with the corresponding pixels in the depth image. This can be achieved by mapping the key point coordinates in the 2D image to the depth image, and then obtaining the associated depth value from the depth image.

[0065] Using the associated depth information, the three-dimensional skeleton of the human body is reconstructed according to the positions and depth values ​​of the key points of the human body, and finally the human skeleton data to be identified is obtained.

[0066] S130, preprocessing the skeleton data to be identified to obtain the temporal skeleton data to be identified and the spatial skeleton data to be identified.

[0067] For the skeleton data to be identified, a bilinear interpolation algorithm is used to adjust the skeleton data to be identified to a uniform length.

[0068] By transforming the dimension of the skeleton data to be identified, the time skeleton data X to be identified is obtained. t ∈R T×V×C .

[0069] By performing dimension transformation on the skeleton data to be identified, the spatial skeleton data X to be identified is obtained. j ∈R V×T×C .

[0070] Where T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points in the source dataset, and C represents the x, y, and z coordinates.

[0071] S140, inputting the time skeleton data to be recognized and the space skeleton data to be recognized into a pre-trained and fine-tuned feature extractor and a linear classifier in sequence to obtain an action recognition result.

[0072] The temporal skeleton data to be identified and the spatial skeleton data to be identified are respectively input into the pre-trained feature extractor to obtain the temporal features and spatial features. The temporal features and spatial features are concatenated and input into the fine-tuned linear classifier to obtain the action recognition result.

[0073] It can be seen that the embodiment of the present application obtains two-dimensional image information and depth information of the human body motion to be measured, converts the two-dimensional image information and the depth information to obtain human skeleton data to be identified, pre-processes the human skeleton data to be identified to obtain time skeleton data to be identified and space skeleton data to be identified, and the time skeleton data to be identified and the space skeleton data to be identified are sequentially input into a pre-trained and fine-tuned feature extractor and a linear classifier to obtain a motion recognition result, thereby realizing a high-accuracy human motion recognition method that can be completed with a small amount of supervised data.

[0074] As a feasible method, before obtaining the two-dimensional image information and depth information of the human body motion to be measured, the method further includes: Figure 2 As shown:

[0075] Directly obtain skeleton data in the preset dataset for model pre-training and fine-tuning;

[0076] Preprocessing the skeleton data in the preset data set to obtain time skeleton data and space skeleton data;

[0077] The unsupervised mask algorithm is used to mine the skeleton features in the skeleton data, and the time-based motion-dependent representation mask learning module is used to mine the temporal skeleton data features and the space-based motion-dependent representation mask learning module is used to mine the spatial skeleton data features to complete the pre-training process;

[0078] The feature extractor is composed of the encoder part of the time-based motion dependency representation learning module and the encoder part of the space-based motion dependency representation learning module, and a linear classifier is added to the bottom layer of the feature extractor to complete the fine-tuning process.

[0079] The preset data set may be the NTU-RGB+D 60 data set, and the method for mining skeleton features in skeleton data using an unsupervised mask algorithm specifically includes: pre-training and fine-tuning processes of feature extractors and linear classifiers; specifically, directly obtaining skeleton data in the NTU-RGB+D 60 data set for model pre-training and fine-tuning processes;

[0080] Preprocess the skeleton data in the data set to obtain temporal skeleton data and spatial skeleton data;

[0081] According to the temporal skeleton data and the spatial skeleton data, feature mining is performed using a time-based motion-dependent representation mask learning module and a space-based motion-dependent representation mask learning module respectively to complete the model pre-training process.

[0082] The feature extractor is composed of the temporal encoder module of the time-based motion dependency representation mask learning module and the spatial encoder module of the space-based motion dependency representation mask learning module in the pre-trained model, and a linear classifier is added at the bottom layer to complete the fine-tuning part.

[0083] As an achievable approach, a time-based motion-dependent representation mask learning module is used to mine temporal skeleton data features, including:

[0084] Using motion skeleton data as a semantic richness prior to guide feature masking embedding, the motion intensity of each time frame is calculated based on the motion intensity algorithm, and the motion intensity is converted into a probability distribution to obtain the first mask probability. The first mask probabilities are arranged in descending order, and the position indexes of the first 20 first mask probabilities are obtained as the first mask index. Among them, the motion intensity algorithm is:

[0085]

[0086] Among them, M :,i,j Represents the motion skeleton data corresponding to the frame number of the skeleton data, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, I represents the motion intensity, i represents the sequence number of the skeleton point, and j represents the corresponding three-dimensional coordinate channel;

[0087] Masking the temporal skeleton data according to the first mask index, and selecting the corresponding temporal skeleton data not masked according to the first mask index;

[0088] Extracting features from the unmasked temporal skeleton data through a temporal encoder to obtain pre-encoded temporal feature information;

[0089] concatenating the pre-coded time feature information with the learnable time mask token according to the first mask index to obtain the time coding feature information;

[0090] The time coding feature information is decoded by a time decoder to obtain time decoding information.

[0091] The first mask probability is a mask probability obtained based on the motion intensity, and the first mask index is a mask index obtained based on the first mask probability.

[0092] As a feasible approach, a space-based motion-dependent representation mask learning module is used to mine spatial skeleton data features, including:

[0093] According to the spatial skeleton data, the entropy of each skeleton point in the time series is calculated by the skeleton point entropy algorithm, and the entropy is converted into a probability distribution to obtain the second mask probability. The second mask probabilities are arranged in descending order, and the position indexes of the first 20 second mask probabilities are obtained as the second mask index. Among them, the skeleton point entropy algorithm is:

[0094]

[0095] G=sum(G(j)) j∈X,Y,Z

[0096] Where V represents the number of skeleton points, P(j) represents the probability frequency distribution of skeleton points in the time series, G(j) represents the entropy in a certain spatial dimension, G represents the sum of the entropies of the three-dimensional space, and sum(G(j)) represents the summation operation of G(j);

[0097] Masking the spatial skeleton data by using the second mask index, and selecting the corresponding spatial skeleton data not shielded by the mask according to the second mask index;

[0098] The spatial skeleton data which is not shielded by the mask is subjected to feature extraction by a spatial encoder to obtain pre-coded spatial feature information;

[0099] concatenating the pre-coded spatial feature information with the learnable spatial mask token according to the second mask index to obtain spatial coded feature information;

[0100] The spatially encoded feature information is decoded according to the temporal decoder to obtain spatially decoded information.

[0101] The second mask probability is a mask probability obtained based on the entropy of the time series, and the second mask index is a mask index obtained based on the second mask probability.

[0102] As an achievable method, the pre-training process specifically includes:

[0103] The loss function is defined as the mean square error function of the temporal decoding information and the spatial decoding information to the motion data, and the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module is completed.

[0104] As a feasible approach, the fine-tuning process specifically includes: utilizing the temporal encoder in the time-based motion dependency representation mask learning module in the pre-trained model and the spatial encoder part in the space-based motion dependency representation mask learning module to jointly form a feature extractor.

[0105] A small amount of supervised skeleton data is used to fine-tune the human recognition module. During the fine-tuning process, the parameters of the temporal encoder and the spatial encoder are frozen, and the parameters of the underlying linear classifier are fine-tuned.

[0106] Combine the following Figure 2 The embodiments of the present invention are further described as follows:

[0107] 1. Preprocessing stage:

[0108] (1) Directly obtain the skeleton data in the NTU-RGB+D 60 dataset for model pre-training and fine-tuning;

[0109] (2) For skeleton data, a bilinear interpolation algorithm is used to adjust the skeleton data to a uniform length.

[0110] (3) For the time skeleton data, the time skeleton data X is obtained by transforming the skeleton data dimensionally. t ∈R T ×V×C .

[0111] (4) For the spatial skeleton data, the spatial skeleton data X is obtained by transforming the skeleton data dimensionally. j ∈R V ×T×C .

[0112] Where T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points in the source data set, and C represents the x, y, and z coordinates in the source data set.

[0113] (5) Based on the motion skeleton formula, the time skeleton data is obtained, and a difference operation is performed on the time skeleton data to obtain the motion skeleton data, wherein the motion skeleton formula is:

[0114]

[0115] Where M represents the motion information of the skeleton data corresponding to the frame number, T represents the fixed time length frame number after bilinear interpolation, and X i,:,:Represents the skeleton data of the i-th frame, where i represents the time frame.

[0116] 2. Time-based motion-dependent representation mask learning:

[0117] Using motion skeleton data as a guide to embed the semantic richness prior of feature masking, the motion intensity of each time frame is calculated, and the motion intensity is converted into a probability distribution to obtain the mask probability. The mask probabilities are then arranged in descending order to obtain the position index of the first 20 mask probabilities as the mask index. The motion intensity calculation formula is:

[0118]

[0119] Among them, M :,i,j Indicates the motion information of the corresponding frame number of the skeleton data, V indicates the number of skeleton points, C indicates the number of three-dimensional coordinate channels, I indicates the motion intensity, i indicates the sequence number of the skeleton point, and j indicates the corresponding three-dimensional coordinate channel;

[0120] The formula for converting motion intensity to mask index is:

[0121] π=Softmax(I),

[0122] g=-log(-log∈),∈∈U[0,1] T ,

[0123] idx mask =Index-of-Top-K(logπ+g),

[0124] Where Π indicates the calculation of motion intensity using the Softmax function, g indicates the regularization term, U indicates the random distribution from 0 to 1, Index-of-Top-K indicates taking out the index with the largest value, idx mask Represents a mask index.

[0125] The time skeleton data is masked according to the obtained mask index, that is, the corresponding time skeleton data that is not masked is selected according to the mask index. In the unmasked time skeleton data part of the corresponding figure, the unmasked data is represented by a solid box, and the masked data is represented by a dotted box.

[0126] The time encoder performs feature extraction on the unmasked time skeleton data to obtain pre-coded time feature information.

[0127] The temporal encoder combines a convolutional neural network with an eight-layer self-attention-based feature extraction block network structure. The self-attention-based feature extraction block consists of a multi-head attention layer, a normalization layer, and a forward propagation layer. Through the stacked self-attention-based feature extraction block network structure extraction operation, higher-level features are obtained. And the unmasked temporal skeleton data features are mined through the temporal encoder.

[0128] According to the pre-coded time feature information extracted by the time encoder, the pre-coded time feature information is combined with the learnable time mask token according to the mask index. The learnable time mask token corresponds to the gray part in the time coding feature information module in the figure. The gray box and the white box together constitute the time coding feature information.

[0129] The time encoding feature information is decoded according to the time decoder to obtain time decoding information.

[0130] The temporal decoder uses the same feature extraction block as the temporal encoder and a three-layer self-attention-based feature extraction block network structure to decode the input temporal encoding feature information.

[0131] 3. Space-based motion-dependent representation mask learning:

[0132] According to the spatial skeleton data, the entropy of each skeleton point in the time series is calculated, and the entropy is converted into a probability distribution to obtain the mask probability. The mask probabilities are then arranged in descending order to obtain the position indexes of the top 20 maximum probabilities as the mask indexes. The calculation formula for the skeleton point entropy is:

[0133]

[0134] G=sum(G(j)) j∈X,Y,Z

[0135] Where V represents the number of skeleton points, P(j) represents the probability frequency distribution of skeleton points in the time series, G(j) represents the entropy in a certain spatial dimension, G represents the sum of the entropies of the three-dimensional space, and sum(G(j)) represents the summation operation of G(j);

[0136] The formula for converting entropy intensity into mask index is:

[0137] π=Softmax(G),

[0138] g=-log(-log∈),∈∈U[0,1] T ,

[0139] idx mask =Index-of-Top-K(logπ+g),

[0140] Among them, Π indicates the calculation of entropy using the Softmax function, g indicates the regular term, U indicates the random distribution from 0 to 1, Index-of-Top-K indicates taking out the index with the larger value, idx mask Represents a mask index.

[0141] The spatial skeleton data is masked according to the obtained mask index, that is, the corresponding spatial skeleton data that is not masked is selected according to the mask index. In the unmasked spatial skeleton data part in the corresponding figure, the unmasked data is represented by a solid box, and the masked data is represented by a dotted box.

[0142] The unmasked spatial skeleton data is subjected to feature extraction according to the spatial encoder to obtain pre-coded spatial feature information.

[0143] The spatial encoder combines a convolutional neural network with an eight-layer self-attention-based feature extraction block network structure. The self-attention-based feature extraction block consists of a multi-head attention layer, a normalization layer, and a forward propagation layer. Through the stacked self-attention-based feature extraction block network structure extraction operation, higher-level features are obtained. The spatial encoder also mines unmasked spatial skeleton data features.

[0144] According to the pre-coded spatial feature information extracted by the spatial encoder, the pre-coded spatial feature information is concatenated with the learnable spatial mask token according to the mask index. The learnable spatial mask token corresponds to the gray part in the spatial coding feature information module in the figure. The gray box and the white box together constitute the spatial coding feature information.

[0145] The spatial encoding feature information is decoded according to the spatial decoder to obtain spatial decoding information.

[0146] The spatial decoder uses the same feature extraction block as the spatial encoder and a three-layer self-attention-based feature extraction block network structure to decode the input spatial encoded feature information.

[0147] 4. Pre-training and fine-tuning phase

[0148] (1) Pre-training process:

[0149] Firstly, the loss function is defined as the mean square error function of the temporal decoding information and the spatial decoding information to the motion data, and the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module is completed.

[0150] Specifically, the setting of the loss function will focus on the masked positions, and only calculate the positions specified by the mask index in the decoding information and motion information according to the mask index, forcing the model to better learn the salient features of time and space.

[0151] (2) Fine-tuning process:

[0152] Obtain a small amount of supervised skeleton data, and apply the same data preprocessing to the supervised data to obtain temporal skeleton data and spatial skeleton data.

[0153] The feature extractor is composed of the temporal encoder part in the time-based motion dependency representation mask learning module and the spatial encoder part in the space-based motion dependency representation mask learning module in the pre-trained model.

[0154] According to the feature extractor extracted during the pre-training process, a linear classifier is added to its bottom layer to form a human action recognition module.

[0155] A small amount of supervised skeleton data is used to fine-tune the human recognition module. During the fine-tuning process, the parameters of the feature extractor are frozen, that is, the parameters of the temporal encoder and the spatial encoder are frozen, and the parameters of the underlying linear classifier are fine-tuned.

[0156] The temporal skeleton data and the spatial skeleton data are used as the input of the human action recognition module, and the loss function is defined as the cross entropy function between the model output and the true category to complete the fine-tuning process.

[0157] Figure 3 The structure of a 3D human motion recognition device 300 based on an unsupervised algorithm provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the device may include: an acquisition unit 301, a conversion unit 302, a pre-processing unit 303 and a recognition unit 304, and may further include: a pre-training fine-tuning unit. The main functions of each component module are as follows:

[0158] An acquisition unit 301 is used to acquire two-dimensional image information and depth information of a human body motion to be measured;

[0159] The conversion unit 302 is used to obtain human skeleton data to be identified by converting the two-dimensional image information and the depth information;

[0160] A preprocessing unit 303 is used to preprocess the human skeleton data to be identified to obtain the time skeleton data to be identified and the space skeleton data to be identified;

[0161] The recognition unit 304 is used to input the time skeleton data to be recognized and the space skeleton data to be recognized into the pre-trained and fine-tuned feature extractor and linear classifier in sequence to obtain the action recognition result.

[0162] As an achievable method, the device includes a pre-training fine-tuning unit. Before obtaining the two-dimensional image information and depth information of the human body action to be measured, the pre-training fine-tuning unit is used to:

[0163] Directly obtain the skeleton data in the preset data set for the model pre-training and fine-tuning process;

[0164] Preprocessing the skeleton data in the preset data set to obtain time skeleton data and space skeleton data;

[0165] The unsupervised mask algorithm is used to mine the skeleton features of the skeleton data in the preset data set, and the time-based motion-dependent representation mask learning module is used to mine the temporal skeleton features of the temporal skeleton data and the space-based motion-dependent representation mask learning module is used to mine the spatial skeleton features of the spatial skeleton data to complete the pre-training process;

[0166] The feature extractor is composed of a time-based motion dependency representation learning module encoder and a space-based motion dependency representation learning module encoder, and a linear classifier is added to the bottom layer of the feature extractor to complete the fine-tuning process.

[0167] As an implementable manner, the preprocessing unit 303 is further configured to:

[0168] The skeleton data in the preset data set is filled using a bilinear interpolation method, and the skeleton data in the preset data set is adjusted to a fixed time length;

[0169] The skeleton data in the preset data set is transformed into time skeleton data X t ∈R T×V×C and space skeleton data X j ∈R V×T×C ,

[0170] Where R represents the skeleton data in the preset data set, T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points in the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, X t represents the time skeleton data, X j Represents spatial skeleton data;

[0171] The time skeleton data is differentially operated based on the motion skeleton formula to obtain the motion skeleton data, where the motion skeleton formula is:

[0172]

[0173] Where M represents the motion information of the skeleton data corresponding to the frame number, T represents the fixed time length frame number after bilinear interpolation, and X i,:,: Represents the skeleton data of the i-th frame, where i represents the time frame.

[0174] As an achievable approach, the pre-trained fine-tuning unit is specifically used for:

[0175] Using motion skeleton data as a semantic richness prior to guide feature masking embedding, the motion intensity of each time frame is calculated based on the motion intensity algorithm, and the motion intensity is converted into a probability distribution to obtain the first mask probability. The first mask probabilities are arranged in descending order, and the position indexes of the first 20 first mask probabilities are obtained as the first mask index. Among them, the motion intensity algorithm is:

[0176]

[0177] Among them, M :,i,j Represents the motion skeleton data corresponding to the frame number of the skeleton data, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, I represents the motion intensity, i represents the sequence number of the skeleton point, and j represents the corresponding three-dimensional coordinate channel;

[0178] Masking the temporal skeleton data according to the first mask index, and selecting the corresponding temporal skeleton data not masked according to the first mask index;

[0179] Extracting features from the unmasked temporal skeleton data through a temporal encoder to obtain pre-encoded temporal feature information;

[0180] concatenating the pre-coded time feature information with the learnable time mask token according to the first mask index to obtain the time coding feature information;

[0181] The time coding feature information is decoded by a time decoder to obtain time decoding information.

[0182] As an achievable approach, the pre-trained fine-tuning unit is specifically used for:

[0183] According to the spatial skeleton data, the entropy of each skeleton point in the time series is calculated by the skeleton point entropy algorithm, and the entropy is converted into a probability distribution to obtain the second mask probability. The second mask probabilities are arranged in descending order, and the position indexes of the first 20 second mask probabilities are obtained as the second mask index. Among them, the skeleton point entropy algorithm is:

[0184]

[0185] G=sum(G(j)) j∈X,Y,Z

[0186] Where V represents the number of skeleton points, P(j) represents the probability frequency distribution of skeleton points in the time series, G(j) represents the entropy in a certain spatial dimension, G represents the sum of the entropies of the three-dimensional space, and sum(G(j)) represents the summation operation of G(j);

[0187] Masking the spatial skeleton data by using the second mask index, and selecting the corresponding spatial skeleton data not shielded by the mask according to the second mask index;

[0188] The spatial skeleton data which is not shielded by the mask is subjected to feature extraction by a spatial encoder to obtain pre-coded spatial feature information;

[0189] concatenating the pre-coded spatial feature information with the learnable spatial mask token according to the second mask index to obtain spatial coded feature information;

[0190] The spatially encoded feature information is decoded according to the temporal decoder to obtain spatially decoded information.

[0191] As a feasible method, the pre-training fine-tuning unit is specifically used to: define the loss function as the mean square error function of the temporal decoding information and the spatial decoding information to the motion data, and complete the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module.

[0192] As an achievable method, the pre-trained fine-tuning unit is specifically used to: utilize the temporal encoder in the time-based motion dependency representation mask learning module and the spatial encoder part in the space-based motion dependency representation mask learning module in the pre-trained model to jointly form a feature extractor.

[0193] A small amount of supervised skeleton data is used to fine-tune the human recognition module. During the fine-tuning process, the parameters of the temporal encoder and the spatial encoder are frozen, and the parameters of the underlying linear classifier are fine-tuned.

[0194] Figure 4 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention is shown.

[0195] The computer device may include a processor 401 and a memory 402 storing computer program instructions.

[0196] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0197] The memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 402 may include a removable or non-removable (or fixed) medium, or the memory 402 is a non-volatile solid-state memory. The memory 402 may be inside or outside the integrated gateway disaster recovery device.

[0198] In one example, the memory 402 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0199] The processor 401 reads and executes the computer program instructions stored in the memory 402 to implement Figure 1 The method / steps S110 to S140 in the embodiment shown in the figure are achieved Figure 1 The corresponding technical effects achieved by executing the methods / steps in the examples shown are not repeated here for the sake of brevity.

[0200] In one example, the computer device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0201] The communication interface 403 is mainly used to implement the communication between the modules, devices, units and / or equipment in the embodiment of the present invention.

[0202] Bus 410 includes hardware, software or both, and couples the components of online data traffic billing equipment to each other. For example, but not limitation, the bus may include Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), Hyper Transport (HT) interconnection, Industry Standard Architecture (ISA) bus, InfiniBand interconnection, Low Pin Count (LPC) bus, Memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the embodiments of the present invention describe and illustrate specific buses, the present invention considers any suitable bus or interconnection.

[0203] The computer device can be used to execute the 3D human action recognition method based on the unsupervised mask algorithm in the embodiment of the present invention, thereby realizing the combination of Figure 1 and Figure 3 The invention describes a 3D human action recognition method and device based on an unsupervised mask algorithm.

[0204] In addition, in combination with the 3D human motion recognition method based on the unsupervised mask algorithm in the above embodiment, the embodiment of the present invention can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the 3D human motion recognition methods based on the unsupervised mask algorithm in the above embodiment is implemented.

[0205] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0206] The functional blocks shown in the above structured block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0207] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0208] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A 3D human action recognition method based on an unsupervised mask algorithm, characterized in that: The method comprises: Obtain two-dimensional image information and depth information of the human body motion to be measured; By converting the two-dimensional image information and the depth information, human skeleton data to be identified is obtained; Preprocessing the human skeleton data to be identified to obtain time skeleton data to be identified and space skeleton data to be identified; Inputting the time skeleton data to be identified and the space skeleton data to be identified into a pre-trained and fine-tuned feature extractor and a linear classifier in sequence to obtain an action recognition result; Wherein, the preprocessing of the human skeleton data to be identified includes: Filling the human skeleton data to be identified using a bilinear interpolation method, and adjusting the human skeleton data to be identified to a fixed time length; The human skeleton data to be identified is transformed into the time skeleton data X to be identified. t1 ∈R1 T×V1×C1 and the spatial skeleton data X to be identified j1 ∈R1 V1×T×C1 , Wherein, R1 represents the human skeleton data to be identified, T represents the number of frames of fixed time length after bilinear interpolation, V1 represents the number of skeleton points of the human skeleton data to be identified, C1 represents the x, y, z coordinates of the human skeleton data to be identified, X t1 represents the time skeleton data to be identified, X j1 represents the spatial skeleton data to be identified; The time skeleton data to be identified is subjected to a differential operation based on a motion skeleton formula to obtain the motion skeleton data to be identified, wherein the motion skeleton formula is: Among them, M1 represents the motion information of the corresponding frame number of the skeleton data to be identified, T represents the fixed time length frame number after bilinear interpolation, and X1 i，：，： Represents the skeleton data to be recognized in the i-th frame, where i represents the time frame.

2. The 3D human action recognition method based on unsupervised mask algorithm according to claim 1, characterized in that: Before obtaining the two-dimensional image information and depth information of the human body motion to be measured, the method further includes: Directly obtain the skeleton data in the preset data set for the model pre-training and fine-tuning process; Preprocessing the skeleton data in the preset data set to obtain temporal skeleton data and spatial skeleton data; An unsupervised mask algorithm is used to mine the skeleton features of the skeleton data in the preset data set, and a time-based motion-dependent representation mask learning module is used to mine the temporal skeleton features of the temporal skeleton data and a space-based motion-dependent representation mask learning module is used to mine the spatial skeleton features of the spatial skeleton data to complete the pre-training process; A feature extractor is formed by using a time-based motion dependency representation learning module encoder and a space-based motion dependency representation learning module encoder, and a linear classifier is added to the bottom layer of the feature extractor to complete the fine-tuning process.

3. The 3D human action recognition method based on unsupervised mask algorithm according to claim 2, characterized in that: The preprocessing of the skeleton data in the preset data set includes: Filling the skeleton data in the preset data set using a bilinear interpolation method, and adjusting the skeleton data in the preset data set to a fixed time length; The skeleton data in the preset data set is dimensionally transformed to obtain the time skeleton data X t ∈R T×V×C and the space skeleton data X j ∈R V×T×C , Where R represents the skeleton data in the preset data set, T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points in the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, X t represents the time skeleton data, X j represents the spatial skeleton data; The time skeleton data is differentially operated based on the motion skeleton formula to obtain the motion skeleton data, where the motion skeleton formula is: Where M represents the motion information of the skeleton data corresponding to the frame number, T represents the fixed time length frame number after bilinear interpolation, and X i,:,: Represents the skeleton data of the i-th frame, where i represents the time frame.

4. The 3D human action recognition method based on unsupervised mask algorithm according to claim 3 is characterized in that: The method of mining time skeleton data features using a time-based motion-dependent representation mask learning module includes: The motion skeleton data is used as a semantic richness prior to guide the embedding feature masking, the motion intensity of each time frame is calculated based on the motion intensity algorithm, and the motion intensity is converted into a probability distribution to obtain a first mask probability, and the first mask probabilities are arranged in order from large to small to obtain the position indexes of the first 20 first mask probabilities as the first mask index, wherein the motion intensity algorithm is: Among them, M :,k,j The motion skeleton data representing the frame number corresponding to the skeleton data, V represents the number of skeleton points, C represents the x, y, z coordinates in the preset data set, I represents the motion intensity, k represents the skeleton point sequence number, and j represents one of the X channel, Y channel, and Z channel in the corresponding three-dimensional coordinate channel; Masking the time skeleton data according to the first mask index, and selecting corresponding time skeleton data that is not masked according to the first mask index; Extracting features from the unmasked time skeleton data using a time encoder to obtain pre-coded time feature information; splicing the pre-coded time feature information with the learnable time mask token according to the first mask index to obtain time coding feature information; The time coding characteristic information is decoded by a time decoder to obtain time decoding information.

5. The 3D human action recognition method based on unsupervised mask algorithm according to claim 4, characterized in that: The method of mining spatial skeleton data features using a space-based motion-dependent representation mask learning module includes: According to the spatial skeleton data, the entropy of each skeleton point in the time series is calculated by the skeleton point entropy algorithm, and the entropy is converted into a probability distribution to obtain a second mask probability, and the second mask probabilities are arranged in descending order to obtain the position indexes of the first 20 second mask probabilities as the second mask indexes, wherein the skeleton point entropy algorithm is: G=sum(G(j)) j∈X,Y,Z Where V represents the number of skeleton points, P(j) represents the probability frequency distribution of the skeleton points in the time series, G(j) represents the entropy in a certain spatial dimension, G represents the sum of the entropies of the three-dimensional space, and sum(G(j)) represents the summation operation of G(j); Masking the spatial skeleton data by using the second mask index, and selecting corresponding spatial skeleton data that is not masked according to the second mask index; Extracting features from the spatial skeleton data not shielded by the mask by using a spatial encoder to obtain pre-coded spatial feature information; concatenating the pre-coded spatial feature information with the learnable spatial mask token according to the second mask index to obtain spatial coded feature information; The spatial coding feature information is decoded according to a temporal decoder to obtain spatial decoding information.

6. The 3D human action recognition method based on unsupervised mask algorithm according to claim 5, characterized in that: The pre-training process specifically includes: The loss function is defined as the mean square error function of the temporal decoding information and the spatial decoding information to the motion data, and the pre-training process of the time-based motion-dependent representation mask learning module and the space-based motion-dependent representation mask learning module is completed.

7. The 3D human action recognition method based on unsupervised mask algorithm according to claim 5, characterized in that: The fine-tuning process specifically includes: The temporal encoder in the time-based motion dependency representation mask learning module and the spatial encoder part in the space-based motion dependency representation mask learning module in the pre-trained model are used to form a feature extractor; A human recognition module is fine-tuned using a small amount of supervised skeleton data, wherein the parameters of the temporal encoder and the spatial encoder are frozen during the fine-tuning process, and the parameters of the underlying linear classifier are fine-tuned.

8. A 3D human motion recognition device based on an unsupervised algorithm, characterized in that: The device comprises: An acquisition unit, used to acquire two-dimensional image information and depth information of the human body motion to be measured; A conversion unit, configured to obtain human skeleton data to be identified by converting the two-dimensional image information and the depth information; A preprocessing unit is used to preprocess the human skeleton data to be identified to obtain time skeleton data to be identified and space skeleton data to be identified. The preprocessing of the human skeleton data to be identified includes: using a bilinear interpolation method to fill the human skeleton data to be identified, adjusting the human skeleton data to be identified to a fixed time length; performing a dimensional transformation on the human skeleton data to be identified to obtain the time skeleton data X to be identified. t1 ∈R1 T×V1×C1 and the spatial skeleton data X to be identified j1 ∈R1 V1×T×C1 , where R1 represents the human skeleton data to be identified, T represents the number of frames of fixed time length after bilinear interpolation, V1 represents the number of skeleton points of the human skeleton data to be identified, C1 represents the x, y, z coordinates of the human skeleton data to be identified, X t1 represents the time skeleton data to be identified, X j1 represents the spatial skeleton data to be identified; performing a differential operation on the time skeleton data to be identified based on a motion skeleton formula to obtain the motion skeleton data to be identified, wherein the motion skeleton formula is: Among them, M1 represents the motion information of the corresponding frame number of the skeleton data to be identified, T represents the fixed time length frame number after bilinear interpolation, and X1 i，：，： represents the skeleton data to be recognized in the i-th frame, where i represents the time frame; The recognition unit is used to input the time skeleton data to be recognized and the space skeleton data to be recognized into a pre-trained and fine-tuned feature extractor and a linear classifier in sequence to obtain an action recognition result.

9. A computer device, characterized in that: The device includes: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the 3D human action recognition method based on an unsupervised mask algorithm as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the 3D human action recognition method based on the unsupervised mask algorithm as described in any one of claims 1 to 7 is implemented.

Images