Unsupervised human action recognition method and device based on information entropy guidance

Abstract

The present invention is applicable to the field of computer vision, and is an unsupervised human motion recognition method and device based on information entropy guidance. The method includes: obtaining time skeleton features and space skeleton features through dimensional transformation of human motion, using information entropy algorithms and multi-layer convolutional networks under a variety of different conditions to obtain time skeleton features and space skeleton features, and then completing the pre-training process of the feature extraction network according to the comparative learning method to obtain an unsupervised human motion recognition model based on information entropy guidance. Since the present invention introduces a variety of information entropy algorithms, the deep neural network can efficiently learn the deep semantic features of human motion. Through the information entropy obtained under different conditions, the feature extraction network can be made directional, the perception depth of the neural network is expanded, the learning speed of the model is accelerated, and the accuracy of motion recognition is greatly improved, so that the deep learning unsupervised algorithm can complete the recognition task more efficiently.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and in particular relates to an unsupervised human motion recognition method and device based on information entropy guidance. Background Art

[0002] Human action recognition uses a deep learning unsupervised algorithm. Unlike a deep learning supervised algorithm, the training sample data does not require manual label annotations. By mining the temporal and spatial dual-dimensional features of the same human action, the deep semantic features of the human action are learned by comparison to achieve human action recognition. However, the difference in information entropy is not distinguished in the dual-dimensional feature mining process, resulting in the inability of the deep neural network to efficiently learn the deep semantic features of human actions. In addition, the two feature representations used for comparative learning often have only a simple and direct linear relationship, which affects the ability of the dual-dimensional feature mining process to obtain deep semantic features and greatly reduces the accuracy of recognition. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an unsupervised human motion recognition method and device based on information entropy guidance, which can use a variety of information entropy algorithms to calculate the information entropy of human motion under different conditions, directionally strengthen the deep semantic features of human motion, and efficiently complete the human motion recognition task.

[0004] To achieve the above object, the present invention proposes an unsupervised human action recognition method based on information entropy guidance, comprising the following steps:

[0005] Obtain two-dimensional image information and depth information of the human body motion to be measured;

[0006] By converting the two-dimensional image information and the depth information, the human skeleton data to be identified is obtained;

[0007] Preprocessing the human skeleton data to be identified to obtain the time skeleton data to be identified and the space skeleton data to be identified;

[0008] Input the time skeleton data to be identified and the space skeleton data to be identified into the feature extraction network in sequence to obtain the time skeleton feature and the space skeleton feature;

[0009] The temporal skeleton features and spatial skeleton features are input into the linear classifier to obtain the action recognition result.

[0010] According to an achievable manner in an embodiment of the present application, before obtaining the two-dimensional image information and depth information of the human body motion to be measured, the method further includes:

[0011] Directly obtain the skeleton data in the preset data set for the model pre-training and fine-tuning process;

[0012] Preprocessing the skeleton data in the preset data set to obtain time skeleton data and space skeleton data;

[0013] An unsupervised contrastive learning algorithm is used to mine skeleton features of skeleton data in a preset data set, specifically including: using a time-based motion feature contrastive learning module to mine temporal skeleton features of temporal skeleton data, and using a space-based motion feature contrastive learning module to mine spatial skeleton features of spatial skeleton data, to complete a pre-training process, wherein the skeleton features include the temporal skeleton features and the spatial skeleton features;

[0014] The feature extractor is composed of a time-based motion feature contrast learning module encoder and a space-based motion feature contrast learning module encoder, and a linear classifier is added to the bottom layer of the feature extractor to complete the fine-tuning process.

[0015] According to an achievable method in an embodiment of the present application, preprocessing the skeleton data in the preset data set includes:

[0016] The skeleton data in the preset data set is filled using a bilinear interpolation method, and the skeleton data in the preset data set is adjusted to a fixed time length;

[0017] The adjusted skeleton data in the preset data set is transformed into time skeleton data Xt∈RT×V×C and space skeleton data Xj∈RV×T×C.

[0018] Where R represents the skeleton data in the preset data set, T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points in the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, Xt represents the time skeleton data, and Xj represents the space skeleton data.

[0019] According to an achievable method in an embodiment of the present application, a time-based motion feature comparison learning module is used to mine the time skeleton features of the time skeleton data, including:

[0020] The first information entropy of the time skeleton data is calculated based on the first information entropy evaluation algorithm, and the first information entropy is converted into a probability distribution to obtain a first reinforcement probability, wherein the first information entropy evaluation algorithm is:

[0021] ;

[0022] Where i represents the time frame, represents the temporal skeleton data of the i-th frame, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, represents the first information entropy of the i-th frame, m represents the skeleton point number, and n represents the corresponding three-dimensional coordinate channel;

[0023] The second information entropy of the time skeleton data is calculated based on the second information entropy evaluation algorithm, and the second information entropy is converted into a probability distribution to obtain a second reinforcement probability, wherein the second information entropy evaluation algorithm is:

[0024] ;

[0025] Where i represents the time frame, represents the temporal skeleton data of the i-th frame, represents the second information entropy of the i-th frame, and T represents the number of fixed time length frames after bilinear interpolation;

[0026] The third information entropy of the time skeleton data is calculated based on the third information entropy evaluation algorithm, and the third information entropy is converted into a probability distribution to obtain a third reinforcement probability, wherein the third information entropy evaluation algorithm is:

[0027] ;

[0028] Where i represents the time frame, represents the temporal skeleton data of the i-th frame, represents the third information entropy of the i-th frame, T represents the number of fixed time length frames after bilinear interpolation;

[0029] The first reinforcement probability, the second reinforcement probability, and the third reinforcement probability are concatenated to obtain time reinforcement information;

[0030] The temporal skeleton features of the temporal skeleton data are obtained by deep learning the temporal reinforcement information through a multi-layer convolutional network.

[0031] According to an achievable method in an embodiment of the present application, a space-based motion feature comparison learning module is used to mine the temporal skeleton features of the spatial skeleton data, including:

[0032] The fourth information entropy of the spatial skeleton data is calculated based on the fourth information entropy evaluation algorithm, and the fourth information entropy is converted into a probability distribution to obtain a fourth reinforcement probability, wherein the fourth information entropy evaluation algorithm is:

[0033] ;

[0034] Among them, i represents the skeleton point, represents the spatial skeleton data of the i-th skeleton point, C represents the number of three-dimensional coordinate channels, T represents the number of fixed time length frames after bilinear interpolation, represents the fourth information entropy of the i-th skeleton point, q represents the frame number, and n represents the corresponding three-dimensional coordinate channel;

[0035] The fifth information entropy of the spatial skeleton data is calculated based on the fifth information entropy evaluation algorithm, and the fifth information entropy is converted into a probability distribution to obtain a fifth reinforcement probability, wherein the fifth information entropy evaluation algorithm is:

[0036] ;

[0037] Among them, i represents the skeleton point, Represents the spatial skeleton data of the i-th skeleton point, represents the fifth information entropy of the i-th skeleton point, V represents the number of skeleton points, The spatial skeleton data representing the last skeleton point;

[0038] The sixth information entropy of the spatial skeleton data is calculated based on the sixth information entropy evaluation algorithm, and the sixth information entropy is converted into a probability distribution to obtain a sixth reinforcement probability, wherein the sixth information entropy evaluation algorithm is:

[0039] ;

[0040] Among them, i represents the skeleton point, Represents the spatial skeleton data of the i-th skeleton point, represents the sixth information entropy of the i-th frame, V represents the number of skeleton points, The spatial skeleton data representing the last skeleton point;

[0041] The fourth reinforcement probability, the fifth reinforcement probability, and the sixth reinforcement probability are concatenated to obtain spatial reinforcement information;

[0042] The spatial skeleton features of the spatial skeleton data are obtained by deep learning the spatial reinforcement information through a multi-layer convolutional network.

[0043] According to an achievable method in an embodiment of the present application, a multi-layer convolutional network includes a feed-forward neural network with a two-dimensional convolutional layer, a one-dimensional convolutional layer, a RELU layer, and a maximum pooling layer; the time reinforcement information is deeply learned through the multi-layer convolutional network to obtain the time skeleton features of the time skeleton data, including:

[0044] Through a feed-forward neural network with a two-dimensional convolutional layer, a one-dimensional convolutional layer, a RELU layer, and a maximum pooling layer, the time reinforcement information is semantically mined to obtain the time reinforcement direction information;

[0045] The time reinforcement direction information is normalized by regularization processing to obtain the time skeleton features of the time skeleton data;

[0046] Through the multi-layer convolutional network, the spatial reinforcement information is deeply learned to obtain the spatial skeleton features of the spatial skeleton data, including:

[0047] Through a feed-forward neural network with a two-dimensional convolutional layer, a one-dimensional convolutional layer, a RELU layer, and a maximum pooling layer, the spatial reinforcement information is semantically mined to obtain the spatial reinforcement direction information;

[0048] The spatial reinforcement direction information is normalized through regularization processing to obtain the spatial skeleton features of the spatial skeleton data.

[0049] According to an achievable method in the embodiment of the present application, the pre-training process specifically includes:

[0050] The softmax layer is used to process the temporal skeleton features and the spatial skeleton features to obtain the time-based prediction classification probability and the space-based prediction classification probability;

[0051] The loss function is defined as the mean square error function between the time-based predicted classification probability and the space-based predicted classification probability of the same action to complete the pre-training process of the feature extraction network.

[0052] Compared with the prior art, the present invention has the following beneficial effects: since the present invention introduces a variety of information entropy algorithms, the deep neural network can efficiently learn the deep semantic features of human body movements. By obtaining the information entropy under different conditions, the feature extraction network can be made directional, the perception depth of the neural network is expanded, the learning speed of the model is accelerated, the accuracy of action recognition is greatly improved, and the deep learning unsupervised algorithm can complete the recognition task more efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of an unsupervised human action recognition method based on information entropy guidance provided by the present invention;

[0054] Figure 2 It is a schematic diagram of the pre-training process and fine-tuning process of the deep learning unsupervised model provided by the present invention;

[0055] Figure 3 It is a schematic diagram of the process of mining time skeleton features by the time-based motion feature comparison learning module provided by the present invention;

[0056] Figure 4 It is a schematic diagram of the process of mining spatial skeleton features by the space-based motion feature comparison learning module provided by the present invention;

[0057] Figure 5 It is a schematic diagram of the structure of a multi-layer convolutional network provided by the present invention;

[0058] Figure 6 is a structural block diagram of an unsupervised human motion recognition device based on information entropy guidance in one embodiment;

[0059] Figure 7 FIG. 4 is a schematic structural diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0061] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0062] Figure 1 The flowchart of an unsupervised human action recognition method based on information entropy guidance provided by the present invention is shown, comprising:

[0063] S110, obtaining two-dimensional image information and depth information of the human body motion to be measured.

[0064] The human body motion to be measured contains a number of two-dimensional image information and depth information. The two-dimensional image information is obtained by the optical sensor, and the depth information is calculated by the ToF camera to measure the time difference of the round trip of light. The two-dimensional image information contains 1920×1080 pixel information, each pixel information contains the size information of the three RGB channels, and the depth information contains 1920×1080 pixel information, each pixel information contains the size information of the depth.

[0065] S120, obtaining human skeleton data to be identified by converting the two-dimensional image information and the depth information.

[0066] Using the trained Heatmap network, each pixel in the two-dimensional image information is assigned the probability of 25 human skeleton points, and the probability convergence is performed through the Max or Average function, and finally the two-dimensional position information of the 25 human skeleton points is obtained. By combining the depth information of the corresponding position, the three-dimensional position information is normalized. The human skeleton data to be identified contains a certain number of 25 human skeleton points, and each human skeleton point contains three-dimensional position information.

[0067] S130, preprocessing the human skeleton data to be identified to obtain time skeleton data to be identified and space skeleton data to be identified.

[0068] The preprocessing process of the human skeleton data to be identified includes data denoising, joint point alignment, bilinear interpolation and dimensional transformation, etc., to obtain the time skeleton data to be identified and the spatial skeleton data to be identified. The time skeleton data to be identified contains 25 human skeleton points in 64 frames, and each human skeleton point contains three-dimensional position information; the spatial skeleton data to be identified contains 64 frames of 25 human skeleton points, and each frame contains three-dimensional position information.

[0069] S140, inputting the time skeleton data to be identified and the space skeleton data to be identified into a feature extraction network in sequence to obtain a time skeleton feature and a space skeleton feature.

[0070] The feature extraction network includes a time-based motion feature contrast learning module, a space-based motion feature contrast learning module, and a multi-layer convolutional network. The time reinforcement information is calculated by three information entropy evaluation algorithms in the time-based motion feature contrast learning module, and the time skeleton feature is obtained by using the trained multi-layer convolutional network; the space reinforcement information is calculated by three information entropy evaluation algorithms in the space-based motion feature contrast learning module, and the space skeleton feature is obtained by using the trained multi-layer convolutional network.

[0071] S150, inputting the temporal skeleton features and the spatial skeleton features into a linear classifier to obtain an action recognition result.

[0072] The trained linear classifier is used to map the input temporal skeleton features to the first category score and the input spatial skeleton features to the second category score through linear mapping. The first category score and the second category score are added to get the total category score. Finally, the action recognition result is obtained through the specified category information.

[0073] Before S110, Figure 2 The schematic diagram of the pre-training process and fine-tuning process of the deep learning unsupervised model provided by the present invention is shown, including:

[0074] S210, preprocessing the skeleton data in the preset data set to obtain temporal skeleton data;

[0075] S220, preprocessing the skeleton data in the preset data set to obtain spatial skeleton data;

[0076] S230, using a time-based motion feature comparison learning module to mine the temporal skeleton features of the temporal skeleton data;

[0077] S240, mining spatial skeleton features of spatial skeleton data using a space-based motion feature comparison learning module;

[0078] S250, using the time-based motion feature contrast learning module encoder and the space-based motion feature contrast learning module encoder to form a feature extractor, and adding a linear classifier at the bottom of the feature extractor to complete the fine-tuning process.

[0079] In S210, it includes:

[0080] The skeleton data in the preset data set is filled using a bilinear interpolation method, and the skeleton data in the preset data set is adjusted to a fixed time length;

[0081] The skeleton data in the preset data set is transformed in dimension to obtain the temporal skeleton data Xt∈RT×V×C;

[0082] Where R represents the skeleton data in the preset data set, T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points in the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, and Xt represents the time skeleton data.

[0083] The preset dataset can be the NTU RGB + D dataset. The NTU RGB + D dataset is a human skeleton behavior recognition dataset proposed by the Rose Lab of Nanyang Technological University. The NTU RGB + D dataset consists of 56,880 action samples, including RGB video, depth map sequence, 3D skeleton data, and infrared video of each sample.

[0084] In S220, it includes:

[0085] The skeleton data in the preset data set is filled using a bilinear interpolation method, and the skeleton data in the preset data set is adjusted to a fixed time length;

[0086] The skeleton data in the preset data set is transformed into spatial skeleton data Xj∈RV×T×C.

[0087] Where R represents the skeleton data in the preset data set, T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points in the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, and Xj represents the spatial skeleton data.

[0088] In S230, Figure 3 The schematic diagram of the process of mining time skeleton features by the time-based motion feature comparison learning module provided by the present invention is shown, including:

[0089] The first information entropy of the time skeleton data is calculated based on the first information entropy evaluation algorithm, and the first information entropy is converted into a probability distribution to obtain a first reinforcement probability. The first information entropy evaluation algorithm is:

[0090] ;

[0091] Where i represents the time frame, represents the temporal skeleton data of the i-th frame, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, represents the first information entropy of the i-th frame, m represents the skeleton point number, and n represents the corresponding three-dimensional coordinate channel;

[0092] The second information entropy of the time skeleton data is calculated based on the second information entropy evaluation algorithm, and the second information entropy is converted into a probability distribution to obtain a second reinforcement probability. The second information entropy evaluation algorithm is:

[0093] ;

[0094] Where i represents the time frame, represents the temporal skeleton data of the i-th frame, represents the second information entropy of the i-th frame, and T represents the number of fixed time length frames after bilinear interpolation;

[0095] The third information entropy of the time skeleton data is calculated based on the third information entropy evaluation algorithm, and the third information entropy is converted into a probability distribution to obtain a third reinforcement probability. The third information entropy evaluation algorithm is:

[0096] ;

[0097] Where i represents the time frame, represents the temporal skeleton data of the i-th frame, represents the third information entropy of the i-th frame, T represents the number of fixed time length frames after bilinear interpolation;

[0098] The first reinforcement probability, the second reinforcement probability, and the third reinforcement probability are concatenated to obtain time reinforcement information;

[0099] The temporal skeleton features of the temporal skeleton data are obtained by deep learning the temporal reinforcement information through a multi-layer convolutional network.

[0100] In S240, Figure 4 The schematic diagram of the process of mining spatial skeleton features by the space-based motion feature comparison learning module provided by the present invention is shown, including:

[0101] The fourth information entropy of the spatial skeleton data is calculated based on the fourth information entropy evaluation algorithm, and the fourth information entropy is converted into a probability distribution to obtain a fourth reinforcement probability. The fourth information entropy evaluation algorithm is:

[0102] ;

[0103] Among them, i represents the skeleton point, represents the spatial skeleton data of the i-th skeleton point, C represents the number of three-dimensional coordinate channels, T represents the number of fixed time length frames after bilinear interpolation, represents the fourth information entropy of the i-th skeleton point, q represents the frame number, and n represents the corresponding three-dimensional coordinate channel;

[0104] The fifth information entropy of the spatial skeleton data is calculated based on the fifth information entropy evaluation algorithm, and the fifth information entropy is converted into a probability distribution to obtain a fifth reinforcement probability. The fifth information entropy evaluation algorithm is:

[0105] ;

[0106] Among them, i represents the skeleton point, Represents the spatial skeleton data of the i-th skeleton point, represents the fifth information entropy of the i-th skeleton point, V represents the number of skeleton points, The spatial skeleton data representing the last skeleton point;

[0107] The sixth information entropy of the spatial skeleton data is calculated based on the sixth information entropy evaluation algorithm, and the sixth information entropy is converted into a probability distribution to obtain a sixth enhancement probability. The sixth information entropy evaluation algorithm is:

[0108] ;

[0109] Among them, i represents the skeleton point, Represents the spatial skeleton data of the i-th skeleton point, represents the sixth information entropy of the i-th frame, V represents the number of skeleton points, The spatial skeleton data representing the last skeleton point;

[0110] The fourth reinforcement probability, the fifth reinforcement probability, and the sixth reinforcement probability are concatenated to obtain spatial reinforcement information;

[0111] The spatial skeleton features of the spatial skeleton data are obtained by deep learning the spatial reinforcement information through a multi-layer convolutional network.

[0112] Figure 5 The schematic diagram of the structure of the multi-layer convolutional network provided by the present invention is shown, including:

[0113] Through a feed-forward neural network with a two-dimensional convolutional layer, a one-dimensional convolutional layer, a RELU layer, and a maximum pooling layer, the temporal reinforcement information is semantically mined to obtain the temporal reinforcement direction information, and then the spatial reinforcement information is semantically mined to obtain the spatial reinforcement direction information;

[0114] The temporal reinforcement direction information is normalized by regularization processing to obtain the temporal skeleton feature, and then the spatial reinforcement direction information is normalized to obtain the spatial skeleton feature.

[0115] Among them, the two-dimensional convolution layer convolves the first reinforcement probability, the second reinforcement probability, and the third reinforcement probability, the one-dimensional convolution layer convolves the data, the RELU layer activates the convolution result, and finally the maximum value in the information entropy receptive field is obtained through the maximum pooling layer, and the normalization process is used to obtain the temporal skeleton feature; the two-dimensional convolution layer convolves the fourth reinforcement probability, the fifth reinforcement probability, and the sixth reinforcement probability, the one-dimensional convolution layer convolves the data, the RELU layer activates the convolution result, and finally the maximum value in the information entropy receptive field is obtained through the maximum pooling layer, and the normalization process is used to obtain the spatial skeleton feature.

[0116] Finally, the softmax layer is used to process the temporal skeleton features and the spatial skeleton features to obtain the time-based prediction classification probability and the space-based prediction classification probability;

[0117] The loss function is defined as the mean square error function between the time-based predicted classification probability and the space-based predicted classification probability of the same action to complete the pre-training process of the feature extraction network.

[0118] In addition, the algorithms mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned above, or in a different order from the above, or several steps can be performed simultaneously.

[0119] Figure 6 The following is a schematic diagram of the structure of an unsupervised human motion recognition device based on information entropy guidance provided in an embodiment of the present application. Figure 6As shown, the device may include: an acquisition unit 610, a conversion unit 620, a preprocessing unit 630, a feature extraction unit 640 and a recognition unit 650, and may further include: a training adjustment module and a mining module. The main functions of each component module are as follows:

[0120] An acquisition unit 610 is used to acquire two-dimensional image information and depth information of the human body motion to be measured;

[0121] A conversion unit 620, configured to obtain human skeleton data to be identified by converting the two-dimensional image information and the depth information;

[0122] A preprocessing unit 630 is used to preprocess the human skeleton data to be identified to obtain time skeleton data to be identified and space skeleton data to be identified;

[0123] A feature extraction unit 640 is used to input the time skeleton data to be identified and the space skeleton data to be identified into a feature extraction network in sequence to obtain a time skeleton feature and a space skeleton feature;

[0124] The recognition unit 650 is used to input the temporal skeleton features and the spatial skeleton features into a linear classifier to obtain an action recognition result.

[0125] As a feasible method, the device also includes a training adjustment module, which is used to directly obtain the skeleton data in the preset data set for the pre-training process and fine-tuning process of the model; pre-process the skeleton data in the preset data set to obtain temporal skeleton data and spatial skeleton data; use an unsupervised contrast learning algorithm to mine the skeleton features of the skeleton data in the preset data set, specifically including: using a time-based motion feature contrast learning module to mine the temporal skeleton features of the temporal skeleton data, and using a space-based motion feature contrast learning module to mine the spatial skeleton features of the spatial skeleton data to complete the pre-training process; using a time-based motion feature contrast learning module encoder and a space-based motion feature contrast learning module encoder to form a feature extractor, and adding a linear classifier at the bottom of the feature extractor to complete the fine-tuning process.

[0126] As an implementable method, the preprocessing unit 630 is also used to fill the skeleton data in the preset data set with a bilinear interpolation method, and adjust the skeleton data in the preset data set to a fixed time length; the skeleton data in the preset data set is dimensionally transformed to obtain time skeleton data Xt∈RT×V×C and spatial skeleton data Xj∈RV×T×C, respectively, where R represents the skeleton data in the preset data set, T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points of the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, Xt represents the time skeleton data, and Xj represents the spatial skeleton data.

[0127] As an implementable manner, the device further includes a mining module for calculating a first information entropy of the time skeleton data based on a first information entropy evaluation algorithm, and converting the first information entropy into a probability distribution to obtain a first reinforcement probability, wherein the first information entropy evaluation algorithm is:

[0128] ;

[0129] Where i represents the time frame, represents the temporal skeleton data of the i-th frame, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, represents the first information entropy of the i-th frame, m represents the skeleton point number, and n represents the corresponding three-dimensional coordinate channel;

[0130] The second information entropy of the time skeleton data is calculated based on the second information entropy evaluation algorithm, and the second information entropy is converted into a probability distribution to obtain a second reinforcement probability, wherein the second information entropy evaluation algorithm is:

[0131] ;

[0132] Where i represents the time frame, represents the temporal skeleton data of the i-th frame, represents the second information entropy of the i-th frame, and T represents the number of fixed time length frames after bilinear interpolation;

[0133] The third information entropy of the time skeleton data is calculated based on the third information entropy evaluation algorithm, and the third information entropy is converted into a probability distribution to obtain a third reinforcement probability, wherein the third information entropy evaluation algorithm is:

[0134] ;

[0135] Where i represents the time frame, represents the temporal skeleton data of the i-th frame, represents the third information entropy of the i-th frame, T represents the number of fixed time length frames after bilinear interpolation;

[0136] The first reinforcement probability, the second reinforcement probability, and the third reinforcement probability are concatenated to obtain time reinforcement information;

[0137] The temporal skeleton features of the temporal skeleton data are obtained by deep learning the temporal reinforcement information through a multi-layer convolutional network.

[0138] As an achievable method, the mining module is also used to mine the temporal skeleton features of the spatial skeleton data based on the spatial motion feature contrast learning module, including:

[0139] The fourth information entropy of the spatial skeleton data is calculated based on the fourth information entropy evaluation algorithm, and the fourth information entropy is converted into a probability distribution to obtain a fourth reinforcement probability, wherein the fourth information entropy evaluation algorithm is:

[0140] ;

[0141] Among them, i represents the skeleton point, represents the spatial skeleton data of the i-th skeleton point, C represents the number of three-dimensional coordinate channels, T represents the number of fixed time length frames after bilinear interpolation, represents the fourth information entropy of the i-th skeleton point, q represents the frame number, and n represents the corresponding three-dimensional coordinate channel;

[0142] The fifth information entropy of the spatial skeleton data is calculated based on the fifth information entropy evaluation algorithm, and the fifth information entropy is converted into a probability distribution to obtain a fifth reinforcement probability, wherein the fifth information entropy evaluation algorithm is:

[0143] ;

[0144] Among them, i represents the skeleton point, Represents the spatial skeleton data of the i-th skeleton point, represents the fifth information entropy of the i-th skeleton point, and V represents the number of skeleton points;

[0145] The sixth information entropy of the spatial skeleton data is calculated based on the sixth information entropy evaluation algorithm, and the sixth information entropy is converted into a probability distribution to obtain a sixth reinforcement probability, wherein the sixth information entropy evaluation algorithm is:

[0146] ;

[0147] Among them, i represents the skeleton point, Represents the spatial skeleton data of the i-th skeleton point, represents the sixth information entropy of the i-th frame, and V represents the number of skeleton points;

[0148] The fourth reinforcement probability, the fifth reinforcement probability, and the sixth reinforcement probability are concatenated to obtain spatial reinforcement information;

[0149] The spatial skeleton features of the spatial skeleton data are obtained by deep learning the spatial reinforcement information through a multi-layer convolutional network.

[0150] As an achievable approach, the training adjustment module is specifically used for

[0151] The time reinforcement information is semantically mined by the feedforward neural network with a two-dimensional convolution layer, a one-dimensional convolution layer, a RELU layer, and a maximum pooling layer to obtain the time reinforcement direction information;

[0152] Normalizing the temporal enhancement direction information by regularization processing to obtain the temporal skeleton feature of the temporal skeleton data;

[0153] The spatial enhancement information is semantically mined by the feedforward neural network with a two-dimensional convolutional layer, a one-dimensional convolutional layer, a RELU layer, and a maximum pooling layer to obtain spatial enhancement direction information;

[0154] The spatial enhancement direction information is normalized by regularization processing to obtain the spatial skeleton features of the spatial skeleton data.

[0155] As a feasible method, a training adjustment module is specifically used to process the temporal skeleton features and the spatial skeleton features using the softmax layer to obtain the time-based predicted classification probability and the space-based predicted classification probability; the loss function is defined as the mean square error function between the time-based predicted classification probability and the space-based predicted classification probability of the same action, thereby completing the pre-training process of the feature extraction network.

[0156] It is to be understood that the implementation of any method or product of the present application does not necessarily require all of the advantages described above to be achieved at the same time.

[0157] The same or similar parts between the above embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] According to an embodiment of the present application, the present application also provides a computer device and a computer-readable storage medium.

[0159] like Figure 7, is a block diagram of a computer device according to an embodiment of the present application. The computer device is intended to represent various forms of digital computers or mobile devices. The digital computer may include a desktop computer, a portable computer, a workbench, a personal digital assistant, a server, a mainframe computer, and other suitable computers. The mobile device may include a tablet computer, a smart phone, a wearable device, etc.

[0160] like Figure 7 As shown, the device 700 includes a computing unit 701, a ROM 702, a RAM 703, a bus 704, and an input / output (I / O) interface 705. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via the bus 704. The input / output (I / O) interface 705 is also connected to the bus 704.

[0161] The computing unit 701 may perform various processes in the method embodiments of the present application according to computer instructions stored in a read-only memory (ROM) 702 or computer instructions loaded from a storage unit 708 into a random access memory (RAM) 703. The computing unit 701 may be various general and / or special processing components with processing and computing capabilities. The computing unit 701 may include, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. In some embodiments, the method provided in the embodiments of the present application may be implemented as a computer software program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 708.

[0162] RAM 703 may also store various programs and data required for the operation of device 700. Part or all of the computer program may be loaded and / or installed on device 700 via ROM 802 and / or communication unit 709.

[0163] The input unit 706, output unit 707, storage unit 708 and communication unit 709 in the device 700 can be connected to the I / O interface 705. The input unit 706 can be, for example, a keyboard, a mouse, a touch screen, a microphone, etc. The output unit 707 can be, for example, a display, a speaker, an indicator light, etc. The device 700 can exchange information, data, etc. with other devices through the communication unit 709.

[0164] It should be noted that the device may also include other components necessary for normal operation, or may only include components necessary for implementing the solution of the present application, rather than all the components shown in the figure.

[0165] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations of them.

[0166] The computer instructions for implementing the method of the present application can be written in any combination of one or more programming languages. These computer instructions can be provided to the computing unit 701, so that when the computer instructions are executed by the computing unit 701 such as a processor, the steps involved in the method embodiment of the present application are executed.

[0167] The computer-readable storage medium provided in the present application may be a tangible medium that may contain or store computer instructions for executing the steps involved in the method embodiments of the present application. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, and other forms of storage media.

[0168] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

[0169] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. An unsupervised human action recognition method based on information entropy guidance, characterized in that: The method comprises: Obtain two-dimensional image information and depth information of the human body motion to be measured; By converting the two-dimensional image information and the depth information, human skeleton data to be identified is obtained; Preprocessing the human skeleton data to be identified to obtain time skeleton data to be identified and space skeleton data to be identified; Inputting the time skeleton data to be identified and the space skeleton data to be identified into a feature extraction network in sequence to obtain a time skeleton feature and a space skeleton feature; Inputting the temporal skeleton feature and the spatial skeleton feature into a linear classifier to obtain an action recognition result; The time skeleton data to be identified is input into a feature extraction network to obtain time skeleton features, including: A first information entropy evaluation algorithm, a second information entropy evaluation algorithm, and a third information entropy evaluation algorithm are respectively used to calculate a first information entropy, a second information entropy, and a third information entropy of the time skeleton data to be identified; The first information entropy, the second information entropy, and the third information entropy of the time skeleton data to be identified are spliced ​​according to the time dimension to obtain the time enhancement information to be identified; Performing deep learning on the time reinforcement information to be identified through a multi-layer convolutional network to obtain the time skeleton features of the time skeleton data to be identified; Wherein, the first information entropy evaluation algorithm is: ； Where i represents the time frame, represents the time skeleton data to be identified in the i-th frame, V represents the number of skeleton points of the time skeleton data to be identified, C represents the number of three-dimensional coordinate channels, represents the first information entropy of the time skeleton data to be identified in the i-th frame, m represents the skeleton point sequence number of the time skeleton data to be identified, and n represents the corresponding three-dimensional coordinate channel; The second information entropy evaluation algorithm is: ； Where i represents the time frame, represents the time skeleton data to be identified in the i-th frame, represents the second information entropy of the time skeleton data to be identified in the i-th frame, and T represents the number of frames of fixed time length after bilinear interpolation; The third information entropy evaluation algorithm is: ； Where i represents the time frame, represents the time skeleton data to be identified in the i-th frame, represents the third information entropy of the time skeleton data to be identified in the i-th frame, and T represents the number of frames of fixed time length after bilinear interpolation.

2. The unsupervised human action recognition method based on information entropy guidance according to claim 1 is characterized in that: Before obtaining the two-dimensional image information and depth information of the human body motion to be measured, the method further includes: Directly obtain the skeleton data in the preset data set for the model pre-training and fine-tuning process; Preprocessing the skeleton data in the preset data set to obtain temporal skeleton data and spatial skeleton data; An unsupervised contrastive learning algorithm is used to mine skeleton features of the skeleton data in the preset data set, specifically including: using a time-based motion feature contrastive learning module to mine the temporal skeleton features of the temporal skeleton data, and using a space-based motion feature contrastive learning module to mine the spatial skeleton features of the spatial skeleton data, to complete a pre-training process, wherein the skeleton features include the temporal skeleton features and the spatial skeleton features; A feature extractor is formed by using a time-based motion feature contrast learning module encoder and a space-based motion feature contrast learning module encoder, and a linear classifier is added to the bottom layer of the feature extractor to complete the fine-tuning process.

3. The unsupervised human action recognition method based on information entropy guidance according to claim 2 is characterized in that: The preprocessing of the skeleton data in the preset data set includes: Filling the skeleton data in the preset data set using a bilinear interpolation method, and adjusting the skeleton data in the preset data set to a fixed time length; Performing dimension transformation on the adjusted skeleton data in the preset data set to obtain the temporal skeleton data Xt∈RT×V×C and the spatial skeleton data Xj∈RV×T×C respectively; Wherein R represents the skeleton data in the preset data set, T represents the number of frames of fixed time length after bilinear interpolation, V represents the number of skeleton points of the skeleton data in the preset data set, C represents the x, y, z coordinates in the preset data set, Xt represents the time skeleton data, and Xj represents the space skeleton data.

4. The unsupervised human action recognition method based on information entropy guidance according to claim 2 is characterized in that: The method of mining the temporal skeleton features of the temporal skeleton data using a time-based motion feature contrast learning module includes: The first information entropy of the time skeleton data is calculated based on a first information entropy evaluation algorithm, and the first information entropy is converted into a probability distribution to obtain a first reinforcement probability, wherein the first information entropy evaluation algorithm is: ； Where i represents the time frame, represents the temporal skeleton data of the i-th frame, V represents the number of skeleton points, C represents the number of three-dimensional coordinate channels, represents the first information entropy of the i-th frame, m represents the skeleton point number, and n represents the corresponding three-dimensional coordinate channel; The second information entropy of the time skeleton data is calculated based on a second information entropy evaluation algorithm, and the second information entropy is converted into a probability distribution to obtain a second reinforcement probability, wherein the second information entropy evaluation algorithm is: ； Where i represents the time frame, represents the temporal skeleton data of the i-th frame, represents the second information entropy of the i-th frame, and T represents the number of fixed time length frames after bilinear interpolation; The third information entropy of the time skeleton data is calculated based on a third information entropy evaluation algorithm, and the third information entropy is converted into a probability distribution to obtain a third reinforcement probability, wherein the third information entropy evaluation algorithm is: ； Where i represents the time frame, represents the temporal skeleton data of the i-th frame, represents the third information entropy of the i-th frame, T represents the number of fixed time length frames after bilinear interpolation; splicing the first reinforcement probability, the second reinforcement probability, and the third reinforcement probability to obtain time reinforcement information; The temporal reinforcement information is deeply learned through a multi-layer convolutional network to obtain the temporal skeleton features of the temporal skeleton data.

5. The method for unsupervised human action recognition based on information entropy guidance according to claim 4, characterized in that: The method of mining the temporal skeleton features of the spatial skeleton data using a space-based motion feature contrast learning module includes: The fourth information entropy of the spatial skeleton data is calculated based on a fourth information entropy evaluation algorithm, and the fourth information entropy is converted into a probability distribution to obtain a fourth enhancement probability, wherein the fourth information entropy evaluation algorithm is: ； Among them, i represents the skeleton point, represents the spatial skeleton data of the i-th skeleton point, C represents the number of three-dimensional coordinate channels, T represents the number of fixed time length frames after bilinear interpolation, represents the fourth information entropy of the i-th skeleton point, q represents the frame number, and n represents the corresponding three-dimensional coordinate channel; The fifth information entropy of the spatial skeleton data is calculated based on a fifth information entropy evaluation algorithm, and the fifth information entropy is converted into a probability distribution to obtain a fifth reinforcement probability, wherein the fifth information entropy evaluation algorithm is: ； Among them, i represents the skeleton point, Represents the spatial skeleton data of the i-th skeleton point, represents the fifth information entropy of the i-th skeleton point, V represents the number of skeleton points, The spatial skeleton data representing the last skeleton point; The sixth information entropy of the spatial skeleton data is calculated based on the sixth information entropy evaluation algorithm, and the sixth information entropy is converted into a probability distribution to obtain a sixth enhancement probability, wherein the sixth information entropy evaluation algorithm is: ； Among them, i represents the skeleton point, Represents the spatial skeleton data of the i-th skeleton point, represents the sixth information entropy of the i-th frame, V represents the number of skeleton points, The spatial skeleton data representing the last skeleton point; splicing the fourth enhancement probability, the fifth enhancement probability, and the sixth enhancement probability to obtain spatial enhancement information; The spatial reinforcement information is deeply learned through a multi-layer convolutional network to obtain the spatial skeleton features of the spatial skeleton data.

6. The method for unsupervised human action recognition based on information entropy guidance according to claim 5, characterized in that: The multi-layer convolutional network includes a feed-forward neural network with a two-dimensional convolutional layer, a one-dimensional convolutional layer, a RELU layer, and a maximum pooling layer; The step of performing deep learning on the time enhancement information through a multi-layer convolutional network to obtain the time skeleton features of the time skeleton data includes: The time reinforcement information is semantically mined by the feedforward neural network with a two-dimensional convolution layer, a one-dimensional convolution layer, a RELU layer, and a maximum pooling layer to obtain the time reinforcement direction information; Normalizing the temporal enhancement direction information by regularization processing to obtain the temporal skeleton feature of the temporal skeleton data; The deep learning of the spatial reinforcement information by a multi-layer convolutional network to obtain the spatial skeleton features of the spatial skeleton data includes: The spatial enhancement information is semantically mined by the feedforward neural network with a two-dimensional convolutional layer, a one-dimensional convolutional layer, a RELU layer, and a maximum pooling layer to obtain spatial enhancement direction information; The spatial enhancement direction information is normalized by regularization processing to obtain the spatial skeleton features of the spatial skeleton data.

7. The unsupervised human action recognition method based on information entropy guidance according to claim 2 is characterized in that: The pre-training process specifically includes: Processing the temporal skeleton features and the spatial skeleton features using a softmax layer to obtain a time-based predicted classification probability and a space-based predicted classification probability; The loss function is defined as the mean square error function between the time-based predicted classification probability and the space-based predicted classification probability of the same action to complete the pre-training process of the feature extraction network.

8. An unsupervised human action recognition device based on information entropy guidance, characterized in that: The device comprises: An acquisition unit, used to acquire two-dimensional image information and depth information of the human body motion to be measured; A conversion unit, configured to obtain human skeleton data to be identified by converting the two-dimensional image information and the depth information; A preprocessing unit, used for preprocessing the human skeleton data to be identified to obtain time skeleton data to be identified and space skeleton data to be identified; A feature extraction unit is used to input the time skeleton data to be identified and the space skeleton data to be identified into a feature extraction network in sequence to obtain a time skeleton feature and a space skeleton feature, wherein the time skeleton data to be identified is input into the feature extraction network to obtain the time skeleton feature, including: A first information entropy evaluation algorithm, a second information entropy evaluation algorithm, and a third information entropy evaluation algorithm are respectively used to calculate a first information entropy, a second information entropy, and a third information entropy of the time skeleton data to be identified; The first information entropy, the second information entropy, and the third information entropy of the time skeleton data to be identified are spliced ​​according to the time dimension to obtain the time enhancement information to be identified; Performing deep learning on the time reinforcement information to be identified through a multi-layer convolutional network to obtain the time skeleton features of the time skeleton data to be identified; Wherein, the first information entropy evaluation algorithm is: ； Where i represents the time frame, represents the time skeleton data to be identified in the i-th frame, V represents the number of skeleton points of the time skeleton data to be identified, C represents the number of three-dimensional coordinate channels, represents the first information entropy of the time skeleton data to be identified in the i-th frame, m represents the skeleton point sequence number of the time skeleton data to be identified, and n represents the corresponding three-dimensional coordinate channel; The second information entropy evaluation algorithm is: ； Where i represents the time frame, represents the time skeleton data to be identified in the i-th frame, represents the second information entropy of the time skeleton data to be identified in the i-th frame, and T represents the number of frames of fixed time length after bilinear interpolation; The third information entropy evaluation algorithm is: ； Where i represents the time frame, represents the time skeleton data to be identified in the i-th frame, represents the third information entropy of the time skeleton data to be identified in the i-th frame, and T represents the number of frames of fixed time length after bilinear interpolation; The recognition unit is used to input the temporal skeleton features and the spatial skeleton features into a linear classifier to obtain an action recognition result.

9. A computer device, characterized in that: The device comprises: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the information entropy-guided unsupervised human motion recognition method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, which, when executed by a processor, implement the information entropy-guided unsupervised human motion recognition method as described in any one of claims 1 to 7.

Images