Human motion posture recognition method based on deep learning and intelligent wearable device

Abstract

The embodiment of the invention provides a human motion posture recognition method based on deep learning and intelligent wearable equipment, and relates to the technical field of image recognition. The method comprises the following steps: S1, obtaining an image of original motion data and carrying out gamma correction processing; s2, obtaining a feature vector after feature extraction; s3, carrying out nonlinear operation; s4, carrying out three-dimensional maximum pooling operation to obtain feature vectors after down-sampling; s5, repeating the three-dimensional maximum pooling operation for a plurality of times; s6, extracting and obtaining graphic features from the feature vectors after down-sampling; step S7, performing three-dimensional TopK image pooling operation to obtain a down-sampled feature matrix; step S8, repeating the three-dimensional TopK image pooling operation for a plurality of times; and S9, mapping the down-sampled feature matrix to a classification space to obtain the classification probability of the motion posture. The technical problems of large calculation amount and low recognition accuracy in human body posture recognition are solved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a human motion posture recognition method and a smart wearable device based on deep learning. Background Art

[0002] Human motion posture recognition is an important research direction in the field of computer vision and artificial intelligence, and has broad application prospects, such as health monitoring, motion analysis, virtual reality, and human-computer interaction. Traditional posture recognition methods mainly rely on image processing and machine learning techniques. However, these methods often show limitations such as large computational complexity and low recognition accuracy when faced with complex motion patterns and diverse environmental changes.

[0003] In recent years, with the development of deep learning technology, convolutional neural networks and graph neural networks have demonstrated powerful capabilities in feature extraction and pattern recognition, but a single network structure still has certain limitations when processing high-dimensional, spatiotemporally correlated motion data.

[0004] Therefore, the large amount of computation and low recognition accuracy in human posture recognition have become technical problems that need to be solved urgently. Summary of the invention

[0005] The present application provides a human motion posture recognition method and an intelligent wearable device based on deep learning to solve the technical problems of large computational complexity and low recognition accuracy in human posture recognition.

[0006] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0007] In a first aspect, the present invention provides a method for human motion posture recognition based on deep learning, comprising the following steps:

[0008] Step S1: obtaining an image of original motion data, and performing gamma correction on the image of the original motion data to obtain motion data after gamma correction processing;

[0009] Step S2: extracting features from the motion data after gamma correction to obtain a feature vector after feature extraction;

[0010] Step S3: performing a nonlinear operation on the feature vector after feature extraction to obtain a feature vector after nonlinear operation;

[0011] Step S4: performing a three-dimensional maximum pooling operation on the feature vector after the nonlinear operation to obtain a downsampled feature vector;

[0012] Step S5: Repeat the above three-dimensional maximum pooling operation several times to obtain the downsampled feature vector output by the three-dimensional maximum pooling operation;

[0013] Step S6: Based on the topological relationship of the graph neural network nodes, the graph features are extracted from the downsampled feature vectors;

[0014] Step S7: Perform a three-dimensional TopK graph pooling operation on the graph features to obtain a downsampled feature matrix;

[0015] Step S8: Repeat the above three-dimensional TopK graph pooling operation several times, and obtain the downsampled feature matrix output after the three-dimensional TopK graph pooling operation;

[0016] Step S9: Map the feature matrix downsampled in step S8 to the classification space to obtain the classification probability of the motion posture.

[0017] A further technical solution is that: in the step S1, gamma correction is performed on the image;

[0018]

[0019] Where X is the motion data after gamma correction; I in is the input image; γ is the gamma value, usually ranging from 0.5 to 2.5.

[0020] A further technical solution is that: in the step S2, the spatiotemporal features are extracted from the input three-dimensional motion data, and the local spatiotemporal features in the motion data are extracted by a convolution operation;

[0021] Y=X*W+b

[0022] Where: Y is the feature vector after feature extraction; X is the motion data after gamma correction; W is the weight of the convolution operation; b is the bias.

[0023] A further technical solution is that: in step S3, introducing an activation function can effectively alleviate the gradient disappearance and help the model converge faster;

[0024] Y′=max(0,x)

[0025] Wherein, Y′ is the feature vector after nonlinear operation of the feature vector Y after feature extraction, which means taking the maximum value between 0 and x; x is the input data, that is, the feature vector Y after feature extraction.

[0026] A further technical solution is that: in the step S4, a three-dimensional maximum pooling operation is introduced, and the three-dimensional maximum pooling operation process includes: dividing the input feature vector Y′ after the nonlinear operation into a plurality of rectangular areas, the rectangular areas are pooling windows or filters, and then selecting the maximum value in the sub-areas in each rectangular area as the output of the rectangular area;

[0027] Yt,h,w,d =max t′,h′,w′ (X t+t′,h+h′,w+w′,d )

[0028] Where Y t,h,w,d is the node feature matrix, that is, the feature vector after downsampling; X t+t′,h+h′,w+w′,d is the characteristic value of each local area; max() means that in each rectangular area, the largest element value is selected as the representative value of the area; max t′,h′,w′ (X t+t′,h+h′,w+w′,d ) represents the sum of the values ​​from each local region X t+t′,h+h′,w+w′,d Select the node with the largest eigenvalue from the matrix to form a new node feature matrix Y t,h,w,d .

[0029] A further technical solution is that: in the step S6, the topological relationship of the graph neural network nodes is obtained according to the human skeleton graph;

[0030] H′=σ(AHW)

[0031] In the formula, H′ is the graph feature; σ is the activation function LeakyReLU; A is the topological relationship of the graph neural network nodes, that is, the adjacency matrix; H is the feature vector after downsampling in step S5, that is, the node feature matrix; W is the graph convolution weight matrix.

[0032] A further technical solution is that the step S7 specifically includes the following steps:

[0033] Step S701: Calculate the importance score of the node;

[0034] s i =MLP(h i )

[0035] In the formula, MLP is a multi-layer perceptron scoring function, h i is the feature vector of the i-th node in the graph feature H′, s i is the importance score of each node;

[0036] Step S702: sort the importance scores from large to small and select the first k nodes;

[0037] Indices = TopK(s i ,k)

[0038] In the formula, Indices is the index of the top k nodes with the highest scores returned by the TopK function, that is, the top k nodes with the highest scores are selected to form a new node set; i is the importance score of each node; k is the node;

[0039] Step S703: reconstruct the selected nodes to form a new adjacency matrix A′ and a node feature matrix H″;

[0040] H″=H′[Indices]

[0041] A′=A[Indices,Indices]

[0042] In the formula, A′ is the new adjacency matrix; H″ is the downsampled feature matrix, i.e., the node feature matrix; Indices is the index of the top k nodes with the highest scores.

[0043] A further technical solution is: in step S9,

[0044] P = softmax(W f H″+b f )

[0045] Where P is the classification probability; W f is the weight matrix of the fully connected layer; b f is the bias for fully connected operation.

[0046] A further technical solution is that the neural network model structure used includes a gamma correction module, a three-dimensional convolutional layer feature extraction module, an activation function ReLU, a three-dimensional maximum pooling layer, a three-dimensional graph convolution layer, a three-dimensional TopK graph pooling layer and a fully connected layer connected in sequence; wherein,

[0047] The image of the obtained original motion data is input into the gamma correction module, the gamma correction module outputs the motion data after gamma correction to the three-dimensional convolutional layer feature extraction module, the three-dimensional convolutional layer feature extraction module outputs the feature vector after feature extraction to the activation function ReLU, the activation function ReLU outputs the feature vector after nonlinear operation to the three-dimensional maximum pooling layer, the three-dimensional maximum pooling layer outputs the downsampled feature vector to the three-dimensional graph convolution layer, the topological relationship of the graph neural network nodes is input into the three-dimensional graph convolution layer, the three-dimensional graph convolution layer outputs the graphic features to the three-dimensional TopK graph pooling layer, the three-dimensional TopK graph pooling layer outputs the downsampled feature matrix to the fully connected layer, and the fully connected layer outputs the classification probability of the motion posture.

[0048] In a second aspect, the present invention provides a smart wearable device, including a front-end sensor component, a wireless communication module, and a display module;

[0049] The sensor assembly includes: a nine-axis gyroscope, an acceleration sensor, an angular velocity sensor, a magnetometer and a depth camera, which are used to capture the motion data of the wearer's limbs;

[0050] The wireless communication module is used to transmit the captured motion data of the wearer's limbs to a back-end processing system; wherein the processor in the back-end processing system is used to execute the human motion posture recognition method based on deep learning as described in the first aspect, to perform high-precision recognition of the wearer's human motion posture, and transmit the motion posture recognition result back to the wireless communication module via a wireless network;

[0051] The display module is used to display the final wearer's motion posture recognition result.

[0052] The beneficial effects of adopting the above technical solution are:

[0053] The human motion posture recognition method based on deep learning of the present invention achieves high recognition accuracy in human posture recognition through gamma correction, etc., and reduces the amount of calculation in human posture recognition through nonlinear operation, etc.; achieves high recognition accuracy in human posture recognition through gamma correction module, etc., and reduces the amount of calculation in human posture recognition through activation function ReLU, etc. By combining a hybrid model of a three-dimensional convolutional neural network and a three-dimensional graph neural network, the spatiotemporal features in the motion video and the graph structure features of the human motion skeleton are effectively extracted, and high-precision recognition and real-time feedback of human motion posture are achieved, and the accuracy and robustness of recognition are effectively improved by using gamma correction technology. During the recognition process, the wearer's motion posture can be analyzed in real time, and accurate feedback can be given, such as posture correction suggestions, exercise effect evaluation, emotion detection, fall warning, and atrial fibrillation monitoring.

[0054] Please refer to the detailed description of the specific implementation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a method for recognizing human motion postures based on deep learning according to an embodiment of the present invention;

[0056] Figure 2 is a schematic diagram of the structure of a neural network model according to an embodiment of the present invention;

[0057] Figure 3 yes Figure 1 Schematic diagram of the process of step S7;

[0058] Figure 4 Schematic diagram of the structure of a smart wearable device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0061] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0062] First of all, the front-end data of this application uses a smart wearable device that integrates a variety of non-invasive sensors to capture the motion data of the limbs, such as accelerometers, gyroscopes, depth sensors, and inertial measurement units IMU, so as to collect the wearer's motion data in real time; these data are transmitted to the back-end processing system through the wireless communication module; the back-end processing system uses deep learning technology to perform high-precision recognition of human motion postures. Specifically, the system first uses the pre-processing module to crop, fill, and normalize the received raw motion data to obtain standardized data. Then, the standardized data is feature extracted and classified using the posture recognition network model based on deep learning, and finally the wearer's motion posture recognition results are output.

[0063] The back-end processing system uses an improved three-dimensional convolutional graph neural network 3DCNN-GNN to identify motion postures based on human skeletal movements. The three-dimensional convolutional neural network 3DCNN is used to extract spatiotemporal features from motion data, and the graph neural network GNN is used to model these features to better understand the motion relationship between various parts of the human body. In addition, the present invention also uses gamma correction technology to effectively improve the accuracy of target recognition. The posture recognition network model is trained and optimized through a large amount of training data with posture labels, and has strong generalization ability and robustness. During the recognition process, the wearer's motion posture can be analyzed in real time, and accurate feedback can be given, such as posture correction suggestions, exercise effect evaluation, emotion detection, fall warning and atrial fibrillation monitoring.

[0064] This application effectively extracts the spatiotemporal features and graph structural features of human motion skeletons in motion videos by combining a hybrid model of a three-dimensional convolutional neural network and a three-dimensional graph neural network, achieves high-precision recognition and real-time feedback of human motion postures, and uses gamma correction technology to effectively improve the accuracy and robustness of recognition.

[0065] Embodiment 1:

[0066] like Figure 1 to Figure 3 As shown, the present invention discloses a method for human motion posture recognition based on deep learning, and the neural network model structure used in the recognition process is described in detail as follows:

[0067] like Figure 2 As shown, the neural network model structure for recognizing human motion posture is an improved three-dimensional convolutional graph neural network 3DCNN-GNN structure, which includes a gamma correction module, a three-dimensional convolutional layer feature extraction module, an activation function ReLU, a three-dimensional maximum pooling layer, a three-dimensional graph convolution layer, a three-dimensional TopK graph pooling layer and a fully connected layer connected in sequence. The image of the obtained original motion data is input into the gamma correction module, the gamma correction module outputs the motion data processed by gamma correction to the three-dimensional convolutional layer feature extraction module, the three-dimensional convolutional layer feature extraction module outputs the feature vector after feature extraction to the activation function ReLU, the activation function ReLU outputs the feature vector after nonlinear operation to the three-dimensional maximum pooling layer, the three-dimensional maximum pooling layer outputs the downsampled feature vector to the three-dimensional graph convolution layer, the topological relationship of the graph neural network nodes is input into the three-dimensional graph convolution layer, the three-dimensional graph convolution layer outputs the graphic features to the three-dimensional TopK graph pooling layer, the three-dimensional TopK graph pooling layer outputs the downsampled feature matrix to the fully connected layer, and the fully connected layer outputs the classification probability of the motion posture.

[0068] Specifically, Figure 1 As shown, the present invention discloses a method for identifying human body motion posture, comprising the following steps:

[0069] Step S1: obtaining an image of original motion data, and performing gamma correction on the image of original motion data to obtain motion data after gamma correction.

[0070] In step S1, when the gamma correction module obtains the image of the original motion data, the gamma correction is performed on the image of the original motion data to obtain the motion data after the gamma correction processing.

[0071] Perform gamma correction on the image, adjust the brightness of the image, enhance the contrast of the image, and improve the visual quality.

[0072] Input: input motion data, raw sensor data I in , the size is T×H×W×C.

[0073] T: number of time steps.

[0074] H: Feature height.

[0075] W: characteristic brightness.

[0076] C: The number of channels is the number of sensors.

[0077] formula:

[0078] I in : input image.

[0079] γ: Gamma value, usually between 0.5 and 2.5.

[0080] Function: Adjust image brightness and contrast to improve visual quality.

[0081] Output: motion data X after gamma correction.

[0082] Step S2: extracting features from the motion data after gamma correction to obtain a feature vector after feature extraction.

[0083] In step S2, when the three-dimensional convolutional layer feature extraction module obtains the motion data after the gamma correction processing, features are extracted from the motion data after the gamma correction processing to obtain a feature vector after feature extraction.

[0084] The 3D convolutional layer feature extraction is used to extract spatiotemporal features from the input 3D motion data. The local spatiotemporal features in the motion data are extracted through the convolution operation. The size and number of convolution kernels directly affect the effect of feature extraction.

[0085] Input: Gamma-corrected motion data cube X of size T×H×W×C.

[0086] Output: Feature vector Y after feature extraction, size is T′×H′×W′×D1 . Value range: normalized to [0,1]. The formula is:

[0087] Y=X*W+b

[0088] Where W: convolution kernel, the weight of the convolution operation, used to extract features from the input data. The convolution kernel determines which features are extracted, and the appropriate weight can extract key motion features. The weights are continuously adjusted through training to capture important patterns in the data. Value range: During initialization, random sampling is usually performed from a Gaussian distribution or a uniform distribution. The value range varies depending on the initialization strategy. The Gaussian distribution is, for example, N(0,0.01).

[0089] b: Bias, the bias of the convolution operation, used to adjust the output value. The bias value is used to adjust the convolution output to help the model fit the data better. Value range: Usually set to 0 during initialization, or randomly sampled from a uniform distribution, with a smaller value range.

[0090] T′, H′, W′: Dimensions calculated by stride and padding.

[0091] D 1 : Number of output channels.

[0092] Step S3: Perform nonlinear operation on the feature vector after feature extraction to obtain the feature vector after nonlinear operation.

[0093] In step S3, when the activation function ReLU obtains the feature vector after feature extraction, a nonlinear operation is performed to obtain the feature vector after the nonlinear operation.

[0094] The activation function ReLU is introduced to effectively alleviate the gradient vanishing problem and help the model converge faster. The introduction of nonlinearity enables the model to learn complex features.

[0095] Input: Feature vector Y after feature extraction, size is T′×H′×W′×D 1 .

[0096] Output: Feature vector Y′ after nonlinear operation, with the same size as input, T′×H′×W′×D 1 . max(0,x), that is, non-negative.

[0097] Y′=max(0,x)

[0098] Wherein, Y′ is the feature vector after nonlinear operation of the feature vector Y after feature extraction, which means taking the maximum value between 0 and x; x is the input data, that is, the feature vector Y after feature extraction.

[0099] Step S4: Perform a three-dimensional maximum pooling operation on the feature vector after the nonlinear operation to obtain a downsampled feature vector.

[0100] In step S4, when the three-dimensional maximum pooling layer obtains the feature vector after the nonlinear operation, the three-dimensional maximum pooling operation is performed to obtain the feature vector after downsampling.

[0101] The three-dimensional maximum pooling operation is introduced. The pooling operation is used for downsampling. Its operation process can be summarized as follows: the input matrix or feature map is divided into several rectangular areas, which are also called pooling windows or filters. Then, the maximum value in the sub-area of ​​each rectangular area is selected as the output of the rectangular area, thereby reducing the size of the feature map, retaining the main features, and reducing the amount of calculation.

[0102] Input: Feature vector Y′ after nonlinear operation, size is T′×H′×W′×D 1 .

[0103] Output: The downsampled feature vector of size T″×H″×W″×D 1 .

[0104] Y t,h,w,d =max t′,h′,w′ (X t+t′,h+h′,w+w′,d )

[0105] Where Y t,h,w,d is the node feature matrix, that is, the feature vector after downsampling; X t+t′,h+h′,w+w′,d is the characteristic value of each local area; max() means that in each rectangular area, the largest element value is selected as the representative value of the area; max t′,h′,w′ (X t+t′,h+h′,w+w′,d ) represents the sum of the values ​​from each local region X t+t′,h+h′,w+w′,d Select the node with the largest eigenvalue from the matrix to form a new node feature matrix Y t,h,w,d .

[0106] T″, H″, W″ are the sizes calculated by the pooling window and step size. The pooling window and step size control the range and step size of the pooling operation and determine the degree of downsampling. Value range: The common pooling window size is 2×2×2 and the step size is 2.

[0107] Step S5: Repeat the above three-dimensional maximum pooling operation several times until a smaller feature tensor is obtained, corresponding to the downsampled feature vector output by the three-dimensional maximum pooling operation.

[0108] Different experimental samples have different dimensions and parameter thresholds. The specific parameter adjustment depends on the actual experimental situation, corresponding to the downsampled feature vector output by the three-dimensional maximum pooling operation. Taking the input feature tensor size of 8×8×8 as an example, the pooling window size is 2×2×2, the step size is 2, and the feature tensor size changes after one pooling operation are as follows:

[0109] Number of operations Feature tensor size Initial Input 8×8×8 First pooling 4×4×4

[0110] Step S6: Based on the topological relationship of the graph neural network nodes, the graph features are extracted from the downsampled feature vectors.

[0111] In step S6, when the three-dimensional graph convolution layer obtains the downsampled feature vector, the graph features are extracted based on the topological relationship of the graph neural network nodes.

[0112] The topological relationship of the graph neural network nodes is constructed according to the human skeleton graph, and the graphic features are extracted through the three-dimensional graph convolution layer.

[0113] Input: downsampled feature vector Topological relationship between nodes of graph neural network

[0114] T″: The number of nodes, i.e. the number of joints.

[0115] D″: characteristic number.

[0116] H″: Feature height.

[0117] W″: characteristic brightness.

[0118] H: The downsampled feature vector is the node feature matrix.

[0119] A: The topological relationship of the nodes in the graph neural network is the adjacency matrix, which represents the connection relationship of the human skeleton graph, that is, the adjacency matrix of each node and edge. The nodes are joints and the edges are bones. The adjacency matrix defines the graph structure and affects the way features are propagated in the graph. Value range: Determined according to the bone structure of the human skeleton graph, the value is 0 or 1, 0 means no connection, and 1 means connection.

[0120] Output: Graphical features

[0121] H′=σ(AHW)

[0122] In the formula, H′ is the graph feature; σ is the activation function LeakyReLU; A is the adjacency matrix; H is the node feature matrix; W is the graph convolution weight matrix.

[0123] W: The graph convolution weight matrix is ​​used to learn the transfer relationship between node features. The learning effect of weights directly affects the accuracy of posture recognition. Value range: randomly sampled from Gaussian distribution during initialization.

[0124] σ: Introduce the activation function LeakyReLU, which is an improved version of the ReLU function. When the input is negative, the output is no longer zero, but has a small slope, such as 0.01, which helps to avoid the problem of neuron death and increases the robustness of the neural network. The computational complexity is similar to ReLU. The activation function can effectively alleviate the gradient vanishing problem and help the model converge faster. Introducing nonlinearity enables the model to learn complex features. Value range: The output is max(0,x), that is, non-negative.

[0125] H”′: Output feature number.

[0126] Step S7: Perform a three-dimensional TopK graph pooling operation on the graph features to obtain a downsampled feature matrix.

[0127] Step S7: 3D TopK graph pooling layer.

[0128] When the three-dimensional TopK graph pooling layer obtains the graph features, a downsampling operation is performed to obtain the downsampled feature matrix.

[0129] The three-dimensional TopK graph pooling layer performs downsampling operations on the graph structure, reduces the dimension and aggregates the node features, and aggregates the high-dimensional node features into a lower-dimensional representation, thereby retaining the main features, reducing the amount of calculation and improving the generalization ability of the model.

[0130] Input: Graphics features

[0131] Output: downsampled feature matrix

[0132] T”′: the number of nodes after pooling.

[0133] D”′: feature dimension after pooling.

[0134] H”′: feature height after pooling.

[0135] W”′: Feature brightness after pooling.

[0136] like Figure 3 As shown, step S7 specifically includes the following steps:

[0137] Step S701: Calculate the importance score of the node.

[0138] s i =MLP(h i )

[0139] In the formula, MLP is a multi-layer perceptron scoring function, h i is the feature vector of the i-th node in the graph feature H′, s i is the importance score of each node.

[0140] Step S702: Sort the importance scores from large to small and select the first k nodes.

[0141] Indices = TopK(s i ,k)

[0142] In the formula, Indices is the index of the top k nodes with the highest scores returned by the TopK function, that is, the top k nodes with the highest scores are selected to form a new node set; i is the importance score of each node; k is the node.

[0143] Step S703: Reconstruct the selected nodes to form a new adjacency matrix A′ and a node feature matrix H″.

[0144] H″=H′[Indices]

[0145] A′=A[Indices,Indices]

[0146] Where A′ is the new adjacency matrix; H″ is the node feature matrix; Indices is the index of the top k nodes with the highest scores.

[0147] Step S8: Repeat the above three-dimensional TopK graph pooling operation several times, the same as step S5, until a smaller feature representation is obtained, corresponding to the downsampled feature matrix output after the three-dimensional TopK graph pooling operation.

[0148] Step S9: Map the feature matrix downsampled in step S8 to the classification space to obtain the classification probability of the motion posture.

[0149] In step S9, when the fully connected layer obtains the feature matrix H″ after downsampling in step S8, it is mapped to the classification space to obtain the classification probability P of the motion posture.

[0150] Mapping high-dimensional features to the classification space determines the final posture classification result.

[0151] Input: Output of graph convolutional layer.

[0152] Output: Classification probability P of the motion posture.

[0153] Formula: P = softmax(W f H″+b f )

[0154] Where P is the classification probability; W f is the weight matrix of the fully connected layer; b f is the bias for fully connected operation.

[0155] W f : The weight matrix of the fully connected layer, which is used to map high-dimensional features to the weights of the output space. The weights determine how the features are mapped to the classification results. Appropriate weights can improve classification accuracy. Value range: Randomly sampled from a Gaussian distribution during initialization.

[0156] b f : Bias of the fully connected operation. Bias is used to adjust the output of the fully connected layer to help the model fit the data better. Value range: Set to 0 during initialization.

[0157] The above-mentioned embodiments of the present application effectively extract the spatiotemporal features and graph structural features of the human motion skeleton in motion videos by combining a hybrid model of a three-dimensional convolutional neural network and a three-dimensional graph neural network, thereby achieving high-precision recognition and real-time feedback of human motion postures, and using gamma correction technology to effectively improve the accuracy and robustness of recognition.

[0158] Example 2

[0159] like Figure 4 As shown, the present invention discloses a smart wearable device 200, including a front-end sensor component 210, a wireless communication module 220, and a display module 230;

[0160] The sensor assembly 210 includes: a nine-axis gyroscope, an acceleration sensor, an angular velocity sensor, a magnetometer, and a depth camera, which are used to capture the motion data of the wearer's limbs;

[0161] A wireless communication module 220, used to transmit the captured motion data of the wearer's limbs to a back-end processing system; wherein the processor in the back-end processing system is used to execute the human motion posture recognition method based on deep learning as in Example 1, for high-precision recognition of the wearer's human motion posture, and transmit the motion posture recognition result back to the wireless communication module 220 via a wireless network;

[0162] The display module 230 is used to display the final wearer's motion posture recognition results. The displayed information may include the current motion posture and accurate feedback on the motion posture information, such as posture correction suggestions, exercise effect evaluation, emotion detection, fall warning and atrial fibrillation monitoring, etc.

[0163] According to this embodiment, the smart wearable device 200 can analyze the wearer's motion posture in real time and give accurate feedback, such as posture correction suggestions, exercise effect evaluation, emotion detection, fall warning and atrial fibrillation monitoring, etc. By combining the hybrid model of the three-dimensional convolutional neural network and the three-dimensional graph neural network, the spatiotemporal features in the motion video and the graph structure features of the human motion skeleton are effectively extracted, and high-precision recognition and real-time feedback of the human motion posture are achieved, and the gamma correction technology is used to effectively improve the accuracy and robustness of the recognition.

[0164] The specific embodiments are as follows:

[0165] The fairness and repeatability of experimental results can be guaranteed by standardized experimental environment and unified experimental parameter settings. These detailed configuration and parameter descriptions can help other researchers repeat the experiments under the same or similar conditions and verify the results.

[0166] As shown in Table 1, it is the table of the software and hardware experimental environment used by this experimental algorithm, as shown in Table 2, it is the parameter setting of the algorithm, and as shown in Table 3, it is the comparative experimental results.

[0167] Table 1: Experimental environment table

[0168]

[0169]

[0170] Table 2: Experimental parameters

[0171]

[0172]

[0173] Table 3: Comparative experimental results

[0174]

[0175] From the above results, it can be seen that the improved three-dimensional convolutional graph neural network 3DCNN-GNN performs better than the graph convolutional neural network in all evaluation criteria. It can more effectively extract spatiotemporal features and improve the accuracy, recall rate and overall performance of motion posture recognition.

[0176] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0177] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for human motion posture recognition based on deep learning, characterized in that: The steps include: Step S1: obtaining an image of original motion data, and performing gamma correction on the image of the original motion data to obtain motion data after gamma correction processing; Step S2: extracting features from the motion data after gamma correction to obtain a feature vector after feature extraction; Step S3: performing a nonlinear operation on the feature vector after feature extraction to obtain a feature vector after nonlinear operation; Step S4: performing a three-dimensional maximum pooling operation on the feature vector after the nonlinear operation to obtain a downsampled feature vector; Step S5: Repeat the above three-dimensional maximum pooling operation several times to obtain the downsampled feature vector output by the three-dimensional maximum pooling operation; Step S6: Based on the topological relationship of the graph neural network nodes, the graph features are extracted from the downsampled feature vectors; Step S7: Perform a three-dimensional TopK graph pooling operation on the graph features to obtain a downsampled feature matrix; Step S8: Repeat the above three-dimensional TopK graph pooling operation several times, and obtain the downsampled feature matrix output after the three-dimensional TopK graph pooling operation; Step S9: Map the feature matrix downsampled in step S8 to the classification space to obtain the classification probability of the motion posture.

2. The method for human motion posture recognition based on deep learning according to claim 1, characterized in that: In the step S1, gamma correction is performed on the image to enhance the contrast of the image; Where X is the motion data after gamma correction; I in is the input image; γ is the gamma value, ranging from 0.5 to 2.

5.

3. The method for human motion posture recognition based on deep learning according to claim 2, characterized in that: In the step S2, spatiotemporal features are extracted from the input three-dimensional motion data, and local spatiotemporal features in the motion data are extracted by convolution operation; Y=X*W+b Where: Y is the feature vector after feature extraction; X is the motion data after gamma correction; W is the weight of the convolution operation; b is the bias.

4. The method for human motion posture recognition based on deep learning according to claim 3, characterized in that: In step S3, introducing the activation function can effectively alleviate the gradient disappearance and help the model converge faster; Y′=max(0,x) Wherein, Y′ is the feature vector after nonlinear operation of the feature vector Y after feature extraction, which means taking the maximum value between 0 and x; x is the input data, that is, the feature vector Y after feature extraction.

5. The method for human motion posture recognition based on deep learning according to claim 4, characterized in that: In step S4, a three-dimensional maximum pooling operation is introduced, and the three-dimensional maximum pooling operation process includes: dividing the input feature vector Y′ after the nonlinear operation into a plurality of rectangular areas, where the rectangular areas are pooling windows or filters, and then selecting the maximum value in the sub-areas within each rectangular area as the output of the rectangular area; AND t,h,w,d =max t′,h′,w′ (X t+t′,h+h′,w+w′,d ) Where Y t,h,w,d is the node feature matrix, that is, the feature vector after downsampling; X t+t′,h+h′,w+w′,d is the characteristic value of each local area; max() means that in each rectangular area, the largest element value is selected as the representative value of the area; max t′,h′,w′ (X t+t′,h+h′,w+w′,d ) represents the sum of the values ​​from each local region X t+t′,h+h′,w+w′,d Select the node with the largest eigenvalue from the matrix to form a new node feature matrix Y t,h,w,d .

6. The method for human motion posture recognition based on deep learning according to claim 5, characterized in that: In the step S6, the topological relationship of the graph neural network nodes is obtained according to the human skeleton graph; H′=σ(AHW) In the formula, H′ is the graph feature; σ is the activation function LeakyReLU; A is the topological relationship of the graph neural network nodes, that is, the adjacency matrix; H is the feature vector after downsampling in step S5, that is, the node feature matrix; W is the graph convolution weight matrix.

7. The method for human motion posture recognition based on deep learning according to claim 6, characterized in that: The step S7 specifically includes the following steps: Step S701: Calculate the importance score of the node; s i =MLP(h i ) In the formula, MLP is a multi-layer perceptron scoring function, h i is the feature vector of the i-th node in the graph feature H′, s i is the importance score of each node; Step S702: sort the importance scores from large to small and select the first k nodes; Indices=TopK(s i ,k) In the formula, Indices is the index of the top k nodes with the highest scores returned by the TopK function, that is, the top k nodes with the highest scores are selected to form a new node set; i is the importance score of each node; k is the node; Step S703: reconstruct the selected nodes to form a new adjacency matrix A′ and a node feature matrix H″; H″=H′[Indices] A′=A[Indices,Indices] In the formula, A′ is the new adjacency matrix; H″ is the downsampled feature matrix, i.e., the node feature matrix; Indices is the index of the top k nodes with the highest scores.

8. The method for human motion posture recognition based on deep learning according to claim 7, characterized in that: In the step S9, P=softmax(W f H″+b f ) Where P is the classification probability; W f is the weight matrix of the fully connected layer; b f is the bias for fully connected operation.

9. The method for human motion posture recognition based on deep learning according to claim 8, characterized in that: The neural network model structure used includes a gamma correction module, a three-dimensional convolutional layer feature extraction module, an activation function ReLU, a three-dimensional maximum pooling layer, a three-dimensional graph convolution layer, a three-dimensional TopK graph pooling layer and a fully connected layer connected in sequence; among them, The image of the obtained original motion data is input into the gamma correction module, the gamma correction module outputs the motion data after gamma correction to the three-dimensional convolutional layer feature extraction module, the three-dimensional convolutional layer feature extraction module outputs the feature vector after feature extraction to the activation function ReLU, the activation function ReLU outputs the feature vector after nonlinear operation to the three-dimensional maximum pooling layer, the three-dimensional maximum pooling layer outputs the downsampled feature vector to the three-dimensional graph convolution layer, the topological relationship of the graph neural network nodes is input into the three-dimensional graph convolution layer, the three-dimensional graph convolution layer outputs the graphic features to the three-dimensional TopK graph pooling layer, the three-dimensional TopK graph pooling layer outputs the downsampled feature matrix to the fully connected layer, and the fully connected layer outputs the classification probability of the motion posture.

10. A smart wearable device, characterized in that: Including front-end sensor components, wireless communication modules, and display modules; The sensor assembly includes: a nine-axis gyroscope, an acceleration sensor, an angular velocity sensor, a magnetometer, and a depth camera, which are used to capture the motion data of the wearer's limbs; The wireless communication module is used to transmit the captured motion data of the wearer's limbs to a back-end processing system; wherein the processor in the back-end processing system is used to execute the human motion posture recognition method based on deep learning according to any one of claims 1 to 9, for high-precision recognition of the wearer's human motion posture, and transmit the motion posture recognition result back to the wireless communication module through a wireless network; The display module is used to display the final wearer's motion posture recognition result.

Images