Three-dimensional efficient human body sitting posture recognition method based on deep learning

Abstract

The invention is suitable for the technical field of sitting posture recognition, and provides a three-dimensional efficient human body sitting posture recognition method based on deep learning, which comprises the following steps of: recording video data, performing 3d posture estimation on images in a video by using blazeose, and extracting three-dimensional key point coordinates; carrying out data enhancement on the obtained key point coordinates through random jitter and mirror image overturning, and expanding a data set; a KpointNet model based on a point cloud model is constructed; the KpointNet model is trained, and a training model is generated; and finally, the trained model is tested and applied to practice. According to the invention, richer spatial information can be obtained from the RGB image, and the information dimension is improved; the KpointNet model corrected based on the point cloud model improves the reasoning utilization rate of the network model, has better accuracy, directly learns three-dimensional key point data, has more efficient data utilization rate, and can three-dimensionally and efficiently identify the human body sitting posture category, timely correct bad sitting postures and prevent potential health risks.

Description

Technical Field

[0001] The present invention belongs to the technical field of sitting posture recognition, and in particular relates to a three-dimensional efficient human sitting posture recognition method based on deep learning. Background Art

[0002] Incorrect sitting posture is closely related to related diseases. Timely identification and correction of incorrect sitting posture can help people improve their physical health to a certain extent. In addition, identifying incorrect sitting posture can also provide reference for research fields such as motion capture, behavior recognition, and motion prediction.

[0003] Human sitting posture recognition mainly studies how to describe human posture and identify posture categories. There are two common methods: methods based on wearable devices or pressure sensors, and methods based on computer vision. The use of wearable devices or pressure sensors to collect human posture information is subject to large equipment restrictions and has limited application prospects, while computer vision methods for RGB images have broader application prospects. In the sitting posture recognition methods for RGB images, most of them use posture estimation methods to obtain human key points, and then further draw skeleton diagrams, which are then converted into classification problems. The process of making skeleton diagrams is relatively slow, and it creates a lot of redundant information, which slows down the reasoning of the model and is not efficient enough; in addition, the point cloud model is a systematic method for processing point cloud data. Point cloud data is similar to key point data. The difference is that point cloud data is disordered and key point data is ordered. Conventional point cloud models make compromises when processing the disorder of point cloud data, and there will be a waste of resources when applied to key point data; in addition, most traditional methods use 2D posture estimation methods and make planar skeleton diagrams for recognition, which loses three-dimensional depth information, and the recognition ability is not strong enough, and the model does not have sufficient three-dimensional recognition ability. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a three-dimensional efficient human sitting posture recognition method based on deep learning, aiming to solve the problems raised in the above background technology.

[0005] The embodiment of the present invention is implemented as follows: a three-dimensional efficient human sitting posture recognition method based on deep learning, comprising the following steps:

[0006] Step A: Under a certain number of observation angles, the camera position is kept unchanged, and videos of people maintaining a specific sitting posture are recorded respectively. During the entire recording period, the sitting posture type remains unchanged, and the body makes appropriate movements. Then, a frame extraction operation is performed on the video data to adjust the image to a uniform size.

[0007] Step B: Use the Blazepose model to perform 3D pose estimation on the image data, collect the coordinates of the three-dimensional key points, use random jitter and mirror flipping to enhance the key point data, and then normalize the data to obtain the final 3D key point data. Divide the data set of each perspective into three parts, namely, training set, validation set and test set;

[0008] Step C: construct a KpointNet model based on point cloud network for sitting posture recognition;

[0009] Step D: Use the training set and validation set obtained in step B to train the KpointNet model, use the supervision signal during the training process, and generate a training model;

[0010] Step E: Use the test set obtained in step B to test the training model generated in step D.

[0011] Preferably, in step A, the step of keeping the camera position unchanged and recording videos of the subject in a specific sitting posture under a plurality of limited observation angles is specifically: shooting from five fixed angles, namely, the front, left, right, upper left and upper right of the subject, and shooting seven categories, namely, sitting upright, leaning left, leaning right, crossing legs, supporting chin, leaning back and lying down, from each viewing angle;

[0012] The step of adjusting the image to a uniform size is specifically as follows: adjusting the image to a size of 224 pixels×224 pixels, while ensuring that the image maintains its original proportions unchanged, and filling the vacant parts resulting from the resizing with black.

[0013] Preferably, in step B, the blazepose model learns the human body information in the RGB image, and recognizes and outputs the coordinates of 33 human body key points through a multi-layer CNN convolutional layer.

[0014] Preferably, in step C, the KpointNet model based on the point cloud network includes a key point feature extraction module, a multi-scale feature extraction module, a feature fusion and classification module, and a geometric feature learning module. The input key point data first passes through the key point feature extraction module to extract preliminary high-dimensional features of each key point, and then the multi-scale feature extraction module is used to extract rich local features and global features. Finally, the local features and global features are fused through the feature fusion and classification module and the final classification results are summarized and output.

[0015] Preferably, in the key point feature extraction module, the input data will undergo three layers of one-dimensional convolution, and the features of each key point will be gradually upgraded. The feature dimension of the input information is increased from 3 to 128, but the number of key points of the data is not changed, that is, the data gradually changes from 33*3 to 33*128; each layer of one-dimensional convolution will cooperate with the ReLu activation function and BatchNormliazation operation.

[0016] Preferably, the multi-scale feature extraction module comprises:

[0017] The global feature extraction module is used to directly extract the overall information from the key point information after preliminary learning, skipping the local feature extraction stage. It first expands the data from 33*128 to a vector of length 4224 through the flatten operation, and then uses the fully connected layer with the ReLu activation function and BatchNormliazation operation to further extract the global features and obtain the global feature vector;

[0018] The local feature extraction module is used to divide the data into 6 parts according to the spatial division relationship of human body parts. Each part is flattened by the flatten operation, and then the fully connected layer is used with the ReLu activation function and BatchNormliazation operation to extract local features. Finally, the 6 local features are spliced ​​and combined into an overall local feature vector.

[0019] Preferably, in the feature fusion and classification module, the global feature vector and the local feature vector are first concatenated to form a longer feature vector, and then a fully connected layer is used with a ReLu activation function and a BatchNormliazation operation for learning, thereby converting the result into a final classification result.

[0020] Preferably, in step D, the supervisory signal includes the classification of the sitting task and the size of body geometric physical quantities.

[0021] Preferably, step D is specifically: using the training set obtained in step B to train KpointNet, using the cross entropy loss function, SGD optimizer, the learning rate is set to 0.0001, and the batch size is 256.

[0022] Preferably, step E specifically includes: using the test set obtained in step B to test the KpointNet trained in step D, predicting the sitting posture category under different viewing angles, and determining whether the prediction is correct.

[0023] The embodiment of the present invention provides a three-dimensional efficient human sitting posture recognition method based on deep learning, which uses a 3D posture estimation method, can obtain richer spatial information from RGB images, and improves the information dimension; the KpointNet model based on point cloud model correction abandons the traditional point cloud model's processing of the disorder of point cloud data, and improves the reasoning utilization rate of the network model. In addition, the KpointNet model uses a multi-scale fusion method to learn the local and global components of the human body, learns richer feature representations, and has better accuracy. Moreover, through comprehensive learning in a multi-task manner, more information can also be learned, which is more conducive to improving the recognition accuracy. In the traditional skeleton graph + CNN mode, the process of constructing the skeleton graph is relatively cumbersome, and the data utilization rate is not high, while the method of the embodiment of the present invention directly learns the three-dimensional key point data, and has a more efficient data utilization rate;

[0024] The method of the embodiment of the present invention can be used to efficiently identify the human sitting posture category in three dimensions, correct bad sitting posture in time, and prevent potential health risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a three-dimensional efficient human sitting posture recognition method based on deep learning provided by an embodiment of the present invention;

[0026] Figure 2 A structural diagram of a KpointNet model based on a point cloud network provided in an embodiment of the present invention; DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] A three-dimensional efficient human sitting posture recognition method based on deep learning, comprising the following steps:

[0029] Step A, shooting the subject from five fixed angles, namely, the front, left side, right side, upper left side and upper right side. Each angle shoots seven types of postures, namely, sitting upright, leaning left, leaning right, crossing legs, supporting chin, leaning back and lying down. The sitting posture type remains unchanged during the entire recording period, but moderate body movement changes are allowed. Subsequently, frame extraction operations are performed on the video data, such as extracting an image every 5 frames as raw data. Subsequently, each captured image is resized to a size of 224 pixels × 224 pixels, while ensuring that the image maintains its original proportions. For the vacant parts caused by the resizing, black is used for filling.

[0030] Step B: Use the blazepose model to perform 3D pose estimation on the summarized image data. The blazepose model can realize real-time 3D pose estimation and predict the plane position and depth information of key points. This method will learn the human body information in the RGB image, identify and output the coordinates of 33 human key points through multi-layer CNN convolutional layers, and then use random jitter and mirror flipping to enhance the key point data. Then, the data is normalized to obtain the final 3D key point data. Finally, the data set of each perspective is divided into three parts, namely training set, validation set and test set;

[0031] Step C: construct a KpointNet model based on point cloud network;

[0032] Conventional point cloud models such as PointNet and PointNet++ need to first increase the dimension of the point cloud data and then reduce the dimension through the max operation in order to process the disorder of point cloud data, so as to realize the processing of disordered point cloud data. However, the key point data is ordered, so the embodiment of the present invention removes this process and designs a new model KpointNet, which includes a key point feature extraction module, a multi-scale feature extraction module, a feature fusion and classification module, and a geometric feature learning module. The input of the KpointNet model is the key point data. The input key point data first passes through the key point feature extraction module to extract the preliminary high-dimensional features of each key point, and then the multi-scale feature extraction module is used to extract rich local features and global features. The final feature fusion and classification module is responsible for fusing the local features and global features and summarizing and outputting the final classification results.

[0033] In the key point feature extraction module, the input data will go through three layers of one-dimensional convolution, and the features of each key point will be gradually upgraded. The feature dimension of the input information is increased from 3 to 128, but the number of key points of the data will not be changed, that is, the data gradually changes from 33*3 to 33*128, which means that the features of the original three spatial dimensions of xyz are upgraded to the complex representation of high-dimensional space; in addition, each layer of one-dimensional convolution will be combined with ReLu activation function and BatchNormliazation operation; in conventional point cloud models, it is usually necessary to upgrade the data and then reduce the dimension through the max operation to overcome the interference of the disorder of the point cloud data, but the key point data of the human body is ordered, so there is no need to use the max operation to reduce the data dimension;

[0034] In the multi-scale feature extraction module, it mainly includes two parts: a local feature extraction module and a global feature extraction module. The global feature extraction module is responsible for directly extracting the overall information from the key point information after preliminary learning, skipping the local feature extraction stage, first expanding the data from 33*128 to a vector of 4224 length through a flatten operation, and then using a fully connected layer with a ReLu activation function and a BatchNormliazation operation to further extract global features to obtain a global feature vector; the local feature extraction module is responsible for dividing the data into 6 parts according to the spatial division relationship of human body parts, each part is flattened by a flatten operation, and then a fully connected layer with a ReLu activation function and a BatchNormliazation operation is used to extract local features, and finally the 6 local features are spliced ​​to form an overall local feature vector;

[0035] In the feature fusion and classification module, the global feature vector and the local feature vector are first concatenated to form a longer feature vector, and then a fully connected layer is used with a ReLu activation function and a BatchNormliazation operation for learning, thereby converting it into the final classification result;

[0036] Step D, use the training set and validation set obtained in step B to train the KpointNet model. KpointNet is trained using the cross entropy loss function, SGD optimizer, learning rate set to 0.0001, batch size of 256, and dual supervision using sitting posture classification and body physical quantities during training. Considering that geometric information such as the angle and distance of body limbs is very important in the sitting posture classification task, the real body geometry information can even directly determine the classification of posture. The local features extracted by the local feature extraction module are mainly used. Some geometric data tasks are manually designed to standardize the feature space learned by the local feature module. Six fully connected networks are used to form a geometric feature learning module to complete the mapping task of six groups of local features to local tasks. Each group of classification tasks is responsible for learning the Euclidean distance or vector information between certain key points of the group.

[0037] Step E: Use the test set obtained in step B to test the KpointNet trained in step D, predict the sitting posture category under different viewing angles, and determine whether the prediction is correct.

[0038] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0039] like Figure 1 As shown, it is a flowchart of a three-dimensional efficient human sitting posture recognition method based on deep learning provided by an embodiment of the present invention, which includes the following steps:

[0040] (1) Data preprocessing: We collected videos of seven sitting postures from five viewing angles, totaling 35 video files. Each video was about 1 minute long. First, we read the video files using Python’s OpenCV library and extracted images frame by frame in chronological order. During this process, we selected an image every five frames and saved it. We then used the PIL library to resize these images to a size of 224 pixels by 224 pixels. Then, we used blazepose’s posture estimation technology to extract key points from these images. We then used random jitter to enhance the human key point data and performed a mirror flip with a probability of 50%. Finally, we used a normalization algorithm to set the key point data range to [0,1] and divided the samples into a training set, a validation set, and a test set at a ratio of 7:1:1. We finally obtained 17,640 images for the training set, 2,520 images for the validation set, and 2,520 images for the test set.

[0041] In the above process, random jitter is to randomly shift the key point data within a certain range. First, the difference d between the three maximum and minimum values ​​of this set of key point data in the three axes of x, y, and z is calculated. Then, for each key point, it is shifted between [-rd, rd], where r is the jitter coefficient, which is set to 0.01 here; mirror flipping is to flip the key point data left and right with the vertical axis as the center, and the left-leaning and right-leaning classifications should also be swapped;

[0042] (2) Network construction: Construct a network based on the point cloud model. The input of KpointNet is key point data. The input data first passes through the key point feature extraction module to extract the preliminary high-dimensional features of each key point, and then passes through the multi-scale feature extraction module to extract rich local features and global features. The final feature fusion and classification module is responsible for fusing local features with global features and summarizing and outputting the final classification results. In addition, the geometric feature learning module is responsible for mapping local features to six geometric tasks.

[0043] (3) Network training: Use the training set obtained in step (1) to train KpointNet, set the initial learning rate to 0.0001, the batch size to 256, the classification task loss function to the cross entropy loss function, the geometric feature learning module loss to SmoothL1Loss, select the Adam optimizer as the optimizer, set the training epoch to 100, set the dynamic learning rate, and the learning rate value decreases step by step as the number of training times increases, which is [0.0001, 0.00001, 0.000001]. The corresponding training epochs are [30, 50, 70]. After each training round, a verification is performed. When the verification set test effect reaches convergence, stop training;

[0044] In the above process, the six groups of tasks that the multi-task learning module should learn are shown in Table 1. The six groups of vectors learned by the six fully connected layers are then concatenated together to form the overall geometric task vector, so that SmoothL1Loss is used for regression learning:

[0045] Table 1

[0046] Local features Geometry tasks to learn HeadFeature <![CDATA[Vector 10，9 ,Vector 9，8 ]]> BodyFeature <![CDATA[Vector bodyZ ,Vector bodyY ]]> LArmFeature <![CDATA[Vector 13，12 ,Vector 12，11 ,Distance 13，9 ]]> RArmFeature <![CDATA[Vector 16，15 ,Vector 15，14 ,Distance 16，9 ]]> LLegFeature <![CDATA[Vector 4，5 ,Vector 5，6 ]]> RLegFeature <![CDATA[Vector 1，2 ,Vector 2，3 ]]>

[0047] Among them, Vector i，j Represents the vector obtained by subtracting j from key point i, Distance i，j Represents the three-dimensional Euclidean distance between key points i and j, Vector bodyZ , Vector bodyY They represent the normal vector of the plane formed by the torso and the vertical direction vector of the torso respectively;

[0048] (4) The trained model is verified by the test data to determine the test effect: the sitting posture category is predicted for the test set, the model and weights saved in the training phase are loaded, and the test data is input into the trained model to obtain the test results, including the probability of the sitting posture category. The category of the human sitting posture predicted by the model is calculated based on the predicted probability given by the model.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-dimensional efficient human sitting posture recognition method based on deep learning, characterized in that: The following steps are involved: Step A: Under a certain number of observation angles, the camera position is kept unchanged, and videos of people maintaining a specific sitting posture are recorded respectively. During the entire recording period, the sitting posture type remains unchanged, and the body makes appropriate movements. Then, a frame extraction operation is performed on the video data to adjust the image to a uniform size. Step B: Use the Blazepose model to perform 3D pose estimation on the image data, collect the coordinates of the three-dimensional key points, use random jitter and mirror flipping to enhance the key point data, and then normalize the data to obtain the final 3D key point data. Divide the data set of each perspective into three parts, namely, training set, validation set and test set; Step C: construct a KpointNet model based on point cloud network for sitting posture recognition; Step D: Use the training set and validation set obtained in step B to train the KpointNet model, use the supervision signal during the training process, and generate a training model; Step E: Use the test set obtained in step B to test the training model generated in step D.

2. The three-dimensional efficient human sitting posture recognition method based on deep learning according to claim 1 is characterized in that: In step A, the step of recording videos of the subject in a specific sitting posture while keeping the camera position unchanged under a plurality of limited observation angles is specifically as follows: shooting from five fixed angles, namely, the front, left, right, upper left, and upper right of the subject, and shooting seven categories, namely, sitting upright, leaning left, leaning right, crossing legs, supporting chin, leaning back, and lying down, from each viewing angle; The step of adjusting the image to a uniform size is specifically as follows: adjusting the image to a size of 224 pixels×224 pixels, while ensuring that the image maintains its original proportions unchanged, and filling the vacant parts resulting from the resizing with black.

3. The three-dimensional efficient human sitting posture recognition method based on deep learning according to claim 1 is characterized in that: In step B, the blazepose model learns the human body information in the RGB image and recognizes and outputs the coordinates of 33 human body key points through multi-layer CNN convolutional layers.

4. The three-dimensional efficient human sitting posture recognition method based on deep learning according to claim 1 is characterized in that: In step C, the KpointNet model based on the point cloud network includes a key point feature extraction module, a multi-scale feature extraction module, a feature fusion and classification module, and a geometric feature learning module. The input key point data first passes through the key point feature extraction module to extract the preliminary high-dimensional features of each key point, and then the multi-scale feature extraction module is used to extract rich local features and global features. Finally, the local features and global features are fused and summarized through the feature fusion and classification module to output the final classification results.

5. The three-dimensional efficient human sitting posture recognition method based on deep learning according to claim 4 is characterized in that: In the key point feature extraction module, the input data will pass through three layers of one-dimensional convolution, and the features of each key point will be gradually upgraded. The feature dimension of the input information is increased from 3 to 128, but the number of key points of the data will not be changed, that is, the data will gradually change from 33*3 to 33*128; each layer of one-dimensional convolution will cooperate with the ReLu activation function and BatchNormliazation operation.

6. The three-dimensional efficient human sitting posture recognition method based on deep learning according to claim 5 is characterized in that: The multi-scale feature extraction module comprises: The global feature extraction module is used to directly extract the overall information from the key point information after preliminary learning, skipping the local feature extraction stage. It first expands the data from 33*128 to a vector of length 4224 through the flatten operation, and then uses the fully connected layer with the ReLu activation function and BatchNormliazation operation to further extract the global features and obtain the global feature vector; The local feature extraction module is used to divide the data into 6 parts according to the spatial division relationship of human body parts. Each part is flattened by the flatten operation, and then the fully connected layer is used with the ReLu activation function and BatchNormliazation operation to extract local features. Finally, the 6 local features are spliced ​​and combined into an overall local feature vector.

7. The three-dimensional efficient human sitting posture recognition method based on deep learning according to claim 6 is characterized in that: In the feature fusion and classification module, the global feature vector and the local feature vector are first concatenated to form a longer feature vector, and then a fully connected layer is used with the ReLu activation function and the BatchNormliazation operation for learning, so as to be converted into the final classification result.

8. The three-dimensional efficient human sitting posture recognition method based on deep learning according to claim 1 is characterized in that: In step D, the supervisory signal includes the classification of the sitting task and the size of the body geometry.

9. The three-dimensional efficient human sitting posture recognition method based on deep learning according to claim 1 is characterized in that: Step D is as follows: Use the training set obtained in step B to train KpointNet, use the cross entropy loss function, SGD optimizer, set the learning rate to 0.0001, and the batch size to 256.

10. The three-dimensional efficient human sitting posture recognition method based on deep learning according to claim 1, characterized in that: Step E is as follows: Use the test set obtained in step B to test the KpointNet trained in step D, predict the sitting posture category under different viewing angles, and determine whether the prediction is correct.

Images