A Deep Learning-Based 3D Efficient Human Sitting Posture Recognition Method

Abstract

This invention relates to the field of posture recognition technology and provides a deep learning-based, efficient 3D human posture recognition method, comprising the following steps: recording video data; performing 3D posture estimation on all images in the video using BlazePose to extract 3D keypoint coordinates; then performing data augmentation by randomly jittering and mirroring the obtained keypoint coordinates to expand the dataset; constructing a KpointNet model based on a point cloud model; training the KpointNet model to generate a training model; and finally testing the trained model and applying it in practice. This invention can obtain richer spatial information from RGB images, increasing the information dimensionality; the KpointNet model based on a point cloud model improves the inference utilization of the network model and has better accuracy; directly learning from 3D keypoint data results in more efficient data utilization; this invention can efficiently identify human posture categories in 3D, correct poor posture in a timely manner, and prevent potential health risks.

Description

Technical Field

[0001] This invention belongs to the field of sitting posture recognition technology, and in particular relates to a three-dimensional efficient human sitting posture recognition method based on deep learning. Background Technology

[0002] Incorrect sitting posture is closely linked to related diseases. Timely identification and correction of incorrect sitting posture can help people improve their health to some extent. In addition, identifying incorrect sitting posture can also provide a reference for research fields such as motion capture, behavior recognition, and action prediction.

[0003] Human posture recognition primarily studies how to describe human posture and identify posture categories. Common methods fall into two main categories: those based on wearable devices or pressure sensors, and those based on computer vision. Using wearable devices or pressure sensors to collect human posture information is limited by equipment constraints, restricting its application prospects. In contrast, computer vision methods targeting RGB images have broader application potential. Most posture recognition methods for RGB images use posture estimation to obtain key points on the human body, then further draw a skeleton map, transforming it into a classification problem. The process of creating the skeleton map is relatively slow and generates a lot of redundant information, slowing down model inference and making it inefficient. Furthermore, point cloud models are a systematic approach to processing point cloud data. Point cloud data is similar to key point data, but the difference is that point cloud data is unordered, while key point data is ordered. Conventional point cloud models compromise when dealing with the unordered nature of point cloud data, leading to resource waste when applying it to key point data. In addition, most traditional methods use 2D posture estimation methods and create planar skeleton maps for recognition, losing 3D depth information, resulting in insufficient recognition capabilities and models lacking sufficient 3D recognition ability. Summary of the Invention

[0004] The purpose of this invention is to provide a three-dimensional efficient human sitting posture recognition method based on deep learning, which aims to solve the problems mentioned in the background art.

[0005] The present invention is implemented as follows: a method for efficient 3D human sitting posture recognition based on deep learning, comprising the following steps:

[0006] Step A: Under several limited observation angles, keep the camera position unchanged and record videos of maintaining a specific sitting posture. During the entire recording, keep the sitting posture type unchanged and make appropriate changes in body movements. Then, perform frame extraction on the video data and adjust the images to a uniform size.

[0007] Step B: Use the Blazepose model to perform 3D pose estimation on the image data, collect the coordinates of 3D key points, perform data augmentation on the key point data using random jitter and mirror flipping, and then normalize the data to obtain the final 3D key point data. Divide the dataset of each viewpoint into three parts, namely training set, validation set and test set.

[0008] Step C: Construct a KpointNet model based on point cloud networks for posture recognition;

[0009] Step D: Use the training set and validation set obtained in step B to train the KpointNet model. Use supervision signals during the training process to generate the training model.

[0010] Step E: Use the test set obtained in step B to test the training model generated in step D.

[0011] Preferably, in step A, the step of keeping the camera position unchanged and recording videos of maintaining a specific sitting posture under a limited number of observation angles specifically involves: shooting from five fixed angles—the front, left, right, upper left, and upper right—of the subject, and shooting seven categories from each angle: sitting upright, leaning to the left, leaning to the right, crossing legs, resting chin on hand, leaning back, and lying down.

[0012] The step of adjusting the image to a uniform size is as follows: the image is adjusted to a size of 224 pixels × 224 pixels, while ensuring that the image maintains its original aspect ratio. Any missing parts caused by the size adjustment are filled with black.

[0013] Preferably, in step B, the blazepose model learns human information in the RGB image and identifies and outputs the coordinates of 33 human body key points through multiple CNN convolutional layers.

[0014] Preferably, in step C, the KpointNet model based on point cloud network includes a key point feature extraction module, a multi-scale feature extraction module, a feature fusion and classification module, and a geometric feature learning module. The input key point data first passes through the key point feature extraction module to extract preliminary high-dimensional features of each key point, and then through the multi-scale feature extraction module to extract rich local and global features. Finally, the feature fusion and classification module fuses the local and global features and summarizes and outputs the final classification result.

[0015] Preferably, in the key point feature extraction module, the input data will go through three layers of one-dimensional convolution, and the features of each key point will be gradually upgraded in dimensionality. The feature dimension of the input information will be increased from 3 to 128, but the number of key points in the data will not be changed, that is, the data will gradually change from 33*3 to 33*128; each layer of one-dimensional convolution will be combined with the ReLU activation function and BatchNormliazation operation.

[0016] Preferably, the multi-scale feature extraction module includes:

[0017] The global feature extraction module is used to directly extract overall information from the key point information after initial learning, skipping the local feature extraction stage. First, the data is flattened from 33*128 to a vector of length 4224 through the flatten operation. Then, a fully connected layer is used in conjunction with the ReLU activation function and the BatchNormliazation operation to further extract global features and obtain a global feature vector.

[0018] The local feature extraction module is used to divide the data into 6 parts according to the spatial division relationship of human body parts. Each part is flattened by the flatten operation, and then a fully connected layer is used with the ReLU activation function and BatchNormliazation operation to extract local features. Finally, the 6 local features are concatenated to form the overall local feature vector.

[0019] Preferably, in the feature fusion and classification module, the global feature vector and the local feature vector are first concatenated to form a longer feature vector, and then a fully connected layer is used in conjunction with the ReLU activation function and BatchNormliazation operation for learning, thereby transforming it into the final classification result.

[0020] Preferably, in step D, the monitoring signal includes the classification of the sitting task and the magnitude of the body's geometric physical quantities.

[0021] Preferably, step D specifically involves: training KpointNet using the training set obtained in step B, using the cross-entropy loss function, the SGD optimizer, setting the learning rate to 0.0001, and the batch size to 256.

[0022] Preferably, step E specifically involves: using the test set obtained in step B to test the KpointNet trained in step D, predicting the sitting posture category under different viewpoints, and determining whether the prediction is correct.

[0023] This invention provides a deep learning-based, high-efficiency 3D human sitting posture recognition method. It utilizes 3D pose estimation to extract richer spatial information from RGB images, increasing the information dimensionality. The KpointNet model, based on point cloud model correction, eliminates the traditional point cloud model's handling of the disordered nature of point cloud data, improving the inference utilization of the network model. Furthermore, the KpointNet model employs multi-scale fusion to learn the local and global components of the human body, acquiring richer feature representations and achieving better accuracy. Moreover, by integrating learning through multiple tasks, it can learn even more information, further enhancing recognition accuracy. In the traditional skeleton graph + CNN approach, constructing the skeleton graph is relatively cumbersome and has low data utilization. In contrast, the method of this invention directly learns from 3D keypoint data, resulting in more efficient data utilization.

[0024] The method described in this invention can efficiently identify human sitting postures in three dimensions, correct poor postures in a timely manner, and prevent potential health risks. Attached Figure Description

[0025] Figure 1 A flowchart illustrating a deep learning-based, efficient 3D human sitting posture recognition method provided in an embodiment of the present invention;

[0026] Figure 2 The structure diagram of the KpointNet model based on point cloud network provided in the embodiments of the present invention; Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0028] A deep learning-based, efficient 3D human sitting posture recognition method includes the following steps:

[0029] Step A: Shoot the subject from five fixed angles: front, left, right, upper left, and upper right. For each angle, capture seven types of postures: sitting upright, leaning to the left, leaning to the right, crossing legs, resting chin on hand, leaning back, and lying down. Maintain the same sitting posture throughout the recording, but allow for moderate body movements. Then, extract frames from the video data, for example, extract one image every 5 frames as the raw data. Adjust each captured image to 224 pixels × 224 pixels while ensuring that the image maintains its original aspect ratio. Fill any gaps caused by resizing with black.

[0030] Step B: Use the Blazepose model to perform 3D pose estimation on the collected image data. The Blazepose model can achieve real-time 3D pose estimation and predict the planar position and depth information of key points. This method learns human information in RGB images, identifies and outputs the coordinates of 33 human key points through multiple CNN convolutional layers, then performs data augmentation on the key point data using random jitter and mirror flipping, and then normalizes the data to obtain the final 3D key point data. Finally, the dataset for each viewpoint is divided into three parts: training set, validation set and test set.

[0031] Step C: Construct a KpointNet model based on a point cloud network;

[0032] Conventional point cloud models such as PointNet and PointNet++ require dimensionality upscaling and dimensionality reduction via max operations to handle the disordered nature of point cloud data. However, keypoint data is ordered. Therefore, this embodiment of the invention eliminates this process and designs a new model, KpointNet. This model includes a keypoint feature extraction module, a multi-scale feature extraction module, a feature fusion and classification module, and a geometric feature learning module. The input of the KpointNet model is keypoint data. The input keypoint data first passes through the keypoint feature extraction module to extract preliminary high-dimensional features for each keypoint. Then, the multi-scale feature extraction module extracts rich local and global features. Finally, the feature fusion and classification module is responsible for fusing the local and global features and summarizing them to output the final classification result.

[0033] In the keypoint feature extraction module, the input data undergoes three layers of one-dimensional convolution. The features of each keypoint are gradually increased in dimensionality, from 3 to 128, without changing the number of keypoints in the data. That is, the data gradually changes from 33*3 to 33*128, which means that the features are upgraded from the initial three spatial dimensions of xyz to a more complex representation in a higher-dimensional space. In addition, each layer of one-dimensional convolution is combined with the ReLU activation function and BatchNormliazation operation. In conventional point cloud models, it is usually necessary to increase the dimensionality of the data and then reduce it using the max operation to overcome the interference of the disorder of point cloud data. However, human keypoint data is ordered, so it is not necessary to use the max operation for data dimensionality reduction.

[0034] The multi-scale feature extraction module mainly includes two parts: a local feature extraction module and a global feature extraction module. The global feature extraction module is responsible for directly extracting overall information from the key point information after preliminary learning, skipping the local feature extraction stage. It first flattens the data from 33*128 to a vector of length 4224 using a flatten operation, and then uses a fully connected layer with ReLU activation function and BatchNormliazation operation to further extract global features and obtain a global feature vector. The local feature extraction module is responsible for dividing the data into 6 parts according to the spatial division relationship of human body parts. Each part is flattened by a flatten operation, and then a fully connected layer with ReLU activation function and BatchNormliazation operation is used to extract local features. Finally, the 6 local features are concatenated to form the overall local feature vector.

[0035] In the feature fusion and classification module, the global feature vector and the local feature vector are first concatenated to form a longer feature vector. Then, a fully connected layer is used in conjunction with the ReLU activation function and BatchNormliazation operation to learn the feature vector and transform it into the final classification result.

[0036] Step D: Train the KpointNet model using the training and validation sets obtained in Step B. Train KpointNet using the cross-entropy loss function, SGD optimizer, learning rate set to 0.0001, and batch size set to 256. During training, dual supervision is provided by posture classification and body physical quantities. Considering that geometric information such as the angle and distance of body limbs is very important in the posture classification task, and that real body geometric information can even directly determine the posture classification, the local features extracted by the local feature extraction module are mainly used. Some geometric data tasks are manually designed to standardize the feature space learned by the local feature module. A geometric feature learning module composed of 6 fully connected networks is used to complete the mapping task from 6 groups of local features to local tasks. Each group of classification tasks is responsible for learning the Euclidean distance or vector information between certain key points of the group.

[0037] Step E: Use the test set obtained in step B to test the KpointNet trained in step D, predict the sitting posture category under different viewpoints, and determine whether the prediction is correct.

[0038] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0039] like Figure 1 The diagram shown is a flowchart of a deep learning-based 3D efficient human sitting posture recognition method according to an embodiment of the present invention, including the following steps:

[0040] (1) Data preprocessing: 35 video files were collected from 7 sitting posture categories from 5 perspectives. Each video was about 1 minute long. First, the video files were read using the OpenCV library in Python and the images were extracted frame by frame in chronological order. During this process, one image was selected and saved every 5 frames. The PIL library was used to adjust these images to a size of 224 pixels by 224 pixels. Then, the pose estimation technology of Blazepose was used to extract key points from these images. The obtained human key point data was augmented with random jitter and mirrored with a 50% probability. Finally, the normalization algorithm was used to define the range of key point data between [0,1]. The samples were divided into training set, validation set and test set in a ratio of 7:1:1. The final results were: 17640 images in the training set, 2520 images in the validation set and 2520 images in the test set.

[0041] In the above process, random jittering involves randomly displacing the keypoint data within a certain range. First, the difference d between the three maximum and minimum values ​​of this set of keypoint data along the x, y, and z axes is calculated. Then, for each keypoint, it is displaced between [-rd, rd], where r is the jitter coefficient, which is set to 0.01 here. Mirror flipping involves flipping the keypoint data left and right around the vertical axis center, and the left-tilt and right-tilt classifications should also be reversed.

[0042] (2) Network Construction: A network based on a point cloud model is constructed. The input of KpointNet is key point data. The input data first passes through the key point feature extraction module to extract the preliminary high-dimensional features of each key point. Then, the multi-scale feature extraction module extracts rich local and global features. Finally, the feature fusion and classification module is responsible for fusing the local and global features and summarizing them to output the final classification result. In addition, the geometric feature learning module is responsible for mapping the local features to 6 geometric tasks.

[0043] (3) Network training: Using the training set obtained in step (1), KpointNet is trained. The initial learning rate is set to 0.0001, the batch size is 256, the loss function for the classification task is set to the cross-entropy loss function, the loss for the geometric feature learning module is set to SmoothL1Loss, the optimizer is selected as Adam optimizer, the training epoch is set to 100, and the dynamic learning rate is set. The value of the learning rate decreases stepwise as the number of training times increases, which is [0.0001, 0.00001, 0.000001], and the corresponding training epochs are [30, 50, 70]. After each training round, a validation is performed. When the validation set test results reach convergence, training is stopped.

[0044] In the above process, the six sets of tasks that the multi-task learning module should learn are shown in Table 1. The six fully connected layers learn six sets of vectors, which are then concatenated to form the overall geometric task vector, and then SmoothL1Loss is used for regression learning.

[0045] Table 1

[0046] Local features Geometric tasks to be learned HeadFeature <![CDATA[Vector 10，9 ,Vector 9，8 ]]> BodyFeature <![CDATA[Vector bodyZ ,Vector bodyY ]]> LArmFeature <![CDATA[Vector 13，12 ,Vector 12，11 ,Distance 13，9 ]]> RArmFeature <![CDATA[Vector 16，15 ,Vector 15，14 ,Distance 16，9 ]]> LLegFeature <![CDATA[Vector 4，5 ,Vector 5，6 ]]> RLegFeature <![CDATA[Vector 1，2 ,Vector 2，3 ]]>

[0047] Vector i，j Distance represents the vector obtained by subtracting j from keypoint i. i，j Vector represents the 3D Euclidean distance between keypoints i and j. bodyZ Vector bodyY These represent the normal vector of the plane formed by the torso and the vertical vector of the torso, respectively.

[0048] (4) Test data to verify the trained model and determine the test effect: predict the sitting posture category on the test set, load the model and weights saved in the training stage, input the test data into the trained model, obtain the test results, including the probability of the sitting posture category, and calculate the category of human sitting posture predicted by the model based on the predicted probability given by the model.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-dimensional efficient human sitting posture recognition method based on deep learning, characterized in that , comprising the following steps: Step A, keep the camera position unchanged under a limited number of observation angles, record the video of maintaining a specific sitting posture respectively, the type of sitting posture is maintained unchanged during the whole recording period, the body makes moderate action changes, then the video data is frame extracted, and the image is adjusted to a uniform size; Step B, use the blazepose model to perform 3D pose estimation on the image data, collect three-dimensional key point coordinates, use random jitter and mirror flip for data enhancement, then normalize the data to obtain the final 3D key point data, and divide each view data set into three parts, namely training set, validation set and test set; Step C, a KpointNet model based on point cloud network is constructed for posture recognition, the KpointNet model based on point cloud network includes a key point feature extraction module, a multi-scale feature extraction module, a feature fusion and classification module, and a geometric feature learning module, the input key point data is first subjected to the key point feature extraction module to extract the preliminary high-dimensional features of each key point, then the multi-scale feature extraction module is used to extract rich local features and global features, and finally the feature fusion and classification module is used to fuse the local features and global features and output the final classification result; In the key point feature extraction module, the input data is subjected to three one-dimensional convolutions, the feature of each key point is gradually upgraded, the feature dimension of the input information is increased from 3 to 128, but the number of key points of the data is not changed, that is, the data is gradually changed from 33*3 to 33*128; each one-dimensional convolution is matched with a ReLu activation function and a BatchNormliazation operation; since the human body key point data is ordered, max operation is not needed to reduce the dimension of the data to overcome the interference of disorder; The multi-scale feature extraction module includes a global feature extraction module for directly extracting overall information from the preliminary learned key point information, and a local feature extraction module for dividing the data into 6 parts according to the spatial division relationship of human body parts, extracting local features for each part, and finally combining the 6 local features into a whole local feature vector; in the feature fusion and classification module, the global feature vector and the local feature vector are first spliced to form a longer feature vector, then learned, and finally converted into the final classification result; Step D, use the training set and the validation set obtained in step B to train the KpointNet model, use a supervision signal during the training process to generate a training model, the supervision signal includes the classification of the posture task and the size of the body geometric physical quantity; use six fully connected networks to form a geometric feature learning module to complete the mapping task of six groups of local features to local tasks, and each group of classification task is responsible for learning the Euclidean distance or vector information between the key points in the group; Step E, test the training model generated in step D using the test set obtained in step B. 2.The deep learning-based three-dimensional efficient human sitting posture recognition method according to claim 1, characterized in that In step A, the step of keeping the camera position unchanged and recording videos of the specific posture from different angles, specifically, the five fixed angles of the front, left side, right side, upper left and upper right of the subject are shot, and seven types of postures, including sitting straight, leaning left, leaning right, lifting the leg, resting the cheek, leaning back and lying down, are shot from each angle. The step of adjusting the image to a uniform size, specifically, the image is adjusted to a size of 224 pixels x 224 pixels, while ensuring that the picture maintains its original proportion unchanged, and the empty part generated due to the size adjustment is filled with black. 3.The deep learning-based three-dimensional efficient human sitting posture recognition method according to claim 1, characterized in that In step B, the blazepose model learns the human body information in the RGB image, and outputs the coordinates of 33 human body key points through a multi-layer CNN convolutional layer. 4.The deep learning-based three-dimensional efficient human sitting posture recognition method of claim 1, wherein Step D specifically uses the training set obtained in step B to train KpointNet, uses a cross-entropy loss function, an SGD optimizer, and sets the learning rate to 0.0001 and the batch size to 256. 5.The deep learning-based three-dimensional efficient human sitting posture recognition method according to claim 1, characterized in that Step E specifically uses the test set obtained in step B to test the KpointNet trained in step D, predicts the posture category under different angles, and judges whether the prediction is correct.

Images