A 3D human posture recognition method based on photogrammetry and deep learning

Abstract

The present invention discloses a three-dimensional human posture recognition method based on photogrammetry and deep learning, which relates to the field of posture recognition technology, and includes the following steps: collecting original video data, using MediaPipe, a three-dimensional posture estimation method based on deep learning, to perform posture estimation, and obtain the original three-dimensional key point data of the image; then using the key point specification algorithm to perform preliminary constraints on the key point data and correct the recognition error; then using a multi-view simulation algorithm based on collinear equations to perform data enhancement to obtain richer perspective information and expand the data set; constructing a posture recognition model KeyposeNet; training and testing KeyposeNet, and selecting the best model for posture recognition. The present invention adopts the above-mentioned three-dimensional human posture recognition method based on photogrammetry and deep learning, realizes more efficient and accurate human posture classification and recognition, reduces the cost of manual classification, and provides auxiliary means for the recognition and judgment of human posture, so that it can be applied to various occasions.

Description

Technical Field

[0001] The present invention relates to the technical field of posture recognition, and in particular to a three-dimensional human posture recognition method based on photogrammetry and deep learning. Background Art

[0002] Human posture recognition mainly studies two problems: one is how to obtain the coordinate information of human posture, and the other is how to use the coordinate information to identify the posture category of the human body.

[0003] There are two common methods for obtaining human body posture coordinate information: using wearable devices to directly obtain physical information of human body posture and computer vision-based methods. Using wearable devices to obtain human body posture information is subject to large device limitations. Wearable devices are difficult to use in most scenarios, resulting in a limited application prospect. Computer vision methods based on RGB images only require cameras and have broader application prospects.

[0004] Regarding how to use coordinate information to identify human posture categories, posture estimation methods are often used to obtain human key point information from RGB images, and then use human key points to draw skeleton images or 3D voxels, which are then converted into CNN classification problems. However, skeleton images or 3D voxel data have a lot of information redundancy compared to human key point data, which will interfere with neural networks and is not conducive to edge computing on mobile devices. In addition, RGB images are affected by the shooting position and angle, and images from different perspectives and positions are very different. Only studying the front-viewing human sitting posture recognition cannot be well applied to practical problems. Summary of the invention

[0005] The purpose of the present invention is to provide a three-dimensional human posture recognition method based on photogrammetry and deep learning, to solve the problems raised in the above-mentioned background technology, to provide auxiliary means for the recognition and judgment of human posture, and to be applied in various occasions.

[0006] To achieve the above object, the present invention provides a three-dimensional human posture recognition method based on photogrammetry and deep learning, comprising the following steps:

[0007] Step A, shooting videos of specified postures at several fixed viewing angles, keeping the posture type unchanged during the recording process, making body changes, and intercepting frames of the collected video data to obtain image data;

[0008] Step B: Use MediaPipe to perform posture estimation on the collected image data and collect 3D key point data;

[0009] Step C: perform data enhancement on the collected key point data, and use the key point standardization algorithm to preliminarily correct the coordinate information of the key point data, then use the multi-view simulation algorithm to simulate the key point data information of other viewpoints, and then normalize the data to obtain the final key point data, and finally divide the expanded data set under each viewpoint into three parts, including training set, validation set and test set;

[0010] Step D: Build a deep learning-based KeyposeNet model for posture recognition;

[0011] Step E: Use the training set and validation set obtained in step C to train the KeyposeNet model to generate a training model;

[0012] Step F: Use the test set obtained in step C to test the training model generated in step E.

[0013] Preferably, step A specifically includes:

[0014] Shoot videos of the subject in the specified posture from seven fixed viewing angles: the front, left front, right front, left, right, upper left, and upper right of the subject;

[0015] The obtained image data is normalized to a uniform size, adjusted to 224pix*224pix, maintaining the original image ratio, and the empty parts are filled with black.

[0016] Preferably, in step B, MediaPipe uses a multi-layer convolutional neural network to infer the image data, and uses a regression method to predict the three-dimensional coordinate information of 33 key points of the human body in the image.

[0017] Preferably, in step C, the key point standardization algorithm uses a set of standard key point information as a reference, and dynamically adjusts the positions of the key points to make the limb proportions similar, specifically including:

[0018] For a set of key point information And a set of template key point information , where the template information is obtained by taking an additional frontal shot with the limbs spread out, and the mean length of all limbs is calculated , reuse Calculate the normalized limb length distribution and ,use and The difference in force on each limb is calculated ;

[0019] Calculate the forces at all key points , project the forces of the limbs connected to the key point onto the x, y, and z axes and sum them to calculate the displacement of each key point ,The above process is repeated 20 times to complete the key point specification algorithm. The key points are continuously displaced under the action of force and finally reach a steady state.

[0020] Preferably, in step C, the multi-view simulation algorithm includes random rotation of key points and random displacement of cameras, and the initial key point information After random rotation operation, , and then After random displacement of the camera, , complete the perspective transformation of key point information, including:

[0021] The key point random rotation uses the rotation matrix to directly act on the key point information and randomly rotates a certain angle. Assume is the initial key point information, then is the key point information after rotation, They are different rotation matrices to achieve The rotation of itself;

[0022] The random displacement of the camera uses the collinear equation to solve the key point information under different viewing angles by randomly setting the camera's position. Assuming that the key point information is the real world coordinate, randomly set a set of camera poses, and then use the collinear equation to find the coordinates of the image world , obtain a set of key point information under a new perspective.

[0023] Preferably, step D specifically comprises:

[0024] In the KeyposeNet model, the input data is processed through three layers of one-dimensional convolution. The features of each key point are upgraded from 3 to 128. The number of key points of the data is not changed. The data is changed from 33*3 to 33*128. Each layer of one-dimensional convolution is combined with the ReLu activation function and BatchNormliazation operation.

[0025] Then, the overall information is extracted from the key point information after preliminary learning, and the data is expanded from 33*128 to a vector of length 4224 through the flatten operation;

[0026] Then use the fully connected layer with the ReLu activation function and BatchNormliazation operation for learning and convert it into the final classification result.

[0027] Preferably, step E specifically includes: using the training set obtained in step C, training KeyposeNet using the cross entropy loss function and SGD optimizer, setting the learning rate to 0.0001, and setting the batch size to 256.

[0028] Preferably, step F specifically includes: using the test set obtained in step C to test the KeyposeNet trained in step E, predicting seven sitting posture categories under different viewing angles, and determining whether the prediction is correct.

[0029] Therefore, the present invention adopts the above-mentioned three-dimensional human posture recognition method based on photogrammetry and deep learning, which has the following beneficial effects:

[0030] (1) This method uses deep learning to directly extract 3D key point information from RGB images. It does not rely on depth cameras, which makes this method have a wider range of application scenarios.

[0031] (2) This method uses a data enhancement method based on collinear equations to transform the viewpoint, which can expand the data from the perspective of the view as much as possible, simulate data from more perspectives, and enable the model to have better multi-perspective three-dimensional recognition capabilities.

[0032] (3) The key point standardization algorithm of this method can reduce various errors to a certain extent and standardize the key point information.

[0033] (4) The KeyposeNet model of this method directly uses key point data and does not need to construct skeleton graphs or voxel data for learning, which reduces data redundancy and improves data utilization. At the same time, the model models three-dimensional key point data and has better three-dimensional learning representation capabilities, which can better handle real-world problems.

[0034] (5) This method can achieve more efficient and accurate human posture classification and recognition, reduce the cost of manual classification, and provide an auxiliary means for human posture recognition.

[0035] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of an embodiment of a three-dimensional human posture recognition method based on photogrammetry and deep learning according to the present invention;

[0037] Figure 2 A structural diagram of a KeyposeNet model of an embodiment of a three-dimensional human posture recognition method based on photogrammetry and deep learning according to the present invention;

[0038] Figure 3This is a multi-view simulation algorithm effect diagram of an embodiment of a three-dimensional human posture recognition method based on photogrammetry and deep learning of the present invention;

[0039] Figure 4 This is a key point standardization algorithm effect diagram of an embodiment of a three-dimensional human posture recognition method based on photogrammetry and deep learning in the present invention. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0041] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0042] Example

[0043] See also Figure 1-4 The present invention provides a three-dimensional human posture recognition method based on photogrammetry and deep learning, comprising the following steps:

[0044] Step A, respectively shoot videos of the subject in designated postures from seven fixed viewing angles, namely, the front left, the front right, the left, the right, the upper left, and the upper right. During the recording process, the posture type remains unchanged, and the body is appropriately changed. The collected video data is frame-captured, for example, an image is captured every 5 frames as initial data, and each image is adjusted to a 224pix*224pix image, maintaining the original image ratio, and the empty parts are filled with black.

[0045] Step B: Use MediaPipe to perform posture estimation on the collected image data, use a multi-layer convolutional neural network to infer the image data, and use regression to predict the three-dimensional coordinate information of 33 key points of the human body in the image.

[0046] Step C: perform data enhancement on the collected key point data, and use the key point normalization algorithm to preliminarily correct the coordinate information of the key point data, then use the multi-view simulation algorithm to simulate the key point data information of other viewpoints, and then normalize the data to obtain the final key point data. Finally, divide the expanded data set at each viewpoint into three parts, including training set, validation set and test set.

[0047] Among them, the key point standardization algorithm is to better deal with two problems: one is that the key point information predicted by MediaPipe may have a large deviation; the other is that in the multi-view simulation algorithm, the key points are not in the world coordinate system, which is a compromise for the lack of camera pose information. The algorithm mainly uses a set of standard key point information as a reference, and dynamically adjusts the position of the key points to make the limb proportions as close as possible. Specifically: for a set of key point information And a set of template key point information , where the template information is obtained by taking an additional frontal shot with the limbs spread out, and the mean length of all limbs is calculated , reuse Calculate the normalized limb length distribution and , in order to simulate the action of force for displacement, use and The difference in force on each limb is calculated .

[0048] In order to calculate the forces at all key points , project the forces of the limbs connected to the key point onto the x, y, and z axes and sum them up to calculate the displacement of each key point , Repeat the above process 20 times to complete the key point standardization algorithm. Finally, the key points will continue to move under the action of "force" and finally reach a steady state.

[0049] The multi-view simulation algorithm is a data enhancement method based on the collinear equation in photogrammetry. This method uses limited-view data to simulate key point data from more viewpoints through an algorithm, so as to enhance the recognition of human postures from multiple viewpoints. The multi-view simulation algorithm includes random rotation of key points and random displacement of cameras.

[0050] The purpose of random rotation of key points is to change the orientation of the key points themselves. The rotation matrix is ​​used directly on the key point information to randomly rotate a certain angle. The calculation method is:

[0051] ;

[0052] in, is the initial key point information, , , is the key point information after rotation, are different rotation matrices, is the corresponding rotation angle. This method realizes The rotation itself.

[0053] The purpose of random camera displacement is to change the camera's position to enrich the data's perspective while keeping the key point information unchanged. By using the collinear equation, the key point information under different perspectives is solved by randomly setting the camera's position. Assuming the key point information is the real world coordinate, randomly set a set of camera poses, and then use the collinear equation to find the coordinates of the image world, and obtain a set of key point information under a new perspective. ,in In order to reasonably meet the solution conditions of the collinear equation, we first find Maximum size:

[0054] ;

[0055] in, Represents the maximum difference in the three coordinate dimensions, so the camera's pose information is:

[0056] ;

[0057] Image information X0 and Y0 control the center of the generated key points. Here they are both set to 0, so that a set of camera pose data can be randomly generated within a reasonable range. Then, the collinear equation is used to solve the coordinates in the image world. ,in , the solution is:

[0058] ;

[0059] In formula x, and Directly obtained from the collinear equation, since the collinear equation cannot give depth information , so actually using the original key point coordinates By converting the coordinate system and rotating .

[0060] Initial key point information After random rotation operation, , After random displacement of the camera, ,These two steps are combined and called the multi-view simulation algorithm, which completes the perspective transformation of key point information.

[0061] Step D: Construct a KeyposeNet model based on deep learning for posture recognition. The input data will go through three layers of one-dimensional convolution. The features of each key point will be gradually upgraded. The feature dimension of the input information will be increased from 3 to 128, but the number of key points of the data will not change. That is, the data will gradually change from 33*3 to 33*128, which means that the features of the original three spatial dimensions of xyz have risen to a complex representation in a high-dimensional space. In addition, each layer of one-dimensional convolution will be combined with the ReLu activation function and BatchNormliazation operation; then the overall information will be extracted from the key point information after preliminary learning, and the data will be expanded from 33*128 to a vector of 4224 length through the flatten operation; then the fully connected layer will be used with the ReLu activation function and BatchNormliazation operation for learning, so as to convert it into the final classification result.

[0062] Step E: Use the training set and validation set obtained in step C to train the KeyposeNet model. The loss function uses the cross entropy loss function, the optimizer uses the SGD optimizer, the learning rate is set to 0.0001, and the batch size is 256.

[0063] Step F: Use the test set obtained in step C to test the training model generated in step E, predict the posture category under different viewing angles, and determine whether the prediction is correct.

[0064] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0065] like Figure 1 As shown, a flowchart of a method for three-dimensional human posture recognition based on photogrammetry and deep learning provided by an embodiment of the present invention specifically includes the following steps:

[0066] (1) Data preprocessing: Videos of 7 posture categories under 7 viewing angles were collected. Each viewing angle recorded seven categories: standing, leaning left, leaning right, sitting upright, supporting the chin, leaning back, and lying down. A total of 49 video files were obtained, and each video was about 1 minute long. Then, the Opencv and PIL libraries in Python were used to read the video files, and the images of each frame were obtained in time series. The images were saved every 5 frames and saved in 224pix*224pix size, maintaining the original image ratio, and the blank parts were filled with black. The samples were divided into training set, validation set, and test set in a ratio of 7:1:1, and finally: 24,696 training sets, 3,528 validation sets, and 3,528 test sets were obtained. MediaP The ipe posture estimation method is used to extract key points, and the extracted data is saved in the form of numpy for subsequent data enhancement processing; the key point specification algorithm is used to process the data to remove errors to a certain extent; then the obtained human key point data is enhanced using a multi-view simulation algorithm and random flipping. In the multi-view simulation algorithm, it is enhanced with a probability of 75%, and the camera posture is generated according to the algorithm to generate new data, and then mirror flipping is performed with a probability of 50%; finally, the normalization algorithm is used to limit the range of the key point data to [-1,1]; after processing the key point information corresponding to all image data, the corresponding key point information is saved in the form of a txt file for subsequent training.

[0067] (2) Network construction: Use pytorch to build the KeyposeNet model. The input of the model is key point data. The input data is first subjected to three layers of one-dimensional convolution for feature extraction. The key point feature extracts the high-dimensional features of the key point information, and then is learned through the flatten operation combined with the fully connected layer to output the final classification result.

[0068] (3) Network training: Use the training set obtained in step (1) to train KeyposeNet, set the initial learning rate to 0.0001, the batch size to 512, the loss function to the cross entropy loss function, select the SGD optimizer, set the training epoch to 100, and configure the dynamic learning rate strategy so that the learning rate decreases step by step with the increase of training rounds (epoch). The specific values ​​are 0.0001, 0.00001, and 0.000001, respectively. These values ​​take effect at the 30th, 50th, and 70th rounds of training, respectively. After each round of training, perform a validation set test. When the performance indicators on the validation set reach the convergence standard, terminate the training process.

[0069] (4) Network testing: Predict the sitting posture category of the test set, load the model and weights saved in the training phase, input the test data into the trained model, and obtain the test results, including the probability of the sitting posture category. Calculate the category of the human sitting posture predicted by the model based on the predicted probability given by the model. In practical applications, use the Python environment to build the MediaPipe method to perform real-time posture estimation on the video data stream. Each time the key point data is obtained, it is processed through the above method steps and then sent to KeyposeNet for recognition. After obtaining the recognition classification, real-time human posture recognition analysis can be performed on the video stream.

[0070] The accuracy and speed performance comparison is as follows:

[0071] The seven fixed-view videos are used as the training set, and an additional set of videos under moving view are taken as the test set. Since the test set is shot while constantly moving, it contains more viewpoints, which can test the multi-view 3D recognition capability of the method. The traditional posture recognition method of MediaPipe+resnet is selected. The traditional method also uses MediaPipe for posture estimation to obtain key point information, and then draws the key point information into the form of a skeleton diagram, which is then handed over to resnet for image classification to recognize human posture.

[0072] The results show that the accuracy of the method in the embodiment of the present invention reaches 97.45%, which can effectively identify the posture categories under different viewing angles, while the accuracy of the traditional method of MediaPipe+resnet is only 83.81%, which is lower than that of the embodiment of the present invention. In addition, if the multi-view simulation algorithm is not used on the basis of the method in the embodiment of the present invention, the recognition accuracy will drop to 93.77%, and if the key point specification algorithm is no longer used, the accuracy will drop to 92.31%, which shows that the multi-view simulation algorithm and the key point specification algorithm have played their corresponding roles and improved the accuracy of posture recognition.

[0073] In order to test the performance of the method, the win11 system, the cpu is i7-12700H, the gpu is 3070ti, and the pytorch version is 11.4, and the running speed of the method is tested.

[0074] The results show that the inference time of KeyposeNet is 1.09ms, the inference time of resnet is 4.65ms, the time for generating a skeleton graph from key point data is 28.80ms, the time for the multi-view simulation algorithm is 1.13ms, and the time for the key point specification algorithm is 9.59ms. The total time of the method in the embodiment of the present invention is 1.09+1.13+9.59=11.81ms, while the total time of the traditional method is 4.65+28.8=33.45ms. Considering the speed and performance, the embodiment of the present invention also has obvious advantages.

[0075] Therefore, the present invention adopts the above-mentioned three-dimensional human posture recognition method based on photogrammetry and deep learning, realizes efficient and accurate human posture classification and recognition, reduces the cost of manual classification, provides an auxiliary means for the recognition and judgment of human posture, and can be applied to various occasions.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A three-dimensional human posture recognition method based on photogrammetry and deep learning, characterized in that: The following steps are involved: Step A, shooting videos of specified postures at several fixed viewing angles, keeping the posture type unchanged during the recording process, making body changes, and intercepting frames of the collected video data to obtain image data; Step B: Use MediaPipe to perform posture estimation on the collected image data and collect 3D key point data; Step C: perform data enhancement on the collected key point data, and use the key point standardization algorithm to preliminarily correct the coordinate information of the key point data, then use the multi-view simulation algorithm to simulate the key point data information of other viewpoints, and then normalize the data to obtain the final key point data, and finally divide the expanded data set under each viewpoint into three parts, including training set, validation set and test set; Step D: Build a deep learning-based KeyposeNet model for posture recognition; Step E: Use the training set and validation set obtained in step C to train the KeyposeNet model to generate a training model; Step F: Using the test set obtained in step C to test the training model generated in step E; Step A specifically includes: Shoot videos of the subject in the specified posture from seven fixed viewing angles: the front, left front, right front, left, right, upper left, and upper right of the subject; Normalize the obtained image data to a uniform size and adjust it to , keep the original image ratio, and fill the empty parts with black; In step C, the key point standardization algorithm uses a set of standard key point information as a reference and dynamically adjusts the positions of the key points to make the limb proportions similar, specifically including: For a set of key point information And a set of template key point information , where the template information is obtained by taking an additional frontal shot with the limbs spread out, and the mean length of all limbs is calculated , reuse Calculate the normalized limb length distribution and ,use and The difference in force on each limb is calculated ; Calculate the forces at all key points , project the forces of the limbs connected to the key point onto the x, y, and z axes and sum them to calculate the displacement of each key point , repeat the above process 20 times to complete the key point specification algorithm. The key points are constantly displaced under the action of force and finally reach a steady state; In step C, the multi-view simulation algorithm includes random rotation of key points and random displacement of cameras. After random rotation operation, , and then After random displacement of the camera, , complete the perspective transformation of key point information, including: The key point random rotation uses the rotation matrix to directly act on the key point information and randomly rotates a certain angle. Assume is the initial key point information, then is the key point information after rotation, They are different rotation matrices to achieve The rotation of itself; The random displacement of the camera uses the collinear equation to solve the key point information under different viewing angles by randomly setting the camera's position. Assuming that the key point information is the real world coordinate, randomly set a set of camera poses, and then use the collinear equation to find the coordinates of the image world , obtain a set of key point information under a new perspective; Step D specifically includes: In the KeyposeNet model, the input data is convolved through three layers of one-dimensional convolution, and the features of each key point are upgraded. The feature dimension of the input information is increased from 3 to 128 without changing the number of key points of the data. Changes to , each layer of one-dimensional convolution is combined with ReLu activation function and BatchNormliazation operation; Then extract the overall information from the key point information after preliminary learning, and use the flatten operation to convert the data from Expand into a vector of length 4224; Then use the fully connected layer with the ReLu activation function and BatchNormliazation operation for learning and convert it into the final classification result.

2. The method for three-dimensional human posture recognition based on photogrammetry and deep learning according to claim 1, characterized in that: In step B, MediaPipe uses a multi-layer convolutional neural network to infer the image data and uses regression to predict the three-dimensional coordinate information of 33 key points of the human body in the image.

3. The three-dimensional human posture recognition method based on photogrammetry and deep learning according to claim 1, characterized in that: Step E specifically includes: using the training set obtained in step C, training KeyposeNet using the cross entropy loss function and SGD optimizer, with the learning rate set to 0.0001 and the batch size set to 256.

4. A three-dimensional human posture recognition method based on photogrammetry and deep learning according to claim 3, characterized in that: Step F specifically includes: using the test set obtained in step C to test the KeyposeNet trained in step E, predicting seven sitting posture categories under different viewing angles, and determining whether the prediction is correct.

Images