A body posture rapid detection method based on a lightweight convolutional pose machine

Abstract

The application discloses a body posture rapid detection method based on a light-weight convolutional pose machine, comprising the following steps: constructing a posture detection model; loading the skeleton network MobileNet pre-training weight of the posture detection model, and pre-processing a training data set; training the posture detection model; deploying the trained posture detection model on a server as a background service by using Flask, opening an API interface, and applying the posture detection model to mobile terminals or embedded devices to perform body posture rapid detection based on the light-weight convolutional pose machine. The method solves the problem that the traditional HPE algorithm is difficult to be converted into practical application due to poor real-time detection caused by excessive model operation and parameter quantity, and greatly optimizes the training speed and video memory consumption.

Description

TECHNICAL FIELD

[0001] The application relates to a body posture rapid detection method, in particular to a body posture rapid detection method based on a light-weight convolutional pose machine. BACKGROUND

[0002] Since the digital era, while technology changes the way of life, the proportion of bad postures in various groups of people is also increasing, and there is a lack of effective auxiliary means for posture detection at present. In order to solve this problem, a human pose estimation (HPE) algorithm is introduced to detect key points of human postures and calculate bad postures based on human key point skeletons. There are two main methods for multi-person HPE, which are "top-down" and "bottom-up" methods. The "top-down" method first needs to perform target detection, and based on the results of target detection, single person pose estimation (SPPE) is performed on each detected person, the running time is proportional to the number of people, and the results of SPPE are highly dependent on the results of the first step of target detection. The "bottom-up" method predicts the positions of all human body part key points, and then assembles these parts into a human skeleton. The part affinity field (PAF) proposed by Carnegie Mellon University saves the position and direction vector of each body part to ensure the accuracy of the connection between the body parts. At present, the HPE algorithm generally has problems such as large parameter quantity, large calculation quantity and complex model, and it is difficult to convert into practical application due to insufficient computing resources or poor real-time performance for mobile devices. In 2017, Google team proposed a light-weight convolutional neural network MobileNet for mobile terminal or embedded device, and subsequently released multiple subsequent versions. Such network can greatly reduce the parameter quantity and calculation quantity of the network under the condition of sacrificing certain accuracy.

[0003] The above prior art does not solve the problem that the traditional HPE algorithm has a large model operation quantity and parameter quantity, which leads to poor real-time performance of detection and is difficult to convert into practical application, and has problems of slow training speed and excessive consumption of video memory. SUMMARY

[0004] The technical problem solved by the application is to provide a body posture rapid detection method based on a light-weight convolutional pose machine.

[0005] In order to solve the above technical problems, the application discloses a body posture rapid detection method based on a light-weight convolutional pose machine, which comprises the following steps:

[0006] Step 1, constructing a pose detection model, i.e. a convolutional pose machine (CPM) based on MobileNet;

[0007] Step 2, loading the skeleton network MobileNet pre-training weight of the pose detection model, constructing a training data set and pre-processing the training data set, including rotation, translation, flipping and random erasing;

[0008] The skeleton network is completed by modifying the last layer of the skeleton network for downsampling and the convolutional pose machine (CPM).

[0009] Step 3, starting to train the pose detection model: adjusting the hyperparameters, gradually increasing the optimization stages of the convolutional pose machine (CPM) in the pose detection model, using the skeleton network to extract features to obtain a feature map, and predicting human key points according to the feature map and the convolutional pose machine (CPM);

[0010] The optimization stage of the convolutional pose machine (CPM) is 6 stages, including 1 initial optimization stage and 5 optimization stages, and the optimization stages of the convolutional pose machine (CPM) meet the following conditions:

[0011]

[0012]

[0013] Wherein, F represents the feature map extracted by the skeleton network, and Φ t and ρ t respectively represent the branch calculation of the optimization stage of the convolutional pose machine (CPM) for optimizing the heat map and optimizing the PAF, wherein t represents the stage number, S t and L t respectively represent the heat map and the part affinity field (PAF) output by the CPM optimization stage.

[0014] In the training of the pose detection model, the loss function used is a loss function composed of L2 Loss and Focal Loss, wherein L2 Loss and Focal Loss are as follows:

[0015]

[0016] FocalLoss=-(1-p t ) γ log(p t )

[0017] Wherein, y i represents the sample output true value, n represents the total number of samples, f() represents the prediction function of the pose detection model, x i represents the sample input, and p tModulation factor, used to focus on difficult samples, gamma represents a parameter, when gamma is 0, it is cross entropy loss function.

[0018] The parameter gamma ranges from 0 to 5.

[0019] Step 4, using greedy inference method to assemble the predicted human key points, draw human posture skeleton map, and output human part information, the human part information at least includes: number of people, category, whether there is key point and key point coordinate;

[0020] Step 5, two methods are used to detect target body posture and classify, specifically including:

[0021] Method one: according to knowledge base, through calculating the angle between key point skeleton, detecting target body posture and classifying;

[0022] The knowledge base is a pre-constructed knowledge base.

[0023] Method two: construct artificial neural network, set the last layer activation function of the artificial neural network as Softmax function, use the labeled multi-type body posture data to train the network, input the human part information detected and output in step 4, infer the probability that the target belongs to each type of body posture, the specific method includes: taking the human part information obtained in step 4 as the input of artificial neural network, after classification by artificial neural network, the probability that the target belongs to each type of body posture is obtained.

[0024] Step 6, repeat steps 3 to 5 to complete the training of the posture detection model;

[0025] Step 7, using Flask to deploy the trained posture detection model on the server as background service, open API interface, for mobile terminal or embedded device to apply the posture detection model to conduct light convolution posture machine based body posture rapid detection.

[0026] The human part information obtained in step 4 is saved in JSON format.

[0027] In the API interface in step 7, the data exchange format is JSON.

[0028] Beneficial effects:

[0029] The method maintains good real-time performance and robustness of the convolutional pose machine, solves the problem of poor real-time performance of detection caused by excessive model operation and parameter quantity of traditional HPE algorithm, and facilitates switching of model training and application based on specific application scenarios by using a light skeleton network type through hyperparameter control, adopts a mixed precision training method in the training process, and greatly optimizes the training speed and memory consumption. The application can be conveniently applied on a mobile terminal or an embedded device by calling an API, thereby providing an efficient auxiliary means for body posture detection. BRIEF DESCRIPTION OF DRAWINGS

[0030] The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0031] Figure 1 is a flowchart of the present application;

[0032] Figure 2 is a schematic diagram of the body posture detection result, the left diagram in the figure is the first body posture type detection result, which is a normal result, the middle diagram is the second body posture type detection result, and the right diagram is the third body posture type detection result. DETAILED DESCRIPTION

[0033] The present application provides a body posture rapid detection method based on a light convolutional pose machine, solves the problem of poor real-time performance of detection caused by excessive model operation and parameter quantity of traditional HPE algorithm, and provides an efficient auxiliary means for body posture detection. Figure 1 The technical scheme adopted by the present application includes the following steps:

[0034] S1. Construct a human pose estimation model based on a light improved convolutional pose machine (CPM), that is, construct a convolutional pose machine CPM based on MobileNet; MobileNet is a light convolutional neural network proposed by Google for image classification, and CPM is a deep learning algorithm for pose estimation

[0035] S2. Load a skeleton network pre-training model, and pre-process the training data set, including image enhancement such as rotation, translation, flipping, and random erasing

[0036] S3. Train the model, adjust the hyperparameters, gradually increase the CPM optimization stage, and use the features extracted by the skeleton network and the CPM to predict the heat map and PAFs

[0037] S4. Assemble the predicted limb joints using the greedy inference algorithm, draw the human pose skeleton graph, and output the human part information (including the number of people, categories, whether there are key points, key point coordinates, etc.)

[0038] S5. Two methods are proposed to detect the target body posture and classify. The first method: according to the knowledge base, the angle between the important node skeletons is calculated to detect the user's body posture and classify. The second method: an artificial neural network is constructed, the activation function of the last layer of the network is set to Softmax function, and the annotated multi-type body posture data is used to train the network. By inputting the part information detected in S4, the probability that the user belongs to each type of body posture is inferred

[0039] S6. Flask is a Python-based backend web framework. The trained model is deployed on the server as a background service using Flask, and an API interface is opened for mobile or embedded device applications

[0040] Preferably, the skeleton network in S2 controls the use of multiple lightweight skeleton networks through hyperparameters, and the alignment is completed by modifying the last few layers of down-sampling and CPM.

[0041] Preferably, in S3, the calculation of heat maps and PAFs in the CPM optimization stage is combined with the branch except for the last two layers of 1x1 convolution, so that the two branches share this part of the convolution calculation.

[0042] Preferably, the optimization stage of CPM in S3 has 6 in total, including 1 initial optimization stage and 5 optimization stages. The calculation of the CPM optimization stage should satisfy formula (1) and formula (2):

[0043]

[0044]

[0045] where F represents the feature map extracted by the skeleton network, Φ t , ρ t respectively represent the branch calculation of the CPM optimization stage for optimizing the heat map and optimizing the PAF, where t represents the stage number, using S t , L t respectively represent the heat map and PAF output by the CPM optimization stage.

[0046] Preferably, the loss function of model training is changed from the L2 Loss used by the original version to a loss function composed of a mixture of L2 Loss and Focal Loss (Reference: Focal loss for dense object detection [C] / / Proceedings of the IEEE international conference on computer vision.2017:2980-2988.), which focuses on training difficult samples, where L2 Loss and Focal Loss are shown in formula (3) and formula (4):

[0047]

[0048] FocalLoss=-α t (1-p t ) t log(p t ) (4)

[0049] Preferably, in the first way of S5, the body posture type of the detection result is inferred based on the relevant knowledge base.

[0050] Preferably, in the second way of S5, an artificial neural network is constructed, and the activation function of the last layer of the network is set to a Softmax function as shown in formula (5):

[0051]

[0052] The network is trained using artificially labeled multi-type body posture data images, and the detection information output in the previous stage (saving the human key point information in JSON format) is used as the input of the artificial neural network. After classification by the artificial neural network, the probability of the user having each type of body posture is inferred.

[0053] Preferably, in S6, the model is deployed as a background service type mode for Web, APP, desktop, WeChat applet and other development processes through Flask, and the data exchange format is JSON.

[0054] Embodiment:

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0056] A fast posture detection method based on a lightweight convolutional pose machine includes the following steps:

[0057] A lightweight human pose estimation model based on a convolutional pose machine is constructed, incorporating various lightweight skeleton networks. Hyperparameters control the use of lightweight skeleton networks, including MobileNet and other types. A CPM (Continuous Pose Management) network is connected after the skeleton network, with the number of layers in the CPM optimization stage controlled by hyperparameters. The skeleton network is used to extract image features. Adjusting the downsampling of the skeleton network and the stride of the last few layers are adjusted according to the network type, controlling the input image to be downsampled by 8 times before being input into the CPM. The CPM optimization stage is used to predict heatmaps and PAFs (Pattern Indicators and Functions).

[0058] The L2 Loss function used in the original structure has been replaced with a hybrid loss function (L2 Loss and Focal Loss) instead of the original L2 Loss function. The Focal Loss function focuses the training on the samples that are difficult to train, and trains them intensively.

[0059] Let F represent the feature map extracted by the skeleton network. The CPM optimization has six stages, including one initial optimization stage and five optimization stages. Let Φ represent the feature map extracted by the skeleton network. t ρ t These represent the branch calculations used in the CPM optimization stage for optimizing the heatmap and optimizing PAFs, respectively, where t represents the stage number, and S is used. t L t These represent the heatmap and PAFs output after the CPM optimization stage, respectively. The calculations for the CPM optimization stage should satisfy L... t =Φ t (F,L t-1 ) and S t =ρ t (F,S t-1 ).

[0060] Downloading pre-trained model weights from learning framework websites such as the PyTorch official website and using these pre-trained weights to initialize the skeleton network saves training time and achieves better results compared to training from scratch.

[0061] The dataset used is the publicly available MSCOCO 2017 dataset provided by Microsoft. The training set contains approximately 120,000 images. Image augmentation was performed on the training dataset, including image rotation, translation, flipping, and random erasure to expand the dataset. The annotations were adapted for network training preprocessing. The validation set of the MSCOCO 2017 dataset contains 5,000 images. Since performing validation on 5,000 images each time during training is too time-consuming, a smaller validation set of 250 images and annotations was generated to monitor the trend of validation metrics during training.

[0062] The training set images were scaled or expanded proportionally to 368×368 for training, and the Adam optimizer was used for optimization. The initial learning rate for the skeleton network was 2e-4 (adjusted according to the selected skeleton network), and the learning rate for CPM was 4e-5. Learning rate decay strategies were set manually (to verify that mAP remained unchanged or decreased after 5 consecutive epochs) or automatically (halving the learning rate at 25, 35, 45, and 50 epochs) based on training performance, and different learning rate adjustment strategies were applied to the parameters of different network layers. Hyperparameters were continuously adjusted during training, and the number of CPM optimization stages was gradually increased from one to six. The feature maps extracted by the skeleton network and CPM were used to predict heatmaps and PAFs. The last few optimization stages did not significantly improve accuracy; therefore, only the training parameters of the first few layers were used after training. The removed last few layers often had a regularizing effect on the first few layers during training, preventing overfitting.

[0063] The greedy inference algorithm is used to assemble the predicted limb joints, draw a human posture skeleton diagram, and output human body part information including the number of people, category, presence of key points, key point coordinates, etc. A total of 19 key points are predicted, including nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, neck and background. There are 38 PAFs.

[0064] Two methods are proposed for detecting and classifying target body postures. The first method: Based on the angle correspondence between various body postures and the torso skeleton provided by a knowledge base, the user's body posture category is determined by calculating the angles between key skeletal nodes. The second method: An artificial neural network is constructed, with the activation function of the last layer set to the Softmax function, satisfying... The network is trained using pre-annotated images of various body postures. The detection information output from the previous stage (saving human key point information in JSON format) is used as the input to the artificial neural network. After the artificial neural network completes the classification, it infers the probability that the user belongs to each body posture.

[0065] The trained model is deployed on a web server using Flask, providing backend services. Through an open API, it can be called by various applications, including but not limited to apps, web pages, and WeChat mini-programs. The frontend sends the uploaded image, the model to be called, and other key parameters as JSON to the backend. After parsing the JSON, the backend sends the image to the model and infers human key points. The generated key point coordinate information is then fed into an artificial neural network to infer the probability information of the human body in the image belonging to various body postures. Finally, the key point coordinates and the probability information of belonging to various body postures are organized into JSON and returned to the frontend. The frontend processes and applies the returned information according to its needs.

[0066] like Figure 2 As shown (where the person in the picture is the inventor of this invention, used for research in this invention, and for rapid posture detection using the method of this invention), the lightweight convolutional pose machine of this invention can process images (such as...) Figure 2 It extracts the skeleton and quickly detects the body type, which can be user-defined, achieving rapid identification.

[0067] This example, by combining a lightweight network structure, addresses the problem of poor real-time performance and difficulty in practical applications caused by excessive model computation and parameter count in traditional HPE algorithms, while maintaining the good real-time performance and robustness of convolutional pose machines. Furthermore, it facilitates switching model training and application based on specific application scenarios by controlling the use of a lightweight skeleton network type through hyperparameter control. The mixed-precision training method significantly optimizes training speed and memory consumption. This invention can be easily applied to mobile or embedded devices via API calls, providing an efficient auxiliary means for body posture detection.

[0068] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a fast posture detection method based on a lightweight convolutional attitude machine, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0069] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0070] This invention provides a concept and method for rapid body posture detection based on a lightweight convolutional attitude machine. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for rapid body posture detection based on lightweight convolutional pose machine, characterized in that, The method comprises the following steps: Step 1, constructing a pose detection model, i.e. a convolutional pose machine (CPM) based on MobileNet; Step 2, loading the skeleton network MobileNet pre-training weight of the pose detection model, constructing a training data set and pre-processing the training data set, including rotation, translation, flipping and random erasing; Step 3, starting to train the pose detection model: adjusting the hyperparameters, gradually increasing the optimization stages of the convolutional pose machine (CPM) in the pose detection model, using the skeleton network to extract features to obtain a feature map, and predicting human key points according to the feature map and the convolutional pose machine (CPM); The optimization stages of the convolutional pose machine (CPM) in step 3 are six stages, including one initial optimization stage and five optimization stages, and the optimization stages of the convolutional pose machine (CPM) satisfy the following conditions: ； wherein F represents a feature map extracted through a skeleton network, Φ t and ρ t respectively represent branch calculation for optimizing a heat map and optimizing PAFs in an optimization stage of a convolution pose machine (CPM), wherein t represents a stage number, S t and L t respectively represent a heat map and a part affinity field (PAF) output through a CPM optimization stage; In the training of the pose detection model, the loss function used is a loss function composed of L2 Loss and Focal Loss, wherein L2 Loss and Focal Loss are as follows: ； where y i represents the sample output true value, n represents the total number of samples, f() represents the prediction function of the posture detection model, x i represents the sample input, p t represents the modulation factor, which is used to focus on difficult samples, and γ represents a parameter, and when γ is 0, it is a cross-entropy loss function. Step 4, using a greedy inference method to assemble the predicted human key points, drawing a human pose skeleton graph, and outputting human part information, wherein the human part information at least includes: the number of people, the category, whether there are key points and key point coordinates; Step 5, detecting the target body state and classifying by two methods; Step 6, repeating steps 3 to 5 to complete the training of the pose detection model; Step 7, using Flask to deploy the trained pose detection model on a server as a background service, opening an API interface, and applying the pose detection model to mobile terminals or embedded devices for fast body state detection based on a lightweight convolutional pose machine. 2.The body posture fast detection method based on the lightweight convolutional pose machine according to claim 1, characterized in that, The skeleton network in step 2 is aligned by modifying the last layer of the skeleton network. 3.The body posture fast detection method based on lightweight convolutional pose machine according to claim 1, wherein, The parameter γ in step 3 ranges from 0 to 5.

4. The body posture fast detection method based on lightweight convolutional pose machine according to claim 3, characterized in that, The two methods for detecting the target body state and classifying in step 5 specifically include: Method 1: according to the knowledge base, detecting the target body state and classifying by calculating the angle between the key point skeletons; Method 2: constructing an artificial neural network, setting the activation function of the last layer of the artificial neural network to a Softmax function, training the network using labeled multi-type body state data, inputting the human part information detected and outputted in step 4, and inferring the probability that the target belongs to each type of body state.

5. The method of claim 4, wherein the method is based on a lightweight convolutional pose machine. The knowledge base in step 5 is a pre-constructed knowledge base.

6. The method of claim 5, wherein the method is based on a lightweight convolutional pose machine. The probability that the target belongs to each type of body state in step 5 specifically includes: inputting the human part information obtained in step 4 into the artificial neural network, and obtaining the probability that the target belongs to each type of body state after classification by the artificial neural network.

7. The method of claim 6, wherein the method further comprises: The human part information obtained in step 4 is saved in JSON format.

8. The method of claim 7, wherein the method is based on a lightweight convolutional pose machine. In the API interface in step 7, the data exchange format is JSON.

Images