A method and system for real-time human pose estimation on mobile devices

Abstract

This invention relates to the field of posture detection technology, providing a method and system for real-time human posture estimation on a mobile device. The method includes: step 100, acquiring detection images using an image acquisition device; step 200, inputting the detection images into the backbone network of a posture recognition model for feature extraction to obtain a feature map; wherein the posture recognition model includes: a backbone network, multiple inverted convolutional layers, and one convolutional layer; the backbone network adopts a MobileViT network; step 300, sequentially inputting the feature map output by the backbone network into the multiple inverted convolutional layers and the convolutional layer to obtain a heatmap of human key points; step 400, predicting the final human key point information based on the heatmap to achieve human posture estimation. This invention ensures the lightweight nature of the network, significantly reduces the number of model parameters and computational load, and improves detection speed.

Description

Technical Field

[0001] This invention relates to the field of posture detection technology, and in particular to a method and system for real-time human posture estimation on a mobile device. Background Technology

[0002] Human pose estimation (HPE) is a crucial task in computer vision. Its core objective is to detect the joint information of the human body, such as hands, shoulders, and hips, in videos or images, laying the groundwork for many other tasks. HPE is increasingly used in smart communities, secure campuses, and AI-powered interactive games. Improving the speed and accuracy of HPE is a critical task to ensure speed and accuracy across various scenarios.

[0003] In the past decade, deep learning has flourished and has been widely applied to 2D human pose estimation. Well-known and effective human pose keypoint algorithms include the SimpleBaseline method based on ResNet as the backbone, and HRNet, which maintains multi-resolution parallelism, but the multi-scale feature fusion of parallel subnetworks makes it very slow.

[0004] Current 2D human pose estimation primarily employs traditional pure convolutional networks, which suffer from problems such as large parameter count, computational complexity, and slow inference speed, especially in scenarios with limited computing resources, such as mobile devices. When deploying 2D pose estimation on mobile devices, achieving a balance between network performance and the deployment environment is difficult: ensuring high robustness and accuracy while maintaining high detection speed is challenging. Based on traditional computer vision techniques, achieving low power consumption and high accuracy on the latest mobile devices is difficult, making it hard to balance mobile device performance with network computational complexity.

[0005] In summary, existing mobile human pose estimation methods mainly use traditional pure convolutional networks, which cannot provide good robustness to complex scenes and make it difficult to achieve high-precision and high-speed real-time estimation of human pose. Summary of the Invention

[0006] This invention primarily addresses the technical problems of existing mobile human pose estimation methods, which mainly employ traditional pure convolutional networks. These methods lack robustness to complex scenes and struggle to achieve high-precision and high-speed real-time estimation of human pose. The invention proposes a real-time mobile human pose estimation method and system that ensures lightweight network design, significantly reduces the number of model parameters and computational load, and improves detection speed.

[0007] This invention provides a real-time human pose estimation method for mobile devices, comprising the following steps:

[0008] Step 100: Acquire detection images using an image acquisition device;

[0009] Step 200: Input the detected image into the backbone network of the pose recognition model for feature extraction to obtain a feature map; wherein, the pose recognition model includes: a backbone network, multiple inverted convolutional layers and one convolutional layer; the backbone network adopts the MobileViT network;

[0010] Step 300: Input the feature map output by the backbone network into multiple inverted convolutional layers and one convolutional layer in sequence to obtain a heat map of human key points.

[0011] Step 400: Based on the heat map, predict the final key point information of the human body to achieve human posture estimation.

[0012] Preferably, the MobileViT network includes: a 3*3 convolutional module, a Mobilenetv2 module, a Mobilenetv2 module, a Mobilenetv2 module, a Mobilenetv2 module, a MobileViT Block module, a Mobilenetv2 module, a MobileViT Block module, a Mobilenetv2 module, and a MobileViT Block module arranged sequentially.

[0013] Preferably, the 3*3 convolution module is downsampled by 2x, and the second, fifth, sixth and seventh Mobilenetv2 modules are also downsampled by 2x.

[0014] Preferably, the MobileViT Block module includes a Transformer Encoder module.

[0015] Preferably, the Transformer Encoder module has a self-attention mechanism;

[0016] The Transformer Encoder module uses a multi-head attention mechanism, and the output feature vector passes through two fully connected layers before undergoing layer normalization.

[0017] Preferably, the number of the inverted convolutional layers is three, and each convolutional layer is a 1*1 convolutional layer.

[0018] Correspondingly, the present invention also provides a real-time human pose estimation system for mobile devices, including: a data acquisition module, a feature extraction module, a reverse convolution processing module, and an output module;

[0019] The acquisition module is used to acquire and detect images using an image acquisition device;

[0020] The feature extraction module is used to input the detected image into the backbone network of the pose recognition model for feature extraction to obtain a feature map; wherein, the pose recognition model includes: a backbone network, multiple inverted convolutional layers and one convolutional layer; the backbone network adopts the MobileViT network;

[0021] The inverted convolution processing module is used to input the feature map output by the backbone network into multiple inverted convolution layers and one convolution layer in sequence to obtain a heat map of human key points.

[0022] The output module is used to predict the final human body key point information based on the heat map, thereby realizing human body posture estimation.

[0023] This invention provides a real-time human pose estimation method and system for mobile devices. It uses MobileViT as the backbone network to extract data. After five downsampling passes, three inverted convolutional layers and one 1x1 convolutional layer are added at the end of the network to obtain a heatmap of human key points. Based on the heatmap, the final human key point information is predicted, thus achieving human pose estimation on mobile devices. The pose recognition model used in this invention has a very simple network structure, directly generating the final heatmap through multiple inverted convolutions following a backbone network. The MobileViT network incorporates the depthwise separable convolutions of the lightweight Mobilenetv2 module, significantly reducing the number of model parameters and computational cost, ensuring network lightweightness, and greatly improving detection speed. It is a network structure specifically designed for mobile scenarios, achieving real-time performance on mobile devices and facilitating subsequent deployment of the network model on mobile devices. This solves the previous problem of balancing mobile device performance and network computational complexity, effectively improving work efficiency. Furthermore, the network includes a Transformer Encoder module, which better captures global information and rich contextual information, possessing stronger dynamic computing capabilities, more powerful learning and modeling abilities, and the ability to learn hidden features. This further improves the model's robustness to multiple environments and its accuracy in human pose estimation, ensuring network lightweightness while learning better features. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the implementation of the real-time human pose estimation method for mobile devices provided by this invention.

[0025] Figure 2 This is a connection block diagram of the posture recognition model used in this invention;

[0026] Figure 3 This is a connection block diagram of the MobileViT network used in this invention. Detailed Implementation

[0027] To make the technical problems solved by this invention, the technical solutions adopted, and the technical effects achieved clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings, not all of them.

[0028] Example 1

[0029] like Figure 1 As shown, the real-time human pose estimation method for mobile devices provided in this embodiment of the invention includes the following process:

[0030] Step 100: Use an image acquisition device to acquire detection images.

[0031] The image acquisition device is not limited to mobile phones, high-speed action cameras, etc. The acquired images undergo data preprocessing.

[0032] Step 200: Input the detected image into the backbone network of the pose recognition model to extract features and obtain a feature map.

[0033] like Figure 2 As shown, the pose recognition model includes: a backbone network, multiple deconverged layers, and one convolutional layer. Specifically, there are three deconverged layers and one 1x1 convolutional layer.

[0034] The backbone network adopts a lightweight MobileViT network, which includes: a convolutional module, multiple Mobilenetv2 modules, and multiple MobileViT Block modules.

[0035] Specifically, such as Figure 3 As shown, the MobileViT network includes: a 3*3 convolutional module (conv-3*3), a Mobilenetv2 module, a Mobilenetv2 module, a Mobilenetv2 module, a Mobilenetv2 module, a Mobilenetv2 module, a MobileViT Block module, a Mobilenetv2 module, a MobileViT Block module, a Mobilenetv2 module, and a MobileViT Block module arranged sequentially.

[0036] The 3x3 convolutional module is downsampled by a factor of 2, and the second, fifth, sixth, and seventh MobileNetv2 modules are also downsampled by a factor of 2 to increase the receptive field. Because the MobileNetv2 modules have depthwise separable convolutions, the number of parameters and computational cost are very small, thus speeding up model computation.

[0037] Specifically, the process involves first performing a 3x3 convolution on the image and then downsampling it by a factor of 2. The image then passes through seven MobileNetv2 modules. Because MobileNetv2 modules have depthwise separable convolutions, they have very few parameters and low computational cost, thus accelerating model computation. The MobileViT network structure is clear, performing five downsampling operations on the image, reducing its height (H) and width (W) to 1 / 32 of their original values. As the network depth increases, the number of channels also increases.

[0038] The MobileViT Block module contains a classic Transformer Encoder module, and the Transformer Encoder module is repeated 2, 4 and 3 times respectively within the three MobileViT Block modules. After the detection image passes through the MobileViT Block module, the dimension of the feature map does not change, so that subsequent convolution processing can continue.

[0039] The core of the Transformer Encoder module is the self-attention mechanism, and the calculation formula for self-attention is as follows:

[0040]

[0041] Here, Z represents the output of the self-attention mechanism, which requires matrices Q (query), K (key), and V (value) for computation. The input to self-attention is represented by X, and matrices Q (query), K (key), and V (value) are obtained by performing linear transformations on the input. k This is the column number of matrices Q and K, i.e., the vector dimension. To prevent the inner product from becoming too large, it is divided by d. k The square root of the matrix. T represents matrix transpose, and B represents bias. Finally, Softmax is used to calculate the attention coefficients of each word for other words. Here, Softmax is applied to each row of the matrix, meaning the sum of each row becomes 1. The final output is the output Z of the self-attention mechanism.

[0042] The Transformer Encoder module internally uses a multi-head attention mechanism. The output feature vector passes through two fully connected layers and then undergoes LN (Layer Normalization) operation. This is a specific detail of the Transformer Encoder structure.

[0043] The backbone network employed in this invention incorporates, on one hand, the lightweight MobileNetv2 depthwise separable convolution module, which significantly reduces the number of model parameters and computational cost, ensuring a lightweight network structure specifically suited for mobile scenarios. On the other hand, the network includes a Transformer Encoder module, where the self-attention mechanism enables more robust feature learning and modeling, thereby enhancing the network model's robustness and performance. The Transformer Encoder module performs global feature processing; the use of depthwise separable convolution MobileNet-v2 modules further reduces the number of model parameters and computational complexity.

[0044] Step 300: Input the feature map output by the backbone network into multiple inverted convolutional layers and one convolutional layer in sequence to obtain a heat map of human body key points.

[0045] The deconv layers consist of three 1x1 convolutional layers, which produce heatmaps of key points on the human body.

[0046] Step 400: Based on the heat map, predict the final key point information of the human body to achieve human posture estimation.

[0047] The process involves traversing the heatmap to see if each local area matches the characteristics of a key point. The larger the calculated response, the more likely it is to be a key point. This process predicts the final key point information of the human body, which in turn reflects the human posture.

[0048] Specifically, the input detection image size is 256*192. After processing by the backbone, the final feature map size is 8*6. The convolutional kernels of the three deconv layers are all 4*4, and the final heatmap size is 64*48. The loss function used is L2 loss.

[0049] This invention provides a real-time human pose estimation method for mobile devices. It uses MobileViT as the backbone network to extract data. After five downsampling passes, three inverted convolutional layers and one 1x1 convolutional layer are added at the end of the network to obtain a heatmap of human key points. The final human key point information is predicted based on the heatmap. The overall network structure of the pose recognition model is very simple. The MobileViT network integrates a large number of cutting-edge computer vision technologies, improving network performance and enhancing model speed and deployment convenience. This invention, based on the MobileViT network, uses multiple inverted convolutions after the backbone network to generate a heatmap for human pose estimation. Simultaneously, due to the inclusion of a Transformer Encoder module, the network has strong learning capabilities, ensuring lightweight operation while learning better features. Because the backbone network uses a more powerful and lightweight network, the detection speed is greatly improved while maintaining human pose accuracy on mobile devices.

[0050] The pose recognition model for human pose estimation in this invention can be trained using a large amount of training data. The model file format is then converted to a format supported by mobile devices. If necessary, unsupported operators are addressed. Once the model conversion is successful, the converted model is verified to accurately obtain results from test images, thus completing the human pose estimation on mobile devices.

[0051] Example 2

[0052] The present invention also provides a real-time human pose estimation system for mobile devices, comprising: a data acquisition module, a feature extraction module, a reverse convolution processing module, and an output module;

[0053] The acquisition module is used to acquire and detect images using an image acquisition device;

[0054] The feature extraction module is used to input the detected image into the backbone network of the pose recognition model for feature extraction to obtain a feature map; wherein, the pose recognition model includes: a backbone network, multiple inverted convolutional layers and one convolutional layer; the backbone network adopts the MobileViT network;

[0055] The inverted convolution processing module is used to input the feature map output by the backbone network into multiple inverted convolution layers and one convolution layer in sequence to obtain a heat map of human key points.

[0056] The output module is used to predict the final human body key point information based on the heat map, thereby realizing human body posture estimation.

[0057] The mobile real-time human pose estimation system of this embodiment corresponds to the method of Embodiment 1 and can implement the method of Embodiment 1.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions for some or all of the technical features, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time human pose estimation method for mobile devices, characterized in that, Includes the following processes: Step 100: Acquire detection images using an image acquisition device; Step 200: Input the detected image into the backbone network of the pose recognition model for feature extraction to obtain a feature map; wherein, the pose recognition model includes: a backbone network, multiple inverted convolutional layers and one convolutional layer; the backbone network adopts the MobileViT network; the MobileViT network includes: a 3*3 convolutional module, a Mobilenetv2 module, a Mobilenetv2 module, a Mobilenetv2 module, a Mobilenetv2 module, a MobileViT Block module, a Mobilenetv2 module, a MobileViT Block module, a Mobilenetv2 module, a MobileViT Block module, a Mobilenetv2 module, a MobileViT Block module; the 3*3 convolutional module is downsampled by 2x, and the second, fifth, sixth and seventh Mobilenetv2 modules are also downsampled by 2x; the MobileViT Block module contains a TransformerEncoder module; the TransformerEncoder module has a self-attention mechanism; the Transformer... The Encoder module's output feature vector, after passing through a multi-head attention mechanism, goes through two fully connected layers and then undergoes layer normalization. The number of inverted convolutional layers is three, and each convolutional layer is a 1*1 convolutional layer. Step 300: Input the feature map output by the backbone network into multiple inverted convolutional layers and one convolutional layer in sequence to obtain a heat map of human key points. Step 400: Based on the heat map, predict the final key point information of the human body to achieve human posture estimation.

2. A real-time human pose estimation system for mobile devices, characterized in that, include: The module consists of an acquisition module, a feature extraction module, a reverse convolution processing module, and an output module. The acquisition module is used to acquire and detect images using an image acquisition device; The feature extraction module is used to input the detected image into the backbone network of the pose recognition model for feature extraction to obtain a feature map. The pose recognition model includes: a backbone network, multiple inverted convolutional layers, and one convolutional layer. The backbone network uses a MobileViT network. The MobileViT network includes: a 3x3 convolutional module, a Mobilenetv2 module, a Mobilenetv2 module, a Mobilenetv2 module, a Mobilenetv2 module, a MobileViT Block module, a Mobilenetv2 module, a MobileViT Block module, a Mobilenetv2 module, and a MobileViT Block module, arranged sequentially. The 3x3 convolutional module uses a 2x downsampling, and the second, fifth, sixth, and seventh Mobilenetv2 modules also use a 2x downsampling. The MobileViT Block module contains a Transformer Encoder module. The Transformer Encoder module has a self-attention mechanism. The Encoder module's output feature vector, after passing through a multi-head attention mechanism, goes through two fully connected layers and then undergoes layer normalization. The number of inverted convolutional layers is three, and each convolutional layer is a 1*1 convolutional layer. The inverted convolution processing module is used to input the feature map output by the backbone network into multiple inverted convolution layers and one convolution layer in sequence to obtain a heat map of human key points. The output module is used to predict the final human body key point information based on the heat map, thereby realizing human body posture estimation.

Images