Lightweight human body posture estimation method based on Lite GS-HRNet

Abstract

The invention discloses a lightweight human body posture estimation method based on Lite GS-HRNet, and belongs to the technical field of computer vision posture estimation, and the method comprises the following steps: downloading a COCO data set; performing data preprocessing on the color human body images in the data set; the preprocessed image is input into a Lite GS-HRNet network for training; through the trained Lite GS-HRNet network, bone joint point information of the human body is output; according to the method, a Ghost module and a SimAM attention mechanism are combined, network parameters and calculation complexity are effectively reduced, high-precision attitude estimation is kept while real-time performance is ensured, the applicability of augmented reality and intelligent monitoring conditions needing real-time processing is improved, and the method is suitable for popularization and application. And the use accuracy of the mobile equipment, the edge calculation and the real-time processing application condition is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision posture estimation, and in particular to a lightweight human posture estimation method based on Lite GS-HRNet. Background Art

[0002] In the field of computer vision, HRNet (High-Resolution Network) is a high-resolution network that first uses 1 / 4 of the input image resolution as the initial input of the network, and then each step adds 1 / 2 of the previous lowest resolution as a new branch in parallel; high-resolution features contain more spatial detail information, which is crucial for accurately locating skeletal key points; low-resolution features contain more global context information, which helps the network understand the relative position and relationship of various parts of the human body; the core concept of HRNet is to improve the performance of the model by maintaining high-resolution features; in many visual tasks, especially those that require fine positioning and detail recognition, low-resolution features may lead to the loss of key information; HRNet aims to maintain high-resolution feature maps through a novel network architecture, while fusing features from different resolutions to enhance feature representation capabilities;

[0003] Most of the network models in the existing technical solutions are variants of the HRNet network model, typically the Small-HRNet and Lite-HRNet network models. The Small-HRNet network model introduces the Hourglass architecture based on the HRNet network model to reduce the number of network parameters, but the amount of computation of the network model has not changed significantly. On this basis, Lite-HRNet introduces the Shuffle Block to further reduce the number of network parameters, but the detection accuracy of the model decreases, and the amount of computation is still large, and it is impossible to strike a balance between the accuracy of the model and the amount of parameters and computation.

[0004] In summary, although the existing lightweight networks, such as Small-HRNet and Lite-HRNet, reduce the number of model parameters, the detection accuracy has decreased. As a result, the body posture estimation task often faces scenarios with limited computing resources, such as mobile devices, edge computing, and real-time processing application scenarios. The performance is poor. In order to solve this problem, the present application provides a lightweight human body posture estimation method based on Lite GS-HRNet. Summary of the invention

[0005] In order to address the deficiencies in the above-mentioned prior art, the purpose of the present invention is to provide a lightweight human posture estimation method based on Lite GS-HRNet, which combines the Ghost module and the SimAM attention mechanism, effectively reduces the network parameters and computational complexity, ensures that it maintains high-precision posture estimation while ensuring real-time performance, increases the applicability of augmented reality and intelligent monitoring situations that require real-time processing, and improves the accuracy of use of mobile devices, edge computing and real-time processing applications.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A lightweight human posture estimation method based on Lite GS-HRNet is provided, which includes the following steps:

[0008] First, the COCO dataset is downloaded, and then the color human image data in the dataset is preprocessed, including image translation, rotation, scaling, etc., to improve the generalization ability and robustness of the network; then the preprocessed image is input into Lite GS-HRNet for training. Lite GS-HRNet can effectively reduce the amount of calculation and parameters through the lightweight convolution operation of the Ghost module, but it will cause the feature extraction ability of the network to decrease. The present invention introduces the parameter-free SimAM attention mechanism for adjacent weight calculation, which can enhance the feature extraction ability of the network without significantly increasing the number of parameters and calculation of the network, and effectively reduce the computing resource overhead of the network while achieving a slight improvement in model accuracy; finally, the skeletal joint information of the human body is output through the trained Lite GS-HRNet network.

[0009] When implementing the Ghost lightweight convolution operation, in order to design a lightweight posture estimation network, this project introduced the lightweight module Ghost to replace the BasicBlock in Lite-HRNet; Ghost divides each convolution layer in the neural network into two parts: the first part is the regular convolution, which generates a partial feature map through a smaller convolution kernel; the second part generates an additional feature map based on the first part of the feature map, and then generates a complete feature map through a simple linear transformation; compared with conventional convolutional neural networks, the Ghost module effectively reduces the total number of required parameters and computational complexity without changing the size of the output feature map; the Ghost module is used to replace BaiscBlock to reduce the number of parameters of the deep network in a compressed manner, and the inter-layer coefficients need to be reconsidered to ensure the feature extraction capability of the network model.

[0010] When implementing SimAM adjacent weighted attention calculation, the feature map generation method using the Ghost module is effective, but it may lose some expressive ability in some complex tasks; this will lead to a slight decrease in performance when extracting skeletal joints; therefore, it is necessary to make up for the weakening of the representation ability of the Ghost module without increasing the number of parameters and calculations; this method introduces SimAM adjacent weight calculation; SimAM is a lightweight, parameter-free convolutional neural network attention mechanism, which generates attention weights by calculating the local self-similarity of feature maps; SimAM does not need to introduce any additional parameters and can effectively improve the performance of convolution.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] 1. The lightweight human posture estimation method based on Lite GS-HRNet of the present invention is exemplified. The Lite GS-HRNet can effectively reduce the amount of calculation and parameters through the lightweight convolution operation of the Ghost module, but the feature extraction ability of the network will decrease. By introducing the parameter-free SimAM attention mechanism for adjacent weight calculation, the feature extraction ability of the network can be enhanced without significantly increasing the number of network parameters and calculation. While achieving a slight improvement in model accuracy, the computing resource overhead of the network is effectively reduced to adapt to the limited computing resources of the device, reduce network parameters and computational complexity, and ensure that it maintains high-precision posture estimation while ensuring real-time performance.

[0013] 2. The lightweight human posture estimation method based on Lite GS-HRNet in the example of the present invention, Lite GS-HRNet combines the Ghost module and SimAM attention mechanism, effectively reduces the network parameters and computational complexity, while maintaining high-precision posture estimation, increasing the applicability of augmented reality and intelligent monitoring situations that require real-time processing, and improving the accuracy of use in mobile devices, edge computing, and real-time processing applications.

[0014] 3. The lightweight human posture estimation method based on Lite GS-HRNet in the example of the present invention has been experimentally verified on the COCO2017 dataset. Lite GS-HRNet has demonstrated its effectiveness in various tasks, providing a feasible solution for the industry and inspiring more researchers to explore and apply deep learning in the field of human posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0016] Figure 1A flow chart of human body posture estimation provided by an embodiment of the present invention;

[0017] Figure 2 Flowchart of the Ghost module and BasicBlock provided by the embodiment of the present invention;

[0018] Figure 3 This is a network structure diagram of Lite GS-HRNet provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0020] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention.

[0021] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0022] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0023] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings.

[0025] Although existing lightweight networks have reduced the number of parameters, they usually need to make a trade-off between performance and complexity; the industry often ignores the demand for computing resources in specific scenarios while pursuing performance; in current practical applications, the computing resources of human posture estimation equipment are often limited; the training and inference processes of complex models require more computing resources, which limits the space for model optimization; and the optimization of deep learning models often involves complex network structure adjustment, hyperparameter tuning and other processes, which requires a lot of experiments and computing resources; at the same time, for different data sets and tasks, the optimal network structure and parameter settings may be different, which increases the difficulty of optimization; the development of new technologies often requires a lot of experiments and verification, especially adaptability and stability testing for different scenarios; if the data set is not large or diverse enough, the model may not be able to fully learn all the necessary features, resulting in insufficient generalization ability.

[0026] It is a challenge to design a network with lower parameters and computational complexity while not reducing the accuracy of human posture. This requires a deep understanding of deep learning models and a large number of experiments to verify the performance of different structures. Finding more efficient and accurate feature extraction methods is also a daunting task. Traditional linear extraction methods may not be able to fully capture the complex features in the image, so more advanced feature extraction techniques need to be introduced, such as attention mechanisms.

[0027] The goal of this invention is to develop a lightweight model suitable for these resource-constrained scenarios, ensuring that it still has high posture estimation accuracy while ensuring real-time performance; in order to overcome the defects of Lite-HRNet, such as large amount of computation and parameters, and insufficient feature expression ability after lightweighting, this invention further optimizes feature extraction and utilization efficiency by introducing Ghost module and SimAM attention mechanism; Ghost module reduces redundant computation, and SimAM attention mechanism helps the model to more effectively concentrate on key features, and the specific method is as follows:

[0028] like Figure 1 As shown in the figure, the human pose estimation process is shown; first download the COCO dataset, then preprocess the color human image data in the dataset. First, the color image size is cropped to 256×192 and the image is translated, rotated, and scaled to improve the generalization ability and robustness of the network. The detailed operations of data preprocessing are as follows:

[0029] Random rotation: Each image will be randomly rotated by an angle between -30° and 30°. This transformation helps the model learn to recognize the same object in different orientations.

[0030] Random scaling: The image size will be randomly adjusted within a certain range. Specifically, the scaling ratio is set from 0.75 to 1.5. This operation enhances the model's adaptability to changes in object size.

[0031] Random translation: The image will be randomly shifted horizontally or vertically, with the range set between -40 pixels and +40 pixels. This helps the model learn to recognize partially occluded or offset objects.

[0032] Random flip: The image will be flipped horizontally with a certain probability (such as 50%). This operation can significantly expand the diversity of the dataset, especially when the object appearance has left-right symmetry.

[0033] Through the above series of data enhancement processing, the present invention can significantly improve the stability and accuracy of the model when facing unknown or complex inputs, thereby enhancing its robustness and generalization ability. These operations not only enrich the diversity of training data, but also effectively prevent the overfitting phenomenon of the model during the training process.

[0034] The preprocessed images are then input into Lite GS-HRNet for training. Lite GS-HRNet can effectively reduce the amount of calculation and parameters through the lightweight convolution operation of the Ghost module, but the feature extraction of the network will be reduced. By introducing the parameter-free SimAM attention mechanism for adjacent weight calculation, the feature extraction capability of the network can be enhanced without significantly increasing the number of network parameters and calculations. While achieving a slight improvement in model accuracy, the computing resource overhead of the network is effectively reduced. Finally, the trained Lite GS-HRNet model generates and outputs 17 predicted heat maps of human key points.

[0035] like Figure 2As shown in the figure, the calculation process of BasicBlock and Ghost modules is shown; in order to design a lightweight posture estimation network, this project introduces a lightweight module Ghost to replace BasicBlock in Lite-HRNet. Compared with the traditional convolutional structure, the advantage of Basicblock is that it adds a shortcut branch to pass the input information directly to the output, which can solve the training difficulties caused by the increase in network depth. The main structure of Basicblock consists of two 3×3 convolutions. In Lite-HRNet, for networks that require multi-scale feature representation, BasicBlock modules are often reused and stacked. For example, a deep network may contain dozens or even hundreds of BasicBlocks. Therefore, this structure causes the Lite-HRNet network to require a large number of parameters and calculations. Compared with BasicBlock, the Ghost module can significantly reduce the amount of calculation and parameters, mainly due to its unique feature generation mechanism. The Ghost module generates feature maps in two steps: first, a small number of convolution kernels are used to generate preliminary intrinsic maps (Primary Feature Maps), which contain the main features of the input data, and then the remaining virtual feature maps (Ghost Feature Maps) are generated through cheap operations (such as depthwise separable convolution). These Ghost maps may be visually similar to the intrinsic maps, but they contain additional information that helps to enhance the representation ability of the model. Finally, the module concatenates the generated intrinsic maps and Ghost maps to obtain the final output feature maps. These output feature maps contain the main features of the input data and add additional information through the Ghost maps, thereby improving the performance of the model. Therefore, the storage space and computational complexity of the model can be reduced while ensuring the performance of the model. The parameter ratio of the Ghost module to the traditional convolution is r P and the computational cost ratio G The formula can be defined mathematically as:

[0036]

[0037]

[0038] Among them, c and n are the number of channels of the input and output feature maps respectively; h′ and w′ are the height and width of the output feature map respectively; k is the size of the traditional convolution kernel, d is the size of the linear operation convolution kernel in the Ghost module, and d≈k; s is the number of redundant features corresponding to each basic feature map in the output feature map, and s<<c; the numerator is the number of parameters and calculations of the traditional convolution, and the denominator is the number of parameters and calculations of the Ghost module. Compared with the traditional convolution module, the Ghost module has lower parameters and calculations, so it can effectively improve the problem of excessive parameter calculations in the backbone network.

[0039] Although the feature map generation method using the Ghost module is effective, it may lose some expressive power in some complex tasks. This will cause a slight decrease in performance when extracting skeletal joints. Therefore, it is necessary to make up for the weakening of the feature expression ability of the Ghost module without increasing the number of parameters and the amount of calculation. The present invention introduces SimAM adjacent weight calculation. SimAM is a lightweight, parameter-free convolutional neural network attention mechanism, which generates attention weights by calculating the local self-similarity of the feature map. SimAM does not need to introduce any additional parameters and can effectively improve the performance of CNN. The SimAM module consists of three steps: calculating local self-similarity, determining attention weights and feature map refinement.

[0040] Calculate local self-similarity: SimAM first calculates the similarity between each pixel in the feature map and its neighboring pixels. This is usually achieved by calculating the negative value of the Euclidean distance between pixel features, because the smaller the distance, the higher the similarity. The calculation formula is as follows:

[0041]

[0042] Determine attention weight: Based on local self-similarity, SimAM assigns an attention weight to each pixel. This weight reflects the importance of the pixel in the feature map. The calculation formula of attention weight is as follows:

[0043]

[0044] Feature map refinement: The original feature map is refined using the calculated attention weights, that is, the weight of each pixel is adjusted according to its importance. This is usually achieved through a scaling operation instead of a simple addition to keep the relative importance of the feature map unchanged.

[0045] The SimAM attention mechanism is applied to the new feature maps generated after each fusion in the network to optimize the new feature maps after each fusion. These new feature maps highlight important feature areas and suppress insignificant feature areas. This can help the model focus more on the key features of the image and ignore irrelevant information, thereby improving the recognition accuracy and performance of the model.

[0046] like Figure 3As shown in the figure, the network structure design of Lite GS-HRNet is demonstrated; first, stage 1 uses 1 / 4 of the input image resolution as the initial input of the network, and then the stage formation process is to add 1 / 2 of the previous lowest resolution as a new branch in parallel. High-resolution features contain more spatial detail information, which is crucial for accurately locating bone key points; low-resolution features contain more global context information, which helps the network understand the relative position and relationship of various parts of the human body. Through this network structure, feature information of different resolutions can be fused with each other, fully utilizing and integrating multi-scale information, improving the network's representation ability, and making Lite GS-HRNet perform well in position-sensitive human bone key point detection tasks.

[0047] The experiment was completed on the Windows 10 operating system, and the Python 3.6 environment was configured. The PyTorch 1.7.0 + cu110 deep learning framework was used, and the Nvidia RTX4060 graphics card was equipped. The dataset used the COCO2017 dataset for training and testing, and visualization experiments were conducted on this basis. The COCO2017 dataset is divided into three subsets: the training dataset train2017 containing 57,000 images and 15,000 person instances, the validation dataset val2017 containing 5,000 images, and the test dataset test-dev2017 containing 20,000 images. Each human body in the dataset has 17 skeletal key points, which are the nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles. This article only focuses on two sizes of images: 256×192 and 384×288, and conducts experiments on them. Lite GS-HRNet was trained on the train2017 dataset, and the model was optimized using the Adam optimizer with an initial learning rate of 2e-3. The total training epochs were 210, and it was tested on the test-dev2017 dataset.

[0048] mAP (mean average precision), AR (average recall), Params (parameters) and FLOPs (floating point operations) are used as evaluation indicators of the experiment. mAP and AR are based on object keypoint similarity (OKS), and OKS is expressed as:

[0049]

[0050] AP means in T OKSThe average precision at 0.50, 0.55, 0.60, ..., 0.90, and 0.95 respectively. mAP is a commonly used evaluation indicator in target detection and pose estimation tasks, which is used to measure the accuracy of the model in detecting or identifying targets. Specifically, mAP is the average precision (AP) of each category calculated at different thresholds, and then the average of all categories is taken. AP 50 and AP 75 T OKS =0.50 and T OKS = 0.75. In addition, AP M represents the detection accuracy of medium-sized targets, while AP L They focus on the detection accuracy of large-scale targets, which helps to analyze the performance of the model on targets of different scales. OKS The average recall rates are 0.50, 0.55, 0.60, …, 0.90, and 0.95 respectively.

[0051] On the other hand, in order to quantitatively evaluate the lightweight degree of the model, the number of parameters (Params) and floating-point operations (FLOPs) in the network are used as evaluation indicators. These two indicators can effectively evaluate the complexity and computational efficiency of the network, providing important references for model optimization and practical applications.

[0052] Table 1

[0053]

[0054]

[0055] This paper fully verifies the performance of the Lite GS-HRNet network by designing comparative experiments. Compared with the commonly used lightweight networks MobileNetV2 and ShuffleNetV2, Lite GS-HRNet has improved the detection accuracy by 0.6 and 5.3, but the number of parameters is much smaller than that of the two networks. This shows that even with smaller computing resources, Lite GS-HRNet can still achieve higher detection accuracy. Compared with the lightweight high-resolution network Lite-HRNet-18, the detection accuracy of the model is improved by 0.4, but the proposed Lite GS-HRNet method has reduced the number of parameters. This means that by using Lite GS-HRNet, the amount of calculation and parameters can be reduced.

[0056] In addition, the present invention sets up an ablation experiment to verify the influence of each module on the present method. The ablation experiment is shown in Table 2:

[0057] Table 2

[0058]

[0059] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

[0060] Except for the technical features described in the specification, the remaining technical features are known technologies to those skilled in the art. In order to highlight the innovative features of the present invention, the remaining technical features will not be described here in detail.

Claims

1. A lightweight human posture estimation method based on Lite GS-HRNet, comprising the following steps: Step 1: Download the COCO dataset. Each color image of a person in the dataset contains a human key point label. The total number of all relevant nodes in each human image is less than or equal to the total number of human joints, which is 17. Step 2, preprocessing the color human image size in the data set; Step 3: Input the preprocessed images into the Lite GS-HRNet network for training; Step 4: Output the human body’s skeletal joint information through the trained Lite GS-HRNet network.

2. The lightweight human posture estimation method based on Lite GS-HRNet according to claim 1, characterized in that: In the step 2, data preprocessing first adjusts the color images in the training set, validation set and test set to make the size of each image 256×192. In addition, it also includes random angle rotation, random scale scaling and random direction flipping of the images. The data preprocessing aims to simulate various deformation conditions that images may encounter in the real world, thereby improving the generalization ability and robustness of the network.

3. The lightweight human posture estimation method based on Lite GS-HRNet according to claim 1, characterized in that: In step 3, the steps for implementing the network structure of Lite GS-HRNet are as follows: Step 1, stage 1 uses 1 / 4 of the input image resolution as the initial input of the network; Step 2, the subsequent stage formation process is to add 1 / 2 of the lowest resolution of the previous branch as a new branch in parallel. In step three, each branch performs a feature fusion process to integrate features from different branches and generate a feature map. Step 4. Finally, the feature maps after fusion of all branches are used to generate a heat map of key points. The predicted position of each key point is extracted from the generated heat map, and the key point coordinates are determined by finding the maximum value in the heat map.

4. The lightweight human posture estimation method based on Lite GS-HRNet according to claim 1, characterized in that: The Lite GS-HRNet network uses lightweight convolution operations of the Ghost module.

5. The lightweight human posture estimation method based on Lite GS-HRNet according to claim 4, characterized in that: The Lite GS-HRNet network performs adjacent weight calculation by introducing the parameter-free SimAM (Similarity-based Attention Module) attention mechanism.

6. The lightweight human posture estimation method based on Lite GS-HRNet according to claim 4, characterized in that: The Ghost module divides each convolutional layer in the neural network into two parts: the Ghost module first performs partial ordinary convolution to obtain an intrinsic feature map, then performs point-by-point convolution on the intrinsic feature map, and finally concatenates the feature maps through cheap operations (linear operations) to reduce the number of channels, the number of parameters and the amount of calculation, thereby reducing memory overhead.

7. The lightweight human posture estimation method based on Lite GS-HRNet according to claim 5, characterized in that: The SimAM attention mechanism generates attention weights by calculating the local self-similarity of feature maps. Regenerate the feature map based on the attention weight to improve the expressiveness of the features. This method can help the model focus more on the key areas of the image and ignore irrelevant information, thereby improving the recognition accuracy and performance of the model.

8. The lightweight human posture estimation method based on Lite GS-HRNet according to claim 1, characterized in that: In the step three, the Lite GS-HRNet network is trained and tested using the COCO2017 dataset and a visualization experiment is performed; the COCO2017 dataset is divided into three subsets: a training dataset train2017 containing 57,000 images and 15,000 person instances, a validation dataset val2017 containing 5,000 images, and a test dataset test-dev2017 containing 20,000 images; each human body in the dataset has 17 skeletal key points, namely, nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles.

Images