Lightweight model building method and system for detecting key points of human body
By adopting a lightweight model construction method in human posture estimation, using the improved LiteHRNet backbone network and extrusion excitation module, combined with the head function based on the convolutional layer and a specific training strategy, the problems of low accuracy and difficulty in running on ordinary devices in the existing technology are solved, and efficient detection of key points in humans are achieved.
Patent Information
- Application Number
- CN202411720613.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-03-18
AI Technical Summary
The existing human pose estimation methods rely on convolutional neural networks, which are difficult to capture global context information in images, and are difficult to achieve high accuracy with very few parameters, and are difficult to run on ordinary devices.
Using the lightweight model construction method, the real-time detector RTMDet or the improved LiteHRNet are selected as the backbone network, and an extrusion excitation module is introduced between deep convolution and point-by-point convolution. The model is trained by building a convolutional layer-based head functions and using training strategies such as unbiased data processing, cosine annealing and two-stage training strategies.
While maintaining the inference speed, it achieves accuracy comparable to existing advanced methods, solving the problem of small amount of parameters and difficulty in running on ordinary devices.
Smart Images
Figure CN119649188A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of human body posture estimation, and in particular to a lightweight model building method and system for detecting key points of a human body. Background Art
[0002] Human pose estimation technology is an analysis method used in the field of computer vision to analyze human actions, postures, and motion states from images or videos. It builds a mathematical model of human structure and dynamics by identifying specific key points of the human body in the image, such as the endpoints of joints and limbs. This model can describe the position of various parts of the human body and how they move in space.
[0003] This technology can provide deep insights into human behavior because it can quantify how the body moves. By analyzing consecutive frames, pose estimation can track changes in human movements over time, which is very useful for analyzing complex movements and predicting movement trends.
[0004] The application of human posture estimation technology is very extensive. It can be used in many scenarios, such as motion analysis, health monitoring, human-computer interaction and intelligent monitoring. In these scenarios, posture estimation can be used as a tool to improve automation and efficiency, while also providing a new perspective for understanding human behavior.
[0005] The most mainstream solutions in the field of human posture estimation are top-down and bottom-up. For the top-down method, engineers need to first select a suitable human detector to detect a single human in the image, then build a backbone network to extract features related to the key points of the human body in the image, then select a suitable head function to convert the features into key points, and then use the data set to train the backbone network and head function, and finally deploy the trained model on a suitable platform to achieve real-time or near real-time application requirements. For the bottom-up method, engineers do not need to use a human detector to extract a single human in the image, but instead build a high-performance backbone network to directly extract the features of all human bodies in the image, and then use the head function to process these features to output the key points of all human bodies in the image, and then use the data set to train the backbone network and head function, and finally deploy the trained model on a suitable platform.
[0006] Both the top-down and bottom-up methods rely heavily on convolutional neural networks. When processing images, convolutional neural networks mainly rely on local receptive fields, which may not be enough to capture the global contextual information in the image. Moreover, it is difficult for mainstream models for detecting key points of the human body to achieve high accuracy with very few parameters.
[0007] In recent years, the emergence of large models has also brought about major changes in the field of human posture estimation. Large models used to detect key points of the human body generally stack a large number of self-attention modules and use ultra-large data sets to train the models. Therefore, although large models can achieve high accuracy, they require a lot of computing power support and are difficult to run on ordinary devices. For some scenarios that require timely output of results, large models are also difficult to use. Summary of the invention
[0008] In view of the above-mentioned problems, the present invention is proposed.
[0009] Therefore, the technical problem solved by the present invention is that the existing human posture estimation method relies on convolutional neural networks, which is not sufficient to capture the global contextual information in the image, and it is difficult to achieve high accuracy with very few parameters, and it is difficult to run on ordinary devices.
[0010] To solve the above technical problems, the present invention provides the following technical solutions: a lightweight model building method for detecting key points of the human body, comprising selecting a human body detector to build a backbone network; building a head function based on a convolutional layer; and training the model through a training strategy.
[0011] As a preferred solution of the lightweight model building method for detecting key points of the human body described in the present invention, wherein: the selection of a human body detector to build a backbone network includes selecting a real-time detector RTMDet.
[0012] As a preferred solution of the lightweight model building method for detecting key points of the human body described in the present invention, the selection of a human body detector to build a backbone network includes selecting LiteHRNet as a base, modifying the Separable Conv module in LiteHRNet, and introducing a squeezing excitation module between the deep convolution and the point-by-point convolution.
[0013] As a preferred solution of the lightweight model building method for detecting key points of the human body described in the present invention, wherein: the selection of the human body detector to build a backbone network includes a squeezing excitation module and includes two steps:
[0014] Compress each channel of the feature map output by the deep convolution into a single value to obtain the overall activation strength of each channel;
[0015] Use two layers of 1×1 convolution to simulate the fully connected layer and then use the activation function ReLU and h-sigmoid to learn a weight vector;
[0016] The first fully connected layer maps the compressed global feature vector to a low-dimensional space, and the second fully connected layer maps it back to the space with the original number of channels.
[0017] As a preferred solution of the lightweight model building method for detecting key points of the human body described in the present invention, the head function constructed based on the convolutional layer includes constructing a convolutional layer, which converts the feature map output by the backbone network into a heat map of key points of the human body, and extracts the maximum value point in each heat map as the coordinate of the key point.
[0018] As a preferred solution of the lightweight model building method for detecting key points of the human body described in the present invention, the training of the model through the training strategy includes selecting COCO2017 as the training data set, adopting unbiased data processing as the data processing strategy, setting the initial learning rate to 4e-3, 210 epoches, the first 105 epoches use Linear to adjust the learning rate for warm-up, and the last 105 epoches use the cosine annealing strategy.
[0019] As a preferred solution of the lightweight model building method for detecting key points of the human body described in the present invention, wherein: the training of the model by the training strategy includes using a two-stage training strategy, in the first 170 epoches, the image scaling range is set to [0.6, 1.4], and the random rotation factor is set to 80;
[0020] In the next 40 epoches, we reduce the random rotation factor to 60 and the probability of CoarseDropout to 0.5.
[0021] Another object of the present invention is to provide a lightweight model building system for detecting key points of the human body, which can solve the problem of low accuracy of current human posture estimation methods by constructing a head function based on a convolutional layer.
[0022] As a preferred solution of the lightweight model building system for detecting key points of the human body described in the present invention, it includes: a backbone network construction module, a head function construction module, and a model training module; the backbone network construction module is used to select a human detector to build a backbone network; the head function construction module is used to construct a heat map of key points of the human body; and the model training module is used to train the model using a training strategy.
[0023] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a lightweight model building method for detecting key points of a human body.
[0024] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a lightweight model building method for detecting key points of a human body.
[0025] Beneficial effects of the present invention: The lightweight model building method for detecting key points of the human body provided by the present invention selects a lightweight real-time detector. The lightweight model improves Lite-HRNet and will serve as its own backbone network. The lightweight model introduces a squeeze excitation module between deep convolution and point-by-point convolution to improve the model's ability to extract information between channels and improve Lite-HRNet performance. The lightweight model uses unbiased data processing to reduce model inference errors, and uses cosine annealing and a two-stage training strategy to improve the model's learning ability. The lightweight model achieves an accuracy rate comparable to existing advanced methods while maintaining the inference speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0027] Figure 1 An overall flow chart of a lightweight model building method for detecting key points of a human body provided in the first embodiment of the present invention.
[0028] Figure 2 A Lite-HRNet structure diagram of a lightweight model building method for detecting key points of the human body provided in the second embodiment of the present invention.
[0029] Figure 3 A deep convolutional graph of a lightweight model building method for detecting key points of the human body provided in the second embodiment of the present invention.
[0030] Figure 4 A schematic diagram of an extrusion excitation module of a lightweight model building method for detecting key points of a human body provided in a second embodiment of the present invention.
[0031] Figure 5 An overall flow chart of a lightweight model building system for detecting key points of a human body provided in the third embodiment of the present invention. DETAILED DESCRIPTION
[0032] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0033] Example 1, reference Figure 1-Figure 4 , as an embodiment of the present invention, provides a lightweight model building method for detecting key points of a human body, comprising:
[0034] S1: Select a human detector to build a backbone network.
[0035] Going further, choose a human detector: choose a lightweight real-time detector - RTMDet.
[0036] It should be noted that the backbone network was constructed: LiteHRNet was selected as the basis, and the following modifications were made to the Separable Conv module in LiteHRNet: a squeeze excitation module was introduced between the deep convolution and the point-by-point convolution. The squeeze excitation module has two steps. First, each channel of the feature map output by the deep convolution is compressed into a separate value to obtain the overall activation intensity of each channel. This is equivalent to creating a global feature vector with a length equal to the number of input channels, and each element represents the global information of the corresponding channel. Second, a weight vector is learned by simulating a fully connected layer through two layers of 1×1 convolution and then using the activation functions ReLU and h-sigmoid. The first fully connected layer maps the compressed global feature vector to a low-dimensional space, and the second fully connected layer maps it back to the space of the original number of channels.
[0037] Furthermore, the structure of Lite-HRNet is as follows Figure 2 As shown in Figure 1, the Separable Conv module consists of a depthwise convolution and a point-by-point convolution.
[0038] S2: Build the head function based on the convolutional layer.
[0039] Furthermore, we construct the head function: first, we construct a 1×1 convolutional layer with a weight tensor of 17×40×1×1. This convolutional layer converts the feature map output by the backbone network into a heat map corresponding to the 17 key points of the human body. Finally, we extract the maximum value point in each heat map and use it as the coordinate of the corresponding key point.
[0040] It should be noted that depthwise convolution is used to extract spatial features, and point-by-point convolution is used to adjust the number of channels, such as Figure 3 As shown in the figure, the Separable Conv module can significantly reduce the number of parameters, but it also has disadvantages. Since the depthwise separable convolution performs spatial convolution and pointwise convolution separately, this limits the model's ability to learn both spatial features and channel features in a single convolution step. In order to improve the model's ability to extract inter-channel information, this lightweight model introduces a squeeze excitation module between the depthwise convolution and the pointwise convolution, as shown in the figure. Figure 4As shown in Figure 2, the squeeze-excitation module adaptively recalibrates the responses of feature channels by explicitly modeling the interdependencies between channels. This improves the performance of Lite-HRNet.
[0041] S3: Train the model using the training strategy.
[0042] Furthermore, a series of training strategies are used to train the model: COCO2017 is selected as the training dataset, unbiased data processing (UDP) is used as the data processing strategy, the initial learning rate is set to 4e-3, 210 epoches, the first 105 epoches use Linear to adjust the learning rate for warm-up, and the last 105 epoches use the cosine annealing strategy. Using a two-stage training strategy, in the first 170 epoches, the image scaling range is set to [0.6, 1.4], and the random rotation factor is set to 80. In the last 40 epoches, the random rotation factor is reduced to 60, and the probability of CoarseDropout is reduced to 0.5.
[0043] Example 2, an embodiment of the present invention, provides a lightweight model building method for detecting key points of the human body. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.
[0044] First, an ablation experiment and a comparative experiment are conducted. The ablation experiment is used to prove that the improvement of this model on LiteHRNet is effective, and the comparative experiment is used to prove that this model is superior to similar models. (2) Dataset: The COCO dataset is widely regarded as the most popular benchmark dataset for human pose estimation tasks. In the experiment, the 2017 version of the training set (train2017) and the validation set (val2017) are used; (3) Training environment and configuration: The experiment is conducted on an Nvidia RTX 4090 graphics card. The relevant parameter settings of the ablation experiment are shown in Table 1. The relevant parameter settings of the comparative experiment are shown in Table 2. (4) The final model is compared with the advanced methods in the current field. The experimental results are shown in Table 3. These models include a variety of different architectures and methods, which have achieved remarkable results in pose estimation tasks. By comparing with these models, the performance of the model can be evaluated more accurately.
[0045] Table 1
[0046] parameter set up optimizer Adam baselearningrate 0.0005 learningrateschedule MultiStep batchsize 128 warm-upiterations 500 trainingepochs 210
[0047] Table 2
[0048] parameter set up optimizer Adam baselearningrate 0.0004 learningrateschedule Flat-Cosine batchsize 128 warm-upiterations 1000 trainingepochs 210
[0049] Table 3
[0050]
[0051]
[0052] Example 3, reference Figure 5 , as an embodiment of the present invention, provides a lightweight model building system for detecting key points of the human body, including a backbone network building module, a head function building module, and a model training module.
[0053] The backbone network construction module is used to select a human detector to build a backbone network, the head function construction module is used to build a human key point heat map, and the model training module is used to train the model using a training strategy.
[0054] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0055] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0056] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.
[0057] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.
[0058] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A lightweight model building method for detecting key points of the human body, characterized in that: include: Select human detector to build the backbone network; Build the head function based on the convolutional layer; The model is trained using a training strategy.
2. The lightweight model building method for detecting key points of the human body according to claim 1, characterized in that: The selecting of a human detector to construct a backbone network includes selecting a real-time detector RTMDet.
3. The lightweight model building method for detecting key points of the human body as claimed in claim 2, characterized in that: The selection of a human detector to construct a backbone network includes selecting LiteHRNet as a base, modifying the SeparableConv module in LiteHRNet, and introducing a squeeze excitation module between the depth convolution and the point-by-point convolution.
4. The lightweight model building method for detecting key points of the human body as claimed in claim 3, characterized in that: The method of selecting a human detector to construct a backbone network including a squeeze excitation module includes two steps: Compress each channel of the feature map output by the deep convolution into a single value to obtain the overall activation strength of each channel; Use two layers of 1×1 convolution to simulate the fully connected layer and then use the activation function ReLU and h-sigmoid to learn a weight vector; The first fully connected layer maps the compressed global feature vector to a low-dimensional space, and the second fully connected layer maps it back to the space with the original number of channels.
5. The lightweight model building method for detecting key points of the human body according to claim 4, characterized in that: The convolutional layer-based construction of the head function includes constructing a convolutional layer, which converts the feature map output by the backbone network into a human body key point heat map, and extracts the maximum value point in each heat map as the coordinate of the key point.
6. The lightweight model building method for detecting key points of the human body according to claim 5, characterized in that: The training of the model through the training strategy includes selecting COCO2017 as the training data set, adopting unbiased data processing as the data processing strategy, setting the initial learning rate to 4e-3, 210 epoches, using Linear to adjust the learning rate for warm-up in the first 105 epoches, and using the cosine annealing strategy in the last 105 epoches.
7. The lightweight model building method for detecting key points of a human body according to claim 6, characterized in that: The training of the model by the training strategy includes using a two-stage training strategy, in which the image scaling range is set to [0.6, 1.4] and the random rotation factor is set to 80 in the first 170 epoches; In the next 40 epoches, we reduce the random rotation factor to 60 and the probability of CoarseDropout to 0.
5.
8. A system using the lightweight model building method for detecting key points of a human body as claimed in any one of claims 1 to 7, characterized in that: Including backbone network building module, head function building module, model training module; The backbone network construction module is used to select a human detector to construct a backbone network; The head function building module is used to build a human body key point heat map; The model training module is used to train the model using a training strategy.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the lightweight model building method for detecting key points of the human body described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the lightweight model building method for detecting key points of the human body described in any one of claims 1 to 7 are implemented.