Staggered alignment self-distillation training method for multi-person pose estimation

Abstract

Method for staggered alignment self-distillation training for multi-person pose recognition, comprising: constructing two identical pose estimation networks with shared weights for high-resolution tra

Description

The present invention relates to the field of target detection and pose estimation, specifically addressing an staggered alignment self-distillation training method for multi-person pose estimation. Background Technology Human pose estimation is a fundamental task in the field of computer vision, aiming to accurately predict the coordinates of key points of individuals in images. This task has become particularly important due to its widespread applications in action recognition, motion analysis, and various other computer vision systems. To enable a broader application of pose estimation methods, it is essential to lightweight these methods. Lightweight pose estimation models can operate on resource-constrained devices such as smartphones and embedded systems. This allows a more extensive user base to access advanced technologies, breaking free from the limitations of users with high-performance computing resources. Currently, regression-based pose estimation methods have potential advantages for lightweight models as they do not require the generation of large-sized heatmaps. However, their accurate regression-based pose estimation models demand high-resolution images, and their performance sharply declines when using low-resolution images. On the other hand, using high-resolution inputs significantly increases the computational burden of the network. Hence, there is a need to propose a training method that achieves lightweight multi-person pose estimation, enhances model capabilities in low-resolution scenarios, and reduces the computational requirements of the model. Summary of the Invention The present invention introduces an Staggered Alignment Self-distillation training method for multi-person pose estimation. The method involves constructing and adopting two pose estimation networks with shared weights for high-resolution and low-resolution training, facilitating knowledge transfer and feature alignment. Additional strategies such as auxiliary training levels, pruning inference, and lightweight inference are implemented to enhance efficiency without compromising performance. Invention Content To achieve a lightweight pose estimation regression method and enhance model capabilities in low-resolution scenarios, a novel "Staggered Alignment Self-Distillation Method" and its corresponding network architecture are proposed. For the aforementioned purposes, the technical solution adopted by the present invention is an Staggered Alignment Self-distillation training method for multi-person pose estimation, comprising: SI: Constructing and adopting two pose estimation networks with shared weights for high-resolution and low-resolution training, utilizing the same image dataset but with different resolutions as inputs. The dataset includes diverse human body images, encompassing both individual figures and multi-person scenarios to ensure the model can handle various complex situations. S2: Performing 2 times up-sampling on the input of the high-resolution network compared to the base resolution input. This design ensures that, at the same level, the feature map size of the high-resolution network is twice that of the base resolution network, achieving fine feature extraction. S3: Introducing a specifically designed auxiliary training level to address the missing fine-grained levels in the high-resolution network. This level focuses on enhanced learning for the finest features of the high-resolution network while providing necessary supervision for the base resolution network. S4: Achieving feature map staggered alignment and self-distillation. This is accomplished by matching the ith level feature map of the base resolution network with the (i+l)th level feature map of the high-resolution network, training with a unified label assignment strategy to transfer knowledge from the high-resolution network to the base resolution network. S5: Introducing a normalized skeleton loss function for training the multi-person pose estimation network, including: - A novel normalized skeleton loss function capturing the interdependence between human keypoints by computing the skeletal length differences between the ground truth (GT) and predicted keypoints for each person. S6: Implementing a "pruning inference" strategy. During the model's inference stage, removing auxiliary training levels to enhance operational efficiency and resource utilization without compromising performance. S7: Implementing a lightweight inference strategy. As knowledge transfer from the high-resolution network to the base resolution network has been accomplished during training, the model utilizes only the base resolution for inference, improving network runtime efficiency. Furthermore: - The network structures of the two pose estimation networks with shared weights in SI are both backbone networks with an added multi-layer feature pyramid structure. The backbone network follows a common residual network structure. - Data set acquisition in SI includes collecting various images containing human bodies, randomly sampled to cover different scenes, poses, and combinations of people to enhance the model's generalization ability. All input images are in JPG format, sequentially numbered, and neatly stored in a specific image folder for ease of model training and data management. - The high-resolution network design in S2 includes transforming the input resolution of the high-resolution network to high resolution using up-sampling techniques, capturing more detailed human features. - The design and implementation of the auxiliary training level in S3 include an independent layer with a feature pyramid structure, used only during the training stage, focusing on enhancing learning for specific levels of the high-resolution network. - Knowledge transfer in S4 is achieved through a distillation method between features. Specifically, the L2 loss function is computed between the ith level feature map of the base resolution network and the (i+l)th level feature map of the high-resolution network during training, and parameters are updated based on this loss function. - The normalization of the skeleton loss function in S5 adapts to variations in different scales of targets in the multi-person pose estimation model. This normalization involves dynamically normalizing the skeletal length during the loss calculation by taking the square root of the estimated human region, aiding in maintaining consistency and effectiveness of the loss function when handling targets of different sizes. The advantageous effects of the present invention are: The proposed Staggered Alignment Selfdistillation training method for multi-person pose estimation introduces a novel "Staggered Alignment Self-Distillation" method and a corresponding network architecture. This method trains two networks with shared weights, a high-resolution network acting as a teacher guiding the learning process of a low-resolution network through staggered feature alignment. The knowledge from the high-resolution network enhances the performance of the low-resolution network during low-resolution inference. Additionally, a normalized skeleton loss is employed to capture the loss of skeletal structure during training. This method significantly enhances model capabilities in low-resolution scenarios, reduces the required input resolution, thereby lowering the computational burden of pose estimation models. It achieves lightweighting of pose estimation methods, improving the universality and compatibility of pose models with lower computational platforms. Figure Description Figure 1: Flowchart of the Staggered Alignment Self-Distillation Training Method for Multi-Person Pose Estimation provided in Example 1 of the present invention. Figure 2: Schematic representation of the network structure for Multi-Person Pose Estimation using the Staggered Alignment Self-Distillation Training Method provided in Example 1 of the present invention. Figure 3: Illustration of the normalized skeleton loss function in the Staggered Alignment SelfDistillation Training Method for Multi-Person Pose Estimation provided in Example 1 of the present invention. In Figure 1, the process flow of the proposed training method is outlined, depicting the sequential steps involved in achieving multi-person pose estimation through Staggered Alignment Selfdistillation. Figure 2 provides a visual representation of the network structure employed in Example 1 for the purpose of multi-person pose estimation using the proposed Staggered Alignment Self-Distillation Training Method. The diagram illustrates the architecture, including shared-weight networks, upsampling, and feature alignment strategies. Figure 3 focuses on the normalized skeleton loss function, offering a schematic depiction of its formulation within the context of the Staggered Alignment Self-Distillation Training Method for Multi-Person Pose Estimation provided in Example 1. The diagram visually explains how this novel loss function captures the length differences between ground truth and predicted keypoints, emphasizing its role in modeling the interdependence between human keypoints. Detailed Description of the Invention To elucidate the objectives, technical content, and benefits of the invention more clearly, this document will provide a detailed explanation of the technical solution with the aid of figures. It should be noted that these embodiments represent only a part of the invention, not the entirety. Based on these embodiments, all other embodiments obtained by those skilled in the art without inventive labor prior to the cutoff date are within the scope of protection of the invention. First Embodiment: In the first embodiment, the invention provides a Staggered alignment self-distillation training method for multi-person pose estimation. The key steps are as follows: Step SI: Construction and use of two pose estimation networks with shared weights for high and low-resolution training: - Input Preprocessing: Resize input images to a uniform size. - Backbone Network (ShuffleNet): Utilize ShuffleNet as the backbone network, extracting features through repeated residual blocks and employing channel shuffle structures to reduce computational load. - Neck (PANet and FPN): PANet and FPN structures aggregate features from different levels, enhancing information flow between features. - Head: Responsible for final pose estimation, including coordinates of keypoints and bounding box regression. The complete network structure of our method is composed of the aforementioned Backbone network, Neck network, and Head network. Additionally, it consists of two identical network structures with shared weights. Both networks share weights and use a carefully selected dataset containing diverse human images to adapt to various scenarios. The dataset includes single-person images, showcasing different poses and expressions, as well as multi-person scenes, ensuring the model's versatility. Step S2: Upsample basic resolution input by a factor of 2 as the input for the high-resolution network. - Specifically, step S2 is used to prepare the input data for the high-resolution network. The core of step S2 involves upsampling the base resolution images in the dataset by a factor of two to serve as input for the high-resolution network. The upsampling process is achieved by applying the bilinear interpolation method, which aims to enhance the image resolution without significantly compromising the original visual quality. The principle behind the bilinear interpolation method is based on the intensity values of adjacent pixels in the image. By calculating and inserting new pixel points, the image resolution is increased. Specifically, bilinear interpolation considers the four nearest pixels surrounding an original pixel. Based on the intensity values of these neighboring pixels, it calculates the value of a new pixel through linear interpolation. When the image is enlarged, the newly generated pixels are not mere simple copies of the original pixels; instead, they reflect the average intensity values of the surrounding pixels. This improves the image resolution while preserving the overall structure of the image. In practice, for each base resolution image in the dataset, the original dimensions of each image are first determined. Each base resolution image is then scaled up to twice its original size. In this way, each dimension (height and width) of the base resolution image is doubled, resulting in new pixel points around each original pixel. During this process, the bilinear interpolation algorithm ensures that the values of the new pixels appropriately reflect the characteristics of their neighboring pixels. Thus, it retains the original details and textures of the image as much as possible while enlarging it. Step S3: Introduce a specifically designed auxiliary training level. - This invention introduces a carefully designed auxiliary training level to address specific challenges encountered in the self-distillation training method. Before performing auxiliary level training on two identical pose estimation networks with shared weights, the following issue must be addressed: the discrepancy in feature map dimensions between the high-resolution network and the base resolution network. This issue complicates direct knowledge transfer, especially when transferring knowledge between high-resolution training and low-resolution training. To overcome this challenge, the auxiliary training level introduced in step S3 is used to enhance the learning capability of the high-resolution network and provide more precise supervision for the base resolution network in self-distillation. The auxiliary training level exists independently within the network hierarchy, supplementing the missing finest-grained level in the high-resolution network and providing effective training targets for the corresponding levels in the base resolution network. This auxiliary training level is a layer of feature pyramid structure independent of the neck network and is used only during the training phase, focusing on enhancing specific levels of the high-resolution network. With the introduction of the auxiliary training level, the i-th level feature map of the base resolution network can perfectly match the (i+l)-th level feature map of the high-resolution network. Step S4: Implement Staggered alignment and self-distillation of feature maps: - Utilize two networks with shared weights: a low-resolution network O operating at basic resolution and a high-resolution network O' obtained through 2x up sampling. - Assign different scale targets to P3, P4, and P5 levels of network O during learning, and apply the same labeling strategy to levels P'4, P'5, and P'6 of network O'. Ensure consistency in target assignment between Pi level and P'i+l level. - Implement knowledge transfer through feature map distillation, where the L2 loss between P'i+l and Pi is calculated during training, as shown in equation(l). 5 Step S5: Use a normalized skeleton loss function for training the multi-person pose estimation network, as shown in equation(2) (2) Here, k_n represents the weight of the key point, s represents the scale of the person, Nkpts denotes the number of key points in the human body, oi and gti represent the output coordinates and the corresponding label of the i -th key point, respectively, and vn >0 is the visibility flag for each key point. - Adopt OKS loss function for keypoint regression during training. - Introduce a novel normalized skeleton loss function, capturing normalized skeleton distances between predicted and true labels, to learn interdependence between keypoints, as shown in equation(3). ™ --------""-------------------------------- <(3) Here, setsk represents the set of all adjacent key points in the human body, o represents the network output, and gt represents the assigned labels. By calculating the normalized skeleton loss function, step S5 achieves the process of staggered self-distillation. Step S6: Implement "pruning inference" strategy: - During the inference phase, remove auxiliary training levels to enhance model efficiency and resource utilization without affecting performance. Step S7: Implement lightweight inference strategy: - Given knowledge transfer from high-resolution to basic resolution during training, in the inference phase, abandon high-resolution input and only use basic resolution for inference to improve network efficiency. This embodiment discloses a system for self-distillation training with staggered alignment for multiperson pose estimation, executing the steps of any one of the technical solutions of the above methods for self-distillation training with staggered alignment for multi-person pose estimation. This embodiment discloses an electronic device, comprising: - A processor, and - A memory storing executable instructions for the processor; wherein -The processor is configured to perform the steps of any one of the technical solutions of the above methods for self-distillation training with staggered alignment for multi-person pose estimation via executing the executable instructions. This embodiment discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any one of the technical solutions of the above methods for self-distillation training with staggered alignment for multi-person pose estimation. The following outlines the overall concept of the method, system, electronic device, and storage medium for self-distillation training with staggered alignment for multi-person pose estimation disclosed in this embodiment. Specifically, constructing and using two identical pose estimation networks with shared weights for high-resolution and low-resolution training; upsampling the base resolution input by 2x for the high-resolution network input; introducing a specifically designed auxiliary training level; implementing feature map interleaving and self-distillation to achieve knowledge transfer from the high-resolution network to the base resolution network; and using a normalized skeleton loss function for training the multi-person pose estimation network. During the inference phase of the model, removing the auxiliary training level to improve computational efficiency and resource utilization; implementing a lightweight inference strategy. During the inference phase of the model, abandoning the use of high-resolution input and using only the base resolution for inference to improve network operating efficiency. The following outlines the working principle of the method, system, electronic device, and storage medium for self-distillation training with staggered alignment for multi-person pose estimation disclosed in this embodiment. SI: Construct and use two identical pose estimation networks with shared weights for high-resolution and low-resolution training. The identical pose estimation networks with shared weights use the same image dataset but with different resolutions as input. The dataset contains a variety of human images, including single individuals and multi-person scenarios, to ensure that the model can handle various complex situations. S2: Upsample the base resolution input by 2x for the high-resolution network input. This design ensures that at the same level, the feature map size of the high-resolution network is twice that of the base resolution network, thereby achieving fine feature extraction. S3: To address the issue of the missing finest granularity level in the high-resolution network, introduce a specifically designed auxiliary training level. The auxiliary training level specifically enhances learning for the finest granularity features of the high-resolution network while providing necessary supervision for the base resolution network. S4: Implement feature map interleaving and self-distillation. Match the i-level feature map of the base resolution network with the i+1 level feature map of the high-resolution network, and use a unified label allocation strategy for training to achieve knowledge transfer from the high-resolution network to the base resolution network. S5: Use a normalized skeleton loss function fortraining the multi-person pose estimation network, including: - A novel normalized skeleton loss function that captures the interdependence between human keypoints by calculating the skeletal length differences between each person's ground truth (GT) and predicted results. S6: Implement a pruning inference strategy. During the inference phase of the model, remove the auxiliary training level to improve computational efficiency and resource utilization, ensuring the model is more efficient without affecting performance. S7: Implement a lightweight inference strategy. Since knowledge transfer from the high-resolution network to the base resolution network has already been achieved during the training phase, abandon the use of high-resolution input during the inference phase of the model, and use only the base resolution for inference to improve network operating efficiency. Further, the network architecture of the two identical pose estimation networks with shared weights in SI consists of a backbone network and a multi-layer feature pyramid structure. The backbone network is a residual network architecture. The multi-layer feature pyramid integrates features at different levels, generating multi-scale outputs corresponding to different scale targets. During multi-person pose estimation, a regression approach is used, directly outputting the human keypoint coordinates. Further, the data acquisition in SI includes: collecting various images containing human bodies, with randomly selected images covering different scenes, poses, and human combinations to enhance the model's generalization ability. All input images are in jpg format, named in numerical order, and neatly stored in specific image folders for model training and data management. Further, the high-resolution network design in S2 includes: transforming the base resolution input into high-resolution input through upsampling techniques, thereby capturing more detailed human features. Further, the design and implementation of the auxiliary training level in S3 include: an independent feature pyramid structure layer dedicated to the high-resolution network, used only during the training phase, focusing on enhanced learning for specific levels of the high-resolution network. Further, the knowledge transfer in S4 is achieved through a distillation method between features, specifically by calculating the L2 loss function between the i-level feature map of the base resolution network and the i+1 level feature map of the high-resolution network during the training process, and updating the parameters based on the loss function. Further, the normalized skeleton loss function in S5 accommodates the variation of different scale targets in the multi-person pose estimation model. The loss function dynamically normalizes the skeletal length during the calculation process, adjusting the loss value through the square root of the estimated human area. This normalization method helps maintain the consistency and effectiveness of the loss function when dealing with different size targets. It is worth mentioning that the technical features such as the processor selection involved in this patent application should be considered as prior art. The specific structure, working principle, and possible control methods, spatial arrangement methods, etc., can be conventionally selected in the field and should not be regarded as the invention point of this patent. This patent will not further elaborate on these. For those skilled in the art, modifications to the technical solutions described in the foregoing embodiments, or equivalent replacements of some technical features, can still be made within the spirit and principles of the invention, and all such modifications, equivalent replacements, improvements, etc., should be included in the protection scope of this invention.

Claims

1. A method for staggered alignment self-distillation training for multi-person pose estimation, characterized by comprising the following steps:- Step SI: Construct and employ two identical pose estimation networks with shared weights for high-resolution training and low-resolution training respectively;- Step S2: Perform 2x upsampling of the input at the base resolution as the input for the high-resolution network;- Step S3: Implement auxiliary training levels;- Step S4: Conduct staggered alignment self-distillation of feature maps;- Step S5: Train the multi-person pose estimation network using a normalized skeleton loss function.

2. The method for staggered alignment self-distillation training for multi-person pose estimation according to claim 1, characterized in that, after Step S5, it further includes Step S6:- Step S6: Implement a pruning inference strategy.

3. The method for staggered alignment self-distillation training for multi-person pose estimation according to claim 2, characterized in that, after Step S6, it further includes Step S7:- Step S7: Implement a lightweight inference strategy.

4. The method for staggered alignment self-distillation training for multi-person pose estimation according to claim 1, characterized in that, Step S4 is specifically implemented as follows:- Compute the L2 loss function between the i-th level feature map of the base resolution network and the i+l-th level feature map of the high-resolution network, and update parameters according to the loss function.

5. A system for staggered alignment self-distillation training for multi-person pose estimation, characterized in that it executes the steps of the method for interleaved alignment self-distillation training for multi-person pose estimation according to any of claims 1 to 4.

6. An electronic device, characterized in that it includes:- A processor, and memory, wherein the memory stores executable instructions for the processor; and wherein:- The processor is configured to execute the steps of the method for staggered alignment selfdistillation training for multi-person pose estimation according to any of claims 1 to 4 via executing the executable instructions.

7. A computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method for interleaved alignment self-distillation training for multi-person pose estimation according to any of claims 1 to 4.11

Images