Pose distillation based low resolution human pose estimation method, device and medium

Abstract

The application discloses a low-resolution human posture estimation method and device based on posture distillation and a medium. Steps of the application include: constructing a paired human posture estimation dataset; obtaining a pre-trained and fine-tuned pre-training model as a teacher model in posture distillation; constructing a posture distiller; constructing a low-resolution human posture detection model; using the posture distiller to decompose human posture features obtained by the teacher model on paired high and low resolution training sets, and guiding the low-resolution network to learn human posture features at different levels in stages; and using the trained low-resolution human posture estimation model to perform human posture estimation prediction to obtain human posture estimation prediction results. The posture distiller decomposes human posture features of the teacher model at different scales and transmits the features to a student model, thereby improving the detection capability of the student model in a low-resolution environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human pose estimation, and in particular to a low-resolution human pose estimation method based on pose distillation, a device and a storage medium. BACKGROUND

[0002] Human pose estimation is an important branch of computer vision, which aims to detect the joint positions, orientations and poses of human bodies from images or videos. This technology plays an important role in applications such as human motion recognition, behavior analysis and human-computer interaction. In recent years, human pose estimation methods based on deep learning have emerged, significantly improving the accuracy of prediction.

[0003] However, when the image resolution is reduced, human pose estimation faces great challenges. Due to the lack of detailed information in low-resolution images, the human structure and key points in the image become blurred, making it difficult for existing models to extract effective human pose features. This results in existing human pose estimation network models performing poorly on low-resolution images, severely affecting the detection performance of the algorithm.

[0004] To address the difficulty of human pose feature extraction on low-resolution images, existing methods mainly adopt two strategies: one is to use multi-scale feature fusion technology, which combines feature maps of different resolutions to enhance the expression ability of human pose features; the other is to use feature super-resolution technology, which upsamples low-resolution feature maps to restore the detailed information of human pose features. However, these methods also have some problems, such as large computational overhead, high computational complexity, and excessive processing introducing noise, which reduces the practicality of the network model and affects the prediction accuracy of the network model. SUMMARY

[0005] In order to improve the practicality and detection ability of low-resolution network models and solve the challenges of low-resolution human pose estimation, the present application proposes a low-resolution human pose estimation method based on pose distillation. The core idea of this method is to design a new pose distiller, which can decompose and transfer human pose features of the teacher model at different scales to the student model, thereby improving the pose perception ability of the student model on low-resolution images, and further realizing accurate low-resolution human pose detection.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] The method comprises the following steps:

[0008] S1, construct a pair of human pose estimation dataset, including high and low resolution paired training set, low resolution verification set and test set;

[0009] S2, obtaining the pre-trained and fine-tuned pre-trained model as a teacher model in pose distillation;

[0010] S3, constructing a pose distiller, including a spatial alignment layer, a contour decoupling layer, and a pose encoding layer;

[0011] S4, constructing a low-resolution human pose estimation model, including a plurality of types of neural network computing layers, parameters of each neural network computing layer, and a connection manner between the neural network computing layers;

[0012] S5, using the pose distiller to decompose and transfer human pose features of the teacher model at different scales to the student model on the paired high and low resolution training sets, so that the student model can learn global and local features of human poses at different levels, thereby improving the prediction accuracy on low resolution images;

[0013] S6, using the trained low-resolution human pose estimation model to perform human pose estimation prediction to obtain human pose estimation prediction results.

[0014] Further, step S1 includes the following steps:

[0015] S1.1, constructing a high and low resolution paired training set, the training set is based on a COCO dataset, which contains annotation information of human key points, different scale affine transformations are performed on the training images in the dataset to obtain corresponding high and low resolution images and annotations, and then rotation, blur, translation and other data enhancement operations are performed on the high and low resolution images to improve the generalization ability of the model, forming the final training set;

[0016] S1.2, constructing a low resolution image validation set, the validation set is based on a COCO dataset, different scale affine transformations are performed on the validation images in the dataset to obtain low resolution images, which are used to verify the performance and effect of the model;

[0017] S1.3, constructing a low resolution image test set, the test set is based on a COCO dataset, different scale affine transformations are performed on the test images in the dataset to obtain low resolution images, which are used to test the generalization ability and robustness of the model.

[0018] Further, step S2 includes the following steps:

[0019] The HRNet-W32 network is selected as the backbone network of the teacher model, which can maintain high resolution feature representation throughout the process, and improve the fusion of semantic information and position information through multi-resolution subnetwork and information interaction, thereby obtaining human pose features with more details;

[0020] The network is fine-tuned on the high-resolution COCO dataset after data enhancement, so as to improve the generalization of the teacher model in extracting features.

[0021] Compared with the existing distillation method, the posture distiller can encode complete and relatively pure human posture features from the high-resolution teacher network, reduce the error and noise of the teacher network, maintain the small model parameters, reduce the calculation complexity and memory consumption, provide phased guidance for the student network, realize layer-by-layer learning from shallow to deep, and significantly improve the human posture perception ability of the student model in extreme environments.

[0022] Further, step S3 includes the following steps:

[0023] S3.1, a spatial alignment layer is constructed, which is composed of multiple deconvolution units, convolution units and RELU functions, and is used to reduce the spatial dimension of the high-resolution feature while retaining the spatial position and other information of the feature;

[0024] S3.2, a contour decoupling layer is constructed, which includes a global nested edge detection layer and a multilayer perceptron, the former is used to extract human contour information from the feature, and the latter is used to denoise and optimize the extracted human contour feature to reduce the interference of background noise;

[0025] S3.3, a posture encoding layer is constructed, which is stacked by multiple linear convolution units and nonlinear convolution units, the linear encoding unit is composed of multiple 1x1 convolution and normalization operation for extracting human posture features. The nonlinear is composed of multiple 1x1 convolution, RELU activation function and normalization operation to generate multiple human posture features with different scales

[0026] For an image, a high-resolution human posture feature can be obtained through a high-resolution teacher model, effectively excluding background interference. Since the high-resolution human posture feature still has a small spatial scale, it will affect the effect of subsequent human contour detection and extraction. Therefore, step S3.1 is needed to align the feature in space, and step S3.1 includes the following steps:

[0027] Through the deconvolution unit in the spatial alignment layer, the feature obtained by the teacher model is enlarged by 4 times in space, and the number of channels is halved, so as to increase the spatial scale of the feature, and prepare for the subsequent contour extraction;

[0028] Then, through the convolution unit in the spatial alignment layer, the feature is reorganized for human contour and edge information, which can enrich the spatial position and global information of the feature, and remove the influence of noise channels.

[0029] Further, the contour decoupling layer decouples the recombined features to extract pure and complete human contour features. Step S3.2 includes the following steps:

[0030] First, the high-resolution image features are input into the VGG16 backbone model of HED to extract features of different scales and depths.

[0031] Then, the feature maps at the end of each stage are input into the corresponding hidden layer for multi-scale feature extraction, thereby obtaining multiple human contour prediction maps with richer spatial structure information, improving the adaptability and robustness of the model. Next, in order to maintain global consistency, the feature maps of all hidden layers are fused together to obtain a fused feature map;

[0032] Next, in order to eliminate the differences between different feature maps, a weighted average operation is performed on all feature maps to generate the final human contour prediction map.

[0033] Finally, a multi-layer perceptron is used to denoise and optimize the human contour prediction map to reduce the influence of background noise.

[0034] Further, the pose encoding layer encodes and optimizes the human contour prediction map to generate human pose features that can be used to guide the student model. Step S3.3 includes the following steps:

[0035] The pose encoding layer uses a linear transformation matrix to map each pixel point of the human contour prediction map to a low-dimensional feature, thereby obtaining and enhancing the human pose features;

[0036] Then, a nonlinear convolution unit is used to perform nonlinear transformation on the human pose features to generate pose features of different scales.

[0037] Next, a lightweight low-resolution human pose estimation model is constructed as a student model. Step 4 includes the following steps:

[0038] Select an appropriate backbone network model, which can be any existing convolutional neural network model, such as the ResNetHRNet-W32 convolutional neural network model;

[0039] According to the characteristics of the backbone network, the number of down-sampling is reduced to adapt to low-resolution images.

[0040] Further, the student model and part of the parameters in the pose distiller are trained on the high-low resolution paired training set. During the training process, the features output by the teacher model are defined as , the student model output feature , the training process uses KL divergence to guide the student model learning, and the guiding process is: , wherein, is a hyperparameter that controls the soft signal output of the teacher network and the student network. Step 5 includes the following steps:

[0041] A pair of high and low resolution images is randomly selected from the training set and input into the teacher model and student model for feature extraction, respectively.

[0042] The pose distiller decouples the multi-scale pose features from the teacher model features and uses large-scale features to guide the shallow perception of the student model;

[0043] At the same time, small-scale features are used to guide the deep perception of the student model, thereby comprehensively improving the pose detection ability of the student model in a low resolution environment.

[0044] Finally, the MSELOSS loss function is used to measure the gap between the prediction result and the label, and the parameters of the model are optimized to obtain a trained low resolution human pose estimation model. The MSE is defined as: Wherein, is the number of samples, is the true value, is the model prediction value.

[0045] Further, step S6 includes the following steps:

[0046] An input sample is randomly selected from the test set, and input data is obtained after data preprocessing;

[0047] The input data is input into the trained low resolution human pose estimation model, and the key point coordinates are output;

[0048] According to the provisions of the data set, the key points are connected to obtain a visual result graph.

[0049] In another aspect, the present application also discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to make the processor execute the steps of the above method.

[0050] In another aspect, the present application also discloses a computer device, which comprises a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the above method.

[0051] From the above technical solution can be known, in order to effectively solve the problem of low resolution image human posture feature difficult to extract, improve the practicability of network model.The present application proposes a kind of low resolution human posture estimation method based on posture distillation, the method uses high resolution dataset and model as teacher, low resolution dataset and model as student.Through the posture distiller proposed in the application, the human posture features of teacher model are decomposed at different scales and transmitted to student model, thereby improving the detection capability of student model in low resolution environment.In addition, the student model based on posture distillation does not require excessive training parameters and can be deployed to a wider range of devices.

[0052] Specifically, the present application discloses a low resolution human posture estimation method based on posture distillation, comprising the following steps: constructing a paired human posture estimation dataset; obtaining a pre-trained and fine-tuned pre-trained model as a teacher model in posture distillation; constructing a posture distiller; constructing a low resolution human posture detection model; using the posture distiller to decompose the human posture features obtained by the teacher model on the paired high and low resolution training sets, and guiding the low resolution network to learn human posture features at different levels in stages; using the trained low resolution human posture estimation model to perform human posture estimation prediction to obtain human posture estimation prediction results.

[0053] Compared with the prior art, the present application has the following advantages:

[0054] The low resolution human posture estimation method based on posture distillation proposed by the present application solves the problem that existing models cannot effectively extract posture features in a low resolution environment, and significantly improves the detection capability of the model.

[0055] The present application designs a brand new posture distiller, which can extract complete and pure human posture features from the teacher model, reduce environmental and background interference, and gradually improve the posture perception ability of the student model, significantly improving the detection performance of the student model.

[0056] The posture distiller and student model of the present application have small parameter quantity, low computational complexity and small memory occupation, and are suitable for training and deployment on various devices to realize human posture detection. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a working flowchart of a low resolution human posture estimation method model based on posture distillation provided by an embodiment of the present application;

[0058] Figure 2 is a structural diagram of the posture distiller provided by an embodiment of the present application;

[0059] Figure 3is a structural diagram of a spatial alignment layer in a pose distiller provided by an embodiment of the present application;

[0060] Figure 4 is a structural diagram of a contour decoupling layer in a pose distiller provided by an embodiment of the present application;

[0061] Figure 5 is a structural diagram of a pose encoding layer in a pose distiller provided by an embodiment of the present application;

[0062] Figure 6 is a training flowchart of a low-resolution human pose estimation method based on pose distillation provided by an embodiment of the present application;

[0063] Figure 7 、 Figure 8 are low-resolution human pose estimation result diagrams provided based on student models, respectively. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application.

[0065] As shown in Figure 1 , an embodiment of the present application proposes a low-resolution human pose estimation method based on pose distillation. The method uses a high-resolution dataset and model as a teacher and a low-resolution dataset and model as a student. The human pose features of the teacher model are decomposed and transferred to the student model at different scales through the pose distiller proposed by the present application. The method overcomes the challenge of difficult extraction of human pose features on low-resolution images, improves the practicability of the model, and realizes accurate low-resolution human pose prediction. The specific steps are as follows:

[0066] S1, a paired human pose estimation dataset is constructed, including a high-low resolution paired training set, a low-resolution verification set and a test set;

[0067] S2, a pre-trained and fine-tuned pre-trained model is obtained as a teacher model in pose distillation;

[0068] S3, a pose distiller is constructed, including a spatial alignment layer, a contour decoupling layer and a pose encoding layer;

[0069] S4, a low-resolution human pose detection model is constructed, including a plurality of types of neural network calculation layers, parameters of each neural network calculation layer and a connection mode between each neural network calculation layer;

[0070] S5, using the pose distiller to decompose and transfer the human pose features of the teacher model at different scales to the student model on the paired high and low resolution training sets, so that the student model can learn the global and local features of the human pose at different levels, thereby improving the prediction accuracy thereof on low resolution images;

[0071] S6, using the trained low resolution human pose estimation model to perform human pose estimation prediction to obtain a human pose estimation prediction result.

[0072] Specifically, step S1 comprises the following steps:

[0073] S1.1, constructing a high and low resolution paired training set, the training set being based on a COCO dataset containing annotation information of human key point coordinates, performing affine transformation on training images in the dataset at different scales to obtain corresponding high and low resolution images and annotations, and then performing rotation, blurring, translation and other data enhancement operations on the high and low resolution images to improve the generalization ability of the model, thereby forming a final training set;

[0074] S1.2, constructing a low resolution image validation set, the validation set being based on a COCO dataset, performing affine transformation on validation images in the dataset at different scales to obtain low resolution images, for verifying the performance and effect of the model;

[0075] S1.3, constructing a low resolution image test set, the test set being based on a COCO dataset, performing affine transformation on test images in the dataset at different scales to obtain low resolution images, for testing the generalization ability and robustness of the model.

[0076] Step S2 comprises the following steps:

[0077] The HRNet-W32 network is selected as the backbone network of the teacher model, which can maintain high resolution feature representation throughout the process, and improve the fusion of semantic information and position information through multi-resolution subnetworks and information interaction, thereby obtaining human pose features with more details;

[0078] The network is fine-tuned on the high resolution COCO dataset after data enhancement, thereby improving the generalization of the feature extraction of the teacher model.

[0079] Compared with existing distillation methods, the pose distiller proposed in the present application can encode noise-free human pose features from the high resolution teacher network, reducing the error and noise of the teacher network; meanwhile, the small model parameters are maintained, reducing the computational complexity and memory consumption; the pose distiller can provide phased guidance for the student network, realizing layer-by-layer learning from shallow to deep, and significantly improving the human pose perception ability of the student model in extreme environments.

[0080] As shown in the formula (1), step S3 includes three feature extraction layers, i.e., a spatial alignment layer, a contour decoupling layer and a pose encoding layer. Figure 2

[0081] S3.1, a spatial alignment layer is constructed, which is composed of a deconvolution unit and a convolution unit, and realizes spatial alignment of high-resolution teacher model features to low-resolution student model features while retaining spatial position and other information of the features;

[0082] S3.2, a contour decoupling layer is constructed, which includes a global nested edge detection layer and a multilayer perception, the former being used to extract human body contour information from the features, and the latter being used to denoise and optimize the extracted human body contour features to reduce the interference of background noise;

[0083] S3.3, a pose encoding layer is constructed, which is stacked by multiple linear encoding units and nonlinear convolution units, human body pose features are extracted by linear encoding, and then multi-level human body pose features are generated by nonlinear transformation, and the student model is guided to learn in turn.

[0084] For a high-resolution image , a high-resolution human body pose feature can be obtained through the HRNet-W32 network. Since the high-resolution human body pose feature still has a small spatial scale, it will affect the effect of subsequent human body contour detection and extraction. Therefore, step S3.1 is needed to align the features in space.

[0085] Figure 3 As shown in the formula (2), step S3.1 includes the following steps:

[0086] Through the deconvolution unit in the spatial alignment layer, the high-resolution human body pose feature is enlarged by 4 times in space, and the number of channels is halved, so as to increase the spatial scale of the feature and prepare for subsequent contour extraction operation;

[0087] Then, through the convolution unit in the spatial alignment layer, the feature is reorganized for human body contour and edge information, and a feature with appropriate spatial scale and rich information is obtained, which can enrich the spatial position and global information of the feature and remove the influence of noise channels, and the expression is as follows:

[0088]

[0089] wherein is the feature obtained by the spatial alignment layer, ​CONV is a deconvolution unit in a spatially aligned layer. It is a convolutional unit in a spatially aligned layer.

[0090] like Figure 4 As shown, step S3.2 is used to decouple the recombined features and extract complete and noise-free human contour features. Step S3.2 includes the following steps:

[0091] The globally nested edge detection layer in the contour decoupling layer affects the reconstructed features. The process involves first using the VGG16 backbone model to extract features from the reconstructed features at different scales. These features are then fed into the corresponding hidden layers for multi-scale feature extraction, resulting in multiple human contour prediction maps with richer spatial structural information. This avoids the problems of a single scale and hierarchy. The expression is as follows:

[0092]

[0093] in, It refers to the predicted human body contour map. This refers to a globally nested edge detection layer.

[0094] Then, a weighted fusion strategy is used to linearly overlay all the predicted human body contour maps to obtain a more complete and refined predicted human body contour map. The expression is as follows:

[0095]

[0096] in, It refers to the predicted human body contour map. This means concatenating and recombining the obtained sub-features into a completely new feature. This indicates that the splicing operation is performed along the channel dimension.

[0097] Finally, a multilayer perceptron is used to denoise and optimize the predicted human contour map to reduce the impact of background noise. The expression is as follows:

[0098]

[0099] in, It is a human contour prediction map obtained after optimization in a multilayer perceptron (MLP).

[0100] like Figure 5 As shown, the pose encoding layer optimizes the encoding of the human contour prediction map to generate human pose features that can be used to guide the student model. Step S3.3 includes the following steps:

[0101] The posture encoding layer utilizes a linear transformation matrix to perform low-dimensional feature mapping on each pixel point of the human body contour prediction graph, thereby obtaining and enhancing human body posture features

[0102] Then, a nonlinear convolution unit is utilized to perform multi-scale transformation on the human body posture features, to generate posture features of different levels , and the expression is as follows:

[0103]

[0104] wherein, is the posture feature obtained after encoding by the posture encoding layer, LCONV

[0105] Next, a lightweight low-resolution human body posture estimation model is constructed as a student model. Further, step 4 includes the following steps:

[0106] A suitable backbone network model can be any existing convolutional neural network model, such as a SimpleBaseline model convolutional neural network model.

[0107] According to the characteristics of the backbone network, the number of down-sampling is moderately reduced to adapt to low-resolution images.

[0108] As shown in FIG. 5, the student model and part of the parameters in the posture distiller are trained on the high-low resolution paired training set, and step 5 includes the following steps: Figure 6 A pair of high-resolution images

[0109] and low-resolution images are randomly selected from the training set, and are respectively input into the teacher model to obtain high-resolution features and shallow low-resolution features obtained by the student model.

[0110] The posture distiller decouples the multi-scale human body posture features from the features of the teacher model , and then uses KL divergence to guide the student model to learn at the shallow layer, and the expression is as follows:

[0111]

[0112] wherein, is a hyperparameter that controls the soft signal output of the teacher network and the student network.

[0113] Meanwhile, small-scale features are utilized to guide the learning of the student model at the deep layer, thereby comprehensively improving the posture ability of the student model in the low-resolution environment, and the expression is as follows:

[0114] ​​

[0115] wherein represents the deep features of the student model.

[0116] Finally, the MSELOSS loss function is used to measure the gap between the prediction result and the label, and the parameters of the model are optimized, so as to obtain the trained low-resolution human pose estimation model. The MSE is defined as: wherein, is the number of samples, is the true value, is the model prediction value.

[0117] Step S6 includes the following steps:

[0118] An input sample is randomly selected from the test set, and input data is obtained after data preprocessing;

[0119] The input data is input into the trained low-resolution human pose estimation model, and the key point coordinates are output;

[0120] According to the provisions of the data set, the key points are connected into a visual result graph.

[0121] Table 1

[0122]

[0123] Table 1 shows the average prediction accuracy of the SimpleBaseline model based on pose distillation and the original SimpleBaseline model on the low-resolution verification set. From the table, it can be clearly seen that the SimpleBaseline model based on pose distillation has a significant accuracy improvement compared to the original SimpleBaseline model, and shows higher accuracy.

[0124] Figure 7 、 Figure 8 shows an example of the result of the student model in the low-resolution human pose estimation task. From the example, it can be seen that the low-resolution human pose estimation method provided by the embodiment of the present application accurately locates the key points of each part of the human body, effectively solves the problem that the model is difficult to extract effective human pose features in a low-resolution environment, and significantly improves the detection performance of low-resolution human pose estimation.

[0125] In another aspect, the present application also discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to make the processor execute the steps of the above method.

[0126] In still another aspect, the present application also discloses a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, causes the processor to perform the steps of the above method.

[0127] In yet another embodiment provided in the present application, a computer program product containing instructions, which, when executed on a computer, causes the computer to perform any of the above-mentioned pose distillation based low-resolution human pose estimation methods.

[0128] It can be understood that the system, device and storage medium provided by the embodiments of the present application correspond to the method provided by the embodiments of the present application, and the explanation, examples and beneficial effects of related contents can refer to the corresponding parts in the above method.

[0129] In the above embodiments, the system, device and storage medium provided by the embodiments of the present application can be realized by software, hardware, firmware or any combination thereof, in whole or in part. When realized by software, it can be realized in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, solid state disk (SSD)) and the like.

[0130] It is to be noted that, in the present text, the relative terms such as first and second, and the like are used merely to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0131] Each of the embodiments in the present specification is described in a relevant manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments.

[0132] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features. Such modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A low-resolution human pose estimation method based on pose distillation, characterized in that, Includes the following steps: S1, construct paired human pose estimation datasets, including high-resolution and low-resolution paired training sets, low-resolution validation sets, and test sets; S2, obtain the pre-trained and fine-tuned pre-trained model, which serves as the teacher model in attitude distillation; S3, construct the attitude distiller, including a spatial alignment layer, a contour decoupling layer, and an attitude encoding layer; S4, construct a low-resolution human pose detection model, i.e., a student model, including neural network computation layers, parameters of each neural network computation layer, and connection methods between each neural network computation layer; Choose a backbone network model and reduce the number of downsampling steps based on the characteristics of the backbone network to adapt to low-resolution images; S5 uses a pose distiller on paired high- and low-resolution training sets to decompose the human pose features of the teacher model at different scales and pass them to the student model, enabling the student model to learn the global and local features of human pose at different levels, thereby improving its prediction accuracy on low-resolution images. S6. Use the trained low-resolution human pose estimation model to perform human pose estimation and prediction, and obtain the human pose estimation prediction result. Step S3 includes the following steps: S3.1 Construct a spatial alignment layer, which consists of multiple deconvolution units, convolution units, and ReLU functions, to reduce the spatial dimension of high-resolution features while preserving the spatial location information of the features; S3.2, Construct a contour decoupling layer, which includes a global nested edge detection layer and a multilayer perceptron. The global nested edge detection layer is used to extract human contour information from features, and the multilayer perceptron is used to denoise and optimize the extracted human contour features to reduce the interference of background noise. S3.3 Construct a pose encoding layer, which is composed of multiple linear convolutional units and nonlinear convolutional units stacked together. The linear encoding unit is composed of multiple 1x1 convolutions and normalization operations to extract human pose features. The nonlinear unit is composed of multiple 1x1 convolutions, ReLU activation functions and normalization operations to generate multiple human pose features at different scales.

2. The low-resolution human pose estimation method based on pose distillation according to claim 1, characterized in that: Step S1 includes the following steps: S1.1 Construct a training set with high and low resolution pairs. The training set is based on the COCO dataset and contains annotation information of human key points. Perform affine transformations at different scales on the training images of the training set to obtain corresponding high and low resolution images and annotations. Then perform rotation, blurring, and translation data augmentation operations on the high and low resolution images to improve the generalization ability of the model and form the final training set. S1.2, Construct a low-resolution image validation set. The validation set is based on the COCO dataset. Perform affine transformations at different scales on the validation images in the validation set to obtain low-resolution images, which are used to verify the performance and effectiveness of the model. S1.3 Construct a low-resolution image test set. The test set is based on the COCO dataset. Perform affine transformations at different scales on the test images in the test set to obtain low-resolution images, which are used to test the generalization ability and robustness of the model.

3. The low-resolution human pose estimation method based on pose distillation according to claim 1, characterized in that: Step S2 includes the following steps: The HRNet-W32 network was selected as the backbone network of the teacher model; The network is fine-tuned on a high-resolution training set with augmented data, thereby improving the generalization of features extracted by the teacher model.

4. The low-resolution human pose estimation method based on pose distillation according to claim 1, characterized in that: For a high-resolution image High-resolution human pose features were obtained through the HRNet-W32 network. ; The spatial alignment of features is performed in step S3.1, which specifically includes the following steps: High-resolution human pose features are obtained by using deconvolution units in the spatial alignment layer. The spatial scale is increased by 4 times while the number of channels is halved, thereby increasing the spatial scale of the features and preparing for subsequent contour extraction operations. Then, through the convolutional units in the spatial alignment layer, the features are reorganized to include human contour and edge information, resulting in a feature with an appropriate spatial scale and rich information. The expression is as follows: in These are features obtained from the spatial alignment layer. It is a deconvolution unit in a spatially aligned layer. CONV It is a convolutional unit in a spatially aligned layer; Contour features are extracted by convolutional units, and then optimized and recombined by the ReLU activation function to obtain high-resolution detailed features.

5. The low-resolution human pose estimation method based on pose distillation according to claim 1, characterized in that: Step S3.2 is used to decouple the recombined features and extract complete and noise-free human body contour features. Step S3.2 specifically includes the following steps: The globally nested edge detection layer in the contour decoupling layer affects the reconstructed features. The process involves first using the VGG16 backbone model to extract features from the reconstructed features at different scales. These features are then fed into the corresponding hidden layers for multi-scale feature extraction, resulting in multiple human contour prediction maps with richer spatial structural information. This avoids the problems of a single scale and hierarchy. The expression is as follows: in, It refers to the predicted human body contour map. This refers to a globally nested edge detection layer; Then, a weighted fusion strategy is used to linearly overlay all the predicted human body contour maps to obtain a more complete and refined predicted human body contour map. The expression is as follows: in, It refers to the predicted human body contour map. This means concatenating and recombining the obtained sub-features into a completely new feature. This indicates that the splicing operation is performed along the channel dimension; Finally, a multilayer perceptron is used to denoise and optimize the predicted human contour map to reduce the impact of background noise. The expression is as follows: in, It is a human body contour prediction image obtained after optimization in a multilayer perceptron. MLP 6. The low-resolution human pose estimation method based on pose distillation according to claim 1, characterized in that: The pose encoding layer optimizes the encoding of the human contour prediction map to generate human pose features that can be used to guide student models. Step S3.3 specifically includes the following steps: The pose encoding layer uses a linear transformation matrix to predict the human body contour map. Low-dimensional feature mapping is performed on each pixel to obtain and enhance human pose features; Then, nonlinear convolutional units are used to perform multi-scale transformations on the human pose features to generate pose features at different levels. The expression is as follows: in, These are the pose features obtained after encoding by the pose coding layer. It is a linear convolutional unit. It is a non-linear convolutional unit.

7. The low-resolution human pose estimation method based on pose distillation according to claim 1, characterized in that: Step S5 includes the following steps: Define the features output by the teacher model as follows Student model output features The training process uses KL divergence to guide student model learning. The guidance process is as follows: ,in, It is a hyperparameter that controls the soft signal output of the teacher network and the student network; Specific step S5 includes the following steps: A pair of high-resolution and low-resolution images are randomly selected from the training set, and features are extracted using the teacher model and the student model, respectively. The posture distiller decouples multi-scale posture features from the features of the teacher model and uses large-scale features to guide the shallow perception of the student model. At the same time, small-scale features are used to guide the deep perception of student models, improving the prediction performance and robustness of student models at low resolution. Finally, the MSELOSS loss function is used to measure the gap between the prediction results and the annotations, and the parameters of the model are optimized to obtain the trained low-resolution human pose estimation model. MSE is defined as: in, It is the sample size. It is the actual value. These are model predictions.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images