Low-resolution image human pose estimation method and device based on super-resolution reconstruction

Abstract

The application discloses a kind of low-resolution image human posture estimation method and device based on super-resolution reconstruction, method includes the following steps: to high-resolution human image true value is carried out downsampling processing and obtains low-resolution human image, constructs and includes high and low resolution human image pair training data set;Super-resolution reconstruction-based low-resolution image human posture estimation model is constructed, including super-resolution reconstruction subnetwork and human posture estimation subnetwork, and task-driven hybrid loss function is constructed;After low-resolution human image is input into super-resolution reconstruction subnetwork and obtains super-resolution reconstruction human image, then input into human posture estimation subnetwork and obtain human posture estimation result, based on training data set and hybrid loss function, model is trained and used for human posture estimation.The application not only improves detection accuracy, but also guarantees the real-time performance of the model, is suitable for panoramic monitoring and unmanned aerial vehicle patrol and other scenes, has higher application value and popularization prospect.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and apparatus for estimating human pose in low-resolution images based on super-resolution reconstruction. Background Technology

[0002] Human pose estimation, a widely studied topic in computer vision, is an important step in describing and understanding human behavior patterns in images and videos. Its purpose is to locate human joints such as shoulders, knees, and wrists, and it plays a key role in various tasks such as object detection, human-computer interaction, pedestrian tracking, and action recognition.

[0003] Human pose estimation is categorized into two-dimensional (2D) and three-dimensional (3D) human pose estimation based on the spatial dimension of the predicted results. With the introduction and development of deep learning, 2D pose estimation methods based on convolutional neural networks (CNNs) have, to some extent, overcome challenges faced by earlier research, including occlusion and lack of generalization. However, most 2D human pose estimation methods based on CNNs presuppose that the human body occupies a sufficient number of pixels and provides enough appearance and texture features in the image. Therefore, their accuracy generally decreases when faced with lower-resolution image inputs. Most 2D multi-person human pose estimation methods employ a top-down approach, first detecting the human target and then performing 2D single-person human pose estimation on the detected target. Therefore, if the problem of 2D single-person human pose estimation under low-resolution image input can be solved, it can be easily extended to multi-person human pose estimation.

[0004] Two-dimensional human pose estimation methods based on convolutional neural networks can be divided into two categories based on their output: direct regression methods and heatmap-based methods. The latter generally offers better performance, but the generated heatmap typically has a lower resolution than the input image, leading to positioning errors in the coordinates of key points on the human body. When the input image resolution is high, the positioning error of the output key points remains within acceptable limits. However, when the input image resolution is low, the positioning error of the output key points increases, resulting in a decrease in the accuracy of human pose estimation. Furthermore, in scenarios involving panoramic camera surveillance or drone patrol photography, human pose estimation faces challenges such as low image resolution and small human body size. To address the issue of reduced feature map resolution during the network's forward pass, which in turn lowers pose estimation accuracy, heatmap-based methods employ HigherHRNet (Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020). This network, based on the HRNet architecture, features multiple parallel forward passes, ensuring that the feature map resolution remains consistent with the input image throughout the forward pass. It also incorporates a high-resolution feature pyramid, using multi-resolution supervision to allow features from different layers to learn information at different scales. Furthermore, multi-scale fusion generates high-resolution feature maps to predict high-quality heatmaps, mitigating the information loss caused by the reduced image resolution during the network's forward pass.

[0005] In essence, these methods that use multiple feedforward networks in parallel to minimize the reduction in feature map resolution caused by the network feedforward are basically based on the idea of ​​"reducing loss". They simply make the most of the information in the low-resolution input image to reduce the loss of information during the network feedforward. However, the information content of the already scarce low-resolution image is the real bottleneck that restricts human pose estimation and detection in low-resolution images. Therefore, this type of method does not solve the essential problem. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a method and apparatus for human pose estimation of low-resolution images based on super-resolution reconstruction, which can improve the resolution of low-resolution human images and improve the accuracy of human pose estimation for small targets at a distance. At the same time, it proposes a hybrid loss that includes super-resolution reconstruction loss and human key point detection loss, so that the super-resolution reconstructed human image contains enough high-level semantic information, thereby improving the resolution capability of the human pose estimation network and improving the estimation accuracy. It is suitable for application scenarios such as panoramic monitoring and UAV patrol.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] In a first aspect, embodiments of the present invention provide a method for estimating human pose in low-resolution images based on super-resolution reconstruction, comprising the following steps:

[0009] The high-resolution human images are downsampled to obtain low-resolution human images, and a training dataset containing both high-resolution human images and low-resolution human images is constructed.

[0010] A human pose estimation model for low-resolution images based on super-resolution reconstruction is constructed, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and a task-driven hybrid loss function is constructed.

[0011] The low-resolution human image is input into the super-resolution reconstruction sub-network to obtain the super-resolution reconstructed human image, and then input into the human pose estimation sub-network to obtain the human pose estimation result. The human pose estimation model based on the low-resolution image reconstructed by super-resolution is trained based on the training dataset and the hybrid loss function.

[0012] The low-resolution human image to be detected is input into a trained low-resolution image human pose estimation model based on super-resolution reconstruction, and the human pose estimation result is obtained from the output heatmap.

[0013] Preferably, the task-driven hybrid loss function includes a super-resolution reconstruction loss function and a human keypoint detection loss function, expressed as follows:

[0014] l com =αl rec +βl est

[0015] Among them, l com For the mixed loss function, l rec For the super-resolution reconstruction loss function, l est The loss function for human keypoint detection is defined by α and β, which are the weights of the two loss terms and are used to balance the weights of the two loss terms.

[0016] Preferably, the super-resolution reconstruction loss function includes a pixel-wise L1 loss function and a feature reconstruction loss function;

[0017] The formula for the pixel-wise L1 loss function l1 is expressed as:

[0018]

[0019] Where, f sr (I lr ) x,y,c For low-resolution human body images I lr Reconstructing the subnetwork f using super-resolution sr (·) The reconstructed pixel coordinates are (x, y) and the pixel value of channel c. For high-resolution human body image ground truth I hr The corresponding pixel coordinates are (x, y) and the pixel ground truth of channel c, where H, W and C are the length, width and total number of channels of the image, respectively;

[0020] Feature Reconstruction Loss Function The formula is expressed as:

[0021]

[0022] in, For super-resolution reconstruction of human images, φ j (·) represents the C extracted by the j-th convolutional layer in the perceptual loss feature extraction network φ. j ×H j ×W j Size feature map;

[0023] Super-resolution reconstruction loss function l rec For pixel-wise L1 loss function l1 and feature reconstruction loss function The weighted summation is expressed by the formula:

[0024]

[0025] Here, k1 and k2 are the weights corresponding to the two loss terms, respectively.

[0026] Preferably, the loss function for human keypoint detection is the heatmap M of the ground truth of the high-resolution human image. gt Prediction heatmap M of the human pose estimation subnetwork pre The mean square error between them is expressed by the formula:

[0027]

[0028] Among them, l est Here, H represents the loss function for human keypoint detection, where K is the total number of human keypoints, and H is the loss function for human keypoint detection. ′and W ′ These are the length and width of the heatmap, respectively, and the heatmap M. gt It is a two-dimensional Gaussian distribution centered on the human key points of the true value of a high-resolution human image. Let be the pixel value of the heatmap corresponding to the k-th key point with pixel coordinates (x, y). Heatmap M generated for the human pose estimation subnetwork pre The pixel value of the heatmap corresponding to the k-th key point with pixel coordinates (x, y);

[0029] Heatmap M pre Generated by the human pose estimation subnetwork, the formula is expressed as:

[0030] M pre =f pose (f sr (I lr ))

[0031] Among them, I lr For low-resolution human body images, f sr (·) represents the super-resolution reconstruction subnetwork, f pose (·) represents the human pose estimation subnetwork.

[0032] Preferably, the constructed super-resolution reconstruction subnetwork adopts a cascaded residual network structure, including a cascaded part and an upsampling part;

[0033] In the cascaded part, a convolutional layer is first used to extract the pixels of the low-resolution human image from the spatial dimension to the channel dimension. Then, multiple alternating cascaded residual blocks and convolutional blocks are used for image feature extraction and multi-layer feature fusion to obtain a feature map.

[0034] In the upsampling part, an upsampling layer is used to rearrange pixels to upsample the feature map to a high resolution to obtain a super-resolution reconstructed human image.

[0035] Preferably, the constructed human pose estimation subnetwork adopts the FastPose network based on the AlphaPose network, including: a backbone network part, a dense upsampling part and a detection head part;

[0036] In the backbone network, the super-resolution reconstructed human image is input, and the features of the human pose are extracted using sequentially connected convolutional layers, max pooling layers, and multiple compression excitation modules.

[0037] In the dense upsampling part, the extracted human pose features are input, and multiple alternating convolutional layers and pixel rearrangement layers are used to capture and recover the detailed information of the features to obtain the upsampled feature map.

[0038] In the detection head section, convolutional layers are used to convert the upsampled feature maps into fine-grained human keypoint prediction heatmaps.

[0039] Preferably, in the cascaded part of the super-resolution reconstruction sub-network, the structure of each cascaded residual block includes at least one enhanced residual block and an inner convolutional block, wherein the enhanced residual block adopts grouped convolution, and the outputs of the enhanced residual blocks are concatenated in parallel along the feature dimension and then weighted and averaged by the inner convolutional block for feature fusion.

[0040] Secondly, in order to achieve the above-mentioned objectives, the present invention also provides a low-resolution image human pose estimation device based on super-resolution reconstruction, comprising: a dataset construction module, a model construction module, a model training module, and a model testing module.

[0041] The dataset construction module is used to downsample the ground truth of high-resolution human images to obtain low-resolution human images, and to construct a training dataset containing both high-resolution human image ground truth and low-resolution human images.

[0042] The model building module is used to build a human pose estimation model for low-resolution images based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and to build a task-driven hybrid loss function.

[0043] The model training module is used to input low-resolution human images into the super-resolution reconstruction sub-network to obtain super-resolution reconstructed human images, and then input them into the human pose estimation sub-network to obtain human pose estimation results. The human pose estimation model based on super-resolution reconstruction of low-resolution images is trained based on the training dataset and the hybrid loss function.

[0044] The model testing module is used to input the low-resolution human image to be detected into the trained low-resolution image human pose estimation model based on super-resolution reconstruction, and obtain the human pose estimation result from the output heatmap.

[0045] Thirdly, to achieve the above-mentioned objectives, embodiments of the present invention also provide a low-resolution image human pose estimation device based on super-resolution reconstruction, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned low-resolution image human pose estimation method based on super-resolution reconstruction when the computer program is executed.

[0046] Fourthly, to achieve the above-mentioned objectives, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-mentioned method for estimating human pose in low-resolution images based on super-resolution reconstruction.

[0047] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0048] (1) The present invention reconstructs low-resolution human images through a super-resolution reconstruction subnetwork and then predicts them through a human pose estimation subnetwork, which can fundamentally improve the amount of image information used for pose estimation, thereby improving the detection accuracy of low-resolution human pose estimation.

[0049] (2) The present invention designs a task-driven hybrid loss function to enable the super-resolution reconstruction sub-network to maximize the amount of information that the low-resolution input image can express, and to generate a high-resolution image containing enough low-level image features and high-level semantic features from the low-resolution image. The super-resolution reconstruction sub-network and the human pose estimation sub-network are trained based on the hybrid loss function, so that the model can estimate human pose more accurately from the input low-resolution image. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the low-resolution image human pose estimation method based on super-resolution reconstruction provided in this embodiment of the invention.

[0052] Figure 2 This is a schematic diagram of the framework of the low-resolution image human pose estimation method based on super-resolution reconstruction provided in the embodiments of the present invention;

[0053] Figure 3 This is a schematic diagram of the network structure of the super-resolution reconstruction sub-network provided in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the network structure of the human pose estimation subnetwork provided in an embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram of the structure of a low-resolution image human pose estimation device based on super-resolution reconstruction provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0057] The inventive concept of this invention is as follows: In view of the problem that the accuracy of human pose estimation methods using low-resolution images in the prior art is low, the embodiments of this invention provide a method and apparatus for human pose estimation of low-resolution images based on super-resolution reconstruction. First, a training dataset is constructed by downsampling high-resolution human images. Then, a super-resolution reconstruction sub-network and a human pose estimation sub-network are combined, and a task-driven hybrid loss function is used for model training. This enables the super-resolution reconstructed human images to have sufficient high-level semantic information, thereby improving the accuracy of human pose estimation in low-resolution images.

[0058] Among them, the super-resolution reconstruction subnetwork and the human pose estimation subnetwork can be designed as very lightweight subnetworks, which can make full use of the device's computing resources to achieve better human pose estimation results, so as to ensure the real-time performance of the model in practical applications.

[0059] Figure 1 This is a flowchart illustrating the low-resolution image human pose estimation method based on super-resolution reconstruction provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the framework of a low-resolution image human pose estimation method based on super-resolution reconstruction provided in an embodiment of the present invention. Figure 1 and Figure 2 As shown in the embodiment, a method for human pose estimation of low-resolution images based on super-resolution reconstruction is provided, including the following steps:

[0060] S1, downsample the high-resolution human image ground truth to obtain a low-resolution human image, and construct a training dataset containing the high-resolution human image ground truth and the low-resolution human image.

[0061] S1.1 Collect high-resolution human images from the existing COCO human image dataset, save them as high-resolution human image ground truth, collect the corresponding human pose estimation labels, and generate human keypoint heatmap ground truth using the Gaussian distribution function.

[0062] S1.2, a low-resolution human image is obtained by downsampling the ground truth value of the high-resolution human image. The high-resolution human image ground truth value and the low-resolution human image are then paired to form an image dataset.

[0063] S2. Construct a human pose estimation model for low-resolution images based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and construct a task-driven hybrid loss function.

[0064] S2.1, Construct a super-resolution reconstruction sub-network. The super-resolution reconstruction sub-network is flexibly adjusted, employing common architectures such as DenseNet, ResNet, and Transformer, and selecting appropriate network depth and width to achieve a balance between real-time performance and accuracy while facilitating task migration. Preferably, such as... Figure 3 As shown, this embodiment of the invention employs a super-resolution reconstruction sub-network based on a Cascading Residual Network structure, including a cascaded part and an upsampling part. The cascaded part performs global and local cascaded connections to make information transmission more efficient, while the upsampling part performs convolution operations and pixel rearrangement to better preserve the detailed information of the upsampled image.

[0065] Specifically, in the cascading part of the super-resolution reconstruction subnetwork, a low-resolution human image is input. First, a two-dimensional convolutional layer (Conv-0) with a kernel size of 3, a stride of 1, and padding of 1 is used to extract pixels from the spatial dimension to the channel dimension. Then, three sets of adjacent alternating cascading residual blocks and convolutional blocks (Conv) are used to extract image features and fuse multi-layer features. Each convolutional block (Conv-1, Conv-2, and Conv-3) consists of a two-dimensional convolutional layer with a kernel size of 1 and a stride of 1 and a ReLU activation layer. The role of the convolutional block is to weighted average all inputs between 1×1 convolutions. Each cascading residual block (Cascading Block-1, Cascading Block-2, and Cascading Block-3) consists of an augmented residual block (Residual-E Block) and internal convolutional blocks. The Residual-E Block uses grouped convolutions instead of ordinary 2D convolutions, significantly reducing the number of model parameters. Within each Cascading Block, three Residual-E Blocks are arranged sequentially. The output of the current augmented residual block is concatenated in parallel with the outputs of all previous augmented residual blocks along the feature dimension, and then weighted and averaged by the internal convolutional blocks for feature fusion. Simultaneously, the three augmented residual blocks within each cascading residual block share parameters, forming a recursive-like structure. This breaks down the entire standard convolution process into multiple smaller sub-operations that are performed in parallel, ultimately reducing computation and parameter count. This makes the super-resolution reconstruction sub-network more lightweight, improving the computational speed of super-resolution reconstruction. Furthermore, using grouped convolutions allows for adjustment of the number of groups in the convolutional process according to the specific needs of the scenario, balancing network performance and parameter size.

[0066] In this embodiment, the number of feature channels F of the super-resolution reconstruction sub-network is set to 64. Assuming the input image size is 1×N×M×3, where 1 is the batch size, 3 is the number of channels, and N and M are the length and width of the image, respectively, after the first 3×3 two-dimensional convolutional layer (Conv-0), the output feature map dimension is 1×N×M×F. Since the cascaded residual blocks in the local cascade do not change the number of channels in the feature map, the input of Conv-1 comes from the output of Conv-0 and the output of CascadingBlock-1. Cascading these two outputs, the input feature map dimension of Conv-1 is 1×N×M×2F, and the output after weighted averaging by Conv-1 is 1×N×M×F. Similarly, the input of Conv-2 is the concatenation of the outputs of Conv-0, Cascading Block-1, and Cascading Block-2, i.e., a feature map with a dimension of 1×N×M×3F is used as the input of Conv-2. The input to Conv-3 is a concatenation of the outputs of Conv-0, Cascading Block-1, Cascading Block-2, and Cascading Block-3, resulting in a feature map of dimension 1×N×M×4F as input. After passing through Conv-3, the output of the global concatenation part is 1×N×M×F. In the concatenation part, the aforementioned super-resolution reconstruction sub-network model employs both local and global concatenation. When a layer's feature map is fused with the feature maps of all previous layers, the complexity of the features is increased, allowing for learnable selection channels in the feature information, thereby improving the feature information extraction capability.

[0067] Specifically, in the upsampling part of the super-resolution reconstruction sub-network, a 2D convolutional layer (Conv-4) with a kernel size of 3, a stride of 1, and padding of 1 is first used to expand the dimension of the feature channels while maintaining the feature map resolution. Then, a pixel shrug layer is used to rearrange the pixels to obtain the upsampled feature map. Pixel shrug has a large receptive field and provides rich contextual information to help generate more realistic upsampled image details. Finally, a 2D convolutional layer (Conv-5) with a kernel size of 3, a stride of 1, and padding of 1 restores the number of channels in the feature map, making the number of feature channels in the reconstructed high-resolution image the same as the number of channels in the input low-resolution image.

[0068] S2.2, Construct a human pose estimation sub-network. The human pose estimation sub-network is flexibly adjusted, employing common architectures such as Stacked Hourglass Network, ResNet, and Transformer, and selecting appropriate network depth and width to achieve a balance between real-time performance and accuracy while facilitating task transfer. Preferably, such as... Figure 4 As shown, this embodiment of the invention uses the FastPose network from the AlphaPose network as the human pose estimation sub-network, including: a backbone network, a dense upsampling part, and a detection head part. In the backbone network, the correlation between feature channels is modeled, and important features are enhanced for extracting human pose features. In the dense upsampling part, convolutional operations and pixel rearrangement are performed to capture and recover detailed information from the feature map. In the detection head part, convolutional layers are used to convert the feature map into a fine-grained human keypoint heatmap, ensuring the accuracy of the human pose estimation results.

[0069] Specifically, in the backbone of the human pose estimation subnetwork, the input super-resolution reconstructed human image first passes through a Conv module, which consists of a 2D convolutional layer with a kernel size of 7, a stride of 2, and padding of 3, a BatchNorm layer, and a ReLU activation layer, mapping the input image to the feature space. Then, it passes through a Maxpooling layer with a kernel size of 3×3, a stride of 2, and padding of 1 to reduce sensitivity to spatial downsampling representation. Following this, four SE-Blocks are sequentially used to extract features. These four SE-Blocks are composed of 3, 4, 6, and 3 SE-Layers cascaded, respectively. In a single SE-Layer, the input feature map first undergoes traditional convolution to obtain an intermediate feature map. Then, the intermediate feature map undergoes compression (Squeeze) and excitation (Excitation) operations to obtain a channel-wise scaling value. This channel-wise scaling value is then multiplied with the intermediate feature map, enhancing important features while weakening unimportant features, thus making the extracted features more targeted.

[0070] Specifically, in the dense upsampling part of the human pose estimation subnetwork, three consecutive DUC modules are used to upsample the input feature map. Each DUC module consists of a Conv module and a PixelShuffle module. The Conv module is responsible for expanding the dimension of the input feature map using 2D convolution operations, and the PixelShuffle module is responsible for rearranging the pixels expanded to the feature channel dimension back to the length and width dimensions. Assuming the input image has dimensions H×W×3, the intermediate feature map after feature extraction by the backbone network has dimensions h×w×c, where w = W / d, h = H / d, and d is the downsampling coefficient. The principle of the DUC module is to compensate for the loss in length and width of the intermediate feature map using the number of channels. The three consecutive DUC modules upsample the intermediate feature map of dimension h×w×c to h×w×d. 2 The upsampled feature map is 64x4, and its dimensions are transformed to H×W×64 through pixel rearrangement. Dense upsampling reduces the loss of spatial details of the figure caused by feature map downsampling.

[0071] Specifically, in the detection head part of the human pose estimation subnetwork, a two-dimensional convolutional layer with a kernel size of 3, a stride of 1, and padding of 1 is used to convert the feature map into a fine-grained human key point heatmap, ensuring the accuracy of the human pose estimation results.

[0072] S2.3, Construct a hybrid loss function. Since current super-resolution reconstruction methods struggle to recover information equivalent to that of the real image, a specific problem scenario can be addressed. The super-resolution reconstruction sub-network used and trained does not aim to reconstruct a high-resolution image with pixel-consistent accuracy to the real image, but rather to recover sufficient effective information for human pose estimation from the low-resolution image. This improves the ability of human pose estimation algorithms to handle low-resolution human image inputs. Therefore, as... Figure 2 As shown, this embodiment of the invention simultaneously considers the loss from super-resolution reconstruction and the loss from human keypoint detection, constructing a task-driven hybrid loss function for network training.

[0073] Specifically, the hybrid loss function includes the super-resolution reconstruction loss function and the human keypoint detection loss function.

[0074] Specifically, for super-resolution reconstruction loss functions, the pixel-wise error between the reconstructed high-resolution image and the real image is calculated, aiming to reconstruct a high-resolution image that is completely consistent with the real image. Compared with L2 loss, L1 loss does not excessively penalize large errors, better preserves high-frequency details of the image, and the generated texture is not overly smooth. Super-resolution reconstruction loss functions include the pixel-wise L1 loss function and the feature reconstruction loss function. The formula for the pixel-wise L1 loss function l1 is expressed as:

[0075]

[0076] Where, f sr (I lr ) x,y,c For low-resolution human body images I lr Reconstructing the subnetwork f using super-resolution sr (·) The reconstructed pixel coordinates are (x, y) and the pixel value of channel c. For high-resolution human body image ground truth I hr The corresponding pixel coordinates are (x, y) and the pixel ground value of channel c, where H, W, and C are the length, width, and total number of channels of the image, respectively.

[0077] Since the super-resolution reconstruction subnetwork in this embodiment does not aim to restore high-resolution images as much as possible, but only to restore the semantic information of the original image in the reconstructed image, thereby improving the performance of the human pose estimation subnetwork on low-resolution image input, perceptual loss can capture the perceptual differences between the reconstructed image and the real image. Perceptual loss is divided into feature reconstruction function and style reconstruction function. In the super-resolution reconstruction task, the feature reconstruction function can be used to enhance image details; therefore, a feature reconstruction loss function is introduced. The formula is expressed as:

[0078]

[0079] in, For super-resolution reconstruction of human images, φ j (·) represents the C extracted by the j-th convolutional layer in the perceptual loss feature extraction network φ. j ×H j ×W j Feature map of size.

[0080] The ability of the reconstructed high-resolution image to recover as much semantic information as possible from the original image is crucial to improving the accuracy of human pose estimation. Therefore, the super-resolution reconstruction loss function designed in this embodiment of the invention... rec For pixel-wise L1 loss function l1 and feature reconstruction loss function The weighted summation is expressed by the formula:

[0081]

[0082] Here, k1 and k2 are the weights corresponding to the two loss terms, respectively. Different values ​​of the weights k1 and k2 of L1 loss and perceptual loss (feature reconstruction loss) in the super-resolution reconstruction task will have different effects on the reconstructed image. When k1 is much smaller than k2, the hybrid loss degenerates into perceptual loss, and the reconstructed image is almost only perceptually similar to Ground-Truth; when k1 is much larger than k2, the hybrid loss degenerates into L1 loss, and the reconstructed image almost only strives for pixel-by-pixel similarity to Ground-Truth. If different ReLU layers of the perceptual loss feature extraction network are used as outputs, it will not only lead to differences in edges, textures, shapes, colors, and other image details in the reconstructed image, but also change the semantic information contained in the reconstructed image, thus affecting the accuracy of human pose estimation in downstream tasks. In this embodiment, k1 = 1, k2 = 0.1 are selected, the perceptual loss feature extraction network φ is a VGG16 network, and ReLU3_3 is selected as the feature output.

[0083] Specifically, for human keypoint detection, the loss function is the heatmap M of the ground truth of the high-resolution human image. gt Prediction heatmap M of the human pose estimation subnetwork pre The mean square error between them is expressed by the formula:

[0084]

[0085] Among them, l est Here, H represents the loss function for human keypoint detection, where K is the total number of human keypoints, and H is the loss function for human keypoint detection. ′ and W ′ These are the length and width of the heatmap, respectively, and the heatmap M. gt It is a two-dimensional Gaussian distribution centered on the human key points of the true value of a high-resolution human image. Let be the pixel value of the heatmap corresponding to the k-th key point with pixel coordinates (x, y). Heatmap M generated for the human pose estimation subnetwork pre The pixel value of the heatmap corresponding to the k-th key point with pixel coordinates (x, y).

[0086] Heatmap M pre Generated by the human pose estimation subnetwork, the formula is expressed as:

[0087] M pre =f pose (f sr (I lr ))

[0088] Among them, I lr For low-resolution human body images, fsr (·) represents the super-resolution reconstruction subnetwork, f pose (·) represents the human pose estimation subnetwork.

[0089] Specifically, for the task-driven hybrid loss function, the hybrid loss function uses pose information to train the super-resolution reconstruction sub-network, driving the super-resolution reconstruction sub-network to generate high-resolution reconstructed images that are beneficial for downstream task processing. The hybrid loss function is defined as l rec and l est Weighted summation:

[0090] l com =αl rec +βl est

[0091] Among them, l com For the mixed loss function, l rec For the super-resolution reconstruction loss function, l est The loss function for human keypoint detection is defined by α and β, which are the weights of the two loss terms and are used to balance the weights of the two loss terms.

[0092] Since the super-resolution reconstruction subnetwork and the human pose estimation subnetwork are deep neural networks, both subnetworks are differentiable and can be trained using the backpropagation algorithm. During network training, the chain rule is used to iterate through the network. est Find the parameters θ of the super-resolution network. sr Partial derivatives:

[0093]

[0094] in, Obtained from the super-resolution reconstruction loss function. It is obtained from the outputs of each layer of the super-resolution reconstruction subnetwork. Obtained from the human keypoint detection loss function. This provides the parameters of each layer in the human pose estimation subnetwork. Therefore, it is possible to... The parameter update uses the backpropagation algorithm.

[0095] In the hybrid loss function formula, the impact of super-resolution reconstruction loss and human keypoint detection loss on the reconstructed high-resolution image is controlled by parameters α and β. When β is much smaller than α, the hybrid loss function approximates the super-resolution reconstruction loss function that seeks to recover the semantic information of the original image; when β is much larger than α, the hybrid loss function expects the super-resolution sub-network to reconstruct features that are beneficial for human keypoint detection. Under this setting, since the semantic relationship between the reconstructed high-resolution image and the ground-truth high-resolution image is ignored, it is difficult to intuitively see the location of keypoints in the reconstructed high-resolution image. In the embodiment, setting α = 1 and β = 1000 allows for the training of an ideal super-resolution reconstruction sub-network.

[0096] S3: Input the low-resolution human image into the super-resolution reconstruction sub-network to obtain the super-resolution reconstructed human image, and then input it into the human pose estimation sub-network to obtain the human pose estimation result. The human pose estimation model based on the low-resolution image reconstructed by super-resolution is trained based on the training dataset and the hybrid loss function.

[0097] Specifically, a predefined hybrid loss function is used, and the human pose estimation model based on super-resolution reconstruction of low-resolution images is trained end-to-end on the training dataset. Data augmentation techniques are also employed to improve the model's generalization ability.

[0098] S4. Input the low-resolution human image to be detected into the trained low-resolution image human pose estimation model based on super-resolution reconstruction, and obtain the human pose estimation result from the output heatmap.

[0099] Specifically, the low-resolution human image to be detected is transformed to a specific image resolution of 32×32 using bilinear interpolation. The human image at this specific resolution is then horizontally flipped. Both the unflipped and flipped images are normalized and simultaneously fed into the aforementioned low-resolution image human pose estimation model based on super-resolution reconstruction for inference, resulting in two human keypoint heatmaps. The corresponding human keypoint heatmap of the horizontally flipped image is then horizontally flipped again, and the average of the two heatmaps is taken to obtain the final human keypoint heatmap. The location of the point with the highest probability value on the human keypoint heatmap is taken as the human keypoint location obtained by the model inference, i.e., the result of human pose estimation.

[0100] In summary, a low-resolution image human pose estimation method based on super-resolution reconstruction reconstructs the low-resolution human image using a super-resolution reconstruction sub-network, followed by prediction using a human pose estimation sub-network. This fundamentally increases the amount of image information used for pose estimation, thereby improving the detection accuracy of low-resolution human pose estimation. By designing a task-driven hybrid loss function, the super-resolution reconstruction sub-network maximizes the amount of information that the low-resolution input image can express, enabling it to generate a high-resolution image containing sufficient low-level image features and high-level semantic features. The super-resolution reconstruction sub-network and the human pose estimation sub-network are trained based on the loss function, allowing the model to estimate human pose more accurately from the input low-resolution image.

[0101] Based on the same inventive concept, such as Figure 5 As shown, this embodiment of the invention also provides a low-resolution image human pose estimation device 500 based on super-resolution reconstruction, including: a dataset construction module 510, a model construction module 520, a model training module 530, and a model testing module 540.

[0102] The dataset construction module 510 is used to downsample the ground truth of high-resolution human images to obtain low-resolution human images, and to construct a training dataset containing both high-resolution human image ground truth and low-resolution human images.

[0103] The model building module 520 is used to build a human pose estimation model for low-resolution images based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and to build a task-driven hybrid loss function.

[0104] The model training module 530 is used to input low-resolution human images into the super-resolution reconstruction sub-network to obtain super-resolution reconstructed human images, and then input them into the human pose estimation sub-network to obtain human pose estimation results. The human pose estimation model based on super-resolution reconstruction of low-resolution images is trained based on the training dataset and the hybrid loss function.

[0105] The model testing module 540 is used to input the low-resolution human image to be detected into the trained low-resolution image human pose estimation model based on super-resolution reconstruction, and obtain the human pose estimation result from the output heatmap.

[0106] Based on the same inventive concept, this invention also provides a low-resolution image human pose estimation device based on super-resolution reconstruction, including a memory and one or more processors. The memory is used to store a computer program, and the processor is used to implement the above-described low-resolution image human pose estimation method based on super-resolution reconstruction when the computer program is executed.

[0107] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for estimating human pose in low-resolution images based on super-resolution reconstruction.

[0108] It should be noted that the human pose estimation device, the human pose estimation equipment, and the computer-readable storage medium based on super-resolution reconstruction of low-resolution images provided in the above embodiments all belong to the same inventive concept as the human pose estimation method based on super-resolution reconstruction of low-resolution images. For details of their specific implementation process, please refer to the embodiments of the human pose estimation method based on super-resolution reconstruction of low-resolution images, which will not be repeated here.

[0109] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for human pose estimation in low-resolution images based on super-resolution reconstruction, characterized in that, Includes the following steps: The high-resolution human images are downsampled to obtain low-resolution human images, and a training dataset containing both high-resolution human images and low-resolution human images is constructed. A human pose estimation model for low-resolution images based on super-resolution reconstruction is constructed, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and a task-driven hybrid loss function is built. The formula is expressed as: ， in, The loss function for super-resolution reconstruction is... The loss function for human keypoint detection is... and Set the weights for each. and To train and obtain an ideal super-resolution reconstruction subnetwork; Among them, the super-resolution reconstruction loss function pixel-wise L1 loss function and feature reconstruction loss function The weighted summation is expressed by the formula: ， in, and The weights corresponding to the two loss terms are selected respectively. , Perceptual loss feature extraction network For the VGG16 network, ReLU3_3 is selected as the feature output to obtain the best reconstructed image effect; Pixel-wise L1 loss function The formula is expressed as: ， in, For low-resolution human body images Subnetwork Reconstruction via Super-Resolution The reconstructed pixel coordinates are and channels pixel values, For high-resolution human body image ground truth The corresponding pixel coordinates are and channels The pixel true value, , and These represent the image's length, width, and total number of channels, respectively. Feature Reconstruction Loss Function The formula is expressed as: ， in, To reconstruct human images at super resolution, For perceptual loss feature extraction network The first in Extracted by convolutional layers Size feature map; The low-resolution human image is input into the super-resolution reconstruction sub-network to obtain the super-resolution reconstructed human image, and then input into the human pose estimation sub-network to obtain the human pose estimation result. The human pose estimation model based on the low-resolution image of the super-resolution reconstruction is trained based on the training dataset and the hybrid loss function. The low-resolution human image to be detected is input into a trained low-resolution image human pose estimation model based on super-resolution reconstruction, and the human pose estimation result is obtained from the output heatmap.

2. The method for human pose estimation in low-resolution images based on super-resolution reconstruction according to claim 1, characterized in that, The loss function for human keypoint detection is a heatmap of the ground truth of a high-resolution human image. Prediction heatmap with human pose estimation subnetwork The mean square error between them is expressed by the formula: ， in, The loss function for human keypoint detection is... The total number of key points on the human body and These are the length and width of the heatmap, respectively. It is a two-dimensional Gaussian distribution centered on the human key points of the true value of a high-resolution human image. For the first There are 1 key point and the pixel coordinates are The corresponding heatmap pixel values, Heatmap generated for the human pose estimation subnetwork The Middle There are 1 key point and the pixel coordinates are The corresponding heatmap pixel values; Heat map Generated by the human pose estimation subnetwork, the formula is expressed as: ， in, For low-resolution human body images, To reconstruct the subnetwork for super-resolution, This is a subnetwork for human pose estimation.

3. The method for human pose estimation in low-resolution images based on super-resolution reconstruction according to claim 1, characterized in that, The constructed super-resolution reconstruction subnetwork adopts a cascaded residual network structure, including a cascaded part and an upsampling part; In the cascaded part, a convolutional layer is first used to extract the pixels of the low-resolution human image from the spatial dimension to the channel dimension. Then, multiple alternating cascaded residual blocks and convolutional blocks are used for image feature extraction and multi-layer feature fusion to obtain a feature map. In the upsampling part, an upsampling layer is used to rearrange pixels to upsample the feature map to a high resolution to obtain a super-resolution reconstructed human image.

4. The method for human pose estimation in low-resolution images based on super-resolution reconstruction according to claim 1, characterized in that, The constructed human pose estimation subnetwork adopts the FastPose network based on the AlphaPose network, which includes: a backbone network part, a dense upsampling part and a detection head part; In the backbone network, the super-resolution reconstructed human image is input, and the features of the human pose are extracted using sequentially connected convolutional layers, max pooling layers, and multiple compression excitation modules. In the dense upsampling part, the extracted human pose features are input, and multiple alternating convolutional layers and pixel rearrangement layers are used to capture and recover the detailed information of the features to obtain the upsampled feature map. In the detection head section, convolutional layers are used to convert the upsampled feature maps into fine-grained human keypoint prediction heatmaps.

5. The method for human pose estimation in low-resolution images based on super-resolution reconstruction according to claim 3, characterized in that, In the cascaded part of the super-resolution reconstruction subnetwork, the structure of each cascaded residual block includes at least one enhanced residual block and an inner convolutional block. The enhanced residual block adopts grouped convolution, and the outputs of the enhanced residual blocks are concatenated in parallel along the feature dimension and then weighted and averaged by the inner convolutional block for feature fusion.

6. A human pose estimation device for low-resolution images based on super-resolution reconstruction, implemented using the human pose estimation method for low-resolution images based on super-resolution reconstruction as described in any one of claims 1-5, characterized in that... include: The system includes a dataset building module, a model building module, a model training module, and a model testing module. The dataset construction module is used to downsample the ground truth of high-resolution human images to obtain low-resolution human images, and to construct a training dataset containing both high-resolution human image ground truth and low-resolution human images. The model building module is used to build a human pose estimation model for low-resolution images based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and to build a task-driven hybrid loss function. The model training module is used to input low-resolution human images into the super-resolution reconstruction sub-network to obtain super-resolution reconstructed human images, and then input them into the human pose estimation sub-network to obtain human pose estimation results. The human pose estimation model based on super-resolution reconstruction of low-resolution images is trained based on the training dataset and the hybrid loss function. The model testing module is used to input the low-resolution human image to be detected into the trained low-resolution image human pose estimation model based on super-resolution reconstruction, and obtain the human pose estimation result from the output heatmap.

7. A device for estimating human pose in low-resolution images based on super-resolution reconstruction, comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that, The processor is configured to, when executing the computer program, implement the human pose estimation method for low-resolution images based on super-resolution reconstruction as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the human pose estimation method for low-resolution images based on super-resolution reconstruction as described in any one of claims 1-5.

Images