Low-resolution image human body posture estimation method and device based on super-resolution reconstruction

By applying the combination of super-resolution reconstruction and human pose estimation subnet on low-resolution images, the model is trained using a hybrid loss function, and the problem of low accuracy of human pose estimation in low-resolution images is solved, improving panoramic monitoring and no The accuracy of attitude estimation in scenarios such as human-machine patrol.

CN119027502AActive Publication Date: 2024-11-26ZHEJIANG UNIV

Patent Information

Application Number
CN202411022203.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-26
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

The existing two-dimensional human posture estimation method based on convolutional neural networks decreases accuracy when inputting low-resolution image, especially in panoramic camera monitoring or drone patrol scenarios, facing low image resolution and small human body size Challenges, resulting in reduced accuracy of human posture estimation.

Method used

Using a low-resolution image human pose estimation method based on super-resolution reconstruction, the super-resolution reconstruction subnet and human pose estimation subnet are constructed, combined with a task-driven hybrid loss function, improve the resolution and human pose of low-resolution images. The estimated accuracy is suitable for application scenarios such as panoramic surveillance and drone patrol.

Benefits of technology

After reconstructing low-resolution images through super-resolution reconstruction subnet, the human posture estimation subnet can predict human posture more accurately, improve the detection accuracy of low-resolution human posture estimation, and enhance the model's information expression on low-resolution images Ability to improve the accuracy of posture estimation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027502A_ABST
    Figure CN119027502A_ABST
Patent Text Reader

Abstract

The invention discloses a low-resolution image human body posture estimation method and device based on super-resolution reconstruction, and the method comprises the following steps: carrying out the down-sampling processing of a truth value of a high-resolution human body image, obtaining a low-resolution human body image, and constructing a training data set containing a high-resolution human body image pair and a low-resolution human body image pair; a low-resolution image human body posture estimation model based on super-resolution reconstruction is constructed, the model comprises a super-resolution reconstruction sub-network and a human body posture estimation sub-network, and a task-driven mixed loss function is constructed; and inputting a low-resolution human body image into the super-resolution reconstruction sub-network to obtain a super-resolution reconstruction human body image, then inputting the super-resolution reconstruction human body image into the human body posture estimation sub-network to obtain a human body posture estimation result, and training the model based on the training data set and the mixed loss function for human body posture estimation. The method not only improves the detection precision, but also ensures the real-time performance of the model, is suitable for scenes such as panoramic monitoring and unmanned aerial vehicle patrol, and has high application value and popularization prospect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method and device for estimating human body posture from a low-resolution image based on super-resolution reconstruction. Background Art

[0002] Human posture estimation, as a widely concerned topic in the field of computer vision, is an important step in describing and understanding human behavior patterns in images and videos. Its purpose is to locate human joints such as shoulders, knees, and wrists. It plays a key role in various tasks such as object detection, human-computer interaction, pedestrian tracking, and action recognition.

[0003] Human pose estimation is divided into two-dimensional human pose estimation and three-dimensional human pose estimation according to the spatial dimension of the predicted results. With the introduction and development of deep learning, the two-dimensional pose estimation method based on convolutional neural network has overcome the difficulties faced by early research, including occlusion and lack of generalization. However, most of the two-dimensional human pose estimation methods based on convolutional neural network presuppose that the human body in the image occupies enough pixels and provides enough appearance texture features, and the accuracy will basically decrease when facing the input of lower resolution images. Most of the two-dimensional multi-person human pose estimation adopts a top-down method, first detecting the human target, and then performing two-dimensional single-person human pose estimation on the detected human target. Therefore, if the problem of two-dimensional single-person human pose estimation under low-resolution image input can be solved, it can be easily expanded to the situation of multi-person human pose estimation.

[0004] The two-dimensional human pose estimation method based on convolutional neural network can be divided into two types according to the output: direct regression method and heat map-based method. The latter has better performance, but the resolution of the heat map generated by the latter is usually lower than the resolution of the input image, which leads to positioning errors in the coordinates of the key points of the human body on the heat map. When the resolution of the input image is high, the positioning error of the output key points is still within the acceptable range, but when the resolution of the input image is low, the positioning error of the output key points will become larger, which will lead to a decrease in the accuracy of human pose estimation. In the scene of panoramic camera monitoring or drone patrol shooting, human pose estimation faces challenges such as low image resolution and small human body size. In order to solve the problem that the heat map-based method causes the resolution of the feature map to decrease in the forward process of the network, thereby reducing the accuracy of pose estimation, Cheng et al. adopted HigherHRNet (Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020), which is a network with multiple parallel forward processes based on the HRNet network architecture. It ensures that the resolution of the feature map is always consistent with the input image in the forward process, and also designs a high-resolution feature pyramid. It uses multi-resolution supervision to enable features at different layers to learn information of different scales, and generates high-resolution feature maps through multi-scale fusion to predict high-quality heat maps, reducing the loss of information caused by the reduction of image resolution in the forward process of the network.

[0005] But in essence, these methods that use multiple forward networks in parallel to minimize the loss of feature map resolution caused by the network forward are basically based on the idea of ​​"reducing losses". They just make use of the information of low-resolution input images as much as possible to reduce the loss of information in the network forward process. However, the already scarce information in low-resolution images is the real bottleneck that restricts human posture estimation and detection in low-resolution images. Therefore, this type of method does not solve the essence of the problem. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a method and device for human posture estimation in low-resolution images based on super-resolution reconstruction, which can improve the resolution of low-resolution human body images and improve the accuracy of human posture estimation of small targets at a long distance. At the same time, a hybrid loss including super-resolution reconstruction loss and human key point detection loss is proposed, so that the super-resolution reconstructed human body image contains enough high-level semantic information, thereby improving the resolution ability of the human posture estimation network and improving the estimation accuracy, which is suitable for application scenarios such as panoramic monitoring and drone patrols.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a method for estimating a human body posture from a low-resolution image based on super-resolution reconstruction, comprising the following steps:

[0009] Downsampling the true value of the high-resolution human image to obtain a low-resolution human image, and constructing a training data set including the true value of the high-resolution human image and the low-resolution human image;

[0010] Build a low-resolution image human pose estimation model based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and construct a task-driven hybrid loss function;

[0011] The low-resolution human image is input into the super-resolution reconstruction sub-network to obtain the super-resolution reconstructed human image, which is then input into the human pose estimation sub-network to obtain the human pose estimation result. The low-resolution image human pose estimation model based on super-resolution reconstruction is trained based on the training data set and the mixed loss function.

[0012] The low-resolution human image to be detected is input into the trained low-resolution image human pose estimation model based on super-resolution reconstruction, and the human pose estimation result is obtained from the output heat map.

[0013] Preferably, the task-driven hybrid loss function includes a super-resolution reconstruction loss function and a human key point detection loss function, and the formula is expressed as:

[0014] l com =αl rec +βl est

[0015] Among them, l com is the mixed loss function, l rec is the super-resolution reconstruction loss function, l est is the loss function for detecting key points of the human body, α and β are the weights of the two, which are used to balance the weight parameters of the two loss terms.

[0016] Preferably, the super-resolution reconstruction loss function includes a pixel-by-pixel L1 loss function and a feature reconstruction loss function;

[0017] The formula of the pixel-by-pixel L1 loss function l1 is expressed as:

[0018]

[0019] Among them, f sr (I lr ) x,y,c is a low-resolution human image I lr Through super-resolution reconstruction sub-network f sr (·) The reconstructed pixel coordinates are (x, y) and the pixel value of channel c, is the ground truth value of high-resolution human image I hr The corresponding pixel coordinates are (x, y) and the true pixel value of channel c, H, W and C are the length, width and total number of channels of the image respectively;

[0020] Feature reconstruction loss function The formula is expressed as:

[0021]

[0022] in, To reconstruct human images with super resolution, φ j (·) is C extracted by the jth convolutional layer in the perceptual loss feature extraction network φ. j ×H j ×W j Feature map of size;

[0023] Super-resolution reconstruction loss function l rec is the pixel-by-pixel L1 loss function l1 and feature reconstruction loss function The weighted sum of is expressed as:

[0024]

[0025] Among them, k1 and k2 are the weights corresponding to the two loss terms respectively.

[0026] Preferably, the loss function for detecting key points of a human body is a heat map M of the true value of a high-resolution human body image. gt Prediction heat map M of the human pose estimation subnetwork pre The mean square error between them is expressed as:

[0027]

[0028] Among them, l est is the human key point detection loss function, K is the total number of human key points, H ′and W ′ are the length and width of the heat map, respectively. gt It is a two-dimensional Gaussian distribution centered on the key points of the true value of the high-resolution human image. is the heat map pixel value corresponding to the kth key point with pixel coordinates (x, y), Heatmap M generated for the human pose estimation subnetwork pre The heat map pixel value corresponding to the kth key point in and the pixel coordinate is (x, y);

[0029] Heatmap M pre Generated by the human body posture estimation subnetwork, the formula is expressed as:

[0030] M pre =f pose (f sr (I lr ))

[0031] Among them, I lr is a low-resolution human image, f sr (·) is the super-resolution reconstruction sub-network, f pose (·) is the human pose estimation subnetwork.

[0032] Preferably, the constructed super-resolution reconstruction sub-network adopts a cascaded residual network structure, including a cascade part and an upsampling part;

[0033] In the cascade part, a convolution layer is first used to extract the pixels of the low-resolution human image from the spatial dimension to the channel dimension, and then multiple alternating cascade residual blocks and convolution blocks are used to extract image features and fuse multi-layer features to obtain a feature map;

[0034] In the upsampling part, the upsampling layer is used to perform pixel rearrangement to upsample the feature map to a high resolution to obtain a super-resolution reconstructed human image.

[0035] Preferably, the constructed human posture estimation subnetwork adopts a FastPose network based on the AlphaPose network, including: a backbone network part, a dense upsampling part and a detection head part;

[0036] In the backbone network, super-resolution reconstructed human images are input, and the features of human posture are extracted using sequentially connected convolutional layers, maximum pooling layers, and multiple compression excitation modules;

[0037] In the dense upsampling part, the extracted human posture features are input, and multiple alternating convolutional layers and pixel rearrangement layers are used to capture and restore the detailed information of the features to obtain an upsampled feature map;

[0038] In the detection head, a convolutional layer is used to convert the upsampled feature map into a fine-grained human key point prediction heat map.

[0039] Preferably, in the cascade part of the super-resolution reconstruction subnetwork, the structure of each cascade residual block includes at least one enhanced residual block and an internal convolution block, wherein the enhanced residual block adopts grouped convolution, and the outputs of the enhanced residual blocks are connected in parallel in the feature dimension and then feature fused by weighted averaging of the internal convolution block.

[0040] In a second aspect, to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a low-resolution image human posture estimation device based on super-resolution reconstruction, comprising: a data set construction module, a model construction module, a model training module and a model testing module;

[0041] The data set construction module is used to downsample the true value of the high-resolution human image to obtain a low-resolution human image, and to construct a training data set including the true value of the high-resolution human image and the low-resolution human image;

[0042] The model building module is used to build a low-resolution image human posture estimation model based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human posture estimation subnetwork, and to build a task-driven hybrid loss function;

[0043] The model training module is used to input the low-resolution human body image into the super-resolution reconstruction subnetwork to obtain the super-resolution reconstructed human body image, and then input it into the human body posture estimation subnetwork to obtain the human body posture estimation result, and train the low-resolution image human body posture estimation model based on super-resolution reconstruction based on the training data set and the mixed loss function;

[0044] The model testing module is used to input the low-resolution human image to be detected into a trained low-resolution image human posture estimation model based on super-resolution reconstruction, and obtain the human posture estimation result from the output heat map.

[0045] In the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides a low-resolution image human body posture estimation device based on super-resolution reconstruction, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned low-resolution image human body posture estimation method based on super-resolution reconstruction when executing the computer program.

[0046] In a fourth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned low-resolution image human body posture estimation method based on super-resolution reconstruction is implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The present invention reconstructs low-resolution human body images through a super-resolution reconstruction subnetwork and then predicts them through a human posture estimation subnetwork, which can essentially increase the amount of image information used for posture estimation, thereby improving the detection accuracy of low-resolution human posture estimation.

[0049] (2) The present invention designs a task-driven hybrid loss function so that the super-resolution reconstruction subnetwork can increase the amount of information that can be expressed by the low-resolution input image as much as possible, and is capable of generating a high-resolution image containing enough low-level image features and high-level semantic features from the low-resolution image. The super-resolution reconstruction subnetwork and the human body posture estimation subnetwork are trained based on the hybrid loss function, so that the model can more accurately estimate the human body posture from the input low-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 1 is a flow chart of a method for estimating a human body posture from a low-resolution image based on super-resolution reconstruction provided by an embodiment of the present invention;

[0052] Figure 2 Schematic diagram of the framework of a method for estimating human body posture from a low-resolution image based on super-resolution reconstruction provided by an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of the network structure of a super-resolution reconstruction subnetwork provided by an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of the network structure of a human posture estimation subnetwork provided by an embodiment of the present invention;

[0055] Figure 5 It is a structural schematic diagram of a low-resolution image human posture estimation device based on super-resolution reconstruction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0057] The inventive concept of the present invention is: in response to the problem of low accuracy of human body posture estimation methods using low-resolution images in the prior art, an embodiment of the present invention provides a method and device for human body posture estimation in low-resolution images based on super-resolution reconstruction. First, a training data set is constructed by downsampling high-resolution human body images, and a super-resolution reconstruction subnetwork and a human body posture estimation subnetwork are combined. A task-driven hybrid loss function is used to perform model training, so that the super-resolution reconstructed human body image has sufficient high-level semantic information, thereby improving the accuracy of human body posture estimation in low-resolution images.

[0058] Among them, the super-resolution reconstruction subnetwork and the human pose estimation subnetwork can be designed as very lightweight subnetworks, which can make full use of the computing resources of the device to achieve better human pose estimation effects, thereby ensuring the real-time performance of the model in practical applications.

[0059] Figure 1 is a flow chart of a method for estimating human body posture from a low-resolution image based on super-resolution reconstruction provided by an embodiment of the present invention, Figure 2 Schematic diagram of the framework of the method for estimating human body posture from low-resolution images based on super-resolution reconstruction provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, the embodiment provides a method for estimating human body posture from a low-resolution image based on super-resolution reconstruction, comprising the following steps:

[0060] S1, down-sampling the true value of the high-resolution human image to obtain a low-resolution human image, and constructing a training data set including the true value of the high-resolution human image and the low-resolution human image.

[0061] S1.1, collect high-resolution human images from the existing COCO human image dataset, save them as high-resolution human image truth values, collect the corresponding human posture estimation labels, and use the Gaussian distribution function to generate the true value of the human key point heat map.

[0062] S1.2, by downsampling the true value of the high-resolution human image, a low-resolution human image is obtained. The high-resolution human image true value and the low-resolution human image are combined into image pairs to construct a training data set.

[0063] S2, builds a low-resolution image human pose estimation model based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and constructs a task-driven hybrid loss function.

[0064] S2.1, build a super-resolution reconstruction sub-network. The super-resolution reconstruction sub-network is flexibly adjusted, and common architectures such as DenseNet, ResNet, and Transformer are adopted, and appropriate network depth and width are selected to achieve a balance between real-time performance and accuracy while facilitating task migration. Preferably, Figure 3 As shown, an embodiment of the present invention adopts a super-resolution reconstruction subnetwork based on a cascading residual network (Cascading Residual Network) structure, including a cascading part and an upsampling part. Global and local cascade connections are performed in the cascading part to make information transmission more efficient, and convolution operations and pixel rearrangement are performed in the upsampling part to better retain the detail information of the upsampled image.

[0065] Specifically, in the cascading part of the super-resolution reconstruction subnetwork, a low-resolution human body image is input, and a two-dimensional convolution layer (Conv-0) with a convolution kernel size of 3, a stride of 1, and a padding of 1 is first used to extract pixels from the spatial dimension to the channel dimension. Then, three groups of adjacent alternating cascade residual blocks (Cascading Block) and convolution blocks (Conv) are used in sequence to extract image features and fuse multi-layer features. Each convolution block (Conv-1, Conv-2, and Conv-3) consists of a two-dimensional convolution layer with a convolution kernel size of 1 and a stride of 1 and a ReLU activation layer. The role of the convolution block is to weighted average all inputs between 1×1 convolutions. Each cascaded residual block (Cascading Block-1, Cascading Block-2, and Cascading Block-3) is composed of an enhanced residual block (Residual-E Block) and an internal convolution block. Grouped convolution is used inside the Residual-E Block to replace ordinary two-dimensional convolution, which can greatly reduce the number of model parameters. Three Residual-E Blocks are arranged in sequence inside each Cascading Block. The output of the current enhanced residual block is connected in parallel with the output of all previous enhanced residual blocks in the feature dimension and then weighted averaged through the internal convolution block for feature fusion. At the same time, the three enhanced residual blocks inside each cascaded residual block share parameters, forming a recursive form, splitting the entire standard convolution process into multiple groups of smaller sub-operations to be performed in parallel, ultimately reducing the amount of calculation and parameters, making the super-resolution reconstruction subnetwork lighter, and improving the speed of super-resolution reconstruction. At the same time, the use of grouped convolution can adjust the number of grouped convolutions according to actual scene requirements, balancing network performance and parameter size.

[0066] In the embodiment, the number of feature channels F of the super-resolution reconstruction subnetwork is set to 64. Assuming that the input image size of the network is 1×N×M×3, where 1 is the batch size, 3 is the number of channels, and N and M are the length and width of the image, respectively, after the first 3×3 two-dimensional convolution layer (Conv-0), the output feature map dimension is 1×N×M×F. Since the cascade residual block in the local cascade does not change the number of channels of the feature map, the input of Conv-1 comes from the output of Conv-0 and the output of CascadingBlock-1 respectively. The above two outputs are cascaded, then the input feature map dimension of Conv-1 is 1×N×M×2F, and the output after weighted averaging of Conv-1 is 1×N×M×F. Similarly, the input of Conv-2 is the cascade of the output of Conv-0, the output of Cascading Block-1, and the output of Cascading Block-2, that is, the feature map with a dimension of 1×N×M×3F is used as the input of Conv-2. The input of Conv-3 is the cascade of the output of Conv-0, the output of Cascading Block-1, the output of Cascading Block-2, and the output of Cascading Block-3, that is, the feature map with a dimension of 1×N×M×4F is used as the input of Conv-3. After Conv-3, the output of the global cascade part is 1×N×M×F. In the cascade part, the above super-resolution reconstruction sub-network model adopts local cascade and global cascade. When the layer feature map fuses the feature maps of all previous layers, the complexity of the features is increased, so that the feature information has a learnable selection channel, thereby improving the feature information extraction capability.

[0067] Specifically, in the upsampling part (Upsampling) of the super-resolution reconstruction subnetwork, a two-dimensional convolution layer (Conv-4) with a convolution kernel size of 3, a stride of 1, and a padding of 1 is first used to expand the dimension of the feature channel while keeping the resolution of the feature map unchanged. Then, a pixel rearrangement layer (PixelShuffle) is used to rearrange the pixels to obtain an upsampled feature map. PixelShuffle has a large receptive field and provides rich contextual information to help generate more realistic upsampled image details. Finally, a two-dimensional convolution layer (Conv-5) with a convolution kernel size of 3, a stride of 1, and a padding layer of 1 is used to restore the number of channels of the feature map, so that the number of feature channels of the reconstructed high-resolution image is the same as the number of channels of the input low-resolution image.

[0068] S2.2, construct a human posture estimation subnetwork. The human posture estimation subnetwork is flexibly adjusted, and general architectures such as Stacked Hourglass Network, ResNet, and Transformer are adopted, and appropriate network depth and width are selected to achieve a balance between real-time performance and accuracy while facilitating task migration. Preferably, Figure 4 As shown, the embodiment of the present invention adopts the FastPose network based on the AlphaPose network as the human posture estimation subnetwork, including: a backbone network part, a dense upsampling part and a detection head part. In the backbone network part, the correlation between feature channels is modeled, and important features are enhanced to extract the features of human posture; in the dense upsampling part, convolution operations and pixel rearrangement are performed to capture and restore the detailed information of the feature map; in the detection head part, a convolution layer is used to convert the feature map into a fine-grained human key point heat map to ensure the accuracy of the human posture estimation result.

[0069] Specifically, in the backbone network of the human pose estimation subnetwork, the super-resolution reconstructed human image is input and first passes through a Conv module, which consists of a two-dimensional convolution layer with a convolution kernel size of 7, a stride of 2, and a padding of 3, a BatchNorm layer, and a ReLU activation layer to map the input image to the feature space. Then it passes through a maximum pooling layer (Maxpool) with a kernel size of 3×3, a stride of 2, and a padding of 1 to reduce the sensitivity to spatial downsampling representation. Then, four compression excitation modules (SE-Blocks) are used in sequence to extract features. These four SE-Blocks are composed of 3, 4, 6, and 3 SE-Layers connected in series, respectively. In a single SE-Layer, the input feature map first undergoes traditional convolution to obtain an intermediate feature map, and then the intermediate feature map undergoes compression (Squeeze) and excitation (Excitation) operations to obtain the channel-by-channel scaling value of the intermediate feature map, and then the channel-by-channel scaling value is multiplied with the intermediate feature map, which enhances important features while weakening unimportant features, thereby making the extracted features more directional.

[0070] Specifically, in the dense upsampling part of the human pose estimation subnetwork, three consecutive DUC modules are used to upsample the input feature map. The DUC module consists of a Conv module and a pixel rearrangement (PixelShuffle) module. The Conv module is responsible for expanding the input feature map dimension using a two-dimensional convolution operation, and the PixelShuffle module is responsible for rearranging the pixels expanded to the feature channel dimension to the length and width dimensions. Assuming that the dimension of the input image is H×W×3, the dimension of the intermediate feature map after feature extraction by the backbone network is h×w×c, where w=W / d, h=H / d, and d is the downsampling coefficient. The principle of the DUC module is to compensate for the loss of length and width of the intermediate feature map by using the number of channels. The three consecutive DUC modules will upsample the intermediate feature map of dimension h×w×c to h×w×d. 2 ×64, and deform the upsampled feature map dimension to H×W×64 by rearranging pixels. Dense upsampling reduces the loss of character details in the spatial dimension caused by the downsampling of the feature map to a certain extent.

[0071] Specifically, in the detection head part of the human pose estimation subnetwork, a two-dimensional convolutional layer with a convolution kernel size of 3, a step size of 1, and a padding of 1 is used to convert the feature map into a fine-grained human key point heat map to ensure the accuracy of the human pose estimation results.

[0072] S2.3, construct a hybrid loss function. Since it is difficult for current super-resolution reconstruction methods to recover information equivalent to that of real images, we can start from a specific problem scenario and use and train a super-resolution reconstruction subnetwork that does not aim to reconstruct a high-resolution image that is pixel-consistent with the real image, but to recover enough effective information for human pose estimation in low-resolution images, thereby improving the ability of human pose estimation algorithms to handle low-resolution human image inputs. Therefore, Figure 2 As shown, the embodiment of the present invention considers the loss of super-resolution reconstruction and the loss of human key point detection at the same time, and constructs a task-driven hybrid loss function for network training.

[0073] Specifically, the hybrid loss function includes a super-resolution reconstruction loss function and a human key point detection loss function.

[0074] Specifically, for the super-resolution reconstruction loss function, the pixel-by-pixel error between the reconstructed high-resolution image and the real image is calculated in order to reconstruct a high-resolution image that is completely consistent with the real image. Compared with the L2 loss, the L1 loss does not overly penalize large errors, better preserves the high-frequency details of the image, and the resulting texture is not too smooth. The super-resolution reconstruction loss function includes the pixel-by-pixel L1 loss function and the feature reconstruction loss function. The formula of the pixel-by-pixel L1 loss function l1 is expressed as:

[0075]

[0076] Among them, f sr (I lr ) x,y,c is a low-resolution human image I lr Through super-resolution reconstruction sub-network f sr (·) The reconstructed pixel coordinates are (x, y) and the pixel value of channel c, is the ground truth value of high-resolution human image I hr The corresponding pixel coordinates are (x, y) and the true pixel value of channel c. H, W, and C are the length, width, and total number of channels of the image, respectively.

[0077] Since the super-resolution reconstruction sub-network in the embodiment of the present invention does not aim to restore the high-resolution image as much as possible, but only seeks to restore the semantic information of the original image by reconstructing the image, thereby improving the performance of the human pose estimation sub-network in low-resolution image input. Perceptual loss can capture the perceptual difference between the reconstructed image and the real image, and perceptual loss is divided into feature reconstruction function and style reconstruction function. In the super-resolution reconstruction task, the feature reconstruction function can be used to enhance the details of the image, so the feature reconstruction loss function is introduced. The formula is:

[0078]

[0079] in, To reconstruct human images with super resolution, φ j (·) is C extracted by the jth convolutional layer in the perceptual loss feature extraction network φ. j ×H j ×W j Size feature map.

[0080] Whether the reconstructed high-resolution image can restore as much semantic information of the original image as possible is the key to improving the accuracy of human posture estimation. Therefore, the super-resolution reconstruction loss function designed in the embodiment of the present invention is rec is the pixel-by-pixel L1 loss function l1 and feature reconstruction loss function The weighted sum of is expressed as:

[0081]

[0082] Among them, k1 and k2 are the weights corresponding to the two loss terms respectively. Different values ​​of the weights k1 and k2 of L1 loss and perceptual loss (feature reconstruction loss) in the super-resolution reconstruction task will have different effects on the effect of the reconstructed image. When k1 is much smaller than k2, the mixed loss degenerates into perceptual loss, and the reconstructed image is almost only perceptually similar to Ground-Truth; when k1 is much larger than k2, the mixed loss degenerates into L1 loss, and the reconstructed image almost only pursues pixel-by-pixel closeness to Ground-Truth. If different ReLU layers of the perceptual loss feature extraction network are selected as output, not only will the reconstructed image have differences in edges, textures, shapes, colors and other picture details, but the semantic information contained in the reconstructed image will also change, which will affect the accuracy of human posture estimation in downstream tasks. In the embodiment, k1=1, k2=0.1 are selected, the perceptual loss feature extraction network φ is a VGG16 network, and ReLU3_3 is selected as the feature output.

[0083] Specifically, for the human key point detection loss function, the heat map M of the high-resolution human image ground-truth is gt Prediction heat map M of the human pose estimation subnetwork pre The mean square error (Mean Square Error) between is expressed as:

[0084]

[0085] Among them, l est is the human key point detection loss function, K is the total number of human key points, H ′ and W ′ are the length and width of the heat map, respectively. gt It is a two-dimensional Gaussian distribution centered on the key points of the true value of the high-resolution human image. is the heat map pixel value corresponding to the kth key point with pixel coordinates (x, y), Heatmap M generated for the human pose estimation subnetwork pre The heat map pixel value corresponding to the kth key point in and the pixel coordinate is (x, y).

[0086] Heatmap M pre Generated by the human body posture estimation subnetwork, the formula is expressed as:

[0087] M pre =f pose (f sr (I lr ))

[0088] Among them, I lr is a low-resolution human image, fsr (·) is the super-resolution reconstruction sub-network, f pose (·) is the human pose estimation subnetwork.

[0089] Specifically, for the task-driven hybrid loss function, the hybrid loss function uses pose information to train the super-resolution reconstruction subnetwork, driving the super-resolution reconstruction subnetwork to generate high-resolution reconstructed images that are beneficial to downstream task processing. The hybrid loss function is defined as l rec and l est The weighted sum of :

[0090] l com =αl rec +βl est

[0091] Among them, l com is the mixed loss function, l rec is the super-resolution reconstruction loss function, l est is the loss function for detecting key points of the human body, α and β are the weights of the two, which are used to balance the weight parameters of the two loss terms.

[0092] Since the super-resolution reconstruction sub-network and the human pose estimation sub-network are deep neural networks, the two sub-networks are differentiable and can use the back-propagation algorithm. During the network training process, the chain derivation rule is used to train l est Find the parameters θ of the super-resolution network sr The partial derivative of :

[0093]

[0094] in, It is obtained by the super-resolution reconstruction loss function, It is obtained by the output of each layer of the super-resolution reconstruction sub-network. It is obtained by the loss function of human key point detection. are the parameters of each layer of the human posture estimation subnetwork. Therefore, Used in the parameter update of the back-propagation algorithm.

[0095] In the formula of the hybrid loss function, the influence of super-resolution reconstruction loss and human key point detection loss on the reconstruction of high-resolution images is controlled by parameters α and β. When β is much smaller than α, the hybrid loss function is approximately a super-resolution reconstruction loss function that seeks to restore the semantic information of the original image; when β is much larger than α, the hybrid loss function expects the super-resolution subnetwork to reconstruct features that are conducive to human key point detection. Under this setting, since the semantic association between the reconstructed high-resolution image and the Ground-Truth high-resolution image is ignored, it is difficult to intuitively see the location of the key points in the reconstructed high-resolution image. In the embodiment, setting α=1 and β=1000 can train an ideal super-resolution reconstruction subnetwork.

[0096] S3, inputting the low-resolution human body image into the super-resolution reconstruction sub-network to obtain the super-resolution reconstructed human body image, and then inputting it into the human body posture estimation sub-network to obtain the human body posture estimation result, and training the low-resolution image human body posture estimation model based on super-resolution reconstruction based on the training data set and the mixed loss function.

[0097] Specifically, a predefined hybrid loss function is used to perform end-to-end training on a low-resolution image human pose estimation model based on super-resolution reconstruction based on a training dataset, and data augmentation technology is also used to improve the generalization ability of the model.

[0098] S4, inputting the low-resolution human image to be detected into the trained low-resolution image human posture estimation model based on super-resolution reconstruction, and obtaining the human posture estimation result from the output heat map.

[0099] Specifically, the low-resolution human image to be detected is changed to a specific image resolution through bilinear interpolation, and the specific image resolution is set to 32×32. The human image of the specific image resolution is horizontally flipped, and the images before and after flipping are normalized, and then sent to the above-mentioned low-resolution image human posture estimation model based on super-resolution reconstruction for inference, and two human key point heat maps are obtained respectively. The human key point heat map corresponding to the horizontally flipped image is horizontally flipped, and the average of the two human key point heat maps is taken to obtain the final human key point heat map. The position of the point with the largest probability value on the human key point heat map is taken as the human key point position obtained by model inference, that is, the result of human posture estimation.

[0100] In summary, a method for estimating human pose from low-resolution images based on super-resolution reconstruction is proposed. The low-resolution human image is reconstructed through a super-resolution reconstruction subnetwork and then predicted through a human pose estimation subnetwork. This can essentially increase the amount of image information used for pose estimation, thereby improving the detection accuracy of low-resolution human pose estimation. By designing a task-driven hybrid loss function, the super-resolution reconstruction subnetwork can maximize the amount of information that can be expressed by the low-resolution input image, and is capable of generating high-resolution images containing enough low-level image features and high-level semantic features from the low-resolution image. The super-resolution reconstruction subnetwork and the human pose estimation subnetwork are trained based on the loss function, so that the model can more accurately estimate the human pose from the input low-resolution image.

[0101] Based on the same inventive concept, Figure 5 As shown, an embodiment of the present invention further provides a low-resolution image human posture estimation device 500 based on super-resolution reconstruction, including: a data set construction module 510, a model construction module 520, a model training module 530 and a model testing module 540.

[0102] The data set construction module 510 is used to downsample the true value of the high-resolution human image to obtain a low-resolution human image, and to construct a training data set including the true value of the high-resolution human image and the low-resolution human image.

[0103] The model construction module 520 is used to construct a low-resolution image human pose estimation model based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and to construct a task-driven hybrid loss function.

[0104] The model training module 530 is used to input the low-resolution human body image into the super-resolution reconstruction subnetwork to obtain the super-resolution reconstructed human body image, and then input it into the human body posture estimation subnetwork to obtain the human body posture estimation result, and train the low-resolution image human body posture estimation model based on super-resolution reconstruction based on the training data set and the mixed loss function.

[0105] The model testing module 540 is used to input the low-resolution human image to be detected into the trained low-resolution image human posture estimation model based on super-resolution reconstruction, and obtain the human posture estimation result from the output heat map.

[0106] Based on the same inventive concept, an embodiment of the present invention also provides a low-resolution image human posture estimation device based on super-resolution reconstruction, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned low-resolution image human posture estimation method based on super-resolution reconstruction when executing the computer program.

[0107] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer, the above-mentioned method for estimating human body posture in a low-resolution image based on super-resolution reconstruction is implemented.

[0108] It should be noted that the low-resolution image human body posture estimation device based on super-resolution reconstruction, the low-resolution image human body posture estimation equipment based on super-resolution reconstruction and the computer-readable storage medium provided in the above-mentioned embodiments all belong to the same inventive concept as the low-resolution image human body posture estimation method based on super-resolution reconstruction. The specific implementation process is detailed in the embodiment of the low-resolution image human body posture estimation method based on super-resolution reconstruction, which will not be repeated here.

[0109] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for human pose estimation from low-resolution images based on super-resolution reconstruction. It is characterized in that The following steps are involved: Downsampling the true value of the high-resolution human image to obtain a low-resolution human image, and constructing a training data set including the true value of the high-resolution human image and the low-resolution human image; Build a low-resolution image human pose estimation model based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human pose estimation subnetwork, and construct a task-driven hybrid loss function; The low-resolution human image is input into the super-resolution reconstruction sub-network to obtain the super-resolution reconstructed human image, which is then input into the human pose estimation sub-network to obtain the human pose estimation result. The low-resolution image human pose estimation model based on super-resolution reconstruction is trained based on the training data set and the mixed loss function. The low-resolution human image to be detected is input into the trained low-resolution image human pose estimation model based on super-resolution reconstruction, and the human pose estimation result is obtained from the output heat map.

2. The method for estimating human body posture from low-resolution images based on super-resolution reconstruction according to claim 1, It is characterized in that The task-driven hybrid loss function includes the super-resolution reconstruction loss function and the human key point detection loss function, and the formula is expressed as: l com =αl rec +βl est Among them, l com is the mixed loss function, l rec is the super-resolution reconstruction loss function, l est is the loss function for detecting key points of the human body, α and β are the weights of the two, which are used to balance the weight parameters of the two loss terms.

3. The method for estimating human body posture from low-resolution images based on super-resolution reconstruction according to claim 2, It is characterized in that The super-resolution reconstruction loss function includes pixel-by-pixel L1 loss function and feature reconstruction loss function; Pixel-by-pixel L1 loss function l 1 The formula is: Among them, f sr (I lr ) x,y,c is a low-resolution human image I lr Through the super-resolution reconstruction sub-network f sr (·) The reconstructed pixel coordinates are (x, y) and the pixel value of channel c, is the ground truth value of high-resolution human image I hr The corresponding pixel coordinates are (x, y) and the true pixel value of channel c. H, W, and C are the length, width, and total number of channels of the image, respectively. Feature reconstruction loss function The formula is expressed as: in, To reconstruct human images with super resolution, φ j (·) is C extracted by the jth convolutional layer in the perceptual loss feature extraction network φ j ×H j ×W j Feature map of size; Super-resolution reconstruction loss function l rec is the pixel-by-pixel L1 loss function l 1 And feature reconstruction loss function The weighted sum of is expressed as: Among them, k 1 and k 2 are the weights corresponding to the two loss terms respectively.

4. The method for estimating human body posture from low-resolution images based on super-resolution reconstruction according to claim 2, It is characterized in that The loss function of human key point detection is the heat map M of the true value of the high-resolution human image gt Prediction heat map M of the human pose estimation subnetwork pre The mean square error between them is expressed as: Among them, l est is the human key point detection loss function, K is the total number of human key points, H ′ and W ′ are the length and width of the heat map, respectively. gt It is a two-dimensional Gaussian distribution centered on the key points of the true value of the high-resolution human image. is the heat map pixel value corresponding to the kth key point with pixel coordinates (x, y), Heatmap M generated for the human pose estimation subnetwork pre The heat map pixel value corresponding to the kth key point in and the pixel coordinate is (x, y); Heatmap M pre Generated by the human body posture estimation subnetwork, the formula is expressed as: M pre =f pose (f sr (I lr )) Among them, I lr is a low-resolution human image, f sr (·) is the super-resolution reconstruction sub-network, f pose (·) is the human pose estimation subnetwork.

5. The method for estimating human body posture from low-resolution images based on super-resolution reconstruction according to claim 1, It is characterized in that The constructed super-resolution reconstruction sub-network adopts a cascaded residual network structure, including a cascade part and an upsampling part; In the cascade part, a convolution layer is first used to extract the pixels of the low-resolution human image from the spatial dimension to the channel dimension, and then multiple alternating cascade residual blocks and convolution blocks are used to extract image features and fuse multi-layer features to obtain a feature map; In the upsampling part, the upsampling layer is used to perform pixel rearrangement to upsample the feature map to a high resolution to obtain a super-resolution reconstructed human image.

6. The method for estimating human body posture from low-resolution images based on super-resolution reconstruction according to claim 1, It is characterized in that The constructed human pose estimation subnetwork adopts the FastPose network based on the AlphaPose network, including: the backbone network part, the dense upsampling part and the detection head part; In the backbone network, super-resolution reconstructed human images are input, and the features of human posture are extracted using sequentially connected convolutional layers, maximum pooling layers, and multiple compression excitation modules; In the dense upsampling part, the extracted human posture features are input, and multiple alternating convolutional layers and pixel rearrangement layers are used to capture and restore the detailed information of the features to obtain an upsampled feature map; In the detection head, a convolutional layer is used to convert the upsampled feature map into a fine-grained human key point prediction heat map.

7. The method for estimating human body posture from low-resolution images based on super-resolution reconstruction according to claim 5, It is characterized in that In the cascade part of the super-resolution reconstruction subnetwork, the structure of each cascade residual block includes at least one enhanced residual block and an internal convolution block, wherein the enhanced residual block adopts grouped convolution, and the outputs of the enhanced residual block are connected in parallel in the feature dimension and then weighted averaged through the internal convolution block for feature fusion.

8. A human body posture estimation device based on low-resolution image super-resolution reconstruction, It is characterized in that include: Dataset building module, model building module, model training module and model testing module; The data set construction module is used to downsample the true value of the high-resolution human image to obtain a low-resolution human image, and to construct a training data set including the true value of the high-resolution human image and the low-resolution human image; The model building module is used to build a low-resolution image human posture estimation model based on super-resolution reconstruction, including a super-resolution reconstruction subnetwork and a human posture estimation subnetwork, and to build a task-driven hybrid loss function; The model training module is used to input the low-resolution human body image into the super-resolution reconstruction subnetwork to obtain the super-resolution reconstructed human body image, and then input it into the human body posture estimation subnetwork to obtain the human body posture estimation result, and train the low-resolution image human body posture estimation model based on super-resolution reconstruction based on the training data set and the mixed loss function; The model testing module is used to input the low-resolution human image to be detected into a trained low-resolution image human posture estimation model based on super-resolution reconstruction, and obtain the human posture estimation result from the output heat map.

9. A device for estimating human posture from low-resolution images based on super-resolution reconstruction, comprising a memory and one or more processors, wherein the memory is used to store a computer program, It is characterized in that The processor is used to implement the method for estimating human body posture from a low-resolution image based on super-resolution reconstruction as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a computer, the method for estimating human body posture from a low-resolution image based on super-resolution reconstruction as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Arbitrary ratio image super-resolution method, system and device and storage medium

    CN112907448A

  • Image processing method and apparatus, portrait super-resolution reconstruction method and apparatus, and portrait super-resolution reconstruction model training method and apparatus, electronic device, and storage medium

    WO2022057837A1

Cited By

  • Dam personnel behavior monitoring and early warning method, system and device and medium

    CN119888864A

  • Traditional Chinese medicine tongue diagnosis auxiliary method and system based on image reconstruction and tongue picture detection

    CN119941652A