Human pose prediction method, device, equipment and readable storage medium

Abstract

The application provides a human posture prediction method, device and equipment and a readable storage medium. The method comprises the following steps: extracting a feature map of an input picture; obtaining an offset of a human key point, a heat map and a center heat map based on the feature map; obtaining a predicted human key point according to the offset and the center heat map; generating a local key point expansion window with the predicted human key point as the center, and then converting the local key point expansion window into a key point attraction field; generating a global key point expansion window by using the predicted human key point and the heat map; performing a convolution operation on the global key point expansion window by using the key point attraction field as a convolution kernel to obtain a corrected key point heat map; obtaining two-dimensional coordinates of a predicted human joint point from the corrected key point heat map; and respectively using spatial and timing information of the two-dimensional coordinates of the predicted human joint point to encode and decode to obtain three-dimensional information of the predicted human joint point. Through the application, the human posture prediction accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and readable storage medium for predicting human pose. Background Technology

[0002] One of the important tasks in the field of computer vision research is the detection of key points on the human skeleton. Specifically, it is to enable computers to perceive the position of key points on the human skeleton, providing a foundation for further practical applications such as action recognition, action anomaly detection, intelligent monitoring, and autonomous driving.

[0003] The goal of human skeletal keypoint detection is to take an image as input and output the coordinates of each skeletal keypoint in the image relative to the real world. Currently, mainstream deep learning-based human keypoint detection techniques can be broadly categorized into two types: top-down and bottom-up methods. Top-down methods first detect bounding boxes for the human body and then perform individual pose estimation for each bounding box. While these methods offer high accuracy, their computational efficiency is low because the cost of pose estimation for a single human is proportional to the number of bounding boxes. Furthermore, these methods are limited by the accuracy of human target detection. Bottom-up methods generally involve two steps: first, detecting human keypoints, and then grouping these keypoints. A representative method of this type predicts keypoint positions based on keypoint centers and offsets. This method avoids complex keypoint grouping and is computationally fast. However, its drawback is also significant: it is inaccurate in estimating offset vectors far from the keypoint center, resulting in lower overall model accuracy. Summary of the Invention

[0004] This invention addresses the shortcomings of existing techniques for predicting key point positions based on key point center points and offsets by providing a human posture prediction method, apparatus, device, and readable storage medium.

[0005] In a first aspect, the present invention provides a human posture prediction method, the human posture prediction method comprising:

[0006] For the input image, a feature map is extracted using a convolutional neural network;

[0007] Based on the extracted feature maps, the offsets of human key points, the heatmaps of human key points, and the center heatmaps of human key points are obtained. Based on the offsets of human key points and the center heatmaps of human key points, predicted human key points are obtained, and local key point expansion windows are generated with the predicted human key points as the center.

[0008] Convert the local keypoint expansion window into a keypoint attraction field;

[0009] A global keypoint expansion window is generated by predicting human keypoints and a human keypoint heatmap. The keypoint attraction field is used as a convolution kernel to perform a convolution operation on the global keypoint expansion window to obtain a corrected keypoint heatmap.

[0010] The predicted two-dimensional coordinates of human joints are obtained from the corrected key point heatmap;

[0011] The spatial and temporal information of the predicted two-dimensional coordinates of human joints are encoded and decoded to obtain the predicted three-dimensional information of human joints.

[0012] Optionally, the step of obtaining the offset of human key points, the heat map of human key points, and the heat map of the center of human key points based on the extracted feature map includes:

[0013] The feature map is reduced in dimensionality through two parallel branches, and then convolution is performed using a convolutional layer with a kernel size of 1x1.

[0014] One branch outputs a heatmap of key human body points and a heatmap of the center of key human body points, using... In this representation, H is the heatmap, and h and w depend on the sampling step size of the backbone network. The resulting low-resolution heatmap is then subjected to bilinear interpolation upsampling to obtain a high-resolution heatmap. This indicates that the other branch outputs the offsets of 17 human body key points, using... express.

[0015] Optionally, the step of obtaining predicted human key points based on the offset of human key points and the human key point center heatmap, and generating a local key point expansion window centered on the predicted human key points includes:

[0016] Non-maximum suppression processing is applied to the heatmap of human key point centers, and the top N points with the highest scores are selected as candidate key point centers, where N is a positive integer.

[0017] N candidate human poses are obtained by using the center point of candidate key points and the offset of human key points, while human poses with a center heatmap score of human key points less than a threshold are removed.

[0018] For N candidate human poses, 17 key points of each candidate human pose are used as the predicted human key points.

[0019] Generate a local keypoint expansion window centered on each predicted human keypoint.

[0020] Optionally, the step of converting the local keypoint expansion window into a keypoint attraction field includes:

[0021] The feature map is processed using convolutional layers, batch normalization, and corrected linear units to obtain a feature map of dimension C3;

[0022] A local keypoint expansion window feature map is obtained by bilinear interpolation from a feature map of dimension C3.

[0023] The keypoint attraction field is obtained by processing local keypoint expansion window feature maps using three different convolutional layers, batch normalization, and modified linear units.

[0024] Optionally, the step of generating a global keypoint expansion window using predicted human keypoints and a human keypoint heatmap, and performing a convolution operation on the global keypoint expansion window using the keypoint attraction field as the convolution kernel to obtain a corrected keypoint heatmap includes:

[0025] For each predicted human keypoint, generate a global keypoint expansion window;

[0026] The global keypoint expansion window feature map is obtained by bilinear interpolation from the feature map of dimension C3 using the generated global keypoint expansion window.

[0027] A new global keypoint expansion window feature map is obtained by weighting the global keypoint expansion window feature map using a Gaussian kernel.

[0028] Using the keypoint attraction field as the convolution kernel, a convolution operation is performed on the new global keypoint expansion window feature map to achieve the fusion of local and global contextual information and obtain the corrected keypoint heatmap.

[0029] Optionally, the step of obtaining the predicted two-dimensional coordinates of human joints from the modified keypoint heatmap includes:

[0030] For each predicted human keypoint, select the two positions with the highest scores in the corrected keypoint heatmap, then weight the scores of these two positions, calculate the first product of the weighted scores of the two positions, multiply the first product by the score of the corresponding candidate keypoint center point to obtain the second product, and use the second product as the confidence score of the predicted human keypoint.

[0031] The location of the key point with the highest confidence score is used as the predicted two-dimensional coordinate of the human joint.

[0032] Optionally, the step of encoding and decoding the predicted two-dimensional coordinates of human joints to obtain the predicted three-dimensional information of human joints includes:

[0033] The predicted two-dimensional coordinates of human joints are used as keys and implicit vectors as indices to encode spatial scale, resulting in output features.

[0034] The output features are encoded over time to obtain the output array;

[0035] The predicted three-dimensional coordinates of human joints are obtained by regression based on the output array.

[0036] Secondly, the present invention also provides a human posture prediction device, the human posture prediction device comprising:

[0037] The extraction module is used to extract feature maps from input images using a convolutional neural network.

[0038] The first generation module is used to obtain the offset of human key points, the heat map of human key points and the center heat map of human key points based on the extracted feature map, to obtain the predicted human key points based on the offset of human key points and the center heat map of human key points, and to generate a local key point expansion window with the predicted human key points as the center.

[0039] The conversion module is used to convert local keypoint expansion windows into keypoint attraction fields;

[0040] The second generation module is used to generate a global keypoint expansion window by using the predicted human keypoints and the human keypoint heatmap, and to perform a convolution operation on the global keypoint expansion window by using the keypoint attraction field as the convolution kernel to obtain the corrected keypoint heatmap.

[0041] The third generation module is used to obtain the predicted two-dimensional coordinates of human joints from the corrected key point heatmap;

[0042] The encoding / decoding module is used to encode and decode the spatial and temporal information of the predicted two-dimensional coordinates of human joints to obtain the predicted three-dimensional information of human joints.

[0043] Thirdly, the present invention also provides a human posture prediction device, the human posture prediction device including a processor, a memory, and a human posture prediction program stored in the memory and executable by the processor, wherein when the human posture prediction program is executed by the processor, it implements the steps of the human posture prediction method as described above.

[0044] Fourthly, the present invention also provides a readable storage medium storing a human posture prediction program, wherein when the human posture prediction program is executed by a processor, it implements the steps of the human posture prediction method as described above.

[0045] In this invention, for an input image, a feature map is extracted using a convolutional neural network. Based on the extracted feature map, the offsets of human keypoints, a heatmap of human keypoints, and a heatmap of the center of human keypoints are obtained. Predicted human keypoints are obtained based on the offsets and the heatmap of the center of human keypoints. A local keypoint expansion window is generated centered on the predicted human keypoints. The local keypoint expansion window is converted into a keypoint attraction field. A global keypoint expansion window is generated using the predicted human keypoints and the human keypoint heatmap. The keypoint attraction field is used as a convolution kernel to perform a convolution operation on the global keypoint expansion window to obtain a corrected keypoint heatmap. The two-dimensional coordinates of the predicted human joints are obtained from the corrected keypoint heatmap. The spatial and temporal information of the predicted two-dimensional coordinates of the human joints are encoded and decoded to obtain the three-dimensional information of the predicted human joints. This invention effectively utilizes feature information, combines pre- and post-process feature information, and global and local information to output richer feature information, improving the localization effect of human keypoints and thus improving the accuracy of human pose prediction. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an embodiment of the human posture prediction method of the present invention;

[0047] Figure 2 This is a schematic diagram of the functional modules of an embodiment of the human posture prediction device of the present invention;

[0048] Figure 3 This is a schematic diagram of the hardware structure of the human posture prediction device involved in the embodiment of the present invention.

[0049] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0051] In a first aspect, embodiments of the present invention provide a method for predicting human posture.

[0052] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the human posture prediction method of the present invention. Figure 1 As shown, human posture prediction methods include:

[0053] Step S10: For the input image, feature maps are extracted using a convolutional neural network;

[0054] In this embodiment, for a given input image, a low-dimensional feature map is extracted using a convolutional neural network. The feature extraction network used can be HRNets, and the spatial size of the final output feature map depends on the overall stride of the feature extraction network.

[0055] For example, assuming the original image is H×W, each time it passes through a pooling layer, the length and width of the original input feature become s times their original size. Using the original image as input, after a series of convolutional layers, modified linear units, and pooling layers, a feature map F∈R with feature dimension C is obtained. C×h×w The spatial dimensions h×w depend on the overall step size of the feature extraction module.

[0056] Step S20: Based on the extracted feature map, obtain the offset of human key points, the heat map of human key points and the heat map of the center of human key points. Based on the offset of human key points and the heat map of the center of human key points, obtain the predicted human key points. Generate a local key point expansion window with the predicted human key points as the center.

[0057] In this embodiment, the feature map F∈R is obtained. C×h×w First, a 1x1 convolutional layer, batch normalization, and corrected linear units are used to reduce the dimensionality of the feature map. Then, a 1x1 convolutional layer is used to perform convolution processing, outputting two branches. Branch one outputs a heatmap of keypoints and their centers. To represent, the low-resolution heatmap is upsampled using bilinear interpolation to obtain a high-resolution heatmap, which is then expressed as... To express.

[0058] Branch 2 outputs the offsets of 17 key human body positions, using... To express.

[0059] Based on the calculated keypoint center heatmap and the offset of keypoint positions, a series of candidate human poses are obtained. Specifically, firstly, non-maximum suppression (non-maximum suppression) is applied to the obtained center point heatmap (window size 3x3), and the top N points with the highest scores are selected as candidates for keypoint center points. Then, N candidate human pose estimates are obtained using these candidate keypoint center points and keypoint offsets, while human poses with keypoint center heatmap scores lower than a given threshold (0.01 in this embodiment) are removed.

[0060] A local window for human keypoint expansion is generated. Specifically, for N calculated candidate human poses, a local mesh (11x11 in this embodiment) centered on the 17 keypoints of each pose is generated to compensate for the missing information in estimating human pose using center point offset. This mesh is defined as the window for human keypoint expansion. To express.

[0061] Further, in one embodiment, the step of obtaining the offset of human key points, the human key point heatmap, and the human key point center heatmap based on the extracted feature map includes:

[0062] The feature maps are dimensionality-reduced through two parallel branches, and then convolutional processing is performed using convolutional layers with a kernel size of 1x1. One branch outputs a heatmap of human keypoints and a heatmap of the centers of human keypoints. In this representation, H is the heatmap, and h and w depend on the sampling step size of the backbone network. The resulting low-resolution heatmap is then subjected to bilinear interpolation upsampling to obtain a high-resolution heatmap. This indicates that the other branch outputs the offsets of 17 human body key points, using... express.

[0063] In this embodiment, the specific process is as follows:

[0064]

[0065]

[0066] Where F∈R C×h×w Here, C represents the feature map, C1 and C2 are predefined parameters (this algorithm commonly uses C1=32 and C2=256). Conv is the convolutional processing, BN is batch normalization, and ReLU is the regularized linear unit.

[0067] Further, in one embodiment, the step of obtaining predicted human key points based on the offset of human key points and the human key point center heatmap, and generating a local key point expansion window centered on the predicted human key points includes:

[0068] Non-maximum suppression is applied to the human keypoint center heatmap, and the top N points with the highest scores are selected as candidate keypoint centers, where N is a positive integer. N candidate human poses are obtained using the center points of the candidate keypoints and the offsets of the human keypoints, while human poses with keypoint center heatmap scores below a threshold are removed. For each of the N candidate human poses, 17 keypoints of each candidate human pose are used as predicted human keypoints. Local keypoint expansion windows are generated centered on each predicted human keypoint.

[0069] In this embodiment, the specific process is as follows:

[0070]

[0071] Step S30: Convert the local keypoint expansion window into a keypoint attraction field;

[0072] In this embodiment, the feature map is first processed using convolutional layers, batch normalization, and modified linear units to obtain a feature map with dimension C3 (64 in this embodiment). Then, a local keypoint expansion window feature map is obtained by bilinear interpolation from the processed feature map using local geometric meshes.

[0073] The keypoint attraction field is generated using a convolutional information delivery module. Specifically, three distinct convolutional layers, batch normalization, and modified linear units are used to process the local keypoint expansion window feature map obtained in step one to obtain the keypoint attraction field. To emphasize the specificity of different pose instances in the convolutional information delivery module, attention mechanism normalization is used instead of batch normalization.

[0074] Further, in one embodiment, step S30 includes:

[0075] The feature map is processed using convolutional layers, batch normalization, and modified linear units to obtain a feature map of dimension C3. A local keypoint expansion window feature map is obtained by bilinear interpolation from the feature map of dimension C3. The keypoint attraction field is obtained by processing the local keypoint expansion window feature map using three different convolutional layers, batch normalization, and modified linear units.

[0076] In this embodiment, the specific process is as follows:

[0077]

[0078]

[0079] Among them, Cin, Cout, and C6 are all dimensions.

[0080] Step S40: Generate a global keypoint expansion window using the predicted human body keypoints and the human body keypoint heatmap, and perform a convolution operation on the global keypoint expansion window using the keypoint attraction field as the convolution kernel to obtain the corrected keypoint heatmap.

[0081] In this embodiment, for each predicted human keypoint, an enlarged global grid (size axa) is generated. This grid is used... This grid can be understood as a global keypoint expansion window. Then, we use the generated global keypoint expansion window to obtain a global keypoint window feature map from the predicted global heatmap via bilinear interpolation. Finally, we re-weight it using a Gaussian kernel to obtain the global keypoint expansion window feature map. The specific process is as follows:

[0082]

[0083] in, For Gaussian kernel,

[0084] The obtained keypoint attraction field is used as the convolution kernel to perform a convolution operation on the global keypoint expansion window feature map to obtain the corrected keypoint heatmap. Specifically, the learned keypoint attraction field is used as the convolution kernel to perform a convolution operation on the global keypoint expansion window feature map to achieve the fusion of local and global contextual information, and finally obtain the corrected heatmap of 17 keypoints.

[0085] Further, in one embodiment, step S40 includes:

[0086] For each predicted human keypoint, a global keypoint expansion window is generated. The global keypoint expansion window feature map is obtained by bilinear interpolation from the feature map of dimension C3 using the generated global keypoint expansion window. The global keypoint expansion window feature map is weighted using a Gaussian kernel to obtain a new global keypoint expansion window feature map. The new global keypoint expansion window feature map is convolved using the keypoint attraction field as the convolution kernel to achieve the fusion of local and global contextual information, resulting in a corrected keypoint heatmap.

[0087] Step S50: Obtain the predicted two-dimensional coordinates of human joints from the corrected key point heatmap;

[0088] Further, in one embodiment, step S50 includes:

[0089] For each predicted human keypoint, the two positions with the highest scores in the corrected keypoint heatmap are selected. Then, the scores of these two positions are weighted, and the first product of the weighted scores of the two positions is calculated. The first product is multiplied by the score of the corresponding candidate keypoint center point to obtain the second product. The second product is used as the confidence score of the predicted human keypoint. The position of the keypoint with the highest confidence score is used as the two-dimensional coordinate of the predicted human joint.

[0090] In this embodiment, the specific process is as follows:

[0091]

[0092] It predicts human keypoints, and N′ is the number of pose instances ultimately predicted in an image I.

[0093] Step S60: Encode and decode the spatial and temporal information of the predicted two-dimensional coordinates of human joints to obtain the predicted three-dimensional information of human joints.

[0094] In this embodiment, the predicted two-dimensional coordinates of the human joints are used as keys and implicit vectors as indices for spatial scaling to obtain output features. For the output features, the implicit vectors used as keys and indices are then encoded for temporal scaling to obtain the output array Y. By weighted averaging of Y, and then regressing the joint positions y in the real three-dimensional world through a multilayer perceptron, the predicted three-dimensional information of the human joints is obtained.

[0095] Further, in one embodiment, step S60 includes:

[0096] The predicted two-dimensional coordinates of human joints are used as keys and implicit vectors as indices to encode the spatial scale to obtain output features; the output features are then encoded on a time scale to obtain an output array; and the predicted three-dimensional coordinates of human joints are obtained by regression based on the output array.

[0097] Compared with the prior art, this embodiment has the following advantages:

[0098] I. This embodiment addresses the shortcomings of low accuracy in predicting human pose based on key point centers and offsets. It proposes to construct a local expansion window around the predicted key points to achieve further refined localization of the key points. The entire constructed key point detection and correction network is an end-to-end network.

[0099] II. This embodiment proposes a novel local and global information adaptation module to obtain structured human pose estimation results and to achieve the fusion of local and global structural information.

[0100] This embodiment can solve the problem of multi-person pose estimation with severe self-occlusion through two-stage training.

[0101] Furthermore, the loss function of the fully convolutional network involved in this embodiment includes the following parts:

[0102] Loss function for keypoint heatmaps. (Using...) This represents the ground truth heatmap value for each keypoint and its center, generated using a Gaussian model based on the mean and variance of a given dataset. Let p = (i, x) represent an element in a set of dimension 10 × h × w. This is the heatmap value for the predicted keypoints and their centers. Calculate it and The mean squared error is used as the loss function for the keypoint heatmap, as follows:

[0103]

[0104] Here, w(x) represents the weights of the foreground and background pixels. For foreground pixels, w(x) = 1; for background elements, w(x) = 0.1.

[0105] Keypoint offset field loss function. (Using...) c represents the true value of the offset. GT This represents a non-empty set of keypoint centers in the true value. For the predicted keypoint offset field... The loss function is expressed as follows:

[0106]

[0107] in This represents the area of ​​the human body centered at pixel p, and β represents the truncation threshold (e.g., 1 / 9).

[0108] OKS loss function. In predicting each human pose, a local keypoint attraction field needs to be learned as a convolution kernel to correct the global keypoint heatmap. Specifically, for a given N in the image... GT For each human pose, the ground truth value is calculated, and the similarity score between each candidate keypoint and the ground truth value is obtained within the local keypoint expansion window. This yields the similarity score tensor. Then the tensor is truncated using a threshold of 0.5, i.e. Then, the average of the first three dimensions of the truncated similarity score tensor is taken to obtain its matching score with the ground truth value of each human pose annotation. The human pose ground truth value n with the largest matching score is selected. * Matching scores Let be the representation. Based on the chosen ground truth of the human pose, calculate the similarity score for each keypoint of the predicted human pose, denoted by s. k (k∈[1, 17]) is used to represent this. Therefore, the loss function of the final keypoint attraction field is defined as:

[0109]

[0110] Finally, the overall loss function of the entire network is defined as... To balance the values ​​between different loss terms, for example, λ is set to 0.01. The defined loss function measures the difference between the predicted human pose and the ground truth. This difference is used as an error signal, and the partial derivatives of each parameter in the convolutional layer are calculated using the backpropagation algorithm. The parameters of the neural network are updated based on the calculation results.

[0111] Secondly, embodiments of the present invention also provide a human posture prediction device.

[0112] In one embodiment, reference is made to Figure 2 , Figure 2This is a functional module diagram of an embodiment of the human posture prediction device of the present invention. Figure 2 As shown, the human posture prediction device includes:

[0113] Extraction module 10 is used to extract feature maps from input images using a convolutional neural network;

[0114] The first generation module 20 is used to obtain the offset of human key points, the heat map of human key points and the center heat map of human key points based on the extracted feature map, to obtain the predicted human key points based on the offset of human key points and the center heat map of human key points, and to generate a local key point expansion window with the predicted human key points as the center.

[0115] Conversion module 30 is used to convert a local keypoint expansion window into a keypoint attraction field;

[0116] The second generation module 40 is used to generate a global key point expansion window using the predicted human key points and the human key point heat map, and to perform a convolution operation on the global key point expansion window using the key point attraction field as the convolution kernel to obtain the corrected key point heat map.

[0117] The third generation module 50 is used to obtain the predicted two-dimensional coordinates of human joints from the corrected key point heat map;

[0118] The encoding / decoding module 60 is used to encode and decode the spatial and temporal information of the predicted two-dimensional coordinates of human joints to obtain the predicted three-dimensional information of human joints.

[0119] Furthermore, in one embodiment, the first generation module 20 is configured to:

[0120] The feature map is reduced in dimensionality through two parallel branches, and then convolution is performed using a convolutional layer with a kernel size of 1x1.

[0121] One branch outputs a heatmap of key human body points and a heatmap of the center of key human body points, using... In this representation, H is the heatmap, and h and w depend on the sampling step size of the backbone network. The resulting low-resolution heatmap is then subjected to bilinear interpolation upsampling to obtain a high-resolution heatmap. This indicates that the other branch outputs the offsets of 17 human body key points, using... express.

[0122] Furthermore, in one embodiment, the first generation module 20 is configured to:

[0123] Non-maximum suppression processing is applied to the heatmap of human key point centers, and the top N points with the highest scores are selected as candidate key point centers, where N is a positive integer.

[0124] N candidate human poses are obtained by using the center point of candidate key points and the offset of human key points, while human poses with a center heatmap score of human key points less than a threshold are removed.

[0125] For N candidate human poses, 17 key points of each candidate human pose are used as the predicted human key points.

[0126] Generate a local keypoint expansion window centered on each predicted human keypoint.

[0127] Furthermore, in one embodiment, the conversion module 30 is used for:

[0128] The feature map is processed using convolutional layers, batch normalization, and corrected linear units to obtain a feature map of dimension C3;

[0129] A local keypoint expansion window feature map is obtained by bilinear interpolation from a feature map of dimension C3.

[0130] The keypoint attraction field is obtained by processing local keypoint expansion window feature maps using three different convolutional layers, batch normalization, and modified linear units.

[0131] Furthermore, in one embodiment, the second generation module 40 is used for:

[0132] For each predicted human keypoint, generate a global keypoint expansion window;

[0133] The global keypoint expansion window feature map is obtained by bilinear interpolation from the feature map of dimension C3 using the generated global keypoint expansion window.

[0134] A new global keypoint expansion window feature map is obtained by weighting the global keypoint expansion window feature map using a Gaussian kernel.

[0135] Using the keypoint attraction field as the convolution kernel, a convolution operation is performed on the new global keypoint expansion window feature map to achieve the fusion of local and global contextual information and obtain the corrected keypoint heatmap.

[0136] Furthermore, in one embodiment, the third generation module 50 is used for:

[0137] For each predicted human keypoint, select the two positions with the highest scores in the corrected keypoint heatmap, then weight the scores of these two positions, calculate the first product of the weighted scores of the two positions, multiply the first product by the score of the corresponding candidate keypoint center point to obtain the second product, and use the second product as the confidence score of the predicted human keypoint.

[0138] The location of the key point with the highest confidence score is used as the predicted two-dimensional coordinate of the human joint.

[0139] Furthermore, in one embodiment, the encoding / decoding module 60 is used for:

[0140] The predicted two-dimensional coordinates of human joints are used as keys and implicit vectors as indices to encode spatial scale, resulting in output features.

[0141] The output features are encoded over time to obtain the output array;

[0142] The predicted three-dimensional coordinates of human joints are obtained by regression based on the output array.

[0143] The functions of each module in the above-mentioned human posture prediction device correspond to the steps in the above-mentioned human posture prediction method embodiment, and their functions and implementation processes will not be described in detail here.

[0144] Thirdly, embodiments of the present invention provide a human posture prediction device, which can be a personal computer (PC), laptop computer, server or other device with data processing capabilities.

[0145] Reference Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of the human posture prediction device involved in an embodiment of the present invention. In this embodiment, the human posture prediction device may include a processor 1001 (e.g., a Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize communication between these components; the user interface 1003 may include a display screen or an input unit such as a keyboard; the network interface 1004 may optionally include a standard wired interface or a wireless interface (e.g., Wireless Fidelity, Wi-Fi); the memory 1005 may be high-speed random access memory (RAM) or stable memory (non-volatile memory), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art will understand that… Figure 3 The hardware structure shown does not constitute a limitation of the invention and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0146] Continue to refer to Figure 3 , Figure 3 The memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a human posture prediction program. The processor 1001 can call the human posture prediction program stored in the memory 1005 and execute the human posture prediction method provided in this embodiment of the invention.

[0147] Fourthly, embodiments of the present invention also provide a readable storage medium.

[0148] The present invention provides a human posture prediction program stored on a readable storage medium, wherein when the human posture prediction program is executed by a processor, it implements the steps of the human posture prediction method described above.

[0149] The method implemented when the human posture prediction program is executed can be referred to in various embodiments of the human posture prediction method of the present invention, and will not be repeated here.

[0150] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0151] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of the present invention.

[0153] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for predicting human posture, characterized in that, The human posture prediction method includes: For the input image, a feature map is extracted using a convolutional neural network; Based on the extracted feature maps, the offsets of human key points, the heatmaps of human key points, and the center heatmaps of human key points are obtained. Based on the offsets of human key points and the center heatmaps of human key points, predicted human key points are obtained, and local key point expansion windows are generated with the predicted human key points as the center. Convert the local keypoint expansion window into a keypoint attraction field; A global keypoint expansion window is generated by predicting human keypoints and a human keypoint heatmap. The keypoint attraction field is used as a convolution kernel to perform a convolution operation on the global keypoint expansion window to obtain a corrected keypoint heatmap. The predicted two-dimensional coordinates of human joints are obtained from the corrected key point heatmap; The spatial and temporal information of the predicted two-dimensional coordinates of human joints are encoded and decoded to obtain the predicted three-dimensional information of human joints. The steps of obtaining the offset of human key points, the heat map of human key points, and the center heat map of human key points based on the extracted feature map include: The feature map is reduced in dimensionality through two parallel branches, and then convolution is performed using a convolutional layer with a kernel size of 1x1. One branch outputs a heatmap of key human body points and a heatmap of the center of key human body points, using... In this representation, H is the heatmap, and h and w depend on the sampling step size of the backbone network. The resulting low-resolution heatmap is then subjected to bilinear interpolation upsampling to obtain a high-resolution heatmap. This indicates that the other branch outputs the offsets of 17 human body key points, using... express; The steps of generating a global keypoint expansion window using predicted human body keypoints and human body keypoint heatmaps, and performing a convolution operation on the global keypoint expansion window using the keypoint attraction field as the convolution kernel to obtain a corrected keypoint heatmap include: For each predicted human keypoint, generate a global keypoint expansion window; Expand the window using the generated global key points from dimension 1 Bilinear interpolation in the feature map yields a global keypoint expanded window feature map; A new global keypoint expansion window feature map is obtained by weighting the global keypoint expansion window feature map using a Gaussian kernel. Using the keypoint attraction field as the convolution kernel, a convolution operation is performed on the new global keypoint expansion window feature map to achieve the fusion of local and global contextual information and obtain the corrected keypoint heatmap.

2. The human posture prediction method as described in claim 1, characterized in that, The step of obtaining predicted human key points based on the offset of human key points and the heat map of the center of human key points, and generating a local key point expansion window centered on the predicted human key points includes: Non-maximum suppression processing is applied to the heatmap of human key point centers, and the top N points with the highest scores are selected as candidate key point centers, where N is a positive integer. N candidate human poses are obtained by using the center point of candidate key points and the offset of human key points, while human poses with a center heatmap score of human key points less than a threshold are removed. For N candidate human poses, 17 key points of each candidate human pose are used as the predicted human key points. Generate a local keypoint expansion window centered on each predicted human keypoint. .

3. The human posture prediction method as described in claim 2, characterized in that, The step of converting a local keypoint expansion window into a keypoint attraction field includes: The feature map is processed using convolutional layers, batch normalization, and corrected linear units to obtain a dimension of Feature map; Expand the window using local keypoints from dimension 1 Bilinear interpolation in the feature map yields a local keypoint expanded window feature map; The keypoint attraction field is obtained by processing local keypoint expansion window feature maps using three different convolutional layers, batch normalization, and modified linear units.

4. The human posture prediction method as described in claim 1, characterized in that, The step of obtaining the predicted two-dimensional coordinates of human joints from the corrected keypoint heatmap includes: For each predicted human keypoint, select the two positions with the highest scores in the corrected keypoint heatmap, then weight the scores of these two positions, calculate the first product of the weighted scores of the two positions, multiply the first product by the score of the corresponding candidate keypoint center point to obtain the second product, and use the second product as the confidence score of the predicted human keypoint. The location of the key point with the highest confidence score is used as the predicted two-dimensional coordinate of the human joint.

5. The human posture prediction method as described in claim 4, characterized in that, The step of encoding and decoding the predicted two-dimensional coordinates of human joints to obtain the predicted three-dimensional information of human joints includes: The predicted two-dimensional coordinates of human joints are used as keys and implicit vectors as indices to encode spatial scale, resulting in output features. The output features are encoded over time to obtain the output array; The predicted three-dimensional coordinates of human joints are obtained by regression based on the output array.

6. A human posture prediction device, characterized in that, The human posture prediction device includes: The extraction module is used to extract feature maps from input images using a convolutional neural network. The first generation module performs dimensionality reduction on the feature maps through two parallel branches, followed by convolutional processing using 1x1 convolutional layers. One branch outputs a heatmap of human keypoints and a heatmap of the centers of human keypoints. In this representation, H is the heatmap, and h and w depend on the sampling step size of the backbone network. The resulting low-resolution heatmap is then subjected to bilinear interpolation upsampling to obtain a high-resolution heatmap. This indicates that the other branch outputs the offsets of 17 human body key points, using... This means that the human body key points are predicted based on the offset of the human body key points and the heat map of the human body key point center, and a local key point expansion window is generated with the predicted human body key points as the center. The conversion module is used to convert local keypoint expansion windows into keypoint attraction fields; The second generation module generates a global keypoint expansion window for each predicted human keypoint; it then uses this global keypoint expansion window to expand the viewpoints from a dimension of [missing information]. The global keypoint expansion window feature map is obtained by bilinear interpolation in the feature map; the global keypoint expansion window feature map is weighted by Gaussian kernel to obtain a new global keypoint expansion window feature map; the keypoint attraction field is used as the convolution kernel to perform convolution operation on the new global keypoint expansion window feature map to achieve the fusion of local and global context information and obtain the corrected keypoint heatmap. The third generation module is used to obtain the predicted two-dimensional coordinates of human joints from the corrected key point heatmap; The encoding / decoding module is used to encode and decode the spatial and temporal information of the predicted two-dimensional coordinates of human joints to obtain the predicted three-dimensional information of human joints.

7. A human posture prediction device, characterized in that, The human posture prediction device includes a processor, a memory, and a human posture prediction program stored in the memory and executable by the processor, wherein when the human posture prediction program is executed by the processor, it implements the steps of the human posture prediction method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores a human posture prediction program, wherein when the human posture prediction program is executed by a processor, it implements the steps of the human posture prediction method as described in any one of claims 1 to 5.

Images