Human pose estimation method, device, terminal and computer readable storage medium

Abstract

The present application provides a human posture estimation method, device, terminal and computer readable storage medium, the human posture estimation method comprises: obtaining an image to be detected; the image to be detected at least contains a target human body; feature extraction is carried out on the image to be detected, and the feature map of the image to be detected is obtained; a plurality of key point detection modes are used to detect the feature map of the image to be detected respectively, and a plurality of key point information is obtained; the plurality of key point information at least includes first type key point information and second type key point information, the key point data in the first type key point information is not less than the key point data in the second type key point information; the posture information of the target human body is determined by correcting the second type key point information based on the first type key point information. The present application detects the key points of the feature map, corrects the second type key point information based on the first type key point information in the plurality of key point information, and improves the accuracy of the key point information of the target human body detected.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a human pose estimation method, device, terminal, and computer-readable storage medium. Background Technology

[0002] Human pose estimation by detecting human keypoints in 2D images containing human figures is a valuable research direction in the field of computer vision. Human keypoints have many applications, especially in video, where the detection results can further serve multiple fields such as human-computer interaction, pedestrian tracking, and behavior recognition, and have already been applied to many products.

[0003] Currently, mainstream human keypoint detection methods are mainly divided into two types: top-down human pose estimation methods and bottom-up human pose estimation methods. Top-down methods perform keypoint detection on single-person images. However, when multiple people appear in the detection box, the keypoint information for each person is only output within each box, and keypoint detection may occur on other people's limbs, resulting in poor keypoint detection performance. Bottom-up pose estimation methods, on the other hand, use the entire human image as input. When multiple people are present in the image, keypoint information for all multiple people is detected simultaneously. Post-processing is then used to match keypoints and generate individual human instances. The advantage of this type of method is that the processing time is fixed regardless of the number of people in the image, meaning keypoint detection is performed only once. However, because the input is the entire image, there is a lot of redundant information in the image, and the scale of the human figures in the image varies, which poses a challenge to the network's learning process. Summary of the Invention

[0004] The main technical problem solved by this invention is to provide a human pose estimation method, device, terminal and computer-readable storage medium to solve the problem of low accuracy of human pose estimation in dense scenes in the prior art.

[0005] To address the aforementioned technical problems, the first technical solution adopted by this invention is: providing a human pose estimation method, which includes: acquiring an image to be detected, the image to be detected containing at least a target human body; performing feature extraction on the image to be detected to obtain a feature map of the image to be detected; performing key point detection on the feature map of the image to be detected using multiple key point detection methods to obtain multiple key point information; the multiple key point information includes at least a first type of key point and a second type of key point information, the key point data in the first type of key point information being no less than the key point data in the second type of key point information; and correcting the second type of key point information based on the first type of key point information to determine the pose information of the target human body.

[0006] The process of obtaining the image to be detected includes: obtaining an original image, which includes multiple human bodies; performing human body detection on the original image to obtain human body detection boxes; the human body detection boxes contain at least one human body; and extracting the human body detection boxes from the original image to obtain the image to be detected, which is a sub-image of the original image.

[0007] The process of extracting features from the image to be detected to obtain a feature map of the image to be detected includes: extracting features from the image to be detected through a key point detection network to obtain a feature map of the image to be detected; wherein the key point detection network includes a feature extraction module and a coupling module cascaded in sequence, and the coupling module includes a first detection unit and a second detection unit, which are respectively cascaded with the feature extraction module.

[0008] The first detection unit comprises a cascaded deconvolutional layer, a first convolutional layer, and a second convolutional layer; the second detection unit comprises a cascaded first feature extraction layer and a second feature extraction layer; multiple keypoint detection methods are used to detect keypoints in the feature maps of the image to be detected, resulting in multiple types of keypoint information; the multiple types of keypoint information include at least first-type and second-type keypoint information, including: upsampling the feature map through the deconvolutional layer to obtain a first feature map; extracting features from the first feature map through the first convolutional layer to obtain a second feature map; extracting features from the feature map through the first feature extraction layer to obtain a third feature map; downsampling the second feature map to obtain a fourth feature map; upsampling the third feature map to obtain a fifth feature map; fusing the second and fifth feature maps to obtain a sixth feature map; fusing the third and fourth feature maps to obtain a seventh feature map; extracting features from the sixth feature map through the second convolutional layer to obtain first-type keypoint information of the image to be detected; and extracting features from the seventh feature map through the second feature extraction layer to obtain second-type keypoint information of the image to be detected.

[0009] The method employs multiple keypoint detection techniques to detect keypoints in the feature maps of the image to be detected, resulting in various keypoint information. This information includes at least a first type of keypoint and a second type of keypoint, specifically: first-type keypoint information is obtained by detecting keypoints in the feature maps using a first detection unit; this first type of keypoint information comprises all keypoints of the human body contained in the image to be detected. Second-type keypoint information is obtained by detecting keypoints in the feature maps using a second detection unit; this second type of keypoint information comprises keypoints of the target human body parts contained in the image to be detected.

[0010] Specifically, the second detection unit performs key point detection on the feature map to obtain the second type of key point information, including: performing key point detection on the feature map through the second detection unit to obtain all the key points of the human body parts contained in the image to be detected and the connection relationships between the key points of each part; matching the key points of the parts based on the connection relationships between the key points of each part to obtain the key point data corresponding to each human body; and selecting the target human body from all human bodies based on the key point data corresponding to each human body.

[0011] The key point data includes the number of key points in a body part and the confidence level of the key points in that body part. Based on the key point data corresponding to each human body, target human bodies are selected from all human bodies, including: calculating the score corresponding to each human body based on the number of key points in a body part and the confidence level of the key points in that body part; selecting the human body with the highest score as the target human body, and the key point data of the human body as the second type of key point information of the target human body.

[0012] The second type of key point information includes the position information and error range of each second type of key point; the first type of key point information includes the position information of each first type of key point; the second type of key point information is corrected based on the first type of key point information to determine the posture information of the target human body, including: calculating the error value between the position information of the first type of key points and the position information of the second type of key points at the same location; if the error value exceeds the error range of the corresponding second type of key point, then the second type of key point is used as the key point of the target human body; if the error value does not exceed the error range of the corresponding second type of key point, then the position information of the first type of key point and the position information of the second type of key point are weighted and averaged to obtain the key points of the target human body; based on all the key points of the target human body, the posture information of the target human body is determined.

[0013] To solve the above-mentioned technical problems, the second technical solution adopted by the present invention is: to provide a human posture estimation device, the human posture estimation device comprising: an acquisition module for acquiring an image to be detected; the image to be detected at least contains a target human body; a feature extraction module for performing feature extraction on the image to be detected to obtain a feature map of the image to be detected; a detection module for performing key point detection on the feature map of the image to be detected using multiple key point detection methods to obtain multiple key point information; the multiple key point information includes at least a first type of key point and a second type of key point information, wherein the key point data in the first type of key point information is not less than the key point data in the second type of key point information; and an analysis module for correcting the second type of key point information based on the first type of key point information to determine the posture information of the target human body.

[0014] To solve the above-mentioned technical problems, the third technical solution adopted by the present invention is to provide a terminal, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. The processor is used to execute program data to implement the steps in the above-mentioned human pose estimation method.

[0015] To solve the above-mentioned technical problems, the fourth technical solution adopted by the present invention is to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the above-mentioned human pose estimation method.

[0016] The beneficial effects of this invention are as follows: Unlike existing technologies, this invention provides a human pose estimation method, apparatus, terminal, and computer-readable storage medium. The human pose estimation method includes: acquiring an image to be detected; the image to be detected at least contains a target human body; extracting features from the image to be detected to obtain a feature map of the image to be detected; performing keypoint detection on the feature map of the image to be detected using multiple keypoint detection methods to obtain multiple keypoint information; the multiple keypoint information includes at least a first type of keypoint information and a second type of keypoint information, wherein the keypoint data in the first type of keypoint information is not less than the keypoint data in the second type of keypoint information; and correcting the second type of keypoint information based on the first type of keypoint information to determine the pose information of the target human body. This application improves the accuracy of detecting keypoint information of the target human body by performing keypoint detection on the feature map of the image to be detected using multiple keypoint detection methods, and then correcting the second type of keypoint information based on the first type of keypoint information among the multiple keypoint information. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the human pose estimation method provided by the present invention;

[0019] Figure 2 This is a flowchart illustrating a specific embodiment of the human pose estimation method provided by the present invention;

[0020] Figure 3 This is a schematic block diagram of a specific embodiment of the human posture estimation method provided by the present invention;

[0021] Figure 4 yes Figure 2A flowchart illustrating a specific embodiment of step S201 in the provided human pose estimation method;

[0022] Figure 5 yes Figure 2 A flowchart illustrating a specific embodiment of step S206 in the provided human pose estimation method;

[0023] Figure 6 This is a schematic block diagram of the human posture estimation device provided by the present invention;

[0024] Figure 7 This is a schematic block diagram of an embodiment of the terminal provided by the present invention;

[0025] Figure 8 This is a schematic block diagram of an embodiment of a computer-readable storage medium provided by the present invention. Detailed Implementation

[0026] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0027] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0028] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.

[0029] To enable those skilled in the art to better understand the technical solution of the present invention, the human posture estimation method provided by the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Please see Figure 1 , Figure 1 This is a flowchart illustrating the human posture estimation method provided by the present invention.

[0031] This embodiment provides a human pose estimation method, which is applicable to human pose estimation in dense scenes. The human pose estimation method includes the following steps.

[0032] S11: Acquire the image to be detected.

[0033] Specifically, the image to be detected must contain at least the target human body.

[0034] In one embodiment, an original image is acquired, which includes multiple human bodies; human body detection is performed on the original image to obtain a human body detection box; the human body detection box contains at least one human body; the human body detection box is extracted from the original image to obtain an image to be detected, which is a sub-image of the original image.

[0035] S12: Extract features from the image to be detected to obtain the feature map of the image to be detected.

[0036] Specifically, a keypoint detection network is used to extract features from the image to be detected, resulting in a feature map of the image. The keypoint detection network includes a feature extraction module and a coupling module cascaded together. The coupling module includes a first detection unit and a second detection unit, which are respectively cascaded with the feature extraction module.

[0037] S13: Use multiple key point detection methods to detect key points in the feature maps of the image to be detected, and obtain multiple key point information.

[0038] Specifically, the multiple key point information includes at least a first type of key point information and a second type of key point information, and the key point data contained in the first type of key point information is no less than the key point data contained in the second type of key point information.

[0039] The first detection unit includes a deconvolution layer, a first convolution layer, and a second convolution layer cascaded in sequence; the second detection unit includes a first feature extraction layer and a second feature extraction layer cascaded in sequence.

[0040] The feature map is upsampled using a deconvolutional layer to obtain a first feature map; features are extracted from the first feature map using a first convolutional layer to obtain a second feature map; features are extracted from the feature map using a first feature extraction layer to obtain a third feature map; the second feature map is downsampled to obtain a fourth feature map; the third feature map is upsampled to obtain a fifth feature map; the second and fifth feature maps are fused to obtain a sixth feature map; the third and fourth feature maps are fused to obtain a seventh feature map; features are extracted from the sixth feature map using a second convolutional layer to obtain the first type of key point information of the image to be detected; features are extracted from the seventh feature map using a second feature extraction layer to obtain the second type of key point information of the image to be detected.

[0041] In one embodiment, a first detection unit performs key point detection on the feature map to obtain a first type of key point information of the target human body; the first type of key point information is the key point information of all human body parts contained in the image to be detected; a second detection unit performs key point detection on the feature map to obtain a second type of key point information of the target human body; the second type of key point information is the key point information of the target human body parts contained in the image to be detected.

[0042] In one embodiment, the second detection unit performs key point detection on the feature map to obtain all the key points of the human body parts contained in the image to be detected and the connection relationships between the key points of each part; the key points of the parts are matched based on the connection relationships between the key points of each part to obtain key point data corresponding to each human body; and the target human body is selected from all human bodies based on the key point data corresponding to each human body.

[0043] In one embodiment, the key point data includes the number of key points in a body part and the confidence level of the key points in that body part; based on the number of key points in a body part corresponding to each human body and the confidence level of the key points in that body part, a score corresponding to each human body is calculated; the human body corresponding to the highest score is selected as the target human body, and the key point data of the human body is used as the second type of key point information of the target human body.

[0044] S14: Correct the second type of key point information based on the first type of key point information to determine the posture information of the target human body.

[0045] Specifically, key point information is used to characterize the posture of the target human body.

[0046] In one embodiment, the second type of key point information further includes the position information of each second type of key point and the error range of each second type of key point; the first type of key point information includes the position information of each first type of key point. The error value between the position information of the first type of key points and the position information of the second type of key points at the same location is calculated; if the error value exceeds the error range of the corresponding second type of key point, the second type of key point is used as a key point of the target human body; if the error value does not exceed the error range of the corresponding second type of key point, the position information of the first type of key point and the position information of the second type of key point are weighted and averaged to obtain the key points of the target human body; based on all the key points of the target human body, the posture information of the target human body is determined.

[0047] The human pose estimation method provided in this embodiment includes: acquiring an image to be detected; the image to be detected at least contains a target human body; extracting features from the image to be detected to obtain a feature map of the image to be detected; performing keypoint detection on the feature map of the image to be detected using multiple keypoint detection methods to obtain multiple keypoint information; the multiple keypoint information includes at least a first type of keypoint information and a second type of keypoint information, wherein the keypoint data in the first type of keypoint information is not less than the keypoint data in the second type of keypoint information; correcting the second type of keypoint information based on the first type of keypoint information to determine the pose information of the target human body. This application performs keypoint detection on the feature map of the image to be detected using multiple keypoint detection methods, and then corrects the second type of keypoint information based on the first type of keypoint information among the multiple keypoint information, thereby improving the accuracy of detecting the keypoint information of the target human body.

[0048] Please see Figure 2 and Figure 3 , Figure 2 This is a flowchart illustrating a specific embodiment of the human pose estimation method provided by the present invention; Figure 3 This is a schematic block diagram of a specific embodiment of the human posture estimation method provided by the present invention.

[0049] This embodiment provides a human pose estimation method, which is applicable to human pose estimation in dense scenes. The human pose estimation method includes the following steps.

[0050] S201: The keypoint detection network is trained.

[0051] Specifically, the keypoint detection network includes a feature extraction module and a coupling module.

[0052] Please see Figure 4 , Figure 4 yes Figure 2 A flowchart illustrating a specific embodiment of step S201 in the provided human pose estimation method.

[0053] The specific training steps for the keypoint detection network are shown below.

[0054] S2011: Obtain the training sample set.

[0055] Specifically, the training sample set includes multiple sample images containing human bodies, with each sample image containing at least one human body. That is, a sample image can contain one or multiple human bodies. Each sample image is associated with first and second ground truth keypoint information. The first ground truth keypoint information includes the first ground truth keypoint and its coordinates; the second ground truth keypoint information includes both the first ground truth keypoint and its coordinates. The first ground truth keypoint is a heatmap of the keypoints in each part; the second ground truth keypoint is the keypoints in each part and the connections between them.

[0056] In one embodiment, data augmentation is performed on the sample images to expand the training sample set. Data augmentation includes translating and rotating the sample images to obtain multiple sample images. The dimensions of the sample images are all preset sizes. If the sample images have inconsistent sizes, their dimensions are adjusted to the preset size. The preset size can be 256*256.

[0057] S2012: Key point detection is performed on the sample image using a key point detection network to obtain key point information for the first predicted region and key point information for the second predicted region.

[0058] Specifically, the coupling module in the keypoint detection network includes a first detection unit and a second detection unit. The first detection unit detects keypoint information of a first predicted part of the human body in the sample image, and the second detection unit detects keypoint information of a second predicted part of the human body in the sample image. The keypoint information of the first predicted part includes the first predicted keypoint of the human body and its position coordinates; the keypoint information of the second predicted part includes the second predicted keypoint of the human body and its position coordinates.

[0059] S2013: The key point detection network is iteratively trained based on the sum of the first difference between the first predicted key point information and the first real key point information of the same part in the same sample image and the second difference between the second predicted key point information and the second real key point information of the same part.

[0060] Specifically, the key point detection network calculates the first difference between the key point information of the first predicted part and the key point information of the first real part, and the second difference between the key point information of the second predicted part and the key point information of the second real part. The key point detection network is then iteratively trained based on the sum of the first difference and the second difference.

[0061] In one optional embodiment, the results of the keypoint detection network are backpropagated, and the weights of the keypoint detection network are corrected based on the first difference between the first predicted keypoint information of the same part and the first real keypoint information in the same sample image and the second difference between the second predicted keypoint information of the same part and the second real keypoint information, thereby achieving the training of the keypoint detection network.

[0062] The sample image is input into the keypoint detection network, which detects key points of the human body in the sample image. Training of the keypoint detection network stops when the sum of the first difference between the first predicted keypoint information and the first true keypoint information, and the second difference between the second predicted keypoint information and the second true keypoint information, for the same part of the same sample image, is less than a preset threshold. This preset threshold can be set manually, for example, 1% or 5%.

[0063] S202: Obtain the original image.

[0064] Specifically, an original image containing human bodies is acquired using an image acquisition device. In this embodiment, the original image includes multiple human bodies. That is, the original image is an image containing multiple human bodies acquired by the image acquisition device in a dense scene. In other embodiments, the original image may also be a video frame from a video stream.

[0065] S203: Perform human detection on the original image to obtain the human detection bounding box.

[0066] Specifically, a target detection network is used to detect human bodies in the original image, resulting in human body bounding boxes. For example, the target detection network can be a YOLO network. In one specific embodiment, the original image is input into the target detection network, which detects human bodies in the original image and obtains various human body bounding boxes. Each human body bounding box contains at least one human body. That is, in a dense crowd, multiple human bodies will be present in the detected human body bounding boxes.

[0067] S204: Extract the human body detection box from the original image to obtain the image to be detected, which is a sub-image of the original image.

[0068] Specifically, to filter out redundant information in the original image and reduce computation, the images within the detected human bounding boxes in the original image are extracted, and each image within a human bounding box is used as a sub-image of the original image. The sub-images are smaller in resolution than the original image, reducing the time required for forward inference.

[0069] In one embodiment, the images to be detected are preprocessed. Specifically, the images to be detected are resized, adjusting the size of all images to a preset size. For example, the preset size can be 256*256.

[0070] S205: The feature extraction module extracts features from the image to be detected to obtain the feature map of the image to be detected.

[0071] Specifically, the image to be detected, which is of a preset size, is input into the feature extraction module. The feature extraction module extracts features from the image to be detected to obtain the feature map of the image to be detected.

[0072] In one specific embodiment, the feature extraction module includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a fourth downsampling layer, a fifth downsampling layer, a first deconvolution layer, and a second deconvolution layer, all cascaded sequentially. A pre-sized image to be detected is input to the first downsampling layer, which performs feature extraction and dimensionality reduction on the image to obtain a first sub-feature map. The second downsampling layer further performs feature extraction and dimensionality reduction on the first sub-feature map to obtain a second sub-feature map. The third downsampling layer further performs feature extraction and dimensionality reduction on the second sub-feature map to obtain a third sub-feature map. The fourth downsampling layer further performs feature extraction and dimensionality reduction on the third sub-feature map to obtain a fourth sub-feature map. The fifth downsampling layer further performs feature extraction and dimensionality reduction on the fourth sub-feature map to obtain a fifth sub-feature map. The first convolutional layer extracts features from the fifth sub-feature map and increases its feature dimension to obtain a sixth sub-feature map. The second convolutional layer extracts features from the sixth sub-feature map and increases its feature dimension to obtain a seventh sub-feature map. The seventh sub-feature map has the same dimension as the third sub-feature map. The sixth sub-feature map has the same dimension as the fourth sub-feature map. The seventh sub-feature map serves as the feature map for the image to be detected.

[0073] S206: The first detection unit and the second detection unit in the coupling module perform key point detection on the feature map respectively to obtain the first type of key point information and the second type of key point information of the image to be detected.

[0074] Specifically, inputting the feature map of the image to be detected into the coupling module allows for feature enhancement and further feature extraction. In this embodiment, two keypoint detection methods are used. The first detection unit employs one keypoint detection method, and the second detection unit employs another. In other embodiments, three or more keypoint detection methods may be used, depending on the specific circumstances.

[0075] In one specific embodiment, the coupling module includes a first detection unit and a second detection unit, which are cascaded with the feature extraction module. The first detection unit uses a bottom-up pose estimation method to detect human key points in the image to be detected; the second detection unit uses a top-down pose estimation method to detect human key points in the image to be detected.

[0076] Please see Figure 5 , Figure 5 yes Figure 2 A flowchart illustrating a specific embodiment of step S206 in the provided human pose estimation method.

[0077] S2061: The first detection unit performs key point detection on the feature map to obtain the first type of key point information of the human body in the image to be detected.

[0078] Specifically, the first type of keypoint information consists of keypoint information for all human bodies contained in the image to be detected. Specifically, the first type of keypoint information is a heatmap of the keypoints of the human bodies contained in the image to be detected. The heatmap has a channel dimension of batch*n*w*h, where batch represents the number of channels; n is the number of keypoints of the human body to be detected; and w and h are the resolutions of the feature map of the image to be detected.

[0079] In one specific embodiment, the first detection unit includes a deconvolutional layer, a first convolutional layer, and a second convolutional layer cascaded sequentially. Since the heatmap obtained by the bottom-up pose estimation method is sensitive to the resolution of the feature map, a deconvolutional layer is needed to increase the feature dimension of the feature map of the image to be detected. Afterward, feature extraction and subsequent processing are performed to obtain the first type of keypoints of the human body contained in the image to be detected, as well as the position coordinates of each first type of keypoint. The first convolutional layer is a residual block in RestNet34.

[0080] S2062: The second detection unit performs key point detection on the feature map to obtain the second type of key point information of the target human body in the image to be detected.

[0081] Specifically, the second type of key point information refers to the key point data corresponding to the target human body in the image to be detected. The second type of key point information includes the position coordinates of the second type of key points and the second type of key points.

[0082] The second type of keypoint information includes PIF (Part Intensity Field) and PAF (Part Association Field) corresponding to keypoints of each part of the human body. PIF and PAF are feature maps of two domains. PIF represents the keypoints of each part of the human body, and PAF represents the direction vectors of the keypoints in each part, facilitating the connection of keypoints between parts to determine the keypoint data for each individual body. In other words, the connection edges between keypoints in different parts can be determined based on their direction vectors. The second detection unit includes a first feature extraction layer and a second feature extraction layer cascaded together.

[0083] The dimensionality of PIF and PAF is batch*n1*w1*h1, where n1 includes a 17*5 feature map for PIF and a 19*7 feature map for PAF. 17 represents 17 site keypoints, each represented by a heatmap. Each heatmap is divided into 5 parts, meaning each site keypoint corresponds to 5 feature maps: the confidence score, the X-axis offset, the Y-axis offset, the breadth (b), and the scale factor (σ). 19 represents 19 connection relationships, each represented by 7 parameters, including the confidence score of the connection edge between site keypoint i and site keypoint j. X-axis offset of key point i The offset of the Y coordinate of key point i in the part Breadth of key points i in the part X-axis offset of key point j in the part The offset of the Y coordinate of the key point j in the part Breadth of key points j in the part w1 and h1 are the resolutions of the feature maps of the image to be detected.

[0084] In one specific embodiment, a first feature map is obtained by upsampling the feature map through a deconvolution layer; a second feature map is obtained by extracting features from the first feature map through a first convolution layer; a third feature map is obtained by extracting features from the feature map through a first feature extraction layer; a fourth feature map is obtained by downsampling the second feature map; a fifth feature map is obtained by upsampling the third feature map; a sixth feature map is obtained by fusing the second and fifth feature maps; a seventh feature map is obtained by fusing the third and fourth feature maps; a first type of key point information of the image to be detected is obtained by extracting features from the sixth feature map through a second convolution layer; and a second type of key point information of the image to be detected is obtained by extracting features from the seventh feature map through a second feature extraction layer.

[0085] In one embodiment, a second detection unit performs keypoint detection on the feature map to obtain all body part keypoints corresponding to the human body in the image to be detected and the connection relationships between these keypoints. Based on the connection relationships between these keypoints, the body part keypoints are matched to obtain keypoint data corresponding to each human body. Based on this keypoint data, the target human body is selected from all human bodies. In other words, the direction vectors of each body part keypoint can be used to match all detected body part keypoint information, thereby determining the body part keypoints corresponding to each human body in the image to be detected. Based on all body part keypoints corresponding to each human body, the corresponding human body instance can be determined.

[0086] Based on the number of key points corresponding to each body part and the confidence level of these key points, a score is calculated for each body part. The body with the highest score is selected as the target body. The key point data of the body is used as the second type of key point information for the target body. The body part key points of the target body are used as the second type of key points.

[0087] In one specific embodiment, a score is calculated for each human body based on the number of key points corresponding to each body part and the weighted sum of the confidence levels of each key point. The human body instance with the highest score is selected as the target human body. Then, the key point information of the target human body in the image to be detected is determined. The key point information includes the coordinate positions of the key points.

[0088] The second type of keypoint information also includes the location information and error range of each second type of keypoint; the first type of keypoint information includes the location information of the first type of keypoints. The error range of the second type of keypoints is the scale factor σ of those keypoints.

[0089] S207: Calculate the error value between the location information of the first type of key points and the location information of the second type of key points representing the same location.

[0090] Specifically, the position coordinates of the second type of key points on the target human body are corrected based on the position coordinates of the first type of key points in the image to be detected, thereby improving the detection accuracy of key points on the target human body. When detecting the position coordinates of the same part using different detection methods, the error between the detected coordinates should be within a preset range. Therefore, it is necessary to calculate the error value between the position coordinates of the first type of key points and the position coordinates of the second type of key points on the same part, and then determine whether the first type of key points and the second type of key points belong to the same target human body.

[0091] S208: In response to an error value exceeding the error range of the corresponding second type of key point, the second type of key point is taken as the key point of the target human body.

[0092] Specifically, if the error between the position coordinates of the first type of keypoint and the position coordinates of the second type of keypoint at the same location exceeds the size factor σ of the second type of keypoint, it indicates that the first type of keypoint does not belong to the corresponding location keypoint of the target human body. Therefore, the posture correction unit uses the second type of keypoint as the keypoint of the target human body.

[0093] S209: In response to the error value not exceeding the error range of the corresponding second type of key point, the position information of the first type of key point and the position information of the second type of key point are weighted and averaged to obtain the key points of the target human body.

[0094] Specifically, if the error between the position coordinates of the first type of key point and the position coordinates of the second type of key point in the same part does not exceed the size factor σ of the second type of key point, it indicates that the first type of key point and the second type of key point belong to the same target human body. Therefore, the posture correction unit can correct the position coordinates of the second type of key point based on the position coordinates of the first type of key point to obtain more accurate position coordinates of the key points in the part.

[0095] In one specific embodiment, the position coordinates of the first type of key points and the position coordinates of the second type of key points corresponding to the same part of the target human body can be weighted and summed to obtain the position coordinates of the key points of the part corresponding to that part, and thus obtain the key points of the target human body.

[0096] S210: Based on all key points of the target human body, determine the key points of the parts of the target human body.

[0097] Specifically, the position coordinates of key points corresponding to various parts of the target human body are determined and output, thus identifying the key points of the target human body. The posture of the target human body is then determined based on these key points.

[0098] The human pose estimation method provided in this embodiment includes: acquiring an image to be detected; the image to be detected contains at least a target human body; extracting features from the image to be detected to obtain a feature map of the image to be detected; performing keypoint detection on the feature map of the image to be detected using multiple keypoint detection methods to obtain multiple keypoint information; the multiple keypoint information includes at least a first type of keypoint information and a second type of keypoint information, wherein the keypoint data in the first type of keypoint information is not less than the keypoint data in the second type of keypoint information; correcting the second type of keypoint information based on the first type of keypoint information to determine the pose information of the target human body. This application performs keypoint detection on the feature map of the image to be detected using multiple keypoint detection methods, and then corrects the second type of keypoint information based on the first type of keypoint information among the multiple keypoint information, thereby improving the accuracy of the detected target human body keypoint information.

[0099] See Figure 6 , Figure 6 This is a schematic block diagram of the human posture estimation device provided by the present invention. This embodiment provides a human posture estimation device 60, which includes an acquisition module 61, a feature extraction module 62, a detection module 63, and an analysis module 64.

[0100] The acquisition module 61 is used to acquire the image to be detected; the image to be detected contains at least the target human body;

[0101] Feature extraction module 62 is used to extract features from the image to be detected, and obtain the feature map of the image to be detected;

[0102] The detection module 63 is used to perform key point detection on the feature map of the image to be detected using multiple key point detection methods to obtain multiple key point information; the multiple key point information includes at least a first type of key point and a second type of key point information, and the key point data in the first type of key point information is no less than the key point data in the second type of key point information;

[0103] The analysis module 64 is used to correct the second type of key point information based on the first type of key point information to determine the posture information of the target human body.

[0104] In this embodiment, key point detection is performed on the feature map of the image to be detected using multiple key point detection methods. Then, based on the first type of key point information among the multiple key point information, the second type of key point information is corrected, thereby improving the accuracy of the key point information of the target human body obtained from the detection.

[0105] Please see Figure 7 , Figure 7 This is a schematic block diagram of an embodiment of the terminal provided by the present invention. The terminal 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-described human pose estimation method embodiments. In a specific implementation scenario, the terminal 80 may include, but is not limited to, a microcomputer or a server. In addition, the terminal 80 may also include mobile devices such as laptops and tablets, which are not limited here.

[0106] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described human pose estimation method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.

[0107] Please see Figure 8 , Figure 8This is a schematic block diagram of an embodiment of a computer-readable storage medium provided by the present invention. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor. The program instructions 901 are used to implement the steps of any of the above-described embodiments of the human pose estimation method.

[0108] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0109] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0114] The above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for estimating human pose, characterized in that, The human pose estimation method includes: Acquire an image to be detected; the image to be detected contains at least the target human body; Feature extraction is performed on the image to be detected to obtain a feature map of the image to be detected; Multiple key point detection methods are used to detect key points in the feature map of the image to be detected, resulting in multiple key point information. The multiple key point information includes at least a first type of key point information and a second type of key point information, wherein the key point data contained in the first type of key point information is no less than the key point data contained in the second type of key point information. Based on the first type of key point information, the second type of key point information is corrected to determine the posture information of the target human body; The step of extracting features from the image to be detected to obtain a feature map of the image to be detected includes: Feature extraction is performed on the image to be detected using a keypoint detection network to obtain a feature map of the image to be detected. The keypoint detection network includes a feature extraction module and a coupling module cascaded in sequence. The coupling module includes a first detection unit and a second detection unit, which are respectively cascaded with the feature extraction module. The first detection unit includes a deconvolution layer, a first convolution layer, and a second convolution layer cascaded in sequence. The second detection unit includes a first feature extraction layer and a second feature extraction layer cascaded in sequence. The method employs multiple keypoint detection techniques to detect keypoints in the feature map of the image to be detected, thereby obtaining various keypoint information, including: The feature map is upsampled by the deconvolution layer to obtain the first feature map; The first feature map is extracted by the first convolutional layer to obtain the second feature map; the third feature map is extracted by the first feature extraction layer. The second feature map is downsampled to obtain the fourth feature map; the third feature map is upsampled to obtain the fifth feature map. The second feature map and the fifth feature map are fused to obtain the sixth feature map; the third feature map and the fourth feature map are fused to obtain the seventh feature map. The second convolutional layer extracts features from the sixth feature map to obtain the first type of key point information of the image to be detected. The second feature extraction layer extracts features from the seventh feature map to obtain the second type of key point information of the image to be detected.

2. The human posture estimation method according to claim 1, characterized in that, The acquisition of the image to be detected includes: Acquire an original image, which includes multiple human figures; Human detection is performed on the original image to obtain a human detection box; the human detection box contains at least one human body. The human body detection box is extracted from the original image to obtain the image to be detected, which is a sub-image of the original image.

3. The human posture estimation method according to claim 1, characterized in that, The feature map of the image to be detected is subjected to key point detection using multiple key point detection methods to obtain multiple key point information. The multiple key point information includes at least a first type of key point information and a second type of key point information, including: The first detection unit performs key point detection on the feature map to obtain the first type of key point information. The first type of key point information is the key point information of all the human body contained in the image to be detected; The second detection unit performs key point detection on the feature map to obtain the second type of key point information; the second type of key point information refers to the key points of the target human body parts contained in the image to be detected.

4. The human posture estimation method according to claim 3, characterized in that, The step of detecting key points in the feature map using the second detection unit to obtain the second type of key point information includes: The second detection unit performs key point detection on the feature map to obtain all the key points of the human body parts contained in the image to be detected and the connection relationships between the key points of each part. Based on the connection relationship between the key points of the said parts, the key points of the said parts are matched to obtain the key point data corresponding to each of the said human bodies; Based on the key point data corresponding to each of the human bodies, the target human body is selected from all the human bodies.

5. The human posture estimation method according to claim 4, characterized in that, The key point data includes the number of key points in the part and the confidence level of the key points in the part; The step of selecting the target human body from all human bodies based on the key point data corresponding to each human body includes: Based on the number of key points corresponding to each part of each human body and the confidence level of the key points of each part, a score corresponding to each human body is calculated. The human body corresponding to the highest score is selected as the target human body, and the key point data of the human body is used as the second type of key point information of the target human body.

6. The human pose estimation method according to claim 4, characterized in that, The second type of key point information also includes the location information of each second type of key point and the error range of each second type of key point; the first type of key point information includes the location information of each first type of key point. The step of correcting the second type of key point information based on the first type of key point information to determine the posture information of the target human body includes: The error value between the positional information of the first type of key points and the positional information of the second type of key points in the same location is calculated. If the error value exceeds the error range of the corresponding second type of key point, then the second type of key point is taken as the key point of the target human body; In response to the error value not exceeding the error range of the corresponding second type of key point, the position information of the first type of key point and the position information of the second type of key point are weighted and averaged to obtain the key points of the target human body; Based on all the key points of the target human body, the posture information of the target human body is determined.

7. A human posture estimation device, characterized in that, The human posture estimation device includes: The acquisition module is used to acquire the image to be detected; the image to be detected contains at least the target human body; A feature extraction module is used to extract features from the image to be detected to obtain a feature map of the image to be detected; a keypoint detection network is used to extract features from the image to be detected to obtain a feature map of the image to be detected; the keypoint detection network includes a feature extraction module and a coupling module cascaded in sequence, the coupling module includes a first detection unit and a second detection unit, the first detection unit and the second detection unit are respectively cascaded with the feature extraction module; the first detection unit includes a deconvolution layer, a first convolution layer and a second convolution layer cascaded in sequence; the second detection unit includes a first feature extraction layer and a second feature extraction layer cascaded in sequence; The detection module is used to perform keypoint detection on the feature map of the image to be detected using multiple keypoint detection methods to obtain multiple keypoint information. The multiple keypoint information includes at least a first type of keypoint information and a second type of keypoint information, wherein the keypoint data in the first type of keypoint information is no less than the keypoint data in the second type of keypoint information. Specifically, the feature map is upsampled through the deconvolution layer to obtain a first feature map; features are extracted from the first feature map through the first convolution layer to obtain a second feature map; features are extracted from the feature map through the first feature extraction layer to obtain a third feature map; the second feature map is downsampled to obtain a fourth feature map; the third feature map is upsampled to obtain a fifth feature map; the second feature map and the fifth feature map are fused to obtain a sixth feature map; the third feature map and the fourth feature map are fused to obtain a seventh feature map; features are extracted from the sixth feature map through the second convolution layer to obtain the first type of keypoint information of the image to be detected; and features are extracted from the seventh feature map through the second feature extraction layer to obtain the second type of keypoint information of the image to be detected. The analysis module is used to correct the second type of key point information based on the first type of key point information to determine the posture information of the target human body.

8. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, the processor being used to execute program data to implement the steps in the human pose estimation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the human pose estimation method as described in any one of claims 1 to 6.

Images