Posture estimation device, method and program

Abstract

To determine likelihood of each part of a posture with respect to output of first posture estimation for explicitly handling which portion of information is uncertain or missing, and for performing second posture estimation by selectively using the information.SOLUTION: A first posture estimating unit 30 estimates a first posture of a posture estimation target captured in a camera image. A certainty factor acquisition unit 40 acquires a certainty factor for each part of the first posture. A second posture estimation unit 50 applies the first posture and its certainty factor to an estimation model of a neural network for estimating a second posture. At that time, the second posture estimation unit 50 selectively corrects a part of the first posture with a relatively low certainty factor, based on an estimation result of a part with a relatively high certainty factor, and converts the second posture into the posture estimation result.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a posture estimation device, method, and program for estimating the posture of an object captured in an image. [Background technology]

[0002] A method for estimating the posture of an object captured in an image is disclosed in Patent Document 1. Patent Document 1 performs a first posture estimation from an image of the object captured by a first image capturing unit, performs a second posture estimation using the result and an image captured by a second image capturing unit, and outputs the result. As a result, it may be possible to use the second posture estimation to interpolate a portion where the posture could not be estimated by the first posture estimation due to the influence of occlusion or the like.

[0003] Non-Patent Document 1 discloses a technique for performing a first pose estimation from an image, generating a feature amount that also takes into account the positional relationship of the entire pose by combining the feature amount of the image obtained there, and performing a second pose estimation using this feature amount to output the result. As a result, it is possible to interpolate and correct to a natural shape by the second pose estimation for a portion where the pose could not be estimated by the first pose estimation due to the influence of occlusion, etc.

[0004] Non-Patent Document 2 discloses a technique in which a first pose estimation is performed from an image by some means, and the result is corrected to a more natural form in a second pose estimation unit. For the second pose estimation, a Convolutional Neural Network (CNN) that has learned the distribution of errors in general pose estimation is used.

[0005] In addition, although it is not related to pose estimation, Non-Patent Document 3 discloses a technology for improving the correction performance in the task of correcting an original image with defects into a natural image without defects by replacing each convolutional layer (hereinafter referred to as Conv layer) in a CNN with a Gated Conv layer that has the function of filtering information derived from defective pixels. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-097577 A [Non-patent literature]

[0007] [Non-Patent Document 1] Y. Chen, Z. Wang, Y. Peng, Z. Zhang, G. Yu, and J. Sun, "Cascaded pyramid network for multi-person pose estimation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7103-7112. [Non-Patent Document 2] Moon, Gyeongsik, Ju Yong Chang, and Kyoung Mu Lee. "Posefix: Model-agnostic general human pose refinement network." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019. [Non-Patent Document 3] Yu, Jiahui, et al. "Free-form image inpainting with gated convolution." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019. Summary of the Invention [Problem to be solved by the invention]

[0008] In Patent Document 1 and Non-Patent Documents 1 and 2, the second attitude estimation is performed based on the result of the first attitude estimation, so if the result of the first attitude estimation contains many errors, it is difficult to improve it by the second attitude estimation. In some cases, the effect of the erroneous result may be strong, and the accuracy may decrease due to the second attitude estimation.

[0009] On the other hand, in order to avoid outputting erroneous results in the first pose estimation, restrictions are imposed on the shooting conditions such as the position of the object and the background. Non-Patent Document 3 does not take into consideration application to pose estimation technology.

[0010] An object of the present invention is to solve the above-mentioned technical problems and to provide a posture estimation device, method, and program that determines the certainty of each posture part for the output of a first posture estimation, explicitly determines which parts of information are uncertain or missing, and selects information to refer to when making correction, thereby performing a second posture estimation. [Means for solving the problem]

[0011] In order to achieve the above object, the present invention provides a posture estimation device that estimates the posture of an object captured in a camera image, and is characterized in that it has the following configuration.

[0012] (1) The apparatus includes a means for estimating a first posture of a posture estimation target captured in a camera image, a means for acquiring a degree of certainty for each part of the first posture, and a means for estimating a second posture based on the first posture and its degree of certainty, and the means for estimating the second posture selectively corrects parts of the first posture for which the degree of certainty is relatively low, based on parts for which the degree of certainty is relatively high.

[0013] (2) The means for estimating the second posture includes means for selecting a portion to be corrected in the first posture based on the degree of certainty, and means for correcting the portion to be corrected.

[0014] (3) The means for estimating the second posture is a convolutional neural network, and the convolutional layer does not perform convolution of the part to be corrected.

[0015] (4) The convolutional layer is a gated convolutional layer, and the gated convolutional layer includes: means for applying an activation function to each element of the intermediate feature after convolution of the first pose masked by the confidence level to output a correction target; and means for normalizing each element of the intermediate feature after convolution of the first pose masked by the confidence level by the confidence level to output a confidence level. The convolution of the feature selected from the correction target based on the confidence level is further repeated.

[0016] (5) The means for estimating the second orientation estimates the second orientation at time t based on the first orientations at time t and at least one of the times before and after t and their certainties. Effect of the Invention

[0017] (1) Even in difficult situations that can occur when the shooting conditions are not constrained, such as when part of the target for pose estimation is not captured in the image or when the background contains an image similar to the target for pose estimation, the adverse effect of an erroneous estimation result of the first pose on the estimation of the second pose can be reduced. This makes it possible to realize pose estimation that is more robust and capable of stable operation, without requiring constraints on the shooting conditions.

[0018] (2) Since an erroneously estimated part of the first posture is determined based on the confidence level obtained during posture estimation, the erroneously estimated part can be quantitatively and accurately grasped.

[0019] (3) By replacing the convolutional layer of the CNN model used to estimate the second pose with a gated convolutional layer, the adverse effect of erroneous estimation results of the first pose on the estimation of the second pose can be reduced, thereby improving the accuracy of estimating the second pose using the CNN estimation model.

[0020] (4) The second orientation is estimated based on not only the first orientation at time t but also the first orientation at a plurality of times before and after time t, thereby enabling more accurate orientation estimation. [Brief description of the drawings]

[0021] [Figure 1] 1 is a functional block diagram of a posture estimation device according to a first embodiment of the present invention. [Diagram 2] FIG. 13 illustrates an example of pose estimation. [Diagram 3] FIG. 11 is a block diagram showing an example in which a confidence level is calculated separately and used for pose estimation. [Figure 4] FIG. 13 is a diagram showing an example of estimating a second pose by applying a first pose and its confidence factor to a CNN. [Diagram 5] FIG. 1 is a block diagram showing an example of a Gated Conv layer that can replace the Conv layer of a CNN. [Figure 6] FIG. 11 is a block diagram showing a second embodiment of the present invention. [Figure 7] FIG. 11 is a block diagram showing a modified example of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. Fig. 1 is a functional block diagram showing the configuration of the main parts of a posture estimation device 1 according to a first embodiment of the present invention, and the main components include an image acquisition unit 10, a target region extraction unit 20, a first posture estimation unit 30, a confidence factor acquisition unit 40, a second posture estimation unit 50, and a posture output unit 60.

[0023] Such a posture estimation device 1 can be configured by implementing an application (program) that realizes each function on at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

[0024] The image acquisition unit 10 acquires a camera image (e.g., an RGB image) capturing a posture estimation target. In this embodiment, a case where the posture estimation target is a person will be described as an example. The target area extraction unit 20 extracts a person area as the image area of ​​the posture estimation target from the camera image. The person area may be extracted as a rectangular area circumscribing the person, such as a bounding box, as in Non-Patent Document 1, or may be extracted as a silhouette following the contour of the person.

[0025] The first posture estimation unit 30 estimates a first posture of a person from an image of the extracted person area. In this embodiment, as shown in an example in Fig. 2, feature points related to the structure of the person are extracted from the person area, key points such as the face, hands, torso, and feet and their positions are estimated, and a set of key points is output as an estimation result of the first posture. In this embodiment, the description will be given assuming that the 23 key points shown in Fig. 2 and their position coordinates are estimated.

[0026] The keypoints to be estimated may be only those visible in the camera image, or all types of keypoints including those not visible may be estimated. The positions of the keypoints may be expressed in two-dimensional coordinates on the image, or may be expressed using other appropriate coordinate systems such as a three-dimensional coordinate system. When expressing the positions in three-dimensional coordinates, images from multiple viewpoints are acquired and the three-dimensional position of each keypoint is calculated by triangulation.

[0027] The certainty factor acquisition unit 40 acquires the likelihood of the estimation result for each keypoint of the first posture as a certainty factor. As the certainty factor, the certainty factor of the estimation result of each keypoint calculated from image features at the same time as posture estimation in Non-Patent Document 1 can be used. The certainty factor of each keypoint may be represented by a score that takes a value between 0 and 1, or may be represented by a truth value of 0 or 1 by setting a threshold value.

[0028] If a confidence level cannot be obtained in the estimation process of the first posture in first posture estimation unit 30, or if it is desired to use a confidence level from another viewpoint even if a confidence level can be obtained, a storage unit 70 may be provided as shown in Fig. 3 to store estimation results at another point in time, and the confidence level may be calculated based on whether or not the position transition is appropriate taking into account physical constraints, etc. Alternatively, reference information such as a person's body model may be stored, and the confidence level may be calculated based on whether or not the positional relationship of each key point satisfies a constraint condition specific to a person, etc.

[0029] Second posture estimation section 50 includes correction portion selection section 501 and selective correction section 502, and estimates a second posture of the person based on the first posture estimated by first posture estimation section 30 and its confidence level. The second posture is output from posture output section 60 as a result of posture estimation.

[0030] The correction part selection unit 501 selects parts to be corrected from each part of the first posture based on the confidence level. For example, if the first posture is estimated by a set of position coordinates of each key point as a main part, a method of using the confidence level as a mask and multiplying each key point of the first posture by the mask value can be considered. The selective correction unit 502 estimates the second posture by correcting parts of the first posture with a relatively low confidence level as correction targets and parts with a relatively high confidence level using reference information.

[0031] With this configuration, uncertain parts in the first posture can be explicitly treated, so that the influence of uncertain parts on the estimation result in the second posture estimation can be suppressed. For example, it becomes possible to prevent erroneous correction of a certain part, or to prevent information of an uncertain part from being referred to when correcting the uncertain part.

[0032] FIG. 4 is a diagram showing an example in which the second posture estimation unit 50 is configured based on an encoder-decoder (autoencoder) type CNN model, and in this embodiment, the Conv layer is replaced with a gated convolution layer (Gated Conv layer).

[0033] The encoder receives a heat map showing the image positions of all keypoints and a mask generated for each keypoint using the confidence as prior knowledge. The features compressed by the encoder are expanded by upsampling in the decoder. In this embodiment, the Conv layer in the CNN model is replaced with a Gated Conv layer, so that keypoints with relatively low confidence in the first pose can be corrected by features obtained from keypoints with relatively high confidence.

[0034] FIG. 5 is a block diagram showing an example of the configuration of a Gated Conv layer employed in this embodiment. In the first stage, for each Conv layer in a CNN model, a masked heat map of each keypoint is used as an input and a convolution layer extracts intermediate features. To these intermediate features, an activation function (ReLU: Rectified Linear Unit) / ramp function is applied in the correction target system R1, so that, with "0" as the base point, if the input value is 0 or less, "0" is output, and if the input value is greater than 0, the same value as the input value is output, both as the features to be corrected.

[0035] In the reference information system R2, a softmax function is applied to the intermediate features extracted by the Conv layer to calculate the reliability (0-1) for each element of the feature, and this is output as reference information. Each Conv layer functions as the selective correction unit 502. A multiplication unit 503 functions as the correction part selection unit 501, and multiplies the feature to be corrected by the reference information (reliability) to select a feature with a relatively high reliability, which is then input to the next Gated Conv layer as new input information.

[0036] With this configuration, it is possible to suppress the propagation of information from parts of the CNN model where the estimation results are uncertain, making it possible to make inferences using only strictly certain information.

[0037] 6 is a functional block diagram showing a configuration of a second embodiment of the present invention. In the first embodiment, the second orientation at time t is estimated based on the first orientation at time t and its certainty factor. In contrast, in the second embodiment, the second orientation at time t is estimated by processing the first orientation at multiple times before and after time t and its certainty factor multiple times in a one-dimensional (1D) Conv layer in the time dimension.

[0038] In this embodiment, all Conv layers are replaced with Gated Conv layers, and a vector (23×4=92 dimensions) is calculated for each time t for which the second posture is to be estimated and the times before and after it, t-2, t-1, t+1, t+2, etc., with position information (x, y, z) of each part (23 key points in this embodiment) in the first posture and its confidence level (k) as elements. Then, the 92-dimensional vectors calculated for each time are provided for multiple times as input matrices to the Gated Conv layers, and the second posture at time t is estimated.

[0039] If the first orientations estimated at more time points and vectors of their certainties are to be provided as an input matrix, it is desirable to provide multiple Gated Convolution layers, as shown in an example in FIG.

[0040] In addition, in this embodiment, the first postures estimated at multiple times before and after the time t at which the posture is to be estimated and their certainty factors are used to estimate the second posture, but the present invention is not limited to this, and the second posture may be estimated based on multiple first postures estimated at time t and at least one other time before and after time t and their certainty factors.

[0041] According to this embodiment, the second posture at time t is estimated based not only on the first posture at time t and its certainty factor, but also on the first posture at multiple times before and after time t and its certainty factor, thereby enabling more accurate posture estimation.

[0042] According to the above-mentioned embodiments, it is possible to provide highly accurate pose estimation at low cost, and therefore it is possible to provide a variety of services to many people regardless of geographic or economic disparity. As a result, it is possible to contribute to Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations. [Explanation of symbols]

[0043] 1... Posture estimation device, 10... image acquisition unit, 20... target area extraction unit, 30... first posture estimation unit, 40... certainty factor acquisition unit, 50... second posture estimation unit, 60... posture output unit, 70... storage unit, 501... correction area selection unit, 502... selective correction unit, 503... multiplication unit

Claims

1. In a posture estimation device that estimates the posture of an object captured in a camera image, A means for estimating a first orientation of a posture estimation target captured in a camera image; A means for acquiring a confidence level for each part of the first posture; means for estimating a second attitude based on the first attitude and a confidence factor thereof; The means for estimating the second attitude a means for selecting a portion to be corrected in the first posture based on the degree of certainty; and a means for correcting the correction target portion, The means for correcting is an autoencoder type convolutional neural network including an encoder and a decoder, A portion of the first posture having a relatively low certainty is selectively corrected based on a portion of the first posture having a relatively high certainty. In the encoder, a convolution layer convolves a portion excluding the portion to be corrected to extract a feature amount; The decoder is configured to estimate each part including the part to be corrected by upsampling the feature amount.

2. The convolution layer is a gated convolution layer, The gated convolutional layer comprises: a means for applying an activation function to each element of an intermediate feature after convolution of the first orientation masked by the confidence level, and outputting a correction target; and a means for normalizing each element of the intermediate feature quantity after convolution of the first posture masked by the confidence level by the confidence level, and outputting a confidence level; The posture estimation apparatus according to claim 1 , further comprising: repeating convolution of the feature quantity selected from the correction target based on the reliability.

3. 3. The posture estimation device according to claim 1, wherein the means for estimating the second posture estimates the second posture at time t based on each of the first postures at time t and at least one of the times before and after t and their certainties.

4. A posture estimation method in which a computer estimates the posture of an object captured in a camera image, comprising: A first posture of the posture estimation target captured in the camera image is estimated, Obtain confidence for each part of the first posture, Estimating a second pose based on the first pose and the confidence level thereof; In estimating the second attitude, selecting a portion to be corrected in the first posture based on the degree of certainty; The correction target portion is corrected by an autoencoder type convolutional neural network including an encoder and a decoder; A portion of the first posture having a relatively low certainty is selectively corrected based on a portion of the first posture having a relatively high certainty. In the encoder, a convolution layer convolves a portion excluding the portion to be corrected to extract a feature amount; The posture estimation method, wherein the decoder estimates each part including the part to be corrected by upsampling the feature amount.

5. The convolution layer is a gated convolution layer, The gated convolutional layer comprises: applying an activation function to each element of the intermediate feature after convolution of the first posture masked by the confidence level to output a correction target; Normalizing each element of the intermediate feature after convolution of the first posture masked by the confidence level by the confidence level to output a confidence level; The posture estimation method according to claim 4 , further comprising repeating convolution of the feature quantity selected from the correction target based on the reliability.

6. In estimating the second attitude, 6. The method according to claim 4, further comprising estimating the second orientation at time t based on the first orientations at time t and at least one of the times before and after t and their certainties.

7. In a posture estimation program that estimates the posture of an object captured in a camera image, A step of estimating a first pose of a pose estimation target captured in a camera image; A procedure for obtaining confidence levels for each part of the first posture; and estimating a second pose based on the first pose and a confidence level thereof; In the step of estimating the second attitude, selecting a portion to be corrected in the first posture based on the degree of certainty; and correcting the correction target portion using an autoencoder type convolutional neural network including an encoder and a decoder; A portion of the first posture having a relatively low certainty is selectively corrected based on a portion of the first posture having a relatively high certainty. In the encoder, a convolution layer convolves a portion excluding the portion to be corrected to extract a feature amount; The decoder upsamples the feature amount to estimate each part including the part to be corrected.

8. The convolution layer is a gated convolution layer, The gated convolutional layer comprises: applying an activation function to each element of the intermediate feature after convolution of the first posture masked by the confidence level to output a correction target; and normalizing each element of the intermediate feature quantity after convolution of the first posture masked by the confidence level by the confidence level to output a confidence level; 8. The posture estimation program according to claim 7, further comprising repeating convolution of the feature quantity selected from the correction target based on the reliability.

9. 9. The posture estimation program according to claim 7 or 8, wherein in the step of estimating the second posture, the second posture at time t is estimated based on each of the first postures at time t and at least one of the times before and after t, and their certainties.

Images