Joint pose estimation method, joint pose estimation system, and joint pose estimation program

Abstract

To provide a technique capable of accurately estimating a two-dimensional joint pose of an object.SOLUTION: Inputting an image 10 of a target object to a two-dimensional joint pose model enables obtaining an estimated two-dimensional joint pose 12. The two-dimensional joint pose model is configured to: receive an input of an image of an articulated object; and then output a two-dimensional joint pose of the articulated object with confidence scores of estimated positions for respective joints. Then, a query pose 13 is generated by removing a joint whose confidence score of the estimated position is lower than a threshold from the estimated two-dimensional joint pose 12. Then, a correction two-dimensional joint pose 14 closest to the query pose 13 is obtained from a database 150 in which a plurality of sample two-dimensional joint poses is registered for each basic joint pose of a sample articulated object, Then, the two-dimensional joint pose 15 of the target object is obtained by replacing the joint whose confidence score is lower than the threshold with a corresponding joint of the correction two-dimensional joint pose 14.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a technique suitable for use in estimating the joint posture of an object, including a person. [Background technology]

[0002] Patent Document 1 discloses a technique for estimating the posture of an object. According to this conventional technique, first, joint position candidates of an object are calculated from a range image including a multi-joint object. Next, the consistency of the positions of the joint position candidates is evaluated based on information on the allowable range for the positional relationship between joints in a multi-joint object model corresponding to the object. Then, based on the evaluation, the joint positions of the object are determined from the joint position candidates, thereby estimating the posture of the object.

[0003] However, the above-mentioned conventional techniques do not take occlusion or truncation into consideration, and therefore, in difficult situations including occlusion or truncation, there is a risk that the accuracy of estimating the joint positions may decrease.

[0004] In addition to Patent Document 1, Patent Documents 2 and 3 can be mentioned as examples of documents showing the state of the art in the technical field related to the present disclosure. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6433149 [Patent Document 2] Patent No. 4709723 [Patent Document 3] Patent No. 5555207 Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a technology that can accurately estimate the two-dimensional joint posture of an object even under difficult conditions including occlusion and truncation. [Means for solving the problem]

[0007] The present disclosure provides a joint posture estimation technique for achieving the above object. The joint posture estimation technique of the present disclosure includes a joint posture estimation method, a joint posture estimation system, and a joint posture estimation program.

[0008] The joint posture estimation method of the present disclosure includes the following first to fourth steps. The first step is a step of estimating a two-dimensional joint posture of a target object from an image of the target object belonging to an articulated object that can be represented by a plurality of joints. A two-dimensional joint posture model is used for estimating the two-dimensional joint posture. The two-dimensional joint posture model is a model configured to receive an image of the articulated object and output the two-dimensional joint posture of the articulated object together with the confidence of the estimated position of each joint. The second step is a step of generating a query posture by removing joints whose confidence of the estimated position is less than a threshold from the two-dimensional joint posture of the target object. The third step is a step of acquiring a sample two-dimensional joint posture closest to the query posture from a database. The database has registered therein a plurality of sample two-dimensional joint postures for each basic joint posture of the sample articulated object. The fourth step is a step of correcting the two-dimensional joint posture of the target object by replacing a joint whose confidence of the estimated position is less than a threshold with a corresponding joint of the closest sample two-dimensional joint posture.

[0009] The joint pose estimation system of the present disclosure includes at least one processor and a program memory coupled to the at least one processor and storing a plurality of instructions. The plurality of instructions are configured to cause the at least one processor to execute the following first to fourth processes. The first process is a process of estimating a two-dimensional joint pose of a target object from an image of the target object belonging to an articulated object that can be represented by a plurality of joints using a two-dimensional joint pose model. The second process is a process of generating a query pose by removing joints whose estimated position has a certainty factor less than a threshold from the two-dimensional joint pose of the target object. The third process is a process of obtaining a sample two-dimensional joint pose closest to the query pose from a database. The fourth process is a process of correcting the two-dimensional joint pose of the target object by replacing a joint whose estimated position has a certainty factor less than a threshold with a corresponding joint of the closest sample two-dimensional joint pose.

[0010] The joint posture estimation program of the present disclosure includes a plurality of instructions executable by at least one processor. The plurality of instructions are configured to cause the at least one processor to execute the following first to fourth processes. The first process is a process of estimating a two-dimensional joint posture of a target object from an image of the target object belonging to an articulated object that can be represented by a plurality of joints using a two-dimensional joint posture model. The second process is a process of generating a query posture by removing a joint whose estimated position has a certainty factor less than a threshold from the two-dimensional joint posture of the target object. The third process is a process of acquiring a sample two-dimensional joint posture closest to the query posture from a database. The fourth process is a process of correcting the two-dimensional joint posture of the target object by replacing a joint whose estimated position has a certainty factor less than a threshold with a corresponding joint of the closest sample two-dimensional joint posture. The joint posture estimation program of the present disclosure may be stored in a computer-readable non-transitory storage medium or provided via a network. Effect of the Invention

[0011] According to the joint posture estimation technique of the present disclosure, among the joints constituting the 2D joint posture estimated using the 2D joint posture model, those joints whose estimated positions have a high degree of certainty are maintained in their original positions, and only those joints whose estimated positions have a low degree of certainty are replaced with the corresponding joints of the closest sample 2D joint posture. This makes it possible to accurately estimate the 2D joint posture of a target object even under difficult conditions such as occlusion or truncation. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a system for implementing a joint posture estimation method according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of a two-dimensional joint posture model. [Diagram 3] FIG. 13 is a diagram illustrating a sample two-dimensional joint posture acquisition unit. [Figure 4] FIG. 11 is a diagram showing a specific flow of a joint posture estimation method according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a joint posture estimation system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] FIG. 1 is a diagram showing a system for realizing a joint posture estimation method according to an embodiment of the present disclosure, that is, a configuration of a joint posture estimation system. The joint posture estimation system 100 according to this embodiment is a system for estimating a two-dimensional joint posture 15 of a target object from an image 10 of the target object. The target object is a multi-joint object whose joint posture is to be estimated. Multi-joint objects include not only humans but also anything that can be represented by multiple joints, such as vehicles, desk drawers, and scissors. The joint posture estimation system 100 according to this embodiment includes a two-dimensional joint posture estimation unit 110, a query posture generation unit 130, a sample two-dimensional joint posture acquisition unit 140, and a two-dimensional joint posture correction unit 160.

[0014] First, the two-dimensional joint posture estimation unit 110 will be described. A two-dimensional joint posture model 120 is used for estimating the two-dimensional joint posture. The two-dimensional joint posture model 120 is a model (neural network) used for estimating the two-dimensional joint posture of a multi-joint object. A two-dimensional joint posture model 120 is prepared for each type of target object. In the following description, it is assumed that the target object is a human being, and that the two-dimensional joint posture model 120 has been trained for estimating the two-dimensional joint posture of a human being.

[0015] A joint position model based on a top-down approach is particularly effective as the two-dimensional joint posture model 120. The architectures TransPose, AlphaPose, VitPose, RMPE, and Location-free Human Pose Estimation described in the following papers are examples of top-down joint position models that can be used as the two-dimensional joint posture model 120. (1)TransPose : Sen Yang, Zhibin Quan, Mu Nie, Wankou Yang. “TransPose: Keypoint Localization via Transformer” arXiv:2012.14214 [cs.CV] 1 Sep 2021 (2)AlphaPose : Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu, “AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Tim” arXiv:2211.03375v1 [cs.CV] 7 Nov 2022 (3)VitPose : Yufei Xu, Jing Zhang, Qiming Zhang, Dacheng Tao, “ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation” arXiv:2204.12484v3 [cs.CV] 13 Oct 2022 (4)RMPE : Hao-Shu Fang1, Shuqin Xie, Yu-Wing Tai, Cewu Lu, “RMPE: Regional Multi-Person Pose Estimation” arXiv:1612.00137 [cs.CV] 4 Feb 2018 (5)Location-free Human Pose Estimation : Xixia Xu, Yingguo Gao, Ke Yan, Xue Lin, Qi Zou, “Location-free Human Pose Estimation” arXiv:2205.12619 [cs.CV] 25 May 2022

[0016] FIG. 2 shows an example of a two-dimensional joint posture model 120. In the example shown in FIG. 2, the TransPose architecture is used as the two-dimensional joint posture model 120. The TransPose architecture has a CNN backbone 121, a transformer encoder layer 122 consisting of multiple layers, and a head 123. By inputting an image 10 of a target person to the CNN backbone 121, multiple keypoint heatmaps 11 are output from the head 123. The keypoint heatmap 11 is output for each estimated joint. In each keypoint heatmap 11, the estimated positions of the target person's joints and their confidence levels are shown in a heatmap.

[0017] A 2D joint pose model 120 such as the TransPose architecture can estimate the joint positions of a multi-joint object very accurately under normal conditions, meaning conditions without occlusion or truncation. However, under difficult conditions including occlusion or truncation, the accuracy of the joint position estimation by the 2D joint pose model 120 decreases.

[0018] As illustrated in FIG. 2, the 2D joint posture model 120 is configured to output a confidence level of the estimated position of each joint of the target person. When no occlusion (including self-occlusion in which joints are hidden by the person's own body) or truncation occurs, the positions of all joints are estimated with a high confidence level. On the other hand, when occlusion or truncation occurs, the estimated positions of some joints have a significantly lower confidence level than others. In other words, even when occlusion or truncation occurs, the positions of many joints are estimated with a high confidence level.

[0019] Here, a sample of an articulated object to which a target object belongs is defined as a sample articulated object. Although the number of joint postures that a sample articulated object can take is infinite, they can be classified into a finite number of basic joint postures in three dimensions. The joint posture of the target object corresponds to one of the basic joint postures of the sample articulated objects. Therefore, even if a part of the joint posture of a target person is hidden due to occlusion or truncation in an image, if the basic joint posture of the sample articulated object corresponding to the joint posture of the target person is known, the correct two-dimensional joint posture of the target object can be estimated with a high degree of accuracy from the basic joint posture.

[0020] In the joint posture estimation method according to the present embodiment, a basic joint posture of a sample joint object is projected onto a plane from various directions, so that a large number of sample 2D joint postures from different viewpoints are prepared for one basic joint posture. Each sample 2D joint posture is associated with the corresponding basic joint posture and registered in a database. In the joint posture estimation method according to the present embodiment, a query posture generated from the output of the 2D joint posture model 120 is used as query information for searching the database.

[0021] The query posture is generated by the query posture generation unit 130 shown in FIG. 1. The query posture generation unit 130 generates a query posture from the two-dimensional joint posture of the target person estimated by the two-dimensional joint posture estimation unit 110 (hereinafter referred to as an estimated two-dimensional joint posture). The two-dimensional joint posture estimation unit 110 outputs the estimated two-dimensional joint posture of the target person as well as the confidence of the estimated value of each joint in the estimated two-dimensional joint posture. The query posture generation unit 130 removes joints whose confidence is less than a threshold from the estimated two-dimensional joint posture, and generates a two-dimensional joint posture consisting of only joints whose confidence is equal to or greater than the threshold as a query posture. The query posture generated in this manner has high information accuracy as query information.

[0022] The query posture generated by the query posture generation unit 130 is sent to the sample two-dimensional joint posture acquisition unit 140. The sample two-dimensional joint posture acquisition unit 140 has a 2D / 3D database 150. In the 2D / 3D database 150, three-dimensional posture data representing a basic joint posture of a sample articulated object and two-dimensional posture data representing a sample two-dimensional joint posture are registered. Each three-dimensional posture data is associated with corresponding two-dimensional posture data. For example, 100 sets of three-dimensional posture data are registered in the 2D / 3D database 150. Also, for example, 70 sets of two-dimensional posture data for one set of three-dimensional posture data, that is, a total of 7000 sets of two-dimensional posture data are registered in the 2D / 3D database 150. The sample two-dimensional joint posture acquisition unit 140 searches the 2D / 3D database 150 using the query posture as query information, and acquires a sample two-dimensional joint posture closest to the query posture from the 2D / 3D database 150.

[0023] FIG. 3 is a diagram for explaining the details of the sample two-dimensional joint posture acquisition unit 140. The sample two-dimensional joint posture acquisition unit 140 includes a normalization unit 141 and a search unit 142 in addition to the 2D / 3D database 150. The normalization unit 141 normalizes the query posture 13 to eliminate the influence on the search of the size of the target person in the image and the body shape of the target person. The search unit 142 searches the 2D / 3D database 150 by the normalized query posture 13. A nearest neighbor search is used as a search method. As a result of the nearest neighbor search, a sample two-dimensional joint posture closest to the query posture 13 is acquired. At the same time, a base joint posture related to the query posture 13 can also be acquired. In the example shown in FIG. 3, if the data of the sample two-dimensional joint posture closest to the query posture 13 is the two-dimensional posture data 2-2, the corresponding three-dimensional posture data 2 is acquired as the data of the base joint posture.

[0024] The sample two-dimensional joint posture acquired by the sample two-dimensional joint posture acquisition unit 140 is used to correct the estimated two-dimensional joint posture obtained by the two-dimensional joint posture estimation unit 110. Hereinafter, the sample two-dimensional joint posture used to correct the estimated two-dimensional joint posture is referred to as a corrective two-dimensional joint posture 14. The correction of the estimated two-dimensional joint posture is performed by a two-dimensional joint posture correction unit 160 shown in Fig. 1. The two-dimensional joint posture correction unit 160 corrects the estimated two-dimensional joint posture by replacing only joints whose estimated positions have a certainty below a threshold with the corresponding joints in the corrective two-dimensional joint posture while maintaining the original positions of joints whose estimated positions have a high certainty.

[0025] The position of each joint in the corrected two-dimensional joint posture is a natural joint position corresponding to a basic joint posture that the target object can realistically take. By correcting only joints with low confidence in their estimated positions to such natural positions while keeping the original positions of joints with high confidence in the estimated two-dimensional joint posture, the overall probability of the estimated two-dimensional joint posture is improved. By performing such correction, even if a part of the target person is hidden by occlusion or truncation in the original image 10, the correct two-dimensional joint posture of the target person can be estimated with high accuracy. The joint posture estimation system 100 outputs the estimated two-dimensional joint posture corrected by the two-dimensional joint posture correction unit 160 as the two-dimensional joint posture 15 of the target person.

[0026] Next, a specific flow of the joint posture estimation method performed by the joint posture estimation system 100 will be described with reference to Fig. 4. In the following example, it is assumed that the TransPose architecture is used as the two-dimensional joint posture model 120.

[0027] In step A, an image 10 of a target person is input to a 2D joint posture model 120. In the image 10, the target person's left foot is hidden by an obstacle and is not visible. In other words, occlusion occurs in the image 10. A keypoint heat map 11 corresponding to the image 10 is output from the 2D joint posture model 120.

[0028] In step B, an estimated 2D joint pose 12 is created from the keypoint heat map 11. Data indicating the confidence of the estimated position is attached to each joint in the estimated 2D joint pose 12. In the example shown in Fig. 4, black circles constituting the estimated 2D joint pose 12 indicate joints whose confidence of the estimated position is equal to or higher than a threshold, and white circles indicate joints whose confidence of the estimated position is less than the confidence threshold.

[0029] In step C, a query pose 13 is generated by removing from the estimated 2D joint pose 12 joints whose estimated position confidence is below a threshold.

[0030] In step D, the 2D / 3D database 150 is searched by nearest neighbor search with the normalized query pose 13 as query information.

[0031] In step E, the sample 2D joint postures obtained by searching the 2D / 3D database 150 are obtained as the corrective 2D joint postures 14. At this time, together with the corrective 2D joint postures 14, a 3D basic joint posture that is most related to the query posture 13 may be obtained.

[0032] In step F, among the joints constituting the estimated 2D joint posture 12, those with a confidence level below a threshold, i.e., the joints removed from the query posture 13, are replaced with the corresponding joints in the corrected 2D joint posture 14. As a result, a 2D joint posture 15 is obtained that is estimated with high accuracy not only for the joints of the target person that are shown in the image 10, but also for the joint position of the left foot that is hidden by an obstacle.

[0033] Finally, an example of the hardware configuration of the joint posture estimation system 100 according to this embodiment will be described with reference to FIG.

[0034] The joint pose estimation system 100 includes a computer 200, a display device 220, and an input device 240. The computer 200 includes a processor 202, a program memory 204, and a data storage 208. The processor 202 is coupled to the program memory 204 and the data storage 208.

[0035] The program memory 204 stores a number of executable instructions 206. The data storage 208 may be, for example, a flash memory, an SSD or a HDD, and stores the images 10 and data required for the execution of the instructions 206. Part of the data storage 208 also constitutes the 2D / 3D database 150.

[0036] The instructions 206 constitute a joint posture estimation program. When a part or all of the instructions 206 are executed by the processor 202, the computer 200 is implemented with the functions of the 2D joint posture estimation unit 110, the query posture generation unit 130, the sample 2D joint posture acquisition unit 140, and the 2D joint posture correction unit 160.

[0037] The display device 220 displays the results of calculations by the computer 200. The input device 240 is, for example, a keyboard or a mouse, and accepts operations for the computer 200. Note that the joint posture estimation system 100 may be configured by multiple computers connected via a network, or may be configured by a server on the Internet. [Explanation of symbols]

[0038] 10 image, 11 keypoint heat map, 12 estimated 2D joint pose, 13 query pose, 14 2D joint pose for correction, 15 2D joint pose, 100 joint pose estimation system, 110 2D joint pose estimation unit, 120 2D joint pose model, 130 query pose generation unit, 140 sample 2D joint pose acquisition, 150 2D / 3D database, 160 2D joint pose correction unit, 200 computer, 202 processor, 204 program memory, 206 instructions, 208 data storage, 220 display device, 240 input device

Claims

1. using a model configured to receive an image of an articulated object that can be represented by a plurality of joints and output a two-dimensional joint posture of the articulated object together with a certainty of an estimated position for each joint, estimating a two-dimensional joint posture of the target object from an image of a target object belonging to the articulated object; generating a query pose by removing joints whose confidence is less than a threshold from the two-dimensional joint pose of the target object; obtaining a sample 2D joint pose that is closest to the query pose from a database in which a plurality of sample 2D joint poses are registered for each basic joint pose of a sample articulated object; and correcting the 2D joint poses of the target object by replacing the joints whose confidence is below a threshold with the corresponding joints of the closest sample 2D joint poses. A joint posture estimation method comprising:

2. The joint posture estimation method according to claim 1 , and obtaining from the database a base joint pose most related to the query pose. A joint posture estimation method comprising:

3. The joint posture estimation method according to claim 2, The basic joint posture is a three-dimensional joint posture. A joint posture estimation method comprising:

4. At least one processor; a program memory coupled to the at least one processor and storing a plurality of executable instructions; The plurality of executable instructions may be configured to cause the at least one processor to: using a model configured to receive an image of an articulated object that can be represented by a plurality of joints and output a two-dimensional joint posture of the articulated object together with a certainty of an estimated position for each joint, estimating a two-dimensional joint posture of the target object from an image of a target object belonging to the articulated object; generating a query pose by removing joints whose confidence is less than a threshold from the two-dimensional joint pose of the target object; obtaining a sample 2D joint pose that is closest to the query pose from a database in which a plurality of sample 2D joint poses are registered for each basic joint pose of a sample articulated object; and correcting the 2-D joint poses of the target object by replacing the joints whose confidence is below a threshold with the corresponding joints of the closest sample 2-D joint poses. A joint posture estimation system comprising:

5. A joint posture estimation program including a plurality of instructions executable by at least one processor, The instructions are provided to the at least one processor: using a model configured to receive an image of an articulated object that can be represented by a plurality of joints and output a two-dimensional joint posture of the articulated object together with a certainty of an estimated position for each joint, estimating a two-dimensional joint posture of the target object from an image of a target object belonging to the articulated object; generating a query pose by removing joints whose confidence is less than a threshold from the two-dimensional joint pose of the target object; obtaining a sample 2D joint pose that is closest to the query pose from a database in which a plurality of sample 2D joint poses are registered for each basic joint pose of a sample articulated object; and correcting the 2-D joint poses of the target object by replacing the joints whose confidence is below a threshold with the corresponding joints of the closest sample 2-D joint poses. A joint posture estimation program comprising:

Images