Joint attitude estimation method, joint attitude estimation system, and joint attitude estimation program

Abstract

Provided is a technique with which it is possible to estimate the two-dimensional joint attitude of an object with high accuracy. An estimated two-dimensional joint attitude (12) is obtained by inputting an image (10) of a target object into a two-dimensional joint attitude model. The two-dimensional joint attitude model is configured so as to receive an input of an image of a joint object, and to output a two-dimensional joint attitude of the joint object together with the certainty of the estimated position for each joint. Next, from the estimated two-dimensional joint attitude (12), a joint for which the degree of certainty of the estimated position is less than a threshold value is removed, and a query attitude (13) is generated. Next, from a database (150) in which a plurality of sample two-dimensional joint postures are registered for each basic joint posture of a sample joint object, a correction two-dimensional joint posture (14) closest to the query posture (13) is acquired. Then, the two-dimensional joint attitude (15) of the target object is obtained by replacing the joint for which the degree of certainty of the estimated position is less than the threshold value with the joint corresponding to the correction two-dimensional joint attitude (14).

Description

Technical Field

[0001] The present disclosure relates to a technology suitable for estimating the joint posture of an object including a person. Background Art

[0002] Patent document 1 discloses a technique for estimating the posture of an object. According to this prior art, first, joint position candidates of the object are calculated based on a range image of an object including multiple joints. Next, based on information on the allowable range of the configuration relationship between joints in a multi-joint object model corresponding to the object, the matching of the configuration of the joint position candidates is evaluated. Then, based on the evaluation, the joint positions of the object are determined from the joint position candidates, thereby estimating the posture of the object.

[0003] However, the above-mentioned prior art does not take occlusion and truncation into consideration. Therefore, in difficult situations including occlusion and truncation, the estimation accuracy of the joint positions may be reduced.

[0004] In addition, as documents showing the technical level of the technical field related to the present disclosure, in addition to Patent Document 1, Patent Document 2 and Patent Document 3 can be exemplified. Prior art literature Patent Literature

[0005] Patent Document 1: Japanese Patent No. 6433149 Patent Document 2: Japanese Patent No. 4709723 Patent Document 3: Japanese Patent No. 5555207 Summary of the invention Technical problem to be solved by the invention

[0006] The present disclosure has been made in view of the above-mentioned problems, and an object thereof is to provide a technology capable of estimating a two-dimensional joint posture of an object with high accuracy even in difficult situations including occlusion or truncation. Means for solving technical problems

[0007] The present disclosure provides a joint posture estimation technology for achieving the above-mentioned purpose. The joint posture estimation technology of the present disclosure includes a joint posture estimation method, a joint posture estimation system and a joint posture estimation program.

[0008] The joint posture estimation method disclosed in the present invention includes the following first to fourth steps. The first step is a step of estimating the two-dimensional joint posture of the target object based on the image of the target object belonging to the joint object, and the joint object can be represented by multiple joints. The two-dimensional joint posture is estimated using a two-dimensional joint posture model. The two-dimensional joint posture model is a model constructed as follows: receiving the input of the image of the joint object, and outputting the two-dimensional joint posture of the joint object together with the confidence of the estimated position of each joint. The second step is a step of removing the joints whose estimated position confidence is less than a threshold from the two-dimensional joint posture of the target object to generate a query posture. The third step is a step of obtaining the sample two-dimensional joint posture closest to the query posture from the database. In the database, for each basic joint posture of the sample joint object, multiple sample two-dimensional joint postures are registered. The fourth step is a step of replacing the joint whose estimated position confidence is less than the threshold with the corresponding joint of the closest sample two-dimensional joint posture, thereby correcting the two-dimensional joint posture of the target object.

[0009] The joint posture estimation system disclosed in the present invention comprises: at least one processor, and a program memory coupled to the at least one processor and storing a plurality of instructions. The plurality of instructions are configured to cause the at least one processor to execute the following first to fourth processes. The first process is a process of estimating the two-dimensional joint posture of an object object based on an image of the object object belonging to a joint object using a two-dimensional joint posture model, and the joint object can be represented by a plurality of joints. The second process is a process of removing joints whose estimated position confidence is less than a threshold from the two-dimensional joint posture of the object object to generate a query posture. The third process is a process of obtaining a sample two-dimensional joint posture closest to the query posture from a database. The fourth process is a process of replacing a joint whose estimated position confidence is less than a threshold with a corresponding joint of the closest sample two-dimensional joint posture, thereby correcting the two-dimensional joint posture of the object object.

[0010] The joint posture estimation program disclosed in the present invention includes a plurality of instructions that can be executed by at least one processor. The plurality of instructions are configured to cause at least one processor to execute the following first to fourth processes. The first process is a process of estimating the two-dimensional joint posture of an object object based on an image of the object object belonging to a joint object using a two-dimensional joint posture model, and the joint object can be represented by multiple joints. The second process is a process of removing joints whose estimated position confidence is less than a threshold from the two-dimensional joint posture of the object object to generate a query posture. The third process is a process of obtaining a sample two-dimensional joint posture closest to the query posture from a database. The fourth process is a process of replacing a joint whose estimated position confidence is less than a threshold with a corresponding joint of the closest sample two-dimensional joint posture, thereby correcting the two-dimensional joint posture of the object object. The joint posture estimation program disclosed in the present invention can be stored in a computer-readable non-temporary storage medium, or can be provided via a network. Effects of the Invention

[0011] According to the joint posture estimation technology disclosed in the present invention, among the joints constituting the two-dimensional joint posture estimated using the two-dimensional joint posture model, the joints with high confidence in the estimated position maintain their original positions, and only the joints with low confidence in the estimated position are replaced with the corresponding joints of the closest sample two-dimensional joint posture. Thus, even in difficult conditions such as occlusion or truncation, the two-dimensional joint posture of the target object can be estimated with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a diagram showing the configuration of a system that realizes the joint posture estimation method according to the embodiment of the present disclosure. Figure 2 This is a diagram showing an example of a two-dimensional joint posture model. Figure 3 This is a diagram for explaining a sample two-dimensional joint posture acquisition unit. Figure 4 It is a diagram showing a specific flow of the joint posture estimation method according to the embodiment of the present disclosure. Figure 5 This is a diagram showing an example of the hardware configuration of the joint posture estimation system according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] Figure 1 2 is a diagram showing a system for implementing the joint posture estimation method involved in the embodiment of the present disclosure, that is, a structure of a joint posture estimation system. The joint posture estimation system 100 involved in the present embodiment is a system for estimating the two-dimensional joint posture 15 of an object based on an image 10 of the object. The object object is a multi-joint object whose joint posture is to be estimated. Multi-joint objects include not only people, but also all objects that can be represented by multiple joints, such as vehicles, table drawers, scissors, etc. The joint posture estimation system 100 involved in the present embodiment includes a two-dimensional joint posture estimation unit 110, a query posture generation unit 130, a sample two-dimensional joint posture acquisition unit 140 and a two-dimensional joint posture correction unit 160.

[0014] First, the two-dimensional joint posture estimation unit 110 is explained. The two-dimensional joint posture estimation uses a two-dimensional joint posture model 120. The two-dimensional joint posture model 120 is a model (neural network) used in the estimation of the two-dimensional joint posture of a multi-joint object. The two-dimensional joint posture model 120 is prepared for each target object. In the following description, the target object is a person, and the two-dimensional joint posture model 120 is learned for the estimation of the two-dimensional joint posture of the person.

[0015] As the two-dimensional joint pose model 120, a joint position model based on the top-down method is particularly effective. The architectures of TransPose, AlphaPose, VitPose, RMPE, and Location-free Human PoseEstimation described in the following paper are examples of top-down joint position models that can be used as the two-dimensional joint pose model 120. (1)TransPose:Sen Yang,Zhibin Quan,Mu Nie,Wankou Yang. "TransPose:Keypoint Localization via Transformer" arXiv:2012.14214[cs.CV]1Sep 2021 (2) AlphaPose: Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu, “AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Tim” arXiv:2211.03375v1[cs.CV]7Nov2022 (3)VitPose: Yufei Xu, Jing Zhang, Qiming Zhang, Dacheng Tao, "ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation" arXiv:2204.12484v3[cs.CV]13Oct 2022 (4) RMPE: Hao-Shu Fang1, Shuqin Xie, Yu-Wing Tai, Cewu Lu, “RMPE: Regional Multi-Person Pose Estimation” arXiv:1612.00137 [cs.CV] 4Feb 2018 (5) Location-free Human Pose Estimation: Xixia Xu, Yingguo Gao, Ke Yan, Xue Lin, Qi Zou, “Location-free Human Pose Estimation” arXiv:2205.12619 [cs.CV] 25 May 2022

[0016] Figure 2 An example of a two-dimensional joint posture model 120 is shown. Figure 2 In the example shown, the TransPose architecture is used as a two-dimensional joint pose model 120. The TransPose architecture has a CNN backbone 121, a Transformer encoder layer 122 composed of multiple layers, and a head 123. By inputting an image 10 of the target person into the CNN backbone 121, a plurality of key point heat maps 11 are output from the head 123. The key point heat map 11 is output for each estimated joint. In each key point heat map 11, the estimated position of the joint of the target person and its confidence are shown by a heat map.

[0017] According to the two-dimensional joint pose model 120 such as the TransPose architecture, the joint positions of a multi-jointed object can be estimated very accurately under normal conditions. Normal conditions refer to conditions without occlusion or truncation. However, in difficult conditions including occlusion or truncation, the estimation accuracy of the joint positions based on the two-dimensional joint pose model 120 will decrease.

[0018] like Figure 2 As illustrated, the two-dimensional joint posture model 120 is configured to output the confidence of the estimated position of each joint of the object person. In the absence of occlusion (including self-occlusion where the joint is hidden by the body) or truncation, the positions of all joints are estimated with high confidence. On the other hand, in the case of occlusion or truncation, the estimated positions of some joints become significantly lower in confidence than others. In other words, even in the case of occlusion or truncation, the positions of many joints are estimated with high confidence.

[0019] Here, a sample of the joint object to which the object belongs is defined as a sample joint object. The joint postures that the sample joint object can take are infinite, but can be classified into a finite number of basic joint postures in three dimensions. The joint posture of the object corresponds to a basic joint posture of the sample joint object. Therefore, even if a part of the image is hidden due to occlusion or truncation, as long as the basic joint posture of the sample joint object corresponding to the joint posture of the object person is known, the correct two-dimensional joint posture of the object can be inferred with high accuracy based on the basic joint posture.

[0020] In the joint posture estimation method involved in the present embodiment, by projecting the basic joint posture of the sample joint object onto a plane from various directions, a plurality of sample two-dimensional joint postures with different viewpoints are prepared for one basic joint posture. Each sample two-dimensional joint posture is registered in a database in association with the corresponding basic joint posture. As query information when searching the database, in the joint posture estimation method involved in the present embodiment, a query posture generated based on the output of the two-dimensional joint posture model 120 is used.

[0021] The query pose is made by Figure 1 The query posture generation unit 130 shown is performed. The query posture generation unit 130 generates a query posture based on the two-dimensional joint posture of the target person estimated by the two-dimensional joint posture estimation unit 110 (hereinafter referred to as the estimated two-dimensional joint posture). The estimated two-dimensional joint posture of the target person and the confidence of the estimated value of each joint in the estimated two-dimensional joint posture are output from the two-dimensional joint posture estimation unit 110. The query posture generation unit 130 removes the joints whose confidence is less than a threshold value from the estimated two-dimensional joint posture, and generates a two-dimensional joint posture consisting only of joints whose confidence is greater than the threshold value as a query posture. The query posture generated in this way has high information accuracy as query information.

[0022] The query posture generated by the query posture generation unit 130 is sent to the sample two-dimensional joint posture acquisition unit 140. The sample two-dimensional joint posture acquisition unit 140 has a 2D / 3D database 150. The 2D / 3D database 150 has registered three-dimensional posture data representing the basic joint posture of the sample joint object and two-dimensional posture data representing the sample two-dimensional joint posture. Each three-dimensional posture data is associated with corresponding two-dimensional posture data. For example, 100 sets of three-dimensional posture data are registered in the 2D / 3D database 150. In addition, for example, in the 2D / 3D database 150, 70 sets of two-dimensional posture data are registered for one set of three-dimensional posture data, that is, a total of 7000 sets of two-dimensional posture data are registered. The sample two-dimensional joint posture acquisition unit 140 uses the query posture as query information to search the 2D / 3D database 150, and obtains the sample two-dimensional joint posture closest to the query posture from the 2D / 3D database 150.

[0023] Figure 3It is a diagram illustrating the details of the sample two-dimensional joint posture acquisition unit 140. In addition to the 2D / 3D database 150, the sample two-dimensional joint posture acquisition unit 140 also includes a standardization unit 141 and a retrieval unit 142. The standardization unit 141 standardizes the query posture 13 to eliminate the influence of the size of the object person in the image, the body shape of the object person, etc. on the retrieval. The retrieval unit 142 searches the 2D / 3D database 150 based on the standardized query posture 13. As a retrieval method, the nearest neighbor retrieval is used. As a result of the nearest neighbor retrieval, the sample two-dimensional joint posture closest to the query posture 13 is obtained. In addition, the basic joint posture related to the query posture 13 can also be obtained at the same time. Figure 3 In the example shown, when the data of the sample two-dimensional joint posture closest to the query posture 13 is the two-dimensional posture data 2-2, the three-dimensional posture data 2 corresponding thereto is acquired as the data of the basic joint posture.

[0024] The sample two-dimensional joint posture obtained by the sample two-dimensional joint posture acquisition unit 140 is used to correct the estimated two-dimensional joint posture obtained by the two-dimensional joint posture estimation unit 110. Hereinafter, the sample two-dimensional joint posture used in the correction of the estimated two-dimensional joint posture is referred to as the correction two-dimensional joint posture 14. Figure 1 The two-dimensional joint posture correction unit 160 shown in the figure maintains the original position of the joints with high estimated position confidence and replaces only the joints with estimated position confidence less than a threshold value with the corresponding joints of the correction two-dimensional joint posture, thereby correcting the estimated two-dimensional joint posture.

[0025] The position of each joint of the corrected two-dimensional joint posture is a natural joint position corresponding to the basic joint posture that the object can actually take. By maintaining the original position of the joints with high confidence in the estimated position in the estimated two-dimensional joint posture, only the joints with low confidence in the estimated position are corrected to such a natural position, thereby improving the overall probability of the estimated two-dimensional joint posture. By performing such a correction, even if a part of the object person in the original image 10 is hidden due to occlusion or truncation, the correct two-dimensional joint posture of the object person can be estimated with high accuracy. The joint posture estimation system 100 outputs the estimated two-dimensional joint posture corrected by the two-dimensional joint posture correction unit 160 as the two-dimensional joint posture 15 of the object person.

[0026] Next, use Figure 4 The specific flow of the joint posture estimation method implemented by the joint posture estimation system 100 will be described. In the following example, the TransPose framework is used as the two-dimensional joint posture model 120 .

[0027] In step A, when the image 10 of the target person is input into the image 10 of the two-dimensional joint posture model 120, the left foot of the target person is hidden by an obstacle and is not reflected. That is, occlusion occurs in the image 10. The key point heat map 11 corresponding to the image 10 is output from the two-dimensional joint posture model 120.

[0028] In step B, an estimated two-dimensional joint posture 12 is generated based on the key point heat map 11. Each joint of the estimated two-dimensional joint posture 12 is provided with data indicating the confidence of its estimated position. Figure 4 In the example shown, solid circles constituting the estimated two-dimensional joint posture 12 represent joints whose estimated position certainty is greater than a threshold value, and hollow circles represent joints whose estimated position certainty is less than the threshold value.

[0029] In step C, the query pose 13 is generated by removing the joints whose estimated positions have a confidence level less than a threshold from the estimated two-dimensional joint poses 12 .

[0030] In step D, the 2D / 3D database 150 is searched using the normalized query pose 13 as a nearest neighbor search for query information.

[0031] In step E, the sample two-dimensional joint posture obtained by searching the 2D / 3D database 150 is obtained as the calibration two-dimensional joint posture 14. At this time, the three-dimensional basic joint posture most related to the query posture 13 may be obtained together with the calibration two-dimensional joint posture 14.

[0032] In step F, the joints constituting the estimated two-dimensional joint pose 12 whose confidence is less than the threshold, that is, the joints removed in the query pose 13, are replaced with the corresponding joints of the correction two-dimensional joint pose 14. Thus, a two-dimensional joint pose 15 is obtained that not only estimates the joints of the target person reflected in the image 10 with high accuracy, but also estimates the joint position of the left foot hidden by the obstacle with high accuracy.

[0033] Finally, refer to Figure 5 An example of the hardware configuration of the joint posture estimation system 100 according to the present embodiment will be described.

[0034] The joint posture estimation system 100 includes a computer 200, a display device 220, and an input device 240. The computer 200 includes a processor 202, a program memory 204, and a data memory 208. The processor 202 is coupled to the program memory 204 and the data memory 208.

[0035] The program memory 204 stores a plurality of executable instructions 206. The data memory 208 is, for example, a flash memory, an SSD, or an HDD, and stores the image 10 and data required for executing the instructions 206. In addition, a part of the data memory 208 constitutes the 2D / 3D database 150.

[0036] The instructions 206 constitute a joint posture estimation program. When the processor 202 executes part or all of the instructions 206 , the computer 200 implements the functions of the two-dimensional joint posture estimation unit 110 , the query posture generation unit 130 , the sample two-dimensional joint posture acquisition unit 140 , and the two-dimensional joint posture correction unit 160 .

[0037] The display device 220 displays the calculation results of the computer 200. The input device 240 is, for example, a keyboard or a mouse, and receives operations on the computer 200. The joint posture estimation system 100 may be composed of a plurality of computers connected via a network, or may be composed of a server on the Internet. [Explanation of Reference Numerals]

[0038] 10: image, 11: key point heat map, 12: estimated two-dimensional joint posture, 13: query posture, 14: two-dimensional joint posture for correction, 15: two-dimensional joint posture, 100: joint posture estimation system, 110: two-dimensional joint posture estimation unit, 120: two-dimensional joint posture model, 130: query posture generation unit, 140: sample two-dimensional joint posture acquisition unit, 150: 2D / 3D database, 160: two-dimensional joint posture correction unit, 200: computer, 202: processor, 204: program memory, 206: instruction, 208: data memory, 220: display device, 240: input device.

Claims

1. A joint posture estimation method, characterized in that: Includes the following: Using a model, inferring a two-dimensional joint posture of a target object based on an image of the target object that is a joint object, the joint object being capable of being represented by a plurality of joints, the model being configured to receive an input of the image of the joint object and output the two-dimensional joint posture of the joint object together with a confidence of an estimated position of each joint; Removing joints whose confidence is less than a threshold from the two-dimensional joint poses of the object to generate a query pose; Obtaining a sample two-dimensional joint pose closest to the query pose from a database in which a plurality of sample two-dimensional joint poses are registered for each basic joint pose of the sample joint object; as well as The two-dimensional joint posture of the target object is corrected by replacing the joint whose confidence is less than a threshold with the corresponding joint of the closest sample two-dimensional joint posture.

2. The joint posture estimation method according to claim 1, characterized in that: The method also includes the following steps: obtaining the basic joint posture most related to the query posture from the database.

3. The joint posture estimation method according to claim 2, characterized in that: The basic joint posture is a three-dimensional joint posture.

4. A joint posture estimation system, characterized in that: have: at least one processor; and a program memory coupled to the at least one processor and storing a plurality of executable instructions, The plurality of executable instructions are configured to cause the at least one processor to execute the following: Using a model, inferring a two-dimensional joint posture of a target object based on an image of the target object that is a joint object, the joint object being capable of being represented by a plurality of joints, the model being configured to receive an input of the image of the joint object and output the two-dimensional joint posture of the joint object together with a confidence of an estimated position of each joint; Removing joints whose confidence is less than a threshold from the two-dimensional joint poses of the object to generate a query pose; Obtaining a sample two-dimensional joint pose closest to the query pose from a database in which a plurality of sample two-dimensional joint poses are registered for each basic joint pose of the sample joint object; as well as The two-dimensional joint posture of the target object is corrected by replacing the joint whose confidence is less than a threshold with the corresponding joint of the closest sample two-dimensional joint posture.

5. A joint posture estimation program, comprising a plurality of instructions executable by at least one processor, characterized in that: The plurality of instructions are configured to cause the at least one processor to execute the following: Using a model, inferring a two-dimensional joint posture of a target object based on an image of the target object that is a joint object, the joint object being capable of being represented by a plurality of joints, the model being configured to receive an input of the image of the joint object and output the two-dimensional joint posture of the joint object together with a confidence of an estimated position of each joint; Removing joints whose confidence is less than a threshold from the two-dimensional joint poses of the object to generate a query pose; Obtaining a sample two-dimensional joint pose closest to the query pose from a database in which a plurality of sample two-dimensional joint poses are registered for each basic joint pose of the sample joint object; as well as The two-dimensional joint posture of the target object is corrected by replacing the joint whose confidence is less than a threshold with the corresponding joint of the closest sample two-dimensional joint posture.

Images