Pose Estimation System

Abstract

To provide a posture estimation system which can improve the accuracy of posture estimation.SOLUTION: A posture estimation system estimates a posture of a person. The system comprises a plurality of RGB-D cameras which image the person from different angles, and a posture estimation device to which image pickup results of the plurality of RGB-D cameras are inputted. The posture estimation device estimates a position of a predetermined portion of the person on each RGB image, by using each RGB image of the plurality of RGB-D cameras. The posture estimation device determines the reliability of depth information at the position, on the basis of a change over time of the depth information at the estimated position. The posture estimation device calculates the position of the predetermined portion of an object in consideration of determination results.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a pose estimation system. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a posture estimation device that estimates a posture of a person is known (see, for example, Patent Document 1).

[0003] The posture estimation device of Patent Document 1 is configured to estimate the position of each part of a person by using a three-dimensional sensor capable of measuring the position of the person in real space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-68424 A Summary of the Invention [Problem to be solved by the invention]

[0005] Here, in the case of estimating the position of a body part of a person using a three-dimensional sensor as described above, if the body part of the person is hidden by an obstacle or the like, it becomes difficult to estimate the position of the body part. For example, when an image of a person is captured using an RGB-D camera, the position (two-dimensional position) of the body part of the person on the RGB image is estimated, and the position (three-dimensional position) of the body part of the person is calculated using depth information of the estimated position, if the body part of the person is hidden by an obstacle or the like, the depth information of the position of the body part estimated on the RGB image becomes the position of the obstacle, making it difficult to calculate the position of the hidden body part. Therefore, there is room for improvement in improving the accuracy of posture estimation.

[0006] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a posture estimation system that is capable of improving the accuracy of posture estimation. [Means for solving the problem]

[0007] The posture estimation system according to the present invention estimates the posture of an object, and includes a plurality of three-dimensional cameras that capture images of the object from different angles, an estimation unit that estimates the position of a predetermined part of the object on each of the two-dimensional images using each of the two-dimensional images from the plurality of three-dimensional cameras, a determination unit that determines the reliability of the depth information of the position estimated by the estimation unit based on changes over time in the depth information of that position, and a calculation unit that calculates the position of the predetermined part of the object taking into account the determination result of the determination unit.

[0008] With this configuration, the position of a predetermined part of the target object is calculated taking into account the reliability of the depth information from the multiple 3D cameras, thereby improving the accuracy of posture estimation.

[0009] In the above posture estimation system, the estimation unit may be configured to learn features of a predetermined part of the object on the two-dimensional image, and track the predetermined part.

[0010] The posture estimation system according to the present invention estimates the posture of an object, and includes three or more three-dimensional cameras that image the object from different angles, a three-dimensional position calculation unit that calculates the three-dimensional position of a specified part of the object based on each of the three-dimensional images from the three or more three-dimensional cameras, and a reliability evaluation unit that evaluates the reliability of the three or more three-dimensional positions based on the three or more three-dimensional positions calculated by the three-dimensional position calculation unit.

[0011] The above posture estimation system may further include an estimation unit that estimates the position of a predetermined part of the object on each of the two-dimensional images using respective two-dimensional images from three or more three-dimensional cameras, and the estimation unit may be configured to learn features of the predetermined part of the object on the two-dimensional image and track the predetermined part, and may be configured to re-learn the features of the predetermined part on the two-dimensional image taken by the three-dimensional camera from which a three-dimensional position evaluated to have low reliability by the reliability evaluation unit was obtained, and to redo the tracking. Effect of the Invention

[0012] According to the posture estimation system of the present invention, it is possible to improve the accuracy of posture estimation. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a schematic configuration of a posture estimation system according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart illustrating an operation of the posture estimation system according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, one embodiment of the present invention will be described.

[0015] First, the configuration of a posture estimation system 100 according to an embodiment of the present invention will be described with reference to FIG.

[0016] The posture estimation system 100 is configured to, for example, calculate the position of each part of a person's skeleton and estimate the posture of the person. Examples of parts of a person's skeleton include, but are not limited to, joints such as shoulders, elbows, and wrists. Note that a person is an example of an "object" in the present invention. As shown in FIG. 1, the posture estimation system 100 includes a posture estimation device 1 and RGB-D cameras 2 and 3.

[0017] The RGB-D cameras 2 and 3 are configured to capture an image of a person located in a predetermined measurement area to obtain an RGB-D image. The RGB-D image includes an RGB image (color image) and a depth image, and has depth information for each pixel in the RGB image. The RGB-D cameras 2 and 3 are examples of the "three-dimensional camera" of the present invention, and the RGB image is an example of the "two-dimensional image" of the present invention.

[0018] The RGB-D cameras 2 and 3 are provided to capture images of a person from different angles. Therefore, even if a specific part of a person is hidden by an obstacle or the like in the RGB image of one of the RGB-D cameras 2 and 3, the specific part is likely to appear in the RGB image of the other RGB-D camera 2 or 3. In other words, the two RGB-D cameras 2 and 3 are provided to prevent each part of a person located in a specific measurement area from becoming a blind spot.

[0019] Pose estimation device 1 is configured to receive RGB-D images from RGB-D cameras 2 and 3, and estimate a person's posture using the RGB-D images. In this posture estimation device 1, information (external parameters) relating to the positions, postures, and the like of RGB-D cameras 2 and 3 is stored in advance, and by merging data from the RGB-D image of RGB-D camera 2 and data from the RGB-D image of RGB-D camera 3 so as to adopt the more reliable one, it is possible to improve the accuracy of posture estimation.

[0020] Specifically, posture estimation device 1 is configured to estimate the position of each part of a person on an RGB image captured by RGB-D camera 2, using the RGB image. Moreover, posture estimation device 1 is configured to estimate the position of each part of a person on an RGB image captured by RGB-D camera 3, using the RGB image. That is, the position of each part on the two-dimensional RGB image captured by RGB-D camera 2 is estimated, and the position of each part on the two-dimensional RGB image captured by RGB-D camera 3 is estimated.

[0021] Moreover, the posture estimation device 1 is configured to learn the features of each part of a person on the RGB images of the RGB-D cameras 2 and 3, and to track each part. That is, each part of a person is extracted by image processing, and each extracted part (image feature) is tracked. Then, the posture estimation device 1 is configured to estimate the position (two-dimensional position) of a hidden part using a known algorithm, for example, when a part of a person is hidden by an obstacle or the like. However, the depth information of a part hidden by an obstacle does not indicate the depth of the part, but indicates the depth of the obstacle. Therefore, if the depth information of a part hidden by an obstacle is used, there is a risk that the estimation accuracy of the position (three-dimensional position) of the part will decrease. Therefore, in this embodiment, the reliability of the depth information is determined, and the position of the part is calculated taking the reliability into consideration.

[0022] Pose estimation device 1 is configured to determine the reliability of depth information of each part of a person from RGB-D camera 2 based on a change over time in depth information about the position of each part of a person estimated on the RGB image from RGB-D camera 2. Pose estimation device 1 is also configured to determine the reliability of depth information of each part of a person from RGB-D camera 3 based on a change over time in depth information about the position of each part of a person estimated on the RGB image from RGB-D camera 3. That is, the reliability of the depth information of each part of a person on the RGB image input from RGB-D camera 2 is determined, and the reliability of the depth information of each part of a person on the RGB image input from RGB-D camera 3 is determined. Note that the change over time in depth information is, for example, a change in depth information during a period from a predetermined time ago to the present time.

[0023] For example, when a specific part of a person is not occluded, the depth information of the specific part changes continuously (linearly) due to the movement of the person (change in posture). On the other hand, when a specific part of a person is occluded, the depth information changes from that due to the specific part to that due to an obstacle, resulting in a sudden change in the depth information (the distance from the camera suddenly becomes shorter). Thus, posture estimation device 1 is configured to determine that the smaller the change over time in the depth information of a part of a person estimated on an RGB image, the higher the reliability of the depth information of that part.

[0024] Pose estimation device 1 is configured to calculate the position (three-dimensional position) of each part of the person using highly reliable depth information from the RGB-D images of RGB-D cameras 2 and 3. That is, for each part of the person, the position (three-dimensional position) of the part is calculated using the more reliable input from RGB-D cameras 2 and 3. For example, if the reliability of RGB-D camera 2 is high for the left shoulder of the person, the position of the left shoulder is calculated using the RGB-D image from RGB-D camera 2, and if the reliability of RGB-D camera 3 is high for the right shoulder of the person, the position of the right shoulder is calculated using the RGB-D image from RGB-D camera 3.

[0025] Moreover, posture estimation device 1 includes a calculation unit 11, a storage unit 12, and an input unit 13. The calculation unit 11 is configured to control posture estimation device 1 by executing calculation processing based on a program stored in storage unit 12, etc. The storage unit 12 stores a program for estimating a posture of a person, and the positions and postures of RGB-D cameras 2 and 3, etc. The RGB-D cameras 2 and 3 are connected to the input unit 13, and the imaging results (RGB-D images) of the RGB-D cameras 2 and 3 are input. Note that the calculation unit 11 executes the program stored in storage unit 12 to realize the "estimation unit," "determination unit," and "calculation unit" of the present invention.

[0026] - Operation of the attitude estimation system - Next, the operation (posture estimation method) of posture estimation system 100 according to this embodiment will be described with reference to Fig. 2. Before the start of this posture estimation operation, each part of the person located in the measurement area (all parts of the measurement target) is made to appear to RGB-D cameras 2 and 3, and the initial position of each part (the position at the start of the operation) is accurately calculated. This allows the characteristics of each part of the person on the RGB images of RGB-D cameras 2 and 3 to be learned, making it possible to track each part. The following flow is repeated from the start of the posture estimation operation until it is ended.

[0027] 2, a person located in a measurement area is captured by the RGB-D cameras 2 and 3. Then, the RGB-D images acquired by the RGB-D cameras 2 and 3 are output from the RGB-D cameras 2 and 3 to the posture estimation device 1.

[0028] Next, in step S2, posture estimation device 1 estimates the position of each part of the person on the RGB image captured by RGB-D camera 2, and also estimates the position of each part of the person on the RGB image captured by RGB-D camera 3. For example, the position of each part of the person is estimated by tracking each learned part of the person on the RGB image.

[0029] Next, in step S3, posture estimation device 1 determines the reliability of the depth information of a part of a person estimated on the RGB image taken by RGB-D camera 2 based on a time-dependent change in the depth information for that part. This reliability determination is performed for each part of a person estimated on the RGB image taken by RGB-D camera 2. Furthermore, posture estimation device 1 determines the reliability of the depth information of that part from RGB-D camera 3 based on a time-dependent change in the depth information for that part. This reliability determination is performed for each part of a person estimated on the RGB image taken by RGB-D camera 3.

[0030] Next, in step S4, the posture estimation device 1 calculates the positions (three-dimensional positions) of the body parts of the person using the highly reliable depth information. Specifically, the positions of the body parts of the person are calculated based on the RGB-D image captured by the camera from which the highly reliable depth information has been obtained. This position calculation is performed for each body part of the person, thereby estimating the posture of the person.

[0031] -effect- In this embodiment, as described above, the accuracy of posture estimation can be improved by providing the RGB-D cameras 2 and 3 and calculating the positions of each part of the person in consideration of the reliability of the depth information from the RGB-D cameras 2 and 3. In other words, the accuracy of posture estimation can be improved by not using low-reliability depth information due to occlusion of parts of the person.

[0032] Furthermore, in this embodiment, when a specific part of a person falls outside the angle of view of one of the RGB-D cameras 2 and 3, pose estimation can be performed appropriately if the specific part is within the angle of view of the other of the RGB-D cameras 2 and 3.

[0033] Furthermore, in this embodiment, by learning the features of each part of a person on an RGB image, it is possible to improve the accuracy of estimating the position of each part on an RGB image.

[0034] -Other embodiments- It should be noted that the embodiments disclosed herein are illustrative in all respects and are not intended to be limiting. Therefore, the technical scope of the present invention is not interpreted solely by the above-described embodiments, but is defined by the claims. The technical scope of the present invention includes all modifications within the scope and meaning equivalent to the claims.

[0035] For example, in the above embodiment, an example has been shown in which the posture of a person is estimated, but the present invention is not limited to this, and the posture of an object other than a person may be estimated.

[0036] In the above embodiment, an example in which two RGB-D cameras 2 and 3 are provided is shown, but the present invention is not limited to this, and three or more RGB-D cameras may be provided. In this case, the posture estimation device may be configured to calculate the three-dimensional position of a predetermined part of a person based on the RGB-D images (three-dimensional images) of the three or more RGB-D cameras, and to evaluate the reliability of the three or more calculated three-dimensional positions. For example, in a case in which three RGB-D cameras are provided, if the three-dimensional positions of the predetermined part based on the RGB-D images of two RGB-D cameras are the same and the three-dimensional positions of the predetermined part based on the RGB-D image of the remaining RGB-D camera are different, the reliability of the three-dimensional position based on the two RGB-D cameras is evaluated as high, and the reliability of the three-dimensional position based on the remaining RGB-D camera is evaluated as low. That is, based on the three-dimensional positions based on the RGB-D images of the three RGB-D cameras, it is evaluated whether each RGB-D camera appropriately captures the predetermined part. That is, since the two RGB-D cameras properly capture the predetermined part, the three-dimensional positions obtained by the two RGB-D cameras are the same, whereas the remaining RGB-D camera does not properly capture the predetermined part, so the three-dimensional position obtained by the remaining RGB-D camera is different. Therefore, the three-dimensional positions obtained by the two RGB-D cameras with high reliability are adopted as the positions of the predetermined part of the person. Even with this configuration, the accuracy of posture estimation can be improved. Furthermore, the posture estimation device may also determine the reliability based on the above-mentioned change over time in the depth information. That is, the posture estimation device may be configured to estimate the positions of the predetermined parts of the person on the RGB images using the RGB images obtained by each RGB-D camera, and to determine the reliability of the depth information of the predetermined part obtained by the RGB-D camera based on the change over time in the depth information for the estimated positions of the predetermined parts.The posture estimation device may be configured to learn the characteristics of a specific part of a person on the RGB images of each RGB-D camera, track the specific part, and re-learn the characteristics of the specific part on the two-dimensional image by the RGB-D camera from which the three-dimensional position evaluated to be unreliable was obtained (the remaining one RGB-D camera in the above example), and redo the tracking. In other words, for the RGB-D camera that cannot capture the specific part properly, the specific part is re-learned and tracking is redo. Note that the "estimation unit", "determination unit", "calculation unit", "three-dimensional position calculation unit" and "reliability evaluation unit" of the present invention are realized by the calculation unit executing the program stored in the storage unit in the posture estimation device.

[0037] In addition, in the above embodiment, an example was shown in which the smaller the change over time of the depth information, the higher the reliability is determined to be. However, without being limited to this, the reliability may be determined to be high when the change over time of the depth information is within a specified range, and the reliability may be determined to be low when the change over time of the depth information is outside the specified range.

[0038] In the above embodiment, the reliability of the depth information may be determined by taking into consideration other factors in addition to the change over time. For example, the greater the distance between a specific part and the part closest to the specific part, the higher the reliability may be determined. Also, the higher the image quality of the periphery of the specific part in the RGB image (there is no blur and the contrast is clear), the higher the reliability may be determined. Also, the closer the distance from the camera to the specific part, the higher the reliability may be determined.

[0039] In the above embodiment, when the reliability of the depth information of a predetermined portion from both the RGB-D cameras 2 and 3 is low, it may be possible to output that the predetermined portion is unmeasurable.

[0040] In addition, in the above embodiment, the RGB-D cameras 2 and 3 may have an RGB image acquisition unit that acquires an RGB image and a depth image acquisition unit that acquires a depth image integrally provided in a single housing, or may each be provided in a separate housing. [Industrial Applicability]

[0041] The present invention can be used in a posture estimation system for estimating the posture of an object. [Explanation of symbols]

[0042] 1 Posture estimation device 2 RGB-D camera (3D camera) 3 RGB-D camera (3D camera) 100 Pose Estimation System

Claims

1. A posture estimation system for estimating a posture of an object, comprising: A plurality of 3D cameras for capturing images of the object from different angles; an estimation unit that estimates a position of a predetermined portion of the object on each of the two-dimensional images by using the two-dimensional images of the plurality of three-dimensional cameras; a determination unit that performs a binary determination as to whether the reliability of the depth information of the position estimated by the estimation unit is high or low based on a time-dependent change in the depth information of the position; a calculation unit that calculates a position of a predetermined part of the object using only depth information that is determined to be highly reliable by a result of the determination unit.

2. 2. The posture estimation system according to claim 1, The posture estimation system according to claim 1, wherein the estimation unit is configured to learn features of a predetermined part of the object on a two-dimensional image and to track the predetermined part.

3. 2. The posture estimation system according to claim 1, Three or more of the three-dimensional cameras are provided, a three-dimensional position calculation unit that calculates a three-dimensional position of a predetermined portion of the object based on each of the three-dimensional images captured by the three or more three-dimensional cameras; a reliability evaluation unit that evaluates reliability of the three or more three-dimensional positions based on the three or more three-dimensional positions calculated by the three-dimensional position calculation unit.

4. 2. The posture estimation system according to claim 1, a posture estimation system, characterized in that the estimation unit is configured to learn features of a specified part of the object on a two-dimensional image and track the specified part, and is configured to re-learn features of the specified part on a two-dimensional image taken by a three-dimensional camera from which a three-dimensional position determined to be unreliable by the determination unit was obtained, and to redo tracking.

5. 5. The posture estimation system according to claim 1, The posture estimation system is characterized in that the determination unit determines that the reliability is high when the change over time of the depth information is within a predetermined range, and determines that the reliability is low when the change over time of the depth information is outside the predetermined range.

Images