Information processor, information processing method and program

Abstract

To make it possible to detect multiple positions on an object more accurately.SOLUTION: An information processor has estimation means for estimating the positions of multiple feature points of an object in three-dimensional space, determination means for determining the likelihood that the shape of the object is a specific shape based on the positions of the multiple feature points estimated by the estimation means, and correction means that corrects the positions of the multiple feature points estimated by the estimation means so that the shape corresponding to the positions of the multiple feature points estimated by the estimation means approaches the specific shape, in a first case where it is determined that the likelihood that the shape of the object is the specific shape is at a first level.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 describes a hand gesture recognition method using deep learning, which enables hand gesture recognition based on hand information (including the fingertips and joint points of the user's hand) using an RGB sensor image connected to a wearable device, thereby enabling control of the wearable device.

[0003] Patent Document 2 describes a technique for preserving a skeletal model of a hand and comparing the detection results with the skeletal model in order to interpolate the detection performance of deep learning. As a result, Patent Document 2 proposes a method for correcting the detection results. Patent Document 3 describes a technique for predicting the movement of joint points using detection results of multiple temporally consecutive images and correcting the detection results according to the predicted results. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-71048 A [Patent Document 2] JP 2017-107503 A [Patent Document 3] JP 2010-140283 A Summary of the Invention [Problem to be solved by the invention]

[0005] Since an augmented reality space generally constitutes a three-dimensional space, it is desirable to be able to detect the three-dimensional positions of the fingertips and joint points of the user's hand in order to represent the user's hand in the augmented reality space. Although there are methods for detecting the position of an object in three-dimensional space using deep learning based on a still image captured by a wearable device in real space, the accuracy of position detection may be low in some cases.

[0006] The present invention aims to provide a technique that enables more accurate detection of multiple positions on an object. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a composition comprising the steps of: An estimation means for estimating positions of a plurality of feature points of an object in a three-dimensional space; a determination means for determining a possibility that the shape of the object is a specific shape based on the positions of the plurality of feature points estimated by the estimation means; a correction means for correcting positions of the plurality of feature points estimated by the estimation means in a first case in which it is determined that the possibility that the shape of the object is the specific shape is at a first level, such that the shape corresponding to the positions of the plurality of feature points estimated by the estimation means approaches the specific shape; The information processing device is characterized by having:

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: an estimation step of estimating positions of a plurality of feature points of an object in a three-dimensional space; a determination step of determining a possibility that the shape of the object is a specific shape based on the positions of the plurality of feature points estimated in the estimation step; a correction step of correcting positions of the plurality of feature points estimated in the estimation step so that a shape corresponding to the positions of the plurality of feature points estimated in the estimation step approaches the specific shape in a first case in which it is determined that the possibility that the shape of the object is the specific shape is at a first level; The information processing method is characterized by having the following features. Effect of the Invention

[0009] According to the present invention, a plurality of positions on an object can be detected more accurately. [Brief description of the drawings]

[0010] [Figure 1] 1 is a functional block diagram of an HMD device according to a first embodiment. [Diagram 2] 5A to 5C are diagrams illustrating correction of three-dimensional positions of feature points according to the first embodiment. [Diagram 3] 4 is a flowchart of a process for detecting three-dimensional positions of feature points according to the first embodiment. [Figure 4] 11 is a functional block diagram of an HMD device according to a second embodiment. [Diagram 5] 13A to 13C are diagrams illustrating correction of three-dimensional positions of feature points according to the second embodiment. [Figure 6] 11 is a flowchart of a process for detecting three-dimensional positions of feature points according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0012] <Embodiment 1> In the first embodiment, a head mounted display device (hereinafter, referred to as "HMD device") equipped with a camera will be described as an example of an information processing device (image recognition device). Fig. 1 is a functional block diagram of the HMD device 101. The HMD device 101 has an imaging unit 102, a system control unit 103, a non-volatile memory 104, a system memory 105, and a detection unit 110.

[0013] The imaging unit 102 has an optical system and an image sensor (e.g., a CCD or CMOS sensor). The optical system is equipped with multiple lenses. The multiple lenses are oriented in a direction that allows them to capture an image of the hand (fingers) of the user wearing the HMD device 101. The imaging unit 102 generates a detection image by performing image processing (such as noise removal or resizing) on ​​the image of the user's hand captured by the image sensor.

[0014] The system control unit 103 is a control unit having at least one processor or circuit. The system control unit 103 controls the entire HMD device 101. The system control unit 103 executes a program recorded in the non-volatile memory 104 to realize each process of this embodiment described later. In addition, the detected image obtained from the imaging unit 102 and the output from the detection unit 110 are recorded in the system memory 105.

[0015] The non-volatile memory 104 is a memory that can be electrically erased and recorded. For example, a Flash-ROM is used as the non-volatile memory 104. Constants and programs for the operation of the system control unit 103 are stored in the non-volatile memory 104. The programs here include programs for the system control unit 103 to execute various flowcharts described later.

[0016] For example, a RAM (Random Access Memory) is used as the system memory 105. In the system memory 105, at least variables for the operation of the system control unit 103, constants and programs read from the non-volatile memory 104, etc. are expanded.

[0017] The detection unit 110 obtains (outputs) information on the three-dimensional position of each feature point of the hand captured in the detected image. Here, the "feature points" refer to the joint points of each finger, the joint points of the wrist, the tips of each finger, and the like. After acquiring the three-dimensional positions of each feature point of the user's hand, the detection unit 110 corrects the three-dimensional positions. The detection unit 110 includes a position estimation unit 111, a shape determination unit 112, a correction determination unit 113, and a shape correction unit 114. The internal configuration of the detection unit 110 will be described in detail with reference to FIG. 2.

[0018] Fig. 2 is a diagram showing the three-dimensional position of each feature point before correction and the three-dimensional position of each feature point after correction. In Fig. 2, a detected image 200 is an image stored in the system memory 105 by the system control unit 103. Here, the detected image 200 includes (photographs) a hand 201 of a user wearing the HMD device 101. In the following, it is assumed that the tip of the thumb and the tip of the index finger are in contact with each other in the hand 201, and a gesture of pinching an object is being made. In other words, it is assumed that the shape of the hand 201 at the time of photographing the hand 201 is a shape for pinching an object.

[0019] The position estimation unit 111 includes a deep learning model. The deep learning model is a model for estimating the three-dimensional position of each feature point of the user's hand included in the detected image. When the detected image includes the user's hand, the position estimation unit 111 estimates the two-dimensional position (u, v) of each feature point of the hand in the detected image (image space of the detected image; two-dimensional coordinate system of the detected image). In this case, the position estimation unit 111 sets the depth coordinate of the wrist to a reference value = 0 in the coordinates (depth coordinates) indicating the position in the direction perpendicular to the image sensor of the imaging unit 102. Then, the position estimation unit 111 estimates the depth coordinate z of each feature point of the hand. As a result, the position estimation unit 111 estimates the three-dimensional position (u, v, z) of each feature point of the hand.

[0020] For example, image 202 is an image in which points indicating the positions of each feature point of the hand estimated by position estimation unit 111 are superimposed on user's hand 201 in detected image 200. Plot image 205 is an image in which points indicating the estimated three-dimensional positions of each feature point (joint points, etc.) of the wrist, thumb, and index finger are plotted on a uv plane (a two-dimensional coordinate system of detected image 200). Plot image 206 is an image in which points indicating the estimated three-dimensional positions of each feature point of the wrist, thumb, and index finger are plotted on a zv plane (a two-dimensional coordinate system perpendicular to the two-dimensional coordinate system of detected image 200).

[0021] In plot image 205, the distance (Euclidean distance) between tip 203 of index finger and tip 204 of thumb is very short. On the other hand, in plot image 206, tip 203 of index finger and tip 204 of thumb are separated in the z direction. Since an actual user's hand 201 is making a gesture of pinching an object with the index finger and thumb (hereinafter referred to as a "grasping gesture"), position estimation unit 111 cannot correctly estimate the positions of the characteristic points of the index finger and the thumb.

[0022] The shape determination unit 112 acquires the three-dimensional position of each feature point estimated by the position estimation unit 111, and calculates the depth difference (difference in the z direction) between the tip of the thumb and the tip of the index finger. Next, the shape determination unit 112 calculates the position (s, t, r) of each feature point in a three-dimensional space with the position of the HMD device 101 as the origin from the three-dimensional position (u, v, z) estimated by the position estimation unit 111 based on the size of the detected image and the focal length of the imaging unit 102.

[0023] The shape determination unit 112 also calculates the distance in three-dimensional space between the tip of the thumb and each of the other feature points, based on the three-dimensional position (s, t, r) of each feature point with the position of the HMD device 101 as the origin. Furthermore, based on the distance in three-dimensional space between the tip of the thumb and each of the other feature points, the shape determination unit 112 compares the position of the tip of the index finger relative to the tip of the thumb with the positions of the other feature points relative to the tip of the thumb. In this way, the shape determination unit 112 determines whether the user's hand is gripping the thumb or not. Determine whether the person is performing a grasping gesture (a gesture of picking up an object with the index finger and thumb).

[0024] For example, in the detected image 200, referring to the image 202, it can be seen that the distance between the tip of the thumb 204 and the tip of the index finger 203 is shorter than the distance between the tip of the thumb 204 and other feature points. Also, referring to the plot image 205, it can be seen that the distance between the tip of the index finger 203 and the tip of the thumb 204 is very short. On the other hand, referring to the plot image 206, it can be seen that the tip of the index finger 203 and the tip of the thumb 204 are separated in the z direction. In this case, the shape determination unit 112 determines that there is a "possibility" that the hand is making a grasping gesture (the possibility that the hand is making a grasping gesture is at the first level).

[0025] When it is determined that there is a "possibility" that the hand is performing a grasping gesture and the distance in three-dimensional space between the tip of the thumb and the tip of the index finger is within a certain distance, the correction determination unit 113 determines to correct the three-dimensional position of each of the estimated feature points of the hand. For example, since the shape determination unit 112 has determined that there is a "possibility" for the detected image 200 and the distance in three-dimensional space between the tip of the index finger 203 and the tip of the thumb 204 is close, the correction determination unit 113 determines to correct the three-dimensional position of each of the estimated feature points.

[0026] When correction determination unit 113 determines that the three-dimensional position (u, v, z) should be corrected, shape correction unit 114 calculates a correction value in the depth direction of the tip of the thumb based on the depth difference between the tip of the thumb and the tip of the index finger. Then, shape correction unit 114 corrects the depth coordinate of the tip of the thumb according to the calculated correction value so that the tip of the thumb approaches the tip of the index finger.

[0027] Furthermore, shape correction unit 114 corrects the depth coordinates of the feature points of the thumb other than the tip so that the “corrected shape of the thumb from the tip to the base” approaches the “pre-correction shape of the thumb from the tip to the base”. For example, shape correction unit 114 multiplies the pre-correction depth coordinate values ​​of the feature points of the thumb other than the tip by a constant ratio.

[0028] Image 208 is an image in which points indicating the three-dimensional positions (u, v, z) of each feature point after correction by shape correction unit 114 are plotted on user's hand 201 in detection image 200. Plot image 209 is an image in which points indicating the three-dimensional positions (u, v, z) of each feature point of the wrist, thumb, and index finger after correction are plotted on a uv plane. Plot image 210 is an image in which points indicating the three-dimensional positions (u, v, z) of each feature point of the wrist, thumb, and index finger after correction are plotted on a zv plane.

[0029] The shape correction unit 114 does not correct the three-dimensional position with respect to the uv plane (u coordinate and v coordinate). Therefore, the plot positions are the same in the plot images 205 and 209. In the plot images 206 and 210, the frame 207 indicates a range surrounding the feature points from the wrist to the tip of the thumb. Here, the length in the z direction of the frame 207 in the plot image 210 indicating the three-dimensional position after correction is shorter than the length in the z direction of the frame 207 in the plot image 206 indicating the three-dimensional position before correction. This allows the shape correction unit 114 to correct the three-dimensional position (the shape of the user's hand 211) so that the three-dimensional position estimated by the position estimation unit 111 corresponds to the gripping gesture.

[0030] Shape correction unit 114 outputs information on the three-dimensional position of each feature point of the hand after correction from detection unit 110. System control unit 103 may generate CG (computer graphics) of the hand based on the information (the three-dimensional position of each feature point of the hand after correction) output from shape correction unit 114. System control unit 103 may also display the generated CG of the hand on the display unit of HMD device 101. This allows the user to see CG in which the hand is expressed in a more accurate shape.

[0031] The process of detecting the three-dimensional positions of feature points of the hand in the HMD device 101 will be described with reference to the flowchart of Fig. 3. The process of the flowchart of Fig. 3 is executed each time the imaging unit 102 obtains a detection image (each time imaging is performed).

[0032] In step S301, the position estimation unit 111 estimates the three-dimensional position (u, v, z) of each feature point (including the wrist joint point) of the user's hand contained in the detected image.

[0033] In step 302, shape determination unit 112 measures (acquires) the distance (hereinafter referred to as the "measured distance") in the detected image (image space) between the tip of the thumb and the tip of the index finger estimated by position estimation unit 111. For example, if it is estimated that the coordinates of the tip of the thumb are (u, v) = (100, 100) and the coordinates of the tip of the index finger are (u, v) = (110, 110) in detected image 200, the measured distance is approximately 14.1.

[0034] In step S303, the shape determination unit 112 determines whether the measured distance measured in step S302 is smaller than a threshold Th1. If it is determined that the measured distance is smaller than the threshold Th1, the process proceeds to step S304. If it is determined that the measured distance is equal to or greater than the threshold Th1, the process proceeds to step S311. For example, assuming that the measured distance is 11.4 and the threshold Th1 is 25, the measured distance is smaller than the threshold Th1, and therefore the process proceeds to step S304.

[0035] In step S304, shape determination unit 112 measures the depth difference (distance in the depth direction) between the tip of the thumb and the tip of the index finger. For example, if position estimation unit 111 detects that the depth coordinate (z coordinate) of the tip of the thumb is 50 and the depth coordinate of the tip of the index finger is 20, the depth difference is 30 (=50-20) mm.

[0036] In step S305, the shape determination unit 112 determines whether the depth difference measured in step S304 is greater than a threshold value Th2. If it is determined that the depth difference is greater than the threshold value Th2, the process proceeds to step S306. If it is determined that the depth difference is equal to or less than the threshold value Th2, the process proceeds to step S309. For example, assuming that the depth difference is measured to be 30 mm in step S304 and the threshold value Th2 is 25 mm, the depth difference is greater than the threshold value Th2, and the process proceeds to step S306.

[0037] In step S306, the shape determination unit 112 calculates the three-dimensional position (p, q, r) of each feature point with the position of the HMD device 101 as the origin (0, 0, 0) from the three-dimensional position (u, v, z) estimated by the position estimation unit 111 based on the size and focal length of the detected image. Next, the shape determination unit 112 calculates the distance in three-dimensional space between the tip of the thumb and each of all other feature points based on the calculated positions (p, q, r) of each feature point in three-dimensional space.

[0038] In step S307, shape determination unit 112 compares all the distances calculated in step S306 to determine whether the feature point closest to the tip of the thumb is the tip of the index finger. If it is determined that the feature point closest to the tip of the thumb is the tip of the index finger, the process proceeds to step S308. If it is determined that the feature point closest to the tip of the thumb is not the tip of the index finger, the process proceeds to step S311.

[0039] For example, if the distance between the tip of the index finger and the tip of the thumb is measured to be 25 mm, and the distance between each of the other feature points and the tip of the thumb is measured to be greater than 40 mm, the tip of the index finger is closest to the tip of the thumb, so the process proceeds to step S308.

[0040] In step S308, shape determination unit 112 determines whether or not there is a feature point other than the tip of the index finger (other than the specific feature point) within a distance of threshold Th3 from the tip of the thumb, based on all the distances found in step S306. If it is determined that there is no feature point other than the tip of the index finger within a distance of threshold Th3 from the tip of the thumb, the process proceeds to step S310. If it is determined that there is a feature point other than the tip of the index finger within a distance of threshold Th3 from the tip of the thumb, the process proceeds to step S311.

[0041] For example, if the distance between the tip of the index finger and the tip of the thumb is measured to be 25 mm, and the distance between each of the other feature points and the tip of the thumb is measured to be greater than 40 mm, and the threshold value TH3 is assumed to be 40 mm, the distance between each of the feature points other than the tip of the index finger and the tip of the thumb is greater than the threshold value Th3, so the process proceeds to step S310.

[0042] In step S309, shape determination unit 112 determines that it is "highly likely" that the hand is making a grasping gesture (the likelihood is at a second level). Note that when it is determined that it is "highly likely" that the hand is making a grasping gesture (the likelihood is at a second level), the positions of the feature points of the hand are not corrected because it is highly likely that the positions of the feature points of the hand have been accurately estimated.

[0043] In step S310, the shape determination unit 112 determines that there is a "possibility" that the hand is making a grasping gesture (the possibility is at a first level that is lower than a second level).

[0044] In step S311, shape determination unit 112 determines that there is "no possibility" that the hand is making a grasping gesture (the possibility is a third level lower than the first level). Note that when it is determined that there is "no possibility" that the hand is making a grasping gesture (the possibility is a third level), the positions of the feature points of the hand should not be corrected to correspond to the grasping gesture, and therefore the positions of the feature points of the hand are not corrected.

[0045] In step S312, the correction determination unit 113 determines whether the distance between the tip of the thumb and the tip of the index finger in the three-dimensional space is equal to or less than a threshold value Th4. If it is determined that the distance between the tip of the thumb and the tip of the index finger is equal to or less than the threshold value Th4, the process proceeds to step S313. If it is determined that the distance between the tip of the thumb and the tip of the index finger is greater than the threshold value Th4, the three-dimensional position of each feature point is not corrected and the process of this flowchart ends. For example, if the distance between the tip of the index finger and the tip of the thumb in the three-dimensional space is measured to be 25 mm and the threshold value Th4 is set to 30 mm, the distance between the tip of the thumb and the tip of the index finger is equal to or less than the threshold value Th4, and the process proceeds to step S313. Note that if the process of step S310 ends without executing the process of step S312, the process of step S313 may start.

[0046] In step S313, shape correction unit 114 calculates a correction amount (hereinafter referred to as "depth correction amount") for the depth coordinate (z coordinate) of the tip of the thumb based on the depth difference between the tip of the thumb and the tip of the index finger. For example, assume that the depth difference is measured to be 30 mm in step S304, and the ratio of "depth difference between the tip of the thumb before correction and the tip of the thumb after correction" to "depth difference between the tip of the thumb after correction and the tip of the index finger after correction" is preset to 7:3. In this case, shape correction unit 114 calculates 21 mm (=30 mm×7 / 10) as the depth correction amount for the tip of the thumb.

[0047] In step S314, shape correcting unit 114 corrects the depth coordinate of the tip of the thumb so that the tip of the thumb approaches the tip of the index finger by the amount of depth correction calculated in step S313.

[0048] For example, if the depth coordinate of the tip of the thumb before correction is 50 and the depth correction amount is calculated to be 21 mm in step S313, the depth coordinate of the tip of the thumb after correction is 50-21=29.

[0049] In step S315, shape correction unit 114 acquires a value obtained by multiplying the depth correction amount calculated in step S313 by a correction magnification as the depth correction amount for the feature points of the thumb other than the tip.

[0050] The correction magnification may be different for each feature point or may be the same for multiple feature points. For example, as shown in a frame 207 in FIG. 2, the thumb has four feature points (including the wrist) other than the tip 204, and the four feature points are the first joint point, the second joint point, the third joint point, and the wrist in the order of proximity to the tip 204. In this case, the correction targets other than the depth coordinate of the tip of the thumb are the depth coordinates of the first joint point, the second joint point, and the third joint point of the thumb. The ratio of the correction magnifications of the depth coordinates of the tip, the first joint point, the second joint point, and the third joint point of the thumb is set to 21:15:9:3. Then, if the depth correction amount is calculated to be 21 mm in step S313, the depth correction amount of the first joint point is 15 mm, the depth correction amount of the second joint point is 9 mm, and the depth correction amount of the third joint point is 3 mm.

[0051] In step S316, the shape correction unit 114 corrects the depth coordinates of each feature point of the thumb other than the tip in a direction approaching the tip of the index finger by the depth correction amount calculated in step S315. For example, assume that the depth correction amount of the first joint point is 15 mm, the depth correction amount of the second joint point is 9 mm, and the depth correction amount of the third joint point is 3 mm in step S315. Here, the corrected depth coordinate is calculated, for example, by subtracting the depth correction amount from the depth coordinate before correction. Therefore, for example, the corrected depth coordinate z of the first joint point is 20, the corrected depth coordinate z of the second joint point is 16, and the corrected depth coordinate z of the third joint point is 12. By such processing, as shown in FIG. 2, the positional relationship shown in the plot image 206 is corrected to the positional relationship shown in the plot image 217.

[0052] In the first embodiment, it is determined that the hand is possibly performing a grasping gesture, but it may be determined that the hand is possibly performing another gesture (for example, a V sign gesture with only the middle finger and index finger raised). In this case, the shape correction unit 114 corrects the three-dimensional position of the hand so as to approach the shape of the hand corresponding to the other gesture (for example, so that the tip of the thumb, the tip of the little finger, and the tip of the ring finger approach each other). In addition, the position of other objects (such as a human body, a home appliance, or a toy) that may change the position of the feature points may be estimated and corrected, not limited to the hand. In this case, "the hand is performing a grasping gesture" described in the first embodiment above may be read as "the shape of the object is a specific shape".

[0053] As described above, according to the first embodiment, the HMD device 101 estimates the positions of a plurality of feature points of an object, and then corrects the positions of the plurality of feature points so as to approach the particular shape if there is a particular level of possibility that the shape of the object is a particular shape. This allows the HMD device 101 to more appropriately correct the positions of the feature points of the object, and therefore to more accurately detect the positions of the plurality of feature points on the object (such as a hand).

[0054] <Embodiment 2> 4 is a functional block diagram of an HMD device 401 according to embodiment 2. The HMD device 401 has an imaging unit 102, a system control unit 103, a non-volatile memory 104, a system memory 105, and a detection unit 410. The imaging unit 102, the system control unit 103, the non-volatile memory 104, and the system memory 105 have the same configurations as those in embodiment 1, and therefore their description will be omitted.

[0055] The detection unit 410 detects the three-dimensional position (u, v, z) of each feature point of the hand included in the detected image stored in the system memory 105. The detection unit 410 has a position estimation unit 111, a shape determination unit 412, a correction determination unit 413, and a shape correction unit 414. The detection unit 410 will be described in detail with reference to FIG. 5.

[0056] 5 is a diagram for explaining information on the three-dimensional position (u, v, z) of each feature point before correction and the three-dimensional position (u, v, z) of each feature point after correction. In the following, the current frame (period) is referred to as the "N frame," and the frame immediately preceding the N frame is referred to as the "N-1 frame." More specifically, the "N frame" refers to the period during which processing is performed on the latest (current) detected image, and the "N-1 frame" refers to the period during which processing is performed on the detected image immediately preceding the latest (previous) detected image.

[0057] 5, a detected image 500 is the detected image of the Nth frame stored in the system memory 105. Here, the detected image 500 includes a hand 501 of a user wearing the HMD device 401. The hand 501 is making a grasping gesture (a gesture in which the tips of the thumb and index finger are touching and an object is being grasped). Due to the presence of the back of the user's hand, the tips of both fingers (thumb and index finger) are not visible to the user and the HMD device 101.

[0058] The position estimation unit 111 performs the same process as in the first embodiment, and therefore detailed description will be omitted. An image 502 is an image in which points indicating the positions (u, v, z) of each feature point of the hand estimated by the position estimation unit 111 based on the detected image of the Nth frame are superimposed on the user's hand 501 in the detected image 500. A plot image 503 is an image in which points indicating the feature points of the thumb and index finger and the three-dimensional position of the wrist among the estimated positions are plotted on the zv plane. In the plot image 503, the tip 504 of the index finger and the tip 505 of the thumb are separated. That is, the position estimation unit 111 cannot correctly estimate the positions of the feature points of the hand making the grasping gesture.

[0059] The shape determination unit 412 acquires the three-dimensional position (u, v, z) estimated by the position estimation unit 111, and calculates the "distance in image space (detected image)" and the "depth difference" between the tip of the thumb and the tip of the index finger. Next, the shape determination unit 412 calculates the positions (p, q, r) of each feature point of the hand in a three-dimensional space with the position of the HMD device 401 as the origin from the three-dimensional position (u, v, z) estimated by the position estimation unit 111 based on the size of the detected image and the focal length of the imaging unit 102. The shape determination unit 412 calculates the distances in the three-dimensional space between the tip of the thumb and each of the other feature points based on the positions (p, q, r) of each feature point of the hand in the three-dimensional space.

[0060] Furthermore, the shape determination unit 412 uses the calculated distance to compare the position in the three-dimensional space of the second joint point of the index finger relative to the tip of the thumb with the positions in the three-dimensional space of other feature points relative to the tip of the thumb. In this way, the shape determination unit 412 determines the possibility (level of possibility) that the user's hand is performing a grasping gesture.

[0061] For example, with respect to detected image 500, referring to image 502 and plot image 503, it can be seen that second joint point 506 of the index finger is closest to tip 505 of the thumb. Also, referring to plot image 503, it can be seen that the depth difference between tip 505 of the thumb and tip 504 of the index finger is small. In this case, shape determination unit 412 determines that there is a "possibility" that hand 501 is making a grasping gesture.

[0062] The correction determination unit 413 first determines whether the result of the determination by the shape determination unit 412 is "possible" and whether the distance in three-dimensional space between the tip of the thumb and the tip of the index finger calculated by the shape determination unit 412 is within a certain distance. Then, the correction determination unit 413 determines whether both conditions are satisfied and whether the result of the determination by the shape determination unit 412 in the N-1 frame is "possible". is "highly likely", it is determined that the three-dimensional position of the feature point estimated by the position estimation unit 111 is to be corrected.

[0063] For example, the shape determination unit 412 has determined that the user's hand 501 in the detected image 500 of the Nth frame is "probably" performing a gripping gesture, and the distance in three-dimensional space between the tip 504 of the index finger and the tip 505 of the thumb is short. Furthermore, it is assumed that the shape determination unit 412 has determined that the gripping gesture is "probably" performed in the N-1th frame based on the image 208. In this case, the correction determination unit 413 determines to correct the three-dimensional positions of the feature points of the user's hand 501 estimated by the position estimation unit 111.

[0064] When the correction determination unit 413 determines that the three-dimensional positions are to be corrected, the shape correction unit 414 compares the "three-dimensional positions of each feature point of the hand" estimated by the position estimation unit 111 in the N frame with the "three-dimensional positions of each feature point of the hand" estimated in the N-1 frame. When the result of the comparison satisfies a certain condition (criterion), the shape correction unit 414 performs specific processing (translation, rotation, and interval adjustment processing) on ​​the "three-dimensional positions of each feature point of the hand" estimated in the N-1 frame. Then, the shape correction unit 414 acquires the "three-dimensional positions of each feature point of the hand" after the specific processing as the three-dimensional positions of each feature point in the N frame. In this way, the shape correction unit 414 corrects the three-dimensional positions of each feature point in the N frame.

[0065] For example, in Fig. 5, shape correction unit 414 translates the three-dimensional positions of each feature point of the hand estimated in the N-1 frame (positions of all feature points) so that wrist position 510 in the N-1 frame overlaps with wrist position 509 in the N frame. Next, shape correction unit 414 rotates the three-dimensional positions of each translated feature point (each feature point estimated in the N-1 frame) by a certain angle, using an axis extending perpendicularly from wrist position 509 in the v-axis direction as the rotation axis.

[0066] Image 511 is an image in which points indicating the three-dimensional positions of each feature point in the N-1 frame, after such processing, are plotted on the detected image 500. Here, plot image 512 is an image in which points indicating the three-dimensional positions of each feature point of the wrist, thumb, and index finger in image 511 are plotted on the uv plane. Plot image 513 is an image in which points indicating the three-dimensional positions of each feature point of the wrist, thumb, and index finger in image 511 are plotted on the zv plane.

[0067] In image 511, there is a difference between the actual positions of the feature points of user's hand 501 and the plot positions. However, referring to tip 504 of index finger and tip 505 of thumb in plot image 512 and plot image 513, it can be seen that the positions of the feature points are corrected to correspond to the gesture of pinching with thumb and index finger. Therefore, the three-dimensional positions (intervals) of all feature points are corrected based on the position of the wrist so that "distance 508 between the third joint point of the index finger and the wrist after correction" becomes equal to "distance 507 between the third joint point of the index finger and the wrist before correction". As a result, shape correction according to the size of user's hand 501 is possible, as in image 514.

[0068] The detection unit 410 outputs information on the three-dimensional position of each feature point of the user's hand after the shape correction unit 414 has corrected it.

[0069] The process of detecting the three-dimensional position of the feature point of the hand in the HMD device 401 according to the second embodiment will be described with reference to the flowchart in Fig. 6. The process of the flowchart in Fig. 6 is executed for each frame, but may be executed at regular intervals. In the following, it is assumed that the process is executed for N frames. In Fig. 6, steps S301 to S304 in Fig. 3 are executed. Although steps S306 and S309 are not shown, the processes from steps S301 to S306 and S309 are performed in the same manner as in embodiment 1. When the process of step S306 ends, the process of step S607 starts.

[0070] In step S607, shape determination unit 412 refers to all the distances found in step S306 and determines whether the feature point closest to the tip of the thumb is the second joint point of the index finger. If it is determined that the feature point closest to the tip of the finger is the second joint point of the index finger, the process proceeds to step S308. If it is determined that the feature point closest to the tip of the finger is not the second joint point of the index finger, the process proceeds to step S311.

[0071] For example, assume that in step S306, the distance between the second joint point of the index finger and the tip of the thumb is 10 mm, the distance between the tip of the index finger and the tip of the thumb is 25 mm, and the distance between the other feature points and the tip of the thumb is greater than 40 mm. In this case, the second joint point of the index finger is closest to the tip of the thumb, so proceed to step S308.

[0072] Description of steps S308 to S312 will be omitted since they are similar to those in embodiment 1. If it is determined in step S312 that the distance between the tip of the thumb and the tip of the index finger is equal to or smaller than threshold value Th4, the process proceeds to step S613.

[0073] In step S613, correction determination unit 413 determines whether or not it has been determined by shape determination unit 412 that it is "highly likely" that the hand is making a grasping gesture in frame N-1. If it has been determined that it is "highly likely" that the hand is making a grasping gesture, the process proceeds to step S614. If it has not been determined that it is "highly likely" that the hand is making a grasping gesture, the process of this flowchart ends.

[0074] In step S614, the shape correction unit 414 calculates a first reference vector a having the three-dimensional position of the wrist estimated by the position estimation unit 111 in step S301 as a start point and the three-dimensional position of the third joint point of the index finger as an end point. Next, the shape correction unit 414 calculates a second reference vector b having the three-dimensional position of the wrist in the "three-dimensional positions of each feature point of the hand" estimated by the position estimation unit 111 in the N-1 frame as a start point and the three-dimensional position of the third joint point of the index finger as an end point.

[0075] For example, if the three-dimensional position a0 of the wrist estimated by the position estimation unit 111 is (50,50,50) and the three-dimensional position a1 of the third joint point of the index finger is (70,120,50), the first reference vector a is a1-a0=(20,70,0). The second reference vector b can also be calculated by a method similar to the calculation method of the first reference vector a.

[0076] In step S615, the shape correction unit 414 calculates the angle (hereinafter referred to as the "formed angle θ") formed by the two reference vectors (the first reference vector and the second reference vector) obtained in step S614 and the norm ratio. Here, cosθ is a value obtained by dividing the "inner product of the first reference vector a and the second reference vector b" by the "product of the norm La of the first reference vector a and the norm Lb of the second reference vector b". For example, when the first reference vector a is (20, 70, 0) and the second reference vector b is (10, 50, 0), the norm La is 72.8 and the norm Lb is 51.0 as shown in the following formula. Therefore, the inner product of the first reference vector a and the second reference vector b is 3700, and the product of the norm La and the norm Lb is 3712.8. Therefore, since cos θ is 0.9965 (= 3700 / 3712.8), the formed angle θ is 0.00831 [rad] = 4.76°. Note that the norm ratio is norm La / norm Lb = 72.8 / 51.0 = 1.43.

number

[0077] In step S616, the correction determination unit 413 determines whether the formation angle θ calculated in step S615 is equal to or less than the threshold value Th5. If it is determined that the formation angle θ is equal to or less than the threshold value Th5, the process proceeds to step S617. If it is determined that the formation angle θ is greater than the threshold value Th5, the process of this flowchart ends. For example, if the formation angle θ is calculated to be 4.76° and the threshold value Th5 is 30°, the formation angle θ is equal to or less than the threshold value Th5, and the process proceeds to step S617.

[0078] In step S617, the correction determination unit 413 translates the three-dimensional positions of all feature points estimated in the N-1 frame so that the wrist position estimated in the N-1 frame moves to the wrist position estimated by the position estimation unit 111 in the N frame.

[0079] For example, suppose that the wrist position (u, v, z) estimated by the position estimation unit 111 in the N frame is (50, 50, 50), and the wrist position (u, v, z) estimated in the N-1 frame is (100, 200, 300). In this case, the correction determination unit 413 translates the three-dimensional positions of all feature points estimated in the N-1 frame by 100-50=50 in the negative direction of the u axis, 200-50=150 in the negative direction of the v axis, and 300-50=250 in the negative direction of the z axis.

[0080] In step S618, shape correction unit 414 rotates the three-dimensional position of each translated feature point by a formation angle θ, using an axis extended in the v-axis direction (the up-down direction of detected image 500) from the wrist position after the translation in step S617 as the rotation axis.

[0081] In step S619, shape correction unit 414 multiplies the three-dimensional position value (three-dimensional position coordinates) of each feature point after rotational movement by the norm ratio. In this way, shape correction unit 414 matches the vector from the wrist to the base of the index finger after the rotation process in step S618 with the first reference vector obtained in step S614. For example, if the norm ratio is 1.43 and the three-dimensional position obtained in step S618 is (i, j, k), then (1.43i, 1.43j, 1.43k) is the three-dimensional position corrected by shape correction unit 414.

[0082] According to the second embodiment, in certain cases, the three-dimensional position of the current frame can be detected (corrected) using the three-dimensional position estimated with high accuracy in the previous frame. Therefore, in certain cases, it becomes possible to detect the three-dimensional position of the current frame with higher accuracy.

[0083] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0084] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." This may result in a contradiction. Unless otherwise stated, "A or more" may be read as "bigger (higher; longer; more) than A" and "A or less" may be read as "smaller (lower; shorter; less) than A." And "bigger (higher; longer; more) than A" may be read as "A or more," and "smaller (lower; shorter; less) than A" may be read as "A or less."

[0085] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0086] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0087] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An estimation means for estimating positions of a plurality of feature points of an object in a three-dimensional space; a determination means for determining a possibility that the shape of the object is a specific shape based on the positions of the plurality of feature points estimated by the estimation means; a correction means for correcting positions of the plurality of feature points estimated by the estimation means in a first case in which it is determined that the possibility that the shape of the object is the specific shape is at a first level, such that the shape corresponding to the positions of the plurality of feature points estimated by the estimation means approaches the specific shape; 13. An information processing device comprising: (Configuration 2) the correction means does not correct the positions of the feature points estimated by the estimation means when it is determined that the possibility that the shape of the object is the specific shape is not at the first level. 2. The information processing device according to configuration 1. (Configuration 3) the determination means determines a possibility that the shape of the object is the specific shape based on positions of the plurality of feature points in a first two-dimensional coordinate system and positions of the plurality of feature points in a second two-dimensional coordinate system perpendicular to the first two-dimensional coordinate system; 3. The information processing device according to configuration 1 or 2. (Configuration 4) the plurality of feature points includes a first point and a second point; the determination means determines a possibility that the shape of the object is the specific shape based on a distance in a depth direction between the first point and the second point. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) the determination means determines a possibility that the shape of the object is the specific shape based on whether the second point among the plurality of feature points is closest to the first point. 5. The information processing device according to configuration 4. (Configuration 6) the determination means determines a possibility that the shape of the object is the specific shape based on whether or not a feature point other than the second point exists within a first distance from the first point. 6. The information processing device according to configuration 4 or 5. (Configuration 7) the correction means corrects positions of the plurality of feature points estimated by the estimation means when it is determined that the possibility that the shape of the object is the specific shape is at the first level and the distance between the first point and the second point is smaller than a second distance. 7. The information processing device according to any one of configurations 4 to 6. (Configuration 8) the correction means, in the first case, corrects the positions of the plurality of feature points estimated by the estimation means so that the first point and the second point approach each other. 8. The information processing device according to any one of configurations 4 to 7. (Configuration 9) The estimation means estimates positions of a plurality of feature points of the object at regular intervals; the determining means determines, at regular intervals, the possibility that the shape of the object is the specific shape; the correction means, in a second case where the possibility that the shape of the object is the specific shape is determined to be at the first level currently and the possibility that the shape of the object is the specific shape is determined to be at a second level higher than the first level previously, corrects the positions of the plurality of feature points estimated currently by the estimation means based on the positions of the plurality of feature points estimated previously by the estimation means. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 10) In the second case, the correction means corrects the positions of the plurality of feature points currently estimated by the estimation means so as to use positions obtained by performing a translation process, a rotation process, and an adjustment process of an interval between the plurality of feature points on the positions of the plurality of feature points previously estimated by the estimation means as the positions of the plurality of feature points currently estimated by the estimation means. 10. The information processing device according to configuration 9. (Configuration 11) the plurality of feature points includes a third point; the correction means performs the translation process so that the position of the third point previously estimated by the estimation means is moved to the position of the third point currently estimated by the estimation means. 11. The information processing device according to configuration 10. (Configuration 12) the plurality of feature points includes a fourth point and a fifth point; the correction means performs the rotation process and the interval adjustment process based on a first vector from the position of the fourth point to the position of the fifth point, which is previously estimated by the estimation means, and a second vector from the position of the fourth point to the position of the fifth point, which is currently estimated by the estimation means. 12. The information processing device according to configuration 10 or 11. (Configuration 13) the object being a human hand, The plurality of feature points includes a plurality of joint points in a hand. 13. The information processing device according to any one of configurations 1 to 12. (method) an estimation step of estimating positions of a plurality of feature points of an object in a three-dimensional space; The shape of the object is determined based on the positions of the plurality of feature points estimated in the estimation step. determining the likelihood that the shape is a particular shape; a correction step of correcting positions of the plurality of feature points estimated in the estimation step so that a shape corresponding to the positions of the plurality of feature points estimated in the estimation step approaches the specific shape in a first case in which it is determined that the possibility that the shape of the object is the specific shape is at a first level; 13. An information processing method comprising: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 13. [Explanation of symbols]

[0088] 101: HMD device (information processing device), 103: system control unit, 110: detection unit, 111: position estimation unit, 112: Shape determination section, 114: Shape correction section

Claims

1. An estimation means for estimating positions of a plurality of feature points of an object in a three-dimensional space; a determination means for determining a possibility that the shape of the object is a specific shape based on the positions of the plurality of feature points estimated by the estimation means; a correction means for correcting positions of the plurality of feature points estimated by the estimation means in a first case in which it is determined that the possibility that the shape of the object is the specific shape is at a first level, such that the shape corresponding to the positions of the plurality of feature points estimated by the estimation means approaches the specific shape; 13. An information processing device comprising:

2. the correction means does not correct the positions of the plurality of feature points estimated by the estimation means when it is determined that the possibility that the shape of the object is the specific shape is not at the first level.

2. The information processing apparatus according to claim 1,

3. the determination means determines a possibility that the shape of the object is the specific shape based on positions of the plurality of feature points in a first two-dimensional coordinate system and positions of the plurality of feature points in a second two-dimensional coordinate system perpendicular to the first two-dimensional coordinate system; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the plurality of feature points includes a first point and a second point; the determining means determines a possibility that the shape of the object is the specific shape based on a distance in a depth direction between the first point and the second point.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. the determination means determines a possibility that the shape of the object is the specific shape based on whether or not the second point among the plurality of feature points is closest to the first point.

5. The information processing apparatus according to claim 4.

6. the determination means determines a possibility that the shape of the object is the specific shape based on whether or not a feature point other than the second point exists within a first distance from the first point.

5. The information processing apparatus according to claim 4.

7. the correction means corrects positions of the plurality of feature points estimated by the estimation means when it is determined that the possibility that the shape of the object is the specific shape is at the first level and the distance between the first point and the second point is smaller than a second distance.

5. The information processing apparatus according to claim 4.

8. the correction means, in the first case, corrects the positions of the plurality of feature points estimated by the estimation means so that the first point and the second point approach each other.

5. The information processing apparatus according to claim 4.

9. The estimation means estimates positions of a plurality of feature points of the object at regular intervals; the determining means determines, at regular intervals, the possibility that the shape of the object is the specific shape; the correction means, in a second case where the possibility that the shape of the object is the specific shape is determined to be the first level currently and the possibility that the shape of the object is the specific shape is determined to be the second level higher than the first level previously, corrects the positions of the plurality of feature points estimated by the estimation means currently by correcting the positions of the plurality of feature points estimated by the estimation means; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

10. and in the second case, the correction means corrects the positions of the plurality of feature points currently estimated by the estimating means so as to use positions obtained by performing a translation process, a rotation process, and an adjustment process of an interval between the plurality of feature points on the positions of the plurality of feature points previously estimated by the estimating means as the positions of the plurality of feature points currently estimated by the estimating means.

10. The information processing apparatus according to claim 9,

11. the plurality of feature points includes a third point; the correction means performs the translation process so that the position of the third point previously estimated by the estimation means is moved to the position of the third point currently estimated by the estimation means.

11. The information processing apparatus according to claim 10,

12. the plurality of feature points includes a fourth point and a fifth point; the correction means performs the rotation process and the interval adjustment process based on a first vector from the position of the fourth point to the position of the fifth point, which is previously estimated by the estimation means, and a second vector from the position of the fourth point to the position of the fifth point, which is currently estimated by the estimation means.

11. The information processing apparatus according to claim 10,

13. the object being a human hand, The plurality of feature points includes a plurality of joint points in a hand.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

14. an estimation step of estimating positions of a plurality of feature points of an object in a three-dimensional space; a determination step of determining a possibility that the shape of the object is a specific shape based on the positions of the plurality of feature points estimated in the estimation step; a correction step of correcting positions of the plurality of feature points estimated in the estimation step so that a shape corresponding to the positions of the plurality of feature points estimated in the estimation step approaches the specific shape in a first case in which it is determined that the possibility that the shape of the object is the specific shape is at a first level; 13. An information processing method comprising:

15. A program for causing a computer to function as each of the means of the information processing device according to claim 1 or 2.

Images