Information processing device, information processing method, and program

Abstract

To enable a position of a detection object to be estimated with a smaller processing load without using a distance sensor.SOLUTION: An information processing device includes first detection means for detecting a detection object in a first image acquired from a first imaging apparatus, update means for updating reference information about the detection object stored in specific storage means on the basis of the first image and a second image acquired from a second imaging apparatus in a first case and not updating the reference information stored in the specific storage means in a second case, and estimation means for estimating a relative position of the detection object to the first imaging apparatus on the basis of a detection result of the detection object by the first detection means and the reference information stored in the specific storage means.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, research has been conducted on mixed reality (MR) technology, which overlays information from a virtual space onto real space in real time and presents it to users. In mixed reality technology, a synthetic image is displayed in which an image of a virtual space corresponding to the position and orientation of an imaging device such as a Head Mounted Display (HMD) is overlaid on all or part of an image of a real space captured by the imaging device.

[0003] In this case, by detecting (estimating) the position of a detection target such as a real user's hand, it is possible to control the display of a virtual object according to the position and shape of the user's hand. In order to realize such detection (estimation) of the position of a detection target, it is preferable to be able to estimate the distance between the imaging device of the HMD and the hand more accurately, for example.

[0004] Patent Document 1 describes a technology that continues tracking an object, the distance of which is being measured by a stereo camera, using features detected by the stereo camera even when the object moves out of the field of view of the stereo camera and enters the field of view of another single camera. Patent Document 2 describes a system that detects a hand based on a distance image captured by a single distance sensor, and compares the distance image with a sphere that approximates the joint of the hand. The system estimates the distance from the sensor to the joint of the hand based on the result of this comparison and the size of the detected hand. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2008-123462 A [Patent Document 2] U.S. Pat. No. 1,568,601 Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology of Patent Document 1, the stereo camera and the single camera each need to perform detection, which requires processing time. In the technology of Patent Document 2, a distance sensor needs to be installed in addition to an imaging device that captures a general luminance image. For this reason, it has not been possible to estimate the position of a detection target with less processing load without using a distance sensor.

[0007] Therefore, an object of the present invention is to make it possible to estimate the position of a detection target with less processing load without using a distance sensor. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: a first detection means for detecting a detection target in a first image acquired from a first imaging device; an update means for updating reference information related to the detection target stored in a specific storage means based on the first image and a second image acquired from a second imaging device in a first case, and not updating the reference information stored in the specific storage means in a second case; an estimation means for estimating a relative position of the detection target with respect to the first imaging device based on a detection result of the detection target by the first detection means and the reference information stored in the specific storage means; The information processing device is characterized by having:

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: a first detection step of detecting a detection object in a first image acquired from a first imaging device; an updating step of updating, in a first case, reference information related to the detection target stored in a specific storage means based on the first image and a second image acquired from a second imaging device, and not updating the reference information stored in the specific storage means in a second case; an estimation step of estimating a relative position of the detection target with respect to the first imaging device based on a detection result of the detection target in the first detection step and the reference information stored in the specific storage means; The information processing method is characterized by having the following features. Effect of the Invention

[0010] According to the present invention, the position of a detection target can be estimated with less processing load without using a distance sensor. [Brief description of the drawings]

[0011] [Figure 1] 1 is a hardware configuration diagram of an information processing device according to a first embodiment. [Diagram 2] 1 is a functional configuration diagram of an information processing device and the like according to a first embodiment. [Diagram 3] 4 is a flowchart of a hand position estimation process according to the first embodiment. [Figure 4] 11 is a flowchart of a reference information update process according to the first embodiment. [Diagram 5] 4A to 4C are diagrams illustrating detection of hand joint points according to the first embodiment. [Figure 6] 13 is a flowchart of a reference information update process according to the second embodiment. [Figure 7] 13 is a flowchart of a hand position estimation process according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the following, an example in which a hand is detected as a detection target will be described. However, the detection target is not limited to a hand, and may be a part of a user's body other than a hand or the entire body. The detection target may be an object other than a human body.

[0013] <Embodiment 1> The hardware configuration of an information processing device 100, which is an electronic device, will be described with reference to Fig. 1. The information processing device 100 includes a CPU 101, a RAM 102, a ROM 103, an input interface 104, and an output interface 105.

[0014] The CPU 101 controls the entire information processing device 100. The RAM 102 is used as a working area when the CPU 101 controls each unit to perform processing. The ROM 103 stores a control program, various application programs, data, and the like. The CPU 101 loads the control program stored in the ROM 103 into the RAM 102 and executes the loaded control program. This realizes processing of each unit in the information processing device 100.

[0015] The input interface 104 acquires an input signal in a format that can be processed by the information processing device 100 from the HMD 200 (the imaging unit 201 or the sensing unit 202, etc.). The output interface 105 outputs a display image in a format that can be processed by the HMD 200 (the display unit 203).

[0016] Referring to FIG. 2, the functional configuration of the information processing device 100 and the HMD 200 according to the first embodiment will be described. The HMD 200 has an imaging unit 201, a sensing unit 202, and a display unit 203. The information processing device 100 has an imaging processing unit 211, a position and orientation calculation unit 212, a CG drawing unit 213, a main detection unit 214, and a sub detection unit 215. The information processing device 100 also has an update unit 216, a control unit 217, a hand position estimation unit 218, and a synthesis unit 219.

[0017] The imaging unit 201 captures images of real space (real world; outside world) using two or more cameras (imaging devices). The imaging unit 201 has a stereo camera composed of "two cameras that have at least a part of a common field of view." Hereinafter, one of the two cameras will be called the "main camera" and the other will be called the "sub-camera." Which of the two cameras will be the main camera may be determined in advance, or may be changeable by a user setting.

[0018] The sensing unit 202 measures (senses) inertial information (angular velocity, acceleration, etc.) of the HMD 200. The sensing unit 202 outputs the inertial information obtained by measurement as sensing information to the position and orientation calculation unit 212. The sensing unit 202 has a sensor such as an IMU (Inertial Measurement Unit), an acceleration sensor, or an angular velocity sensor. In this embodiment, the sensing unit 202 is an IMU that can measure the angular velocity and acceleration of the HMD 200.

[0019] The display unit 203 displays the display image generated by the composition unit 219. In this way, the display unit 203 presents to the user an image in which the real space image is composed with the CG.

[0020] The imaging processing unit 211 performs image processing on each of the captured image captured by the main camera of real space and the captured image captured by the sub camera of real space. In the following, an image captured by the main camera of real space and subjected to image processing is called a "main image". An image captured by the sub camera of real space and subjected to image processing is called a "sub image". The image processing performed by the imaging processing unit 211 may include processes such as demosaic processing, shading correction, noise reduction, distortion correction, and stereo rectification. In the stereo camera, if a subject is appropriately detected, a specific point in real space is captured at the same height in the main image and the sub image by stereo rectification processing.

[0021] These image processes convert the real image acquired by the imaging unit 201, so that "what a human perceives by looking at a subject directly" and "what a human perceives by looking at an image" become closer. In addition, the processes in other functional blocks (such as the position and orientation calculation unit 212, the main detection unit 214, and the sub detection unit 215) can be performed efficiently.

[0022] The position and orientation calculation unit 212 calculates the relationship between a world coordinate system (a coordinate system in real space) and a camera coordinate system (a coordinate system in an image observed through the HMD 200) based on the main image, the sub-image, or inertial information (information on the acceleration and angular velocity of the HMD 200). Then, the position and orientation calculation unit 212 calculates the position and orientation (position and orientation) of the HMD 200 in the world coordinate system based on the calculated relationship.

[0023] The position and orientation may be calculated, for example, by arranging a reference marker or the like in real space and calculating the position and orientation of the HMD 200 based on the position of the marker in an image acquired by the imaging unit 201. Alternatively, the position and orientation may be calculated by using an external sensor that constantly monitors the position of the world coordinate system of the position and orientation of the HMD 200. In this embodiment, the method of calculating the position and orientation of the HMD 200 is not particularly limited.

[0024] The CG rendering unit 213 reads out CG data from the RAM 102 or the ROM 103. Then, based on the information on the position and orientation of the HMD 200, the CG rendering unit 213 calculates the position and orientation of the CG when superimposing the CG on the main image (or the sub-image). The CG rendering unit 213 renders the CG based on the calculated position and orientation of the CG. Here, the CG rendering unit 213 renders, for example, a CG that reflects the shape of a hand at the position of the hand.

[0025] The main detection unit 214 detects the coordinates (position) of the hand in the main image based on the main image. The main detection unit 214 uses, for example, a detector that has been trained in advance by combining an image in which a hand appears and the correct coordinates of the hand in that image. The detector detects the coordinates of the hand when it receives the main image as an input. Deep Learning (DL) can be used for training this detector. The coordinates of the hand may be the coordinates (center coordinates) of a rectangular frame including the hand, the coordinates (center coordinates) of the wrist, or the coordinates of a joint point of the hand. In this embodiment, any method may be used as a method for the main detection unit 214 to detect the coordinates of the hand in the main image.

[0026] The sub-detection section 215 detects the coordinates (position) of the hand in the sub-image based on the sub-image using a method similar to that used by the main detection section 214.

[0027] The update unit 216 calculates (acquires) reference information (hereinafter referred to as "reference information") for determining the three-dimensional position of the hand based on the coordinates (detection result) of the hand in the main image and the coordinates (detection result) of the hand in the sub-image. For example, the reference information of the hand is information on the distance (length) between multiple joint points in three-dimensional space, information on the size of a rectangular frame including the hand, or information on the size of the hand. Note that the concept of the joint points of the hand includes the joint points of the wrist.

[0028] Note that, until the reference information is calculated, the information processing device 100 uses the initial value of the reference information stored in the storage unit (RAM 102 or ROM 103). The initial value of the reference information is determined in advance. The initial value of the reference information is, for example, information on the statistical size of a hand (average adult hand size), information on the size of a hand input by a user, or information on the size of a rectangular frame calculated from such information. Furthermore, when the update unit 216 calculates the reference information as described above, it updates the initial value of the reference information stored in the RAM 102 or ROM 103 to the calculated reference information.

[0029] Here, the update unit 216 calculates the difference between the coordinates of the hand in the main image and the coordinates of the hand in the sub-image (particularly the difference in the coordinates in the left-right direction (horizontal direction)). Then, the update unit 216 calculates (obtains) reference information based on the difference in coordinates and the camera parameters. The camera parameters are parameters such as the distance between the main camera and the sub camera, and the focal lengths of the main camera and the sub camera.

[0030] Furthermore, the update unit 216 may calculate the depth position of the hand (the difference between the position of the hand in the depth direction relative to the image and the position of the HMD 200) by stereo matching using the main image and the sub-image. The update unit 216 may then calculate the distance (length in three-dimensional space) between multiple joint points as reference information based on the coordinates of the hand in the main image and the position of the hand in the depth direction. Here, the further back the hand is located (the further away it is from the HMD 200), the smaller the hand appears in the main image. For this reason, the update unit 216 calculates the distance between two joint points in three-dimensional space by, for example, multiplying the distance between the two joint points in the main image by a coefficient corresponding to the depth position.

[0031] In order to improve the accuracy of calculating the position in the depth direction by stereo matching, infrared light having a wavelength that can be detected by both the main camera and the sub camera may be projected onto the subject. For example, as a method for improving accuracy using infrared light, a method can be used in which a subject is irradiated with clear infrared light to highlight objects (such as a hand) that are close to the camera. Furthermore, a method can be used in which an object that has no features under visible light (such as a floor or wall that is uniform in color) is given features by irradiating the subject with structured infrared light such as a random dot pattern.

[0032] The control unit 217 controls each unit (for example, determines the execution conditions of each unit).

[0033] The hand position estimation unit 218 estimates the position of the hand relative to the HMD 200 (main camera) based on the coordinates of the hand in the main image and the reference information of the hand. Note that the hand position estimated by the hand position estimation unit 218 may be the position of the center (center of gravity) of the hand or the positions of multiple joint points, so long as it is a three-dimensional position. Furthermore, the hand position estimation unit 218 may estimate the posture of the hand based on the positions of multiple joint points.

[0034] For example, the hand position estimation unit 218 can obtain the relative positional relationship of the three joint points with respect to the HMD 200 (main camera) based on the coordinates of the three joint points in the main image and the distance (length) between the three joint points in a three-dimensional space. In this case, the hand position estimation unit 218 can obtain the relative positional relationship of the three joint points with respect to the HMD 200 (main camera) by solving a P3P (Perspective-3-Points) problem. The hand position estimation unit 218 may solve P3P by a direct solution. The hand position estimation unit 218 may solve P3P by performing optimization calculation by a non-direct solution using the position obtained in the previous frame as an initial value. When solving P3P by a direct solution, a maximum of four solutions may be derived, and it is preferable to select a plausible solution from the derived real solutions based on the inclination of the three points or the distance from the HMD 200 (main camera), etc. It is desirable that the three joint points used in this calculation are points whose distances (lengths) between each point are unlikely to change due to deformation of the hand.

[0035] In addition, when the main detection unit 214 determines the inclination of the hand or the relative positional relationship between the joint points of the hand using a detector trained by deep learning (DL) or the like, a method using the distance between two joint points can be considered. By making the depth of one of the two points an unknown quantity, a quadratic equation can be formulated from the geometric positional relationship, so that a direct solution method can be used. It is desirable that the two points used in this calculation are points whose length is unlikely to change due to deformation of the hand.

[0036] The synthesis unit 219 synthesizes a main image (or a sub-image) obtained by capturing a real space with a CG image acquired from the CG rendering unit 213. The synthesis unit 219 generates a display image through this synthesis.

[0037] The process of estimating the position of the hand will be described with reference to the flowchart in Fig. 3. The process of the flowchart in Fig. 3 is executed every frame (or at a specific cycle) while the HMD 200 is running, by the control unit 217 of the information processing device 100 executing a program. In the following, the process executed for "one hand" of the two hands, left and right, will be described, and this "one hand" will be referred to as the "target hand."

[0038] In step S301, control unit 217 controls main detection unit 214 to detect the "target hand" in the main image and detect the coordinates of the "target hand" in the main image.

[0039] In step S302, the control unit 217 determines whether a specific time T1 has elapsed since the most recent process of step S304 (previous detection of the coordinates of the "target hand" in the sub-image) was performed. If it is determined that the specific time T1 has elapsed since the most recent execution of the process of step S304, the process proceeds to step S303. If it is determined that the specific time T1 has not elapsed since the most recent execution of the process of step S304, the process proceeds to step S306.

[0040] In step S303, the control unit 217 judges whether the "target hand" is stationary. Specifically, it judges whether the coordinates of the "target hand" in the main image detected by the main detection unit 214 have changed. For example, when the control unit 217 detects a plurality of joint points of the "target hand", it obtains the square of the difference between the detection coordinates of the previous frame and the detection coordinates of the current frame for each of the detected joint points as the detection difference. Then, the control unit 217 can judge whether the coordinates of the "target hand" have changed depending on whether the average value of the detection differences of all the detected joint points is equal to or less than a certain value. If it is judged that the coordinates of the "target hand" have not changed (the "target hand" is stationary), the process proceeds to step S304. If it is judged that the coordinates of the "target hand" have changed (the "target hand" is not stationary), the process proceeds to step S306.

[0041] In step S304, the control unit 217 controls the sub-detection unit 215 to detect the "target hand" in the sub-image and to detect the coordinates of the "target hand" in the sub-image.

[0042] In step S305, control unit 217 controls update unit 216 to update the reference information of the "target hand" stored in RAM 102 or ROM 103, based on the coordinates of the "target hand" in the main image and the coordinates of the "target hand" in the sub-image. Details of the process in step S305 will be described later with reference to the flowchart in FIG.

[0043] In step S306, control unit 217 controls hand position estimation unit 218 to estimate the three-dimensional position of the "target hand" based on coordinate information of the "target hand" in the main image and reference information stored in RAM 102 or ROM 103. Specifically, hand position estimation unit 218 estimates (acquires) the relative position of the "target hand" with respect to HMD 200 (main camera).

[0044] In this way, if it is determined in step S302 that a specific time T1 (a fixed time) has not elapsed since the immediately preceding step S304, the process proceeds to step S306. Also, if it is determined in step S303 that the hand is not stationary, the process proceeds to step S306. That is, in these cases, the detection of the coordinates of the "target hand" in the sub-image and the update of the reference information are not performed, and therefore the imaging process by the sub-camera and the calculation process of the reference information are not required. Therefore, according to the process of the flowchart in FIG. 4, the processing load can be reduced and the processing delay can be reduced compared to the case where imaging is always performed by stereo when stationary.

[0045] 4 is a flowchart for explaining the details of the process of step S305. Here, an example will be explained in which "lengths between a plurality of joint points of the hand" are used as "reference information."

[0046] In step S401, the update unit 216 determines whether or not the "target hand" is detected from both the main image and the sub-image. If the "target hand" is detected from both images, the process proceeds to step S402. If the "target hand" is not detected from at least one of the two images, the process of this flowchart ends.

[0047] Here, in steps S402 and S403, processing is performed based on the positions of each joint point in the main image and the sub-image. For this reason, in step S401, whether or not the "target hand" is detected from both the main image and the sub-image is determined based on whether or not a predetermined number of "target hands" are detected. In other words, even if the "target hand" is detected in both the main image and the sub-image, if the "target hand" does not include a predetermined number of joint points or more in both images, the process of this flowchart may end.

[0048] The processes in steps S402 and S403 are executed for each pair of joint points of the “target hand.” Here, the two joint points that are the targets of the processes in steps S402 and S403 are referred to as the “first joint point” and the “second joint point.”

[0049] In step S402, the update unit 216 determines whether the difference between the coordinate (detected coordinate) of the first joint point in the main image in the up-down direction (vertical direction) and the coordinate (detected coordinate) of the first joint point in the sub-image in the up-down direction is equal to or less than a certain value Th1. The update unit 216 also determines whether the difference between the coordinate (detected coordinate) of the second joint point in the main image in the up-down direction (vertical direction) and the coordinate (detected coordinate) of the second joint point in the sub-image in the up-down direction is equal to or less than a certain value Th1. If the difference between the coordinates in the up-down direction for both of the two joint points is equal to or less than the certain value Th1, the process proceeds to step S403. If the difference between the coordinates in the up-down direction for at least one of the two joint points exceeds the certain value Th1, the process for the two joint points (pair of joint points) is terminated, and the process of step S402 is performed for the next pair of joint points. That is, in this case, the "information on the distance between the first joint point and the second joint point" in the reference information is not updated.

[0050] Here, the main image and the sub-image are subjected to stereo rectification processing. Therefore, if a certain joint point is correctly detected, the vertical coordinate of the joint point should be the same in the main image and the sub-image. Therefore, if there is a large difference in the vertical coordinate of the same joint point in two images, it is expected that the detection error of the joint point will be large and the reference information cannot be updated with sufficient accuracy.

[0051] FIG. 5 shows an example of detection of hand joint points in a main image and a sub-image. A right hand is shown in both main image 500 and sub-image 510, and a joint point at the base of the middle finger and a joint point at the wrist are detected. A detection position 501 of the joint point at the base of the middle finger in main image 500 and a detection position 511 of the joint point at the base of the middle finger in sub-image 510 have the same coordinate in the vertical direction. Therefore, it can be said that the detection of the coordinate of the joint point at the base of the middle finger is highly reliable. On the other hand, a detection position 502 of the wrist joint point in main image 500 and a detection position 512 of the wrist joint point in sub-image 510 have significantly different coordinates in the vertical direction. Therefore, there is a possibility that the detection error of the wrist joint point is large.

[0052] In step S403, the update unit 216 calculates the difference in coordinates of the first joint point between the main image and the sub-image (hereinafter referred to as the "first difference"), and the difference in coordinates of the second joint point between the main image and the sub-image (hereinafter referred to as the "second difference"). The update unit 216 also reads out the camera parameters of the main camera and the sub-camera stored in ROM 103. Then, based on the first difference, the second difference, and the camera parameters, the update unit 216 calculates the distance from each camera to the first joint point (hereinafter referred to as the "first distance") and the distance from each camera to the second joint point (hereinafter referred to as the "second distance").

[0053] Thereafter, the update unit 216 calculates three-dimensional coordinates (coordinates based on the position and orientation of the HMD 200) of the first joint point and the second joint point based on the first distance, the second distance, and the coordinates of the two joint points in the main image (the coordinates of the first joint point and the coordinates of the second joint point). The update unit 216 calculates the distance (Euclidean distance) between the first joint point and the second joint point in real space based on the three-dimensional coordinates of the first joint point and the second joint point. The update unit 216 uses the information on the distance between the first joint point and the second joint point as one piece of reference information. and overwrites and saves it in RAM 102 or ROM 103. That is, the update unit 216 updates "information on the distance between the first joint point and the second joint point" of the reference information to the newly calculated "information on the distance between the first joint point and the second joint point."

[0054] According to the first embodiment, the position of the hand, which is the detection target, can be estimated without using a distance sensor. Also, since the hand position can be estimated without using a camera other than a stereo camera, the information processing device can estimate the hand position with a small processing load. Furthermore, since the reference information is updated (acquired) less frequently than the hand position is estimated, the processing load of the information processing device due to the use of the sub-image can also be reduced.

[0055] <Variation 1> In the first embodiment, the process of estimating the position of one of the left and right hands has been described. However, when two hands, one left and one right, are captured in the main image and the sub-image, the flowcharts in Fig. 3 and Fig. 4 may be executed for each of the two hands.

[0056] <Embodiment 2> In the first embodiment, every time information processing device 100 calculates (acquires) reference information of the hand, it updates the reference information of the hand stored in RAM 102 or ROM 103. Here, if an error occurs in the detection result of the coordinates of the hand in the main image or the sub-image, an error also occurs in the reference information, and further, there is a possibility that an error occurs in the estimation of the position of the hand.

[0057] Therefore, in the second embodiment, the information processing device 100 reduces the influence of detection errors on the reference information and the estimation of the hand position by using an average of the distance between two joint points, etc., calculated in multiple frames.

[0058] The flowchart in Fig. 6 shows the process of step S305 according to the second embodiment instead of the flowchart in Fig. 4. Here, an example in which the length between joint points is used as reference information will be described.

[0059] The processes of steps S401 and S402 are similar to those of steps S401 and S402 according to embodiment 1. Note that the processes of steps S402, S603, and S604 are executed for each pair of joint points of the "target hand" as in embodiment 1. Here, the two joint points targeted in the processes of steps S302, S603, and S604 are referred to as the "first joint point" and the "second joint point."

[0060] In step S603, the update unit 216 calculates information on the distance between the first joint point and the second joint point as "joint distance information" by the same method as in step S403 according to the first embodiment. Then, the update unit 216 adds the joint distance information (information on the distance between the first joint point and the second joint point) to the reference information queue. The reference information queue has a data structure capable of storing the calculated joint distance information multiple times (a predetermined number of times) for each pair of joint points. The reference information queue is stored in the RAM 102 or the ROM 103. Note that, when more than a predetermined number of pieces of joint distance information (a predetermined number of frames) are stored in the reference information queue, the update unit 216 deletes the oldest piece of joint distance information from the reference information queue.

[0061] In step S604, the update unit 216 obtains the average value of the multiple pieces of joint distance information registered in the reference information queue. Then, the update unit 216 updates the information on the distance between the first joint point and the second joint point among the reference information stored in the storage unit (RAM 102 or ROM 103) to the average value of the multiple pieces of joint distance information.

[0062] According to the second embodiment, by using an average of a plurality of pieces of joint distance information, it is possible to stabilize the variation due to measurement errors and stabilize the accuracy of the estimation of the hand position.

[0063] <Embodiment 3> In the third embodiment, the information processing device 100 minimizes the load by updating the reference information only when the user wearing the HMD 200 is changed.

[0064] The hand position estimation process according to the third embodiment will be described with reference to the flowchart of Fig. 7. This process is executed for each frame while the HMD 200 is running by the control unit 217 of the information processing device 100. Note that the processes of steps S301 and S303 to S306 in Fig. 7 are similar to the processes of steps S301 and S303 to S306 in Fig. 3, and therefore detailed description thereof will be omitted.

[0065] In step S701, the control unit 217 determines whether or not the user (wearer) wearing the HMD 200 has changed. If it is determined that the wearer of the HMD 200 has changed, the process proceeds to step S702. If it is determined that the wearer of the HMD 200 has not changed, the process proceeds to step S301.

[0066] The control unit 217 detects that the wearer of the HMD 200 has changed, for example, when the object detection state of a proximity sensor mounted on the HMD 200 switches from a non-proximity state to a proximity state. Alternatively, the control unit 217 identifies an individual through iris authentication using a camera mounted on the HMD 200 that captures an image of the wearer's pupils. In this way, the control unit 217 may detect that the wearer of the HMD 200 has changed.

[0067] In step S702, the control unit 217 controls the update unit 216 to set the state of the reference information to an unupdated state, and updates (rewrites) the reference information stored in the RAM 102 or ROM 103 to an initial value.

[0068] In step S704, the control unit 217 judges whether the state of the reference information is unupdated information. If it is judged that the state of the basic information is unupdated, the process proceeds to step S303. If it is judged that the state of the basic information is updated, the process proceeds to step S708.

[0069] In step S708, the control unit 217 determines whether the update unit 216 updated the reference information in step S305. If it is determined that the reference information has been updated, the process proceeds to step S709. If it is determined that the reference information has not been updated, the process proceeds to step S306.

[0070] In step S709, the control unit 217 sets the state of the reference information to "updated" and stores information on the state of the reference information in the RAM 102.

[0071] According to the third embodiment, the information processing device 100 updates the reference information only when there is a change in the wearer of the HMD 200. This can reduce the processing load on the information processing device 100.

[0072] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0073] In the above, "If A is equal to or greater than B, proceed to step S1. If A is smaller than B, proceed to step S2. "If A is greater (higher) than B, proceed to step S1, and if A is less than or equal to B, proceed to step S2." may be read as "If A is greater (higher) than B, proceed to step S1, and if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is less than or equal to B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is less than or equal to B, proceed to step S2." For this reason, unless a contradiction arises, "A or greater" may be read as "greater (higher; longer; more) than A," and "less than or equal to A" may be read as "smaller (lower; shorter; less) than A." Furthermore, "greater (higher; longer; more) than A" may be read as "A or greater," and "smaller (lower; shorter; less) than A" may be read as "A or less."

[0074] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0075] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0076] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) a first detection means for detecting a detection target in a first image acquired from a first imaging device; an update means for updating reference information related to the detection target stored in a specific storage means based on the first image and a second image acquired from a second imaging device in a first case, and not updating the reference information stored in the specific storage means in a second case; an estimation means for estimating a relative position of the detection target with respect to the first imaging device based on a detection result of the detection target by the first detection means and the reference information stored in the specific storage means; 13. An information processing device comprising: (Configuration 2) the first detection means acquires a new first image from the first imaging device at a constant cycle and detects the detection target in the acquired first image; The update means, in the first case, updates the reference information every time the first detection means detects the detection target; the estimation means estimates a relative position of the detection target with respect to the first imaging device every time the first detection means detects the detection target; 2. The information processing device according to configuration 1. (Configuration 3) Further comprising a second detection means for detecting the detection target in the second image in the first case. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The update means, in the first case, updates the reference information based on a detection result of the detection target by the first detection means and a detection result of the detection target by the second detection means. 4. The information processing device according to configuration 3. (Configuration 5) The second case includes a case where a certain time has not elapsed since the second detection means previously detected the detection target in the second image. 5. The information processing device according to configuration 3 or 4. (Configuration 6) The apparatus further includes a first determination means for determining whether the detection target is stationary or not based on a detection result of the detection target by the first detection means, The second case includes a case where it is determined that the detection target is not stationary. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The information processing device further includes a second determination means for determining whether a user who uses the information processing device has changed, The case of 2 includes a case where it is determined that the user of the information processing device has not been changed. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) In the first case, the update means acquires information about the detection target based on the first image and the second image, and updates the reference information to an average of the information about the detection target acquired multiple times. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The detection target is a hand. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 10) the detection result of the detection object indicates coordinates of each joint point of the hand; The reference information is information on the length between a plurality of joint points. 10. The information processing device according to configuration 9. (Configuration 11) The reference information includes information on a length between a first joint point and a second joint point, when at least one of the first difference and the second difference is greater than a certain value, even in the first case, the update means does not update information on the length between the first joint point and the second joint point in the reference information, the first difference is a difference between a vertical position of the first joint point in the first image and a vertical position of the first joint point in the second image; the second difference is a difference between a vertical position of the second joint point in the first image and a vertical position of the second joint point in the second image; 11. The information processing device according to configuration 10. (Configuration 12) In the first case, the update means measuring a depth direction position of the detection target relative to the images based on the first image and the second image; updating the reference information based on the position in the depth direction and a detection result of the detection target by the first detection means; 12. The information processing device according to any one of configurations 1 to 11. (method) A first detection step of detecting a detection target in a first image acquired from a first imaging device. P and an updating step of updating, in a first case, reference information related to the detection target stored in a specific storage means based on the first image and a second image acquired from a second imaging device, and not updating the reference information stored in the specific storage means in a second case; an estimation step of estimating a relative position of the detection target with respect to the first imaging device based on a detection result of the detection target in the first detection step and the reference information stored in the specific storage means; 13. An information processing method comprising: (program) 13. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0077] 100: information processing device, 214: main detection unit, 216: Update section, 218: Hand position estimation section

Claims

1. a first detection means for detecting a detection target in a first image acquired from a first imaging device; an update means for updating reference information related to the detection target stored in a specific storage means based on the first image and a second image acquired from a second imaging device in a first case, and not updating the reference information stored in the specific storage means in a second case; an estimation means for estimating a relative position of the detection target with respect to the first imaging device based on a detection result of the detection target by the first detection means and the reference information stored in the specific storage means; 13. An information processing device comprising:

2. the first detection means acquires a new first image from the first imaging device at a constant cycle and detects the detection target in the acquired first image; The update means, in the first case, updates the reference information every time the first detection means detects the detection target; the estimation means estimates a relative position of the detection target with respect to the first imaging device every time the first detection means detects the detection target; 2. The information processing apparatus according to claim 1,

3. Further comprising a second detection means for detecting the detection target in the second image in the first case.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. The update means, in the first case, updates the reference information based on a detection result of the detection target by the first detection means and a detection result of the detection target by the second detection means.

4. The information processing apparatus according to claim 3.

5. The second case includes a case where a certain time has not elapsed since the second detection means previously detected the detection target in the second image.

4. The information processing apparatus according to claim 3.

6. a first determination means for determining whether the detection target is stationary or not based on a detection result of the detection target by the first detection means, The second case includes a case where it is determined that the detection target is not stationary.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

7. The information processing device further includes a second determination means for determining whether a user who uses the information processing device has changed, The second case includes a case where it is determined that the user of the information processing device has not been changed.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

8. In the first case, the update means acquires information about the detection target based on the first image and the second image, and updates the reference information to an average of information about the detection target acquired multiple times.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

9. The detection target is a hand.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

10. the detection result of the detection object indicates coordinates of each joint point of the hand; The reference information is information on the length between a plurality of joint points.

10. The information processing apparatus according to claim 9,

11. The reference information includes information on a length between a first joint point and a second joint point, when at least one of the first difference and the second difference is greater than a certain value, even in the first case, the update means does not update information on the length between the first joint point and the second joint point, which is included in the reference information; the first difference is a difference between a vertical position of the first joint point in the first image and a vertical position of the first joint point in the second image; the second difference is a difference between a vertical position of the second joint point in the first image and a vertical position of the second joint point in the second image; 11. The information processing apparatus according to claim 10,

12. In the first case, the update means measuring a depth direction position of the detection target relative to the images based on the first image and the second image; updating the reference information based on the position in the depth direction and a detection result of the detection target by the first detection means; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

13. a first detection step of detecting a detection target in a first image acquired from a first imaging device; an updating step of updating, in a first case, reference information related to the detection target stored in a specific storage means based on the first image and a second image acquired from a second imaging device, and not updating the reference information stored in the specific storage means in a second case; an estimation step of estimating a relative position of the detection target with respect to the first imaging device based on a detection result of the detection target in the first detection step and the reference information stored in the specific storage means; 13. An information processing method comprising:

14. A program for causing a computer to function as each of the means of the information processing device according to claim 1 or 2.

Images