Information processing apparatus and hmd

Abstract

To provide a technique which allows for recognizing a hand of a user himself / herself from among hands included in a visual field of an HMD with simple processing and high accuracy.SOLUTION: An information processing apparatus for recognizing a hand of a user wearing an HMD from in an image captured by the HMD includes: detection means which detects a hand from a captured image captured by the HMD; acquisition means which acquires, from a sensor mounted on a device that the user puts on his / her hand, sensing data about a position of the device; and processing operation means which, with respect to the hand detected from the captured image, performs comparison processing of comparing movement of the hand recognized from the captured image with movement of the device recognized from the sensing data and determines that the hand detected from the captured image is the hand of the user in the case of coincidence between the movement of the hand and the movement of the device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technology for recognizing a user's hands contained within the field of view of an HMD. [Background technology]

[0002] Mixed Reality (MR) technology fuses real and virtual spaces, allowing users to interact with virtual objects. MR technology enables interaction by synthesizing and presenting computer graphics (CG) that represent virtual objects with real scenery, and by expressing contact between real and virtual objects.

[0003] With MR technology, it is expected that users will use gestures with their own hands to move virtual objects in real-world scenes. Gesture manipulation allows users to move and manipulate CG objects without using a controller. However, if there are multiple people other than the user in the same space, it is possible that users will be unable to distinguish between their own hands and others' hands, and their own HMD (Head Mounted Display) may be unintentionally operated by gestures from other people's hands.

[0004] Technologies relating to object recognition in three-dimensional space are disclosed in Patent Documents 1 and 2. Patent Document 1 discloses a method for recognizing human movements using images and motion sensors and identifying a target person based on the characteristics of the movements. Patent Document 2 discloses a method for correctly expressing the front-to-back relationship between a real object and a virtual object by measuring the depth position of a real object such as a hand using a stereo camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2015-61577 A [Patent Document 2] JP 2012-13514 A Summary of the Invention [Problem to be solved by the invention]

[0006] The conventional technology disclosed in Patent Document 1 learns the movements of a person to be identified in advance, and judges whether a person currently moving is the person to be identified by comparing the learned features with features extracted from a person moving within a specified area. However, such a learning-based method requires advance preparation such as registering the movements of the person, which is cumbersome. In addition, there is the inconvenience that people whose movements have not been registered cannot use the system.

[0007] The present invention has been made in consideration of the above-mentioned situation, and its object is to provide technology that enables the user's hand to be recognized with simple processing and high accuracy from among the hands contained within the field of view of the HMD. [Means for solving the problem]

[0008] The present disclosure relates to an information processing device that recognizes the hand of a user wearing a head mounted display (HMD) from an image captured by the HMD, the information processing device including a detection means for detecting a hand from an image captured by the HMD, an acquisition means for acquiring sensing data relating to a position of a device from a sensor mounted on the device worn by the user, and a comparison process for comparing the movement of the hand detected from the captured image with the movement of the device recognized from the sensing data. and a processing calculation means for determining that the hand detected from the captured image is the hand of the user if the movement of the hand matches the movement of the device. Effect of the Invention

[0009] According to the present invention, the user's own hand can be recognized with a high degree of accuracy from among the hands contained within the field of view of the HMD through simple processing. [Brief description of the drawings]

[0010] [Figure 1] Block diagram showing the configuration of a mixed reality system [Diagram 2] Block diagram showing the hardware configuration of the HMD [Diagram 3] Block diagram showing the configuration of a small operating device [Figure 4] FIG. 1 shows a usage mode of a mixed reality system. [Diagram 5] 1 is a flowchart of an operation target recognition process according to a first embodiment; [Figure 6] A diagram showing an example of movement of a small operating device [Figure 7] A diagram showing an example of hand positions detected by a small operating device and an HMD. [Figure 8] A diagram showing an example of hand movement in an HMD image. [Figure 9] Flowchart of operation target recognition processing in the second embodiment [Figure 10] Flowchart of operation target recognition processing in the third embodiment [Figure 11] Flowchart of operation target recognition processing in the fourth embodiment [Figure 12] Flowchart of operation target recognition processing in the fifth embodiment [Figure 13] Flowchart of operation target recognition processing in the sixth embodiment [Figure 14] Flowchart of operation target recognition processing in the seventh embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is one example of a means for realizing the present invention, and may be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. In addition, each embodiment may be appropriately combined.

[0012] (First embodiment) The configuration of a mixed reality system according to a first embodiment will be described with reference to Figs. 1 to 4. Fig. 1 is a block diagram showing the overall configuration of the mixed reality system according to the first embodiment. Fig. 2 is a block diagram showing the hardware configuration of an HMD. Fig. 3 is a block diagram showing the configuration of a small-sized operating device. Fig. 4 is a diagram showing a usage mode of the mixed reality system.

[0013] The mixed reality system 1 is composed of an HMD (Head Mount Display) 100 and a small operating device 300. The HMD 100 is a device worn on the head of a user, and is composed of a goggle device 100A and an information processing device 103. The small operating device 300 is a device worn on the hand of the user. Fig. 4 shows an example of a state in which the HMD 100 and the small operating device 300 are worn by a user.

[0014] The CPU 200 is a processor that controls the entire system by executing a program for operating the system. The processing units 104 to 109 shown in Fig. 1 are realized in software form by the CPU 200 executing a program read from the ROM 202. However, some of these processing units may be realized by hardware (e.g., ASIC or FPGA) independent of the CPU 200. A processor (GPU, DSP, etc.) may be provided to assist the arithmetic processing of the CPU 200.

[0015] The goggle device 100A of the HMD 100 is a head-mounted display device having an image capturing unit 101 and an image display unit 102. As shown in the schematic diagram of FIG. 4, the HMD 100 of this embodiment uses a video see-through method in which the image capturing unit 101 is disposed at the user's viewpoint position and the real-life image captured by the image capturing unit 101 is displayed on the image display unit 102. In the case of the HMD 100 of the video see-through method, a method in which the image capturing unit 101 is treated as the viewpoint position and posture of the user is generally used. Note that although the HMD 100 of this embodiment uses a video see-through method, this should be understood as merely an example. For example, the present invention can also be applied to an HMD for virtual reality in which the real-life image is not displayed on the image display unit 102. In that case, the image capturing unit 101 may be used as a camera for capturing an image of a hand, which is a measurement target, rather than as a see-through camera.

[0016] The information processing device 103 of the HMD 100 creates data to be displayed on the image display unit 102 using an image captured by the image capturing unit 101. The information processing device 103 is a small computer equipped with a processor (CPU 200) and memory (RAM 201, ROM 202) shown in FIG. 2. The information processing device 103 may be built in the same housing as the goggle device 100A, or may be configured in a housing independent of the goggle device 100A. When the information processing device 103 and the goggle device 100A are configured in independent housings, the information processing device 103 and the goggle device 100A are connected to each other by wire or wirelessly so that they can communicate with each other.

[0017] The image capturing unit 101 is fixed to the housing of the goggle device 100A, and captures images of real space at a predetermined frame rate (for example, 30 frames / second). Images captured by the image capturing unit 101 are stored in a captured image storage unit 104. The captured image storage unit 104 stores images acquired from the image capturing unit 101 in a RAM 201. This image capturing unit 101 is composed of two real cameras, and a real camera for the left eye and a real camera for the right eye are disposed in positions close to both eyes of the HMD wearer. Hereinafter, a pair of left and right images captured by the image capturing unit 101 will be referred to as stereo camera images.

[0018] The image display unit 102 is fixed to the housing of the goggle device 100A, and displays an image generated by the information processing device 103. This image is, for example, an image for MR in which a real-life video captured by the image capturing unit 101 is combined with digital content (virtual object) by 3DCG. The image display unit 102 has a structure that covers the field of view of a user wearing the goggle device 100A, and can provide a highly immersive MR experience to a user viewing the image. The image display unit 102 is, for example, configured with an organic EL display, a liquid crystal display, or the like.

[0019] The hand detection unit 105 detects hands from each captured image stored in the captured image storage unit 104 by image recognition, and extracts contour points of the fingers. The three-dimensional position calculation unit 106 calculates the three-dimensional position of the contour points extracted by the hand detection unit 105. The x and y coordinates of the contour points are obtained by taking the bottom left of the VRAM area of ​​the image captured by the image capture unit 101 of the HMD 100 as (0,0). The z coordinate in the depth direction is obtained from the stereo camera image. This coordinate system is called a camera coordinate system or a local coordinate system. The origin of the camera coordinate system does not have to be the bottom left of the VRAM area, and may be freely determined. In this embodiment, the hand detection unit 105 (or the hand detection unit 105 and the three-dimensional position calculation unit 106) constitutes a detection means for detecting hands from captured images captured by the HMD 100.

[0020] As an algorithm for detecting hands from an image, a classical machine learning algorithm such as a support vector machine may be used, or an algorithm based on deep learning such as R-CNN, YOLO, SSD, or DCN may be used. A rule-based detection algorithm may also be used.

[0021] In order to correctly display the depth of the detected hand, there is a method of extracting the hand area from the image of the hand captured by a stereo camera and calculating the depth value of the hand relative to the stereo camera. This method uses a method of finding the depth value by triangulation for all corresponding points of the extracted hand outline in the stereo image (Patent Document 2). Patent Document 2 uses a stereo camera mounted on a video see-through type HMD, which is a display device for presenting mixed reality.

[0022] LeapMotion, Inc.'s LeapMotion, can measure the position and orientation of the hand, including the fingers. LeapMotion can detect the area of ​​the hand from a stereo camera (image capture unit 101). Since the position and orientation of the fingers can also be estimated by using a separate depth sensor, a depth sensor may be installed in the HMD 100.

[0023] In this embodiment, one arbitrary point (called a reference point or representative point) is determined from the contour points extracted by hand detection unit 105, and a three-dimensional position relative to the reference point is calculated. The reference point determined here is stored as a reference position during this processing, and the reference three-dimensional position information is periodically stored in RAM 201 in accordance with the movement of the fingers.

[0024] The relative position and orientation calculation unit 107 calculates the position and orientation of each captured image stored by the captured image storage unit 104. The position and orientation can be calculated using a method of creating the position and orientation using deep learning or a method of using an existing public library. The position and orientation of the captured image correspond to the position and orientation of the HMD 100 in real space, that is, the position of the viewpoint and the line of sight of the user wearing the HMD 100. The position and orientation of the captured image are expressed in a global coordinate system.

[0025] The processing calculation unit 109 performs a comparison calculation between the movement of the hand recognized based on the three-dimensional position information calculated from the captured image and the movement of the small-sized operating device 300 recognized based on the sensing data acquired from a sensor 303 mounted on the small-sized operating device 300 described later. Specifically, the processing calculation unit 109 evaluates whether the relative movement amounts of the coordinates of the hand detected by the HMD 100 and the small-sized operating device 300 match. If they match, the processing calculation unit 109 determines that the hand detected by the HMD 100 and the hand on which the small-sized operating device 300 is worn are the same hand, that is, the hand of the user of the HMD 100. Generally, since an error occurs in the coordinates acquired by the sensor, a certain error rate is allowed, for example, an error rate of up to 5% is determined to be the same. The error rate may be freely determined.

[0026] When raw data (unprocessed data) of the sensor is sent from the small-sized operating device 300 to the information processing device 103, the processing calculation unit 109 may convert the raw data into coordinate values ​​so that the raw data can be compared with the values ​​calculated by the three-dimensional position calculation unit 106. Alternatively, the small-sized operating device 300 may convert the raw data into coordinate values ​​and then transmit the coordinate values ​​to the information processing device 103.

[0027] The image drawing unit 108 draws (generates) an image to be displayed on the image display unit 102 of the HMD 100 based on the processing result of the processing calculation unit 109 .

[0028] The near-field communication unit 110 is composed of, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller. The near-field communication unit 110 outputs modulated wireless signals from the antenna and demodulates wireless signals received by the antenna to realize near-field wireless communication in accordance with the IEEE802.15 standard (so-called Bluetooth (registered trademark)). In this embodiment, the Bluetooth (registered trademark) communication employs version 5.1 of Bluetooth (registered trademark) Low Energy, which has low power consumption. The HMD 100 communicates with a small operating device 300 and the like via the near-field communication unit 110. It is possible to exchange data with any external device.

[0029] The RAM 201 is a volatile memory used as a working area for temporarily storing data and programs. The ROM 202 is a non-volatile memory for non-temporarily storing data such as programs executed by the CPU 200, setting information for the HMD 100, and setting information required for the next and subsequent uses.

[0030] As shown in Fig. 3, the small-sized operating device 300 includes a control unit 301, an operating unit 302, a sensor 303, a working memory 304, a power supply control unit 305, and a communication unit 306. As shown in Fig. 4, the small-sized operating device 300 is a wearable device that can be worn on the hand or finger of the user, and a ring-shaped device is used in this embodiment. However, the form of the small-sized operating device 300 is not limited to a finger-worn type, and any type of device may be used as long as it can detect the position and posture of the user's own hand. For example, a wristwatch-type device, a device that is held in the hand, or a device that is attached to the finger may be used.

[0031] The control unit 301 controls the small operating device 300 according to input signals and a program described later. Note that instead of the control unit 301 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

[0032] The working memory 304 is used as a buffer memory for temporarily storing data acquired by the sensor 303, a working area for the control unit 301, and the like.

[0033] The sensor 303 is a sensor for measuring the three-dimensional position, movement, and posture (direction) of the hand of a user wearing the small operating device 300, and is also called a position sensor, a movement sensor, or a posture sensor. For example, an IMU (Inertial Measurement Unit) sensor can be used as the sensor 303. The IMU sensor can calculate posture change, relative direction, and position based on the angular velocity detected by a gyro and the acceleration detected by an accelerometer.

[0034] The operation unit 302 is used to receive instructions from the user for the small-sized operating device 300. The operation unit 302 includes, for example, a power button for instructing ON / OFF of a main power supply including power supply to the control unit 301 of the small-sized operating device 300, and a mode switching button for switching function modes. Furthermore, the operation unit 302 includes a button for starting communication with an external device such as the HMD 100 via the communication unit 306.

[0035] The power supply control unit 305 is a unit for supplying power for the small-sized operating device 300 to operate.

[0036] The communication unit 306 is composed of, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller, similar to the close-proximity wireless communication unit 110 of the HMD 100. The communication unit 306 realizes short-distance wireless communication in accordance with the IEEE802.15 standard (so-called Bluetooth (registered trademark)). The small-sized operating device 300 can exchange data with an external device such as the HMD 100 via the communication unit 306.

[0037] 1, the near-field communication unit 110 of the HMD 100 and the small-sized operating device 300 are connected by Bluetooth (registered trademark) communication. For example, various sensing data acquired by a sensor 303 can be transmitted to the near-field communication unit 110 of the HMD 100 via a communication unit 306 of the small-sized operating device 300. In this embodiment, the near-field communication unit 110 constitutes an acquisition means that acquires sensing data related to the position and attitude of the small-sized operating device 300 from the sensor 303 mounted on the small-sized operating device 300. The data communication between the HMD 100 and the small-sized operating device 300 is not limited to Bluetooth (registered trademark) communication, and may be other communication methods.

[0038] 5 is a flowchart of an operation target recognition process executed in the HMD 100 according to the first embodiment. The operation target recognition process is a process for recognizing the hand of the user wearing the HMD 100 (i.e., the hand (operation target) of the person performing the gesture operation) from one or more hands (targets) included in an image captured by the HMD 100 (i.e., the field of view of the HMD 100). While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the operation target recognition process of FIG. 5 is repeatedly executed in a predetermined cycle in the HMD 100.

[0039] In step S500, the CPU 200 (processing and calculation unit 109) of the HMD 100 judges whether or not there has been a change in the position of the small-sized operating device 300 based on the sensing data acquired from the small-sized operating device 300. The presence or absence of a change in the position can be judged, for example, by comparing sensing data for a plurality of times (i.e., time-series data of the position). Since the position coordinates measured by the sensor 303 have a certain error, if the change in the position is slight, it may be regarded as "no change in the position". If a change in the position of the small-sized operating device 300 is detected in step S500, the CPU 200 assumes that the user of the HMD 100 is performing a gesture operation using the hand wearing the small-sized operating device 300, and proceeds to the process of step S501. On the other hand, if a change in the position of the small-sized operating device 300 is not detected, the CPU 200 assumes that a gesture operation is not being performed, and skips the subsequent processes (recognition of the user's hand by the image of the HMD 100, gesture recognition, etc.). In this way, by not executing subsequent processing when the small operating device 300 does not detect the user's hand movement, it is possible to reduce unnecessary calculation processing and save power consumption, and to assure that the risk of mistakenly recognizing someone else's hand is eliminated as much as possible.

[0040] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 acquires an image captured by the image capturing unit 101 from the captured image storage unit 104, and detects hands from the captured image. If the captured image includes multiple hands, all of the hands are set as targets for determining whether they are the user's own hands or not. Next, the CPU 200 (three-dimensional position calculation unit 106) calculates the three-dimensional position coordinates of the outline point of each of the targets. The captured image used here is an image captured at the same time as the sensing data referenced in step S500 (or the closest time if there is no image captured at exactly the same time).

[0041] If one or more hands to be judged are detected, the hands to be judged are selected in order in step S502, and the process of step S503 is executed for the selected hands to be judged. Since it is unclear which hand in the image is the user's own hand, they are judged one by one. If no hands to be judged are detected, or if the process of step S503 has been executed for all hands to be judged, the result of step S502 is NO, and the process ends.

[0042] In step S503, the CPU 200 (processing and calculation unit 109) of the HMD 100 selects one contour point (reference point) to serve as a reference from the contour points of the hand to be judged. Next, the processing and calculation unit 109 converts the coordinate value of the reference point from the camera coordinate system to the global coordinate system, taking into account the position and orientation of the captured image calculated by the relative position and orientation calculation unit 107. The processing and calculation unit 109 then evaluates whether or not the movement of the hand to be judged (i.e., the change in the position of the reference point) matches the movement of the small-sized operating device 300 (i.e., the change in the position of the small-sized operating device 300). If it is determined that the movement of the hand to be judged matches the movement of the small-sized operating device 300 (i.e., the movement of the same object), the CPU 200 proceeds to step S504. If it is determined that there is no match, the CPU 200 returns to step S502 and evaluates the next target.

[0043] In step S504, the processing calculation unit 109 recognizes that the hand to be determined is the hand (operation target) of the user who wears the HMD 100. Thereafter, the movement information of the operation target is subjected to a gesture operation recognition process (not shown).

[0044] FIG. 6 is an example of the movement of the small-sized operating device 300 for each frame. The position of the small-sized operating device 300 changes from P1 (X1, Y1, Z1) to P2 (X2, Y2, Z2) to P3 (X3, Y3, Z3) in accordance with the movement of the hand. The position of the small-sized operating device 300 for each frame is measured by the sensor 303. Since the small-sized operating device 300 is attached (fixed) to the user's hand, the position of the small-sized operating device 300 measured by the sensor 303 can be regarded as the position of the user's hand. The table on the left side of FIG. 7 is an example of time-series data of the hand position P (i.e., the change in the hand position over time) measured and recorded by the sensor 303 of the small-sized operating device 300.

[0045] FIG. 8 shows an example of the movement of a hand for each frame contained in an image captured by the image capturing unit 101 of the HMD 100. The hand position changes from Q1 (x1, y1, z1) to Q2 (x2, y2, z2) to Q3 (x3, y3, z3). In the example of FIG. 8, the contour point of the tip of the index finger of the left hand is selected as the reference point indicating the hand position. The table on the right side of FIG. 7 shows an example of time-series data of the hand position Q (i.e., temporal changes in the hand position) detected from the image and recorded by the HMD 100. The time-series data of the hand position Q detected from the image is also called hand tracking information.

[0046] As shown in FIG. 7, the information processing device 103 of the HMD 100 acquires information on the hand position P detected by the small operating device 300 and information on the hand position Q detected from the image for each corresponding time t1, t2, .... In the process of step S503 in FIG. 5, the processing calculation unit 109 compares the relative movement amount of the hand position P detected by the small operating device 300 with the relative movement amount of the hand position Q detected from the image for a predetermined period. The relative movement amount is the relative change amount of the hand position in the current frame when the hand position in the previous frame is used as a reference. For example, the relative movement amount of the position P1 (X1, Y1, Z1) at time t1 and the position P2 (X2, Y2, Z2) at time t2 is expressed as (X2-X1, Y2-Y1, Z2-Z1). If the relative movement amount in the predetermined period is the same or within a predetermined error range, the processing calculation unit 109 determines that the hand to be determined is the hand of the user wearing the HMD 100. Here, the "predetermined period" used for comparison may be freely set. In this embodiment, it is set to about 3 to 10 frames. The "predetermined error range" may be set in consideration of the measurement error of the sensor 303 and the error of the three-dimensional position calculated from the image.

[0047] By using the relative movement amount for comparison, there is no need to match the position detected by the small-sized operating device 300 (the mounting position of the sensor 303) with the position of the reference point detected from the image, and there is an advantage that the processing is extremely simplified. For example, as shown in the examples of Fig. 6 and Fig. 8, even if the mounting position of the sensor 303 is the base of the index finger and the reference point is the tip of the index finger, the relative movement amount (see the dashed arrow) of both is almost the same, and simple comparison is possible. If the absolute position were to be compared, calibration would be required to precisely match the reference point detected from the image with the mounting position of the sensor 303 of the small-sized operating device 300, and measurement preparation would be complicated.

[0048] As described above, in this embodiment, the IMU information acquired from the sensor 303 of the small-sized operating device 300 and the hand tracking information acquired from the image captured by the image capturing unit 101 of the HMD 100 are used. Any hand that moves in the same way as the hand movement detected in step 00 is recognized as the hand of the user wearing the HMD 100. This makes it possible to identify the hand of the user with high accuracy through simple processing even if another person's hand is captured within the field of view of the HMD 100, thereby reducing erroneous gesture operation in MR. The method of this embodiment is highly convenient because it does not require learning the skeleton and movements of the user as in the conventional method, and does not require advance preparation such as alignment (calibration) between the HMD 100 and the small-sized operating device 300.

[0049] Second embodiment In the first embodiment, all hands detected by the HMD 100 are compared with the detection information of the small-sized operating device 300. However, in order to determine whether a hand is the user's hand, it is necessary to continue the comparison with each judgment target for a predetermined period of time, so if many hands are included in the field of view of the HMD 100, it may take a long time. In the second embodiment, when one or more hands are detected from the captured image, the hands to be compared with the detection information of the small-sized operating device 300 are narrowed down based on the distance from the HMD 100 to each hand. This makes it possible to reduce the number of candidates for hands to be compared with the detection information of the small-sized operating device 300, and shorten the comparison processing time. Hereinafter, detailed explanations of the same parts as in the first embodiment will be omitted, and the characteristic parts of the second embodiment will be mainly explained.

[0050] Fig. 9 is a flowchart of the operation target recognition process executed in the HMD 100 according to the second embodiment. The same processes as in the first embodiment are given the same step numbers as in the flowchart of Fig. 5. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the operation target recognition process of Fig. 9 is repeatedly executed in a predetermined cycle in the HMD 100.

[0051] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small-sized operating device 300 based on the sensing data acquired from the small-sized operating device 300. The processes from step S501 onward are performed only when the movement of the small-sized operating device 300 is detected.

[0052] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 acquires an image captured by the image capturing unit 101 from the captured image storage unit 104, and detects hands from the captured image. If the captured image contains multiple hands, all of the hands are set as objects to be determined. In step S502, the hands to be determined are selected in order, and the process of step S900 is executed for the selected objects to be determined.

[0053] In step S900, the CPU 200 (processing and calculation unit 109) of the HMD 100 calculates the distance D from the HMD 100 to the hand of the determination target, and compares the distance D with a threshold value ThD. The distance D can be acquired from depth information acquired from the stereo image described in the first embodiment. Alternatively, if the HMD 100 is equipped with a depth sensor, information measured by the depth sensor may be used. The threshold value ThD may be set to a value slightly larger than the reach (reach distance) of the user of the HMD 100, for example, a value of about 0.8 m to 1.0 m. The threshold value ThD is set in advance in the HMD 100.

[0054] If the distance D to the hand to be judged is equal to or greater than the threshold ThD (NO in step S900), the CPU 200 judges that the hand to be judged is not the hand of the user of the HMD 100 but the hand of another person. In this case, the comparison process (step S503) is not performed, and the process proceeds to the process of the next hand to be judged (step S502).

[0055] On the other hand, if the distance D to the hand to be judged is closer than the threshold ThD (YES in step S900), the CPU 200 judges that the hand to be judged may be the hand of the user of the HMD 100, and proceeds to a comparison process (step S503). The subsequent processes are the same as those described in the first embodiment.

[0056] The configuration of this embodiment described above can achieve the same effects as those of the first embodiment. In addition, in this embodiment, since candidates to be compared are narrowed down based on the distance from the HMD 100 to the hand, even if many hands are captured within the field of view of the HMD 100, the processing cost can be reduced, and the processing time can be shortened and power consumption can be suppressed.

[0057] (Third embodiment) In the first embodiment, all hands detected by the HMD 100 are compared with the detection information of the small-sized operating device 300. However, in order to determine that a hand is the user's hand, it is necessary to continue the comparison with each judgment target for a predetermined period of time, so if many hands are included in the field of view of the HMD 100, it may take a long time. In the third embodiment, when one or more hands are detected from a captured image, the hands to be compared with the detection information of the small-sized operating device 300 are narrowed down based on the movement direction of each hand. This makes it possible to reduce the number of candidates for hands to be compared with the detection information of the small-sized operating device 300, and shorten the comparison processing time. Hereinafter, detailed explanations of the same parts as in the first embodiment will be omitted, and the characteristic parts of the third embodiment will be mainly explained.

[0058] Fig. 10 is a flowchart of the operation target recognition process executed in the HMD 100 according to the third embodiment. The same processes as in the first embodiment are given the same step numbers as in the flowchart of Fig. 5. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the operation target recognition process of Fig. 10 is repeatedly executed in a predetermined cycle in the HMD 100.

[0059] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small-sized operating device 300 based on the sensing data acquired from the small-sized operating device 300. The processes from step S501 onward are performed only when the movement of the small-sized operating device 300 is detected.

[0060] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 acquires an image captured by the image capturing unit 101 from the captured image storage unit 104, and detects hands from the captured image. If multiple hands are included in the captured image, all of the hands are set as objects to be determined. In step S502, the hands to be determined are selected in order, and the process of step S1000 is executed for the selected objects to be determined.

[0061] In step S1000, the CPU 200 (processing and calculation unit 109) of the HMD 100 judges whether the moving direction of the hand to be judged and the small operating device 300 match. The moving direction may be calculated from the position information of any two frames. For example, the direction of a moving vector starting from the three-dimensional position of the previous frame and ending at the three-dimensional position of the current frame may be set as the moving direction. A specific example will be described with reference to FIG. 7. Assuming that the current frame is time t2 and the previous frame is time t1, the moving vector V1 of the hand to be judged is obtained as {x2-x1, y2-y1, z2-z1}, and the moving vector V2 of the small operating device 300 is obtained as {X2-X1, Y2-Y1, Z2-Z1}. If the angle θ between the vectors V1 and V2 is smaller than a predetermined threshold value Thθ, the processing and calculation unit 109 judges that the moving direction of the hand to be judged and the small operating device 300 match. The threshold value Thθ may be set in consideration of the measurement error of the sensor 303 and the error of the three-dimensional position calculated from the image. Set it to.

[0062] If the moving directions of the hand to be judged and the small operating device 300 are different (NO in step S1000), the CPU 200 judges that the hand to be judged is not the hand of the user of the HMD 100 but the hand of another person. In this case, the comparison process (step S503) is not performed, and the process proceeds to the process of the next hand to be judged (step S502).

[0063] On the other hand, if the moving direction of the hand to be judged and the small operating device 300 match (YES judgment in step S1000), the CPU 200 judges that the hand to be judged may be the hand of the user of the HMD 100, and proceeds to a comparison process (step S503). The subsequent processes are the same as those described in the first embodiment.

[0064] The configuration of this embodiment described above can achieve the same effects as those of the first embodiment. In addition, since candidates to be compared are narrowed down based on the movement direction of the hand to be judged and the small operating device 300, the processing cost can be reduced even when many hands are captured within the field of view of the HMD 100, and the processing time can be shortened and power consumption can be suppressed. In particular, since the present embodiment employs a method for simply calculating the movement direction from position information for two frames, it has the advantage of being able to immediately narrow down candidates to be compared.

[0065] (Fourth embodiment) In the first embodiment, all hands detected by the HMD 100 are compared with the detection information of the small-sized operating device 300. However, in order to determine that a hand is the user's hand, it is necessary to continue the comparison with each judgment target for a predetermined period of time, so if many hands are included in the field of view of the HMD 100, it may take a long time. In the fourth embodiment, when one or more hands are detected from a captured image, the hands to be compared with the detection information of the small-sized operating device 300 are narrowed down based on the extension direction of each hand. This makes it possible to reduce the number of candidates for hands to be compared with the detection information of the small-sized operating device 300, and shorten the comparison processing time. Hereinafter, detailed explanations of the same parts as in the first embodiment will be omitted, and the characteristic parts of the fourth embodiment will be mainly explained.

[0066] Fig. 11 is a flowchart of the operation target recognition process executed in the HMD 100 according to the fourth embodiment. The same processes as in the first embodiment are given the same step numbers as in the flowchart of Fig. 5. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the operation target recognition process of Fig. 11 is repeatedly executed in a predetermined cycle in the HMD 100.

[0067] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small-sized operating device 300 based on the sensing data acquired from the small-sized operating device 300. The processes from step S501 onward are performed only when the movement of the small-sized operating device 300 is detected.

[0068] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 acquires an image captured by the image capturing unit 101 from the captured image storage unit 104, and detects hands from the captured image. If the captured image contains multiple hands, all of the hands are set as objects to be determined. In step S502, the hands to be determined are selected in order, and the process of step S1100 is executed for the selected objects to be determined.

[0069] In step S1100, the CPU 200 (processing and calculation unit 109) of the HMD 100 judges whether the extension direction of the hand to be judged is within a predetermined range. The hand is a direction toward the tip of the hand, and is expressed by the direction of a vector starting from the wrist and ending at the tip of the fingers, for example. When a user of the HMD 100 performs a gesture operation with his / her right hand, the right hand of the user is usually photographed as extending from bottom to top or from right to left in the field of view of the HMD 100. When a user of the HMD 100 performs a gesture operation with his / her left hand, the left hand of the user is usually photographed as extending from bottom to top or from left to right in the field of view of the HMD 100. Therefore, if a hand detected from an image captured by the HMD 100 extends from top to bottom in the field of view, or extends from the opposite side to the hand performing the gesture operation, it is considered that the possibility that the hand is the hand of the user of the HMD 100 is extremely low. The angle that the extension direction of the user's hand can take is set in advance in the HMD 100 as a "predetermined range." For example, when the upward direction (12 o'clock direction) in the captured image is set as 0 degrees, and 0 to +180 degrees clockwise and 0 to -180 degrees counterclockwise, a hand extending from the bottom to the top falls within a range of approximately -45 degrees to 45 degrees. A hand extending from right to left falls within a range of approximately -90 degrees to -45 degrees, and a hand extending from left to right falls within a range of approximately 45 degrees to 90 degrees. Therefore, in this embodiment, for example, -90 degrees to 90 degrees is set as the predetermined range, and a hand with an extension direction outside the range is determined to be someone else's hand. Note that the method of setting the predetermined range is not limited to this, and can be set arbitrarily. For example, when it is known in advance whether the hand performing the gesture operation is the right hand or the left hand, the predetermined range may be set even narrower. Alternatively, taking into consideration that when the user tilts their head sideways, the field of view (captured image) of the HMD 100 itself rotates, affecting the relative angle between the direction of hand extension and the captured image, a certain margin may be set within a specified range.

[0070] If the extension direction of the hand to be determined is outside the predetermined range (NO in step S1100), the CPU 200 determines that the hand to be determined is not the hand of the user of the HMD 100 but the hand of another person. In this case, the comparison process (step S503) is not performed, and the process proceeds to the process of the next hand to be determined (step S502).

[0071] On the other hand, if the extension direction of the hand to be judged is within the predetermined range (YES in step S1100), the CPU 200 judges that the hand to be judged may be the hand of the user of the HMD 100, and proceeds to a comparison process (step S503). The subsequent processes are similar to those described in the first embodiment.

[0072] The configuration of this embodiment described above can achieve the same effects as those of the first embodiment. In addition, in this embodiment, since candidates to be compared are narrowed down based on the extension direction of the hand to be determined, the processing cost can be reduced even when many hands are captured within the field of view of the HMD 100, and the processing time can be shortened and power consumption can be reduced.

[0073] Fifth embodiment In the first embodiment, whether or not the hand is the user's hand is determined based on whether the hand movement detected by the HMD 100 matches the movement of the small operating device 300. In contrast, in the fifth embodiment, the user's hand is determined by image recognition using an image captured by the image capturing unit 101 of the HMD 100. In the following, detailed descriptions of the same parts as in the first embodiment will be omitted, and the characteristic parts of the fifth embodiment will be mainly described.

[0074] Fig. 12 is a flowchart of the operation target recognition process executed in the HMD 100 according to the fifth embodiment. The same processes as in the first embodiment are given the same step numbers as in the flowchart of Fig. 5. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the operation target recognition process of Fig. 12 is repeatedly executed in a predetermined cycle in the HMD 100.

[0075] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small-sized operating device 300 based on the sensing data acquired from the small-sized operating device 300. The processes from step S501 onward are performed only when the movement of the small-sized operating device 300 is detected.

[0076] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 acquires an image captured by the image capturing unit 101 from the captured image storage unit 104, and detects hands from the captured image. If multiple hands are included in the captured image, all of the hands are set as objects to be determined. In step S502, the hands to be determined are selected in order, and the process of step S1200 is executed for the selected objects to be determined.

[0077] In step S1200, the CPU 200 (processing and calculation unit 109) of the HMD 100 determines whether or not the small operating device 300 is attached to the hand of the object to be determined by image recognition. As the image recognition algorithm, a classic machine learning algorithm represented by a support vector machine may be used, an algorithm based on deep learning such as R-CNN, YOLO, SSD, or DCN may be used, or a rule-based detection algorithm may be used.

[0078] If the small operating device 300 is not attached to the hand to be determined (NO in step S1200), the CPU 200 determines that the hand to be determined is not the hand of the user of the HMD 100 but the hand of another person. In this case, the process proceeds to the process of the next hand to be determined (step S502).

[0079] On the other hand, if the small operating device 300 is attached to the hand to be judged (YES judgment in step S1200), the CPU 200 judges that the hand to be judged is the hand of the user of the HMD 100, and proceeds to step S504. The subsequent processing is the same as that described in the first embodiment.

[0080] The above-described configuration of the present embodiment can achieve the same effects as those of the first embodiment. In addition, the present embodiment has the advantage that it is possible to immediately identify the user's hand because it is possible to determine from one frame of a captured image whether the small operating device 300 is attached to the hand to be judged.

[0081] Sixth embodiment In the first embodiment, whether or not it is the user's hand is determined based on whether the movement of the hand detected by the HMD 100 matches the movement of the small operating device 300. In contrast, in the sixth embodiment, the user's hand is identified based on whether the posture of the hand detected by the HMD 100 matches the posture of the small operating device 300. The posture of the small operating device 300 can be estimated from the angular velocity of the IMU information acquired by the sensor 303. In the following, detailed descriptions of the same parts as in the first embodiment will be omitted, and the characteristic parts of the sixth embodiment will be mainly described.

[0082] Fig. 13 is a flowchart of the operation target recognition process executed in the HMD 100 according to the sixth embodiment. The same processes as in the first embodiment are given the same step numbers as in the flowchart of Fig. 5. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the operation target recognition process of Fig. 13 is repeatedly executed in a predetermined cycle in the HMD 100.

[0083] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small-sized operating device 300 based on the sensing data acquired from the small-sized operating device 300. Step S50 is performed only when the movement of the small-sized operating device 300 is detected. Processing from 1 onwards will be carried out.

[0084] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 acquires an image captured by the image capturing unit 101 from the captured image storage unit 104, and detects hands from the captured image. If the captured image contains multiple hands, all of the hands are set as objects to be determined. In step S502, the hands to be determined are selected in order, and the process of step S1300 is executed for the selected objects to be determined.

[0085] In step S1300, the CPU 200 (processing and calculation unit 109) of the HMD 100 evaluates whether the posture of the hand to be judged matches the posture of the small operating device 300. Specifically, the processing and calculation unit 109 calculates the posture angle of the small operating device 300 from the angular velocity of the IMU information acquired from the sensor 303. The processing and calculation unit 109 also calculates the posture angle of the hand to be judged by estimating the posture of the hand to be judged detected from the captured image. Any posture estimation algorithm may be used. For example, the three-dimensional coordinates of multiple feature points (joints, contour points, etc.) in the hand to be judged may be calculated, and the posture of the hand may be estimated based on the relative positional relationship of the multiple feature points. Alternatively, the posture estimation may be performed from an image of the hand using an algorithm based on deep learning. In addition, in consideration of the calculation error of the posture angle and the posture estimation error, if the difference between the posture angle of the hand to be judged and the posture angle of the small operating device 300 is within a predetermined error range, it may be determined that the postures of the two match.

[0086] If the posture of the hand to be judged is different from the posture of the small operating device 300 (NO in step S1300), the CPU 200 judges that the hand to be judged is not the hand of the user of the HMD 100 but the hand of another person. In this case, the process proceeds to the process of the next hand to be judged (step S502).

[0087] On the other hand, if the posture of the hand to be judged matches the posture of the small operating device 300 (YES judgment in step S1300), the CPU 200 judges that the hand to be judged is the hand of the user of the HMD 100, and proceeds to step S504. The subsequent processing is the same as that described in the first embodiment.

[0088] The configuration of this embodiment described above can achieve the same effects as those of the first embodiment. It is also possible to perform both the "comparison of movements" of the first embodiment and the "comparison of postures" of this embodiment, and determine that the hand is the user's own hand if both the movements and postures match. This is expected to further reduce erroneous determinations (mistaking another person's hand for the user's hand).

[0089] Seventh embodiment In the first embodiment, all hands detected by the HMD 100 are compared with the movement of the small operating device 300. However, there may be cases where the hand tracking information of the user's hand cannot be properly acquired because the user's hand is out of the field of view of the HMD 100 or in the blind spot of the image capturing unit 101. Therefore, in the seventh embodiment, when the user's hand cannot be recognized from the captured image, the user is notified that the hand position is not appropriate, so that the user is encouraged to move the hand to a position where the hand can be detected by the HMD 100. Hereinafter, detailed descriptions of the same parts as in the first embodiment will be omitted, and the characteristic parts of the seventh embodiment will be mainly described.

[0090] 14 is a flowchart of the operation target recognition process executed in the HMD 100 according to the seventh embodiment. The same processes as in the first embodiment are given the same step numbers as in the flowchart of FIG. 5. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the operation target recognition process of FIG. 14 is repeated in a predetermined cycle in the HMD 100. It will be executed.

[0091] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small-sized operating device 300 based on the sensing data acquired from the small-sized operating device 300. The processes from step S501 onward are performed only when the movement of the small-sized operating device 300 is detected.

[0092] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 acquires an image captured by the image capturing unit 101 from the captured image storage unit 104, and detects hands from the captured image. If the captured image contains multiple hands, all of the hands are set as objects to be determined. In step S502, the hands to be determined are selected in order, and the process of step S503 is executed for the selected objects to be determined.

[0093] If no hand to be judged is detected from the captured image, or if the user's own hand is not recognized even though the process of step S503 has been performed for all judgment targets, a NO judgment is made in step S502, and the process proceeds to step S1400. In step S1400, the CPU 200 (processing and calculation unit 109) of the HMD 100 displays a warning guidance on the image display unit 102 to the effect that the user's hand is not displayed on the HMD 100, thereby guiding the user of the HMD 100 to move the position of the hand. The processing and calculation unit 109 may notify the user that the hand position is not appropriate by audio guidance.

[0094] The configuration of the present embodiment described above can achieve the same effects as those of the first embodiment. In addition, in the present embodiment, if the position of the hand when the user performs a gesture operation is inappropriate, it can be detected and the user can be prompted to correct the position of the hand. Therefore, it is possible to reduce failures of gesture operations caused by inappropriate hand positions, and to improve convenience.

[0095] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The configurations described in the first to seventh embodiments may be combined with each other (as long as no technical contradiction occurs).

[0096] The present invention also includes a case where a software program for implementing the functions of the above-mentioned embodiment is supplied to a system or device having a computer capable of executing the program directly from a recording medium or by using wired / wireless communication, and the program is executed. Therefore, the program code itself that is supplied and installed to the computer in order to realize the functional processing of the present invention by the computer also realizes the present invention. In other words, the computer program itself for implementing the functional processing of the present invention is also included in the present invention. In that case, as long as it has the function of the program, the form of the program does not matter, such as object code, a program executed by an interpreter, script data supplied to an OS, etc. The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory. In addition, as a method of supplying the program, a method in which a computer program forming the present invention is stored in a server on a computer network and a connected client computer downloads and executes the computer program is also considered. The present invention can also be realized by a process in which a program for implementing one or more functions of the above-mentioned embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0097] The disclosure of this specification includes the following configurations, methods, and programs. (Configuration 1) An information processing device that recognizes the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, A detection means for detecting hands from an image captured by the HMD; an acquisition means for acquiring sensing data relating to a position of the device from a sensor mounted on the device worn by the user on the hand; a processing and calculation means for performing a comparison process for comparing a movement of a hand detected from the captured image with a movement of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user when the movement of the hand matches the movement of the device; An information processing device having the above configuration. (Configuration 2) An information processing device that recognizes the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, A detection means for detecting hands from an image captured by the HMD; an acquisition means for acquiring sensing data relating to the orientation of the device from a sensor mounted on the device worn by the user on the hand; a processing and calculation means for performing a comparison process of comparing a posture of a hand detected from the captured image with a posture of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user when the posture of the hand and the posture of the device match; An information processing device having the above configuration. (Configuration 3) An information processing device that recognizes the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, A detection means for detecting hands from an image captured by the HMD; an acquisition means for acquiring sensing data relating to a position and an orientation of a device from a sensor mounted on the device worn on the hand of the user; a processing and calculation means for performing a comparison process for comparing a movement and posture of a hand detected from the captured image with a movement and posture of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user when the movement and posture of the hand match the movement and posture of the device; An information processing device having the above configuration. (Configuration 4) When one or more hands are detected from the captured image, the processing calculation means narrows down the number of hands to be subjected to the comparison process based on a distance from the HMD to each hand. The information processing device according to any one of configurations 1 to 3. (Configuration 5) When one or more hands are detected from the captured image, the processing calculation means narrows down the number of hands to be subjected to the comparison process based on the moving direction of each hand. The information processing device according to any one of configurations 1 to 4. (Configuration 6) When one or more hands are detected from the captured image, the processing calculation means narrows down the hands to be subjected to the comparison process based on the extending direction of each hand. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) An information processing device that recognizes the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, A detection means for detecting hands from an image captured by the HMD; a processing calculation means for determining whether or not a device worn by the user is worn on the hand detected from the photographed image by image recognition, and determining that the hand detected from the photographed image is the hand of the user when the device is worn on the hand detected from the photographed image; An information processing device having the above configuration. (Configuration 8) When the hand of the user is not recognized from the captured image, the processing calculation means notifies the user that the position of the hand is not appropriate. The information processing device according to any one of configurations 1 to 7. (Configuration 9) When a change in the position of the device is detected based on sensing data acquired from a sensor mounted on the device, processing is executed by the detection means and the processing calculation means. The information processing device according to any one of configurations 1 to 8. (Configuration 10) In the comparison process, the processing calculation means compares the relative movement amount of the hand position with the relative movement amount of the device during a predetermined period, and determines that the movement of the hand and the movement of the device match when the relative movement amount of the hand position and the relative movement amount of the device are the same or within a predetermined error range. 4. The information processing device according to configuration 1 or 3. (Configuration 11) The sensor is an IMU sensor. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) The device is a wearable device that is worn on the user's hand or finger. 12. The information processing device according to any one of configurations 1 to 11. (Configuration 13) a goggle device having an image capturing unit and an image display unit and worn by a user on the head; An information processing device according to any one of configurations 1 to 12; An HMD having the above configuration. (Configuration 14) a goggle device having an image capturing unit and an image display unit and worn by a user on the head; An information processing device according to any one of configurations 1 to 12; A device that is worn on the user's hand and has a sensor that measures a position and orientation; A system having (Method 15) A method for recognizing the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, comprising: An information processing device, detecting hands from an image captured by the HMD; acquiring sensing data relating to a position of a device worn by the user on a hand from a sensor mounted on the device; performing a comparison process for comparing a movement of the hand detected from the captured image with a movement of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user when the movement of the hand matches the movement of the device; A recognition method to perform. (Method 16) The HMD (Head Mount Display) is selected from the images captured by the HMD. A method for recognizing the hand of a user wearing a handheld ... An information processing device, detecting hands from an image captured by the HMD; acquiring sensing data relating to the orientation of a device worn by the user on a hand from a sensor mounted on the device; performing a comparison process for comparing a posture of the hand detected from the captured image with a posture of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user when the posture of the hand matches the posture of the device; A recognition method to perform. (Method 17) A method for recognizing the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, comprising: An information processing device, detecting hands from an image captured by the HMD; acquiring sensing data relating to a position and an orientation of a device worn by the user from a sensor mounted on the device; performing a comparison process for comparing a movement and posture of a hand detected from the captured image with a movement and posture of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user if the movement and posture of the hand match the movement and posture of the device; A recognition method to perform. (Method 18) A method for recognizing the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, comprising: An information processing device, detecting hands from an image captured by the HMD; determining by image recognition whether or not a device worn by the user is worn on the hand detected from the captured image, and if the device is worn on the hand detected from the captured image, determining that the hand detected from the captured image is the hand of the user; A recognition method to perform. (Program 19) 19. A program for causing an information processing device to execute each step of the recognition method according to any one of Methods 15 to 18. [Explanation of symbols]

[0098] 100 HMD 103 Information processing equipment 105 Hand detection unit 109 Processing Calculation Unit 110 Near field communication unit 300 Small operation device 303 Sensors

Claims

1. An information processing device that recognizes the hands of a user wearing a head-mounted display (HMD) from an image captured by the HMD, a detection means for detecting hands from an image captured by the HMD; an acquisition means for acquiring sensing data relating to the position of the device from a sensor mounted on the device worn by the user on the hand; a processing calculation means for performing a comparison process for comparing the movement of the hand detected from the captured image with the movement of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user when the movement of the hand matches the movement of the device; An information processing device having the above.

2. An information processing device that recognizes the hands of a user wearing a head-mounted display (HMD) from an image captured by the HMD, a detection means for detecting hands from an image captured by the HMD; an acquisition means for acquiring sensing data relating to the orientation of the device from a sensor mounted on the device worn by the user on the hand; a processing calculation means for performing a comparison process of comparing the posture of the hand detected from the captured image with the posture of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user when the posture of the hand matches the posture of the device; An information processing device having the above.

3. An information processing device that recognizes the hands of a user wearing a head-mounted display (HMD) from an image captured by the HMD, a detection means for detecting hands from an image captured by the HMD; an acquisition means for acquiring sensing data relating to the position and orientation of the device from a sensor mounted on the device worn by the user on the hand; For a hand detected from the captured image, a movement and a posture of the hand recognized from the captured image are compared with a movement and a posture of the device recognized from the sensing data. a processing calculation means for performing a comparison process to determine that the hand detected from the captured image is the hand of the user when the movement and posture of the hand match the movement and posture of the device; An information processing device having the above.

4. When one or more hands are detected from the captured image, the processing calculation means narrows down the number of hands to be compared based on the distance from the HMD to each hand.

4. The information processing device according to claim 1.

5. when one or more hands are detected from the captured image, the processing calculation means narrows down the number of hands to be compared based on the movement direction of each hand.

4. The information processing device according to claim 1.

6. when one or more hands are detected from the captured image, the processing calculation means narrows down the number of hands to be compared based on the extending direction of each hand.

4. The information processing device according to claim 1.

7. An information processing device that recognizes the hands of a user wearing a head-mounted display (HMD) from an image captured by the HMD, a detection means for detecting hands from an image captured by the HMD; a processing calculation means for determining by image recognition whether or not a device worn by the user is worn on the hand detected from the photographed image, and determining that the hand detected from the photographed image is the hand of the user if the device is worn on the hand detected from the photographed image; An information processing device having the above.

8. When the hand of the user is not recognized from the captured image, the processing calculation means notifies the user that the position of the hand is not appropriate.

8. The information processing device according to claim 1, wherein the first and second inputs are input to the first and second inputs.

9. When a change in the position of the device is detected based on sensing data acquired from a sensor mounted on the device, processing is performed by the detection means and the processing calculation means.

8. The information processing device according to claim 1, wherein the first and second inputs are input to the first and second inputs.

10. In the comparison process, the processing calculation means compares the amount of relative movement of the hand position with the amount of relative movement of the device over a predetermined period of time, and determines that the movement of the hand and the movement of the device match when the amount of relative movement of the hand position and the amount of relative movement of the device are the same or within a predetermined error range. The information processing device according to claim 1 or 3.

11. the sensor is an IMU sensor; The information processing device according to any one of claims 1 to 3.

12. The device is a wearable device worn on the user's hand or finger. The information processing device according to any one of claims 1 to 3.

13. a goggle device having an image capturing unit and an image display unit and worn by a user on the head; An information processing device according to any one of claims 1 to 3 and 7; An HMD having the above.

14. a goggle device having an image capturing unit and an image display unit and worn by a user on the head; An information processing device according to any one of claims 1 to 3 and 7; a device worn on the user's hand and having a sensor for measuring position and orientation; A system having:

15. A method for recognizing the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, comprising: The information processing device detecting hands from an image captured by the HMD; acquiring sensing data relating to the position of the device from a sensor mounted on the device worn by the user on the hand; performing a comparison process for comparing the movement of the hand detected from the captured image recognized from the captured image with the movement of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user when the movement of the hand matches the movement of the device; A recognition method to perform.

16. A method for recognizing the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, comprising: The information processing device detecting hands from an image captured by the HMD; acquiring sensing data relating to the orientation of the device from a sensor mounted on the device worn by the user on the hand; performing a comparison process for comparing the posture of the hand detected from the captured image with the posture of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user when the posture of the hand matches the posture of the device; A recognition method to perform.

17. A method for recognizing the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, comprising: The information processing device detecting hands from an image captured by the HMD; acquiring sensing data relating to the position and orientation of the device from a sensor mounted on the device worn by the user on the hand; performing a comparison process for comparing the movement and posture of the hand detected from the captured image recognized from the captured image with the movement and posture of the device recognized from the sensing data, and determining that the hand detected from the captured image is the hand of the user if the movement and posture of the hand match the movement and posture of the device; A recognition method to perform.

18. A method for recognizing the hands of a user wearing a head mounted display (HMD) from an image captured by the HMD, comprising: The information processing device detecting hands from an image captured by the HMD; a step of determining by image recognition whether or not the device worn by the user is worn on the hand detected from the photographed image, and determining that the hand detected from the photographed image is the hand of the user if the device is worn on the hand detected from the photographed image; Top and A recognition method to perform.

19. A program for causing an information processing device to execute each step of the recognition method according to any one of claims 15 to 18.

Images