Information processing equipment and HMD

Abstract

To provide a technique which allows for recognizing a hand of a user himself / herself from among hands included in a visual field of an HMD with simple processing and high accuracy.SOLUTION: An information processing apparatus for recognizing a hand of a user wearing an HMD from in an image captured by the HMD includes: detection means which detects a hand from a captured image captured by the HMD; acquisition means which acquires, from a sensor mounted on a device that the user puts on his / her hand, sensing data about a position of the device; and processing operation means which, with respect to the hand detected from the captured image, performs comparison processing of comparing movement of the hand recognized from the captured image with movement of the device recognized from the sensing data and determines that the hand detected from the captured image is the hand of the user in the case of coincidence between the movement of the hand and the movement of the device.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for recognizing a user's hand included in the field of view of an HMD.

Background Art

[0002] There is a Mixed Reality (MR) technology that fuses real space and virtual space to allow an experiencer to interact with virtual objects. In the MR technology, interaction is realized by synthesizing and presenting computer graphics (CG) representing virtual objects on a real scene, or expressing contact between real objects and virtual objects.

[0003] In the MR technology, it is assumed that a user performs gesture operations with their own hand to move virtual objects relative to the real scene. In gesture operations, movement, manipulation, etc. of CG objects can be performed without a controller. However, when there are multiple people other than the experiencer in the same space, it is not possible to distinguish between the user's own hand and others' hands, and there is a possibility that the user's own HMD (Head Mount ed Display) may be unintentionally operated by gesture operations of others' hands.

[0004] Patent Documents 1 and 2 disclose techniques related to object recognition in three-dimensional space. Patent Document 1 discloses a method of recognizing a person's motion using an image or a motion sensor, and identifying a target person based on features of the motion. Patent Document 2 discloses a method of correctly representing the front-back relationship between a real object and a virtual object by measuring the depth position of a real object such as a hand with a stereo camera.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

[0006] The prior art disclosed in Patent Document 1 involves pre-learning the actions of the target person to be identified, and then determining whether the person currently acting is the target person by comparing the learned characteristics with characteristics extracted from people acting within a defined area. However, such learning-based methods are cumbersome because they require prior preparation, such as registering the actions of the target person. Furthermore, there is the inconvenience that people whose actions have not been registered cannot use the system.

[0007] This invention has been made in view of the above circumstances, and its purpose is to provide a technology that enables the recognition of the user's own hand from among the hands included in the field of view of the HMD with simple processing and high accuracy. [Means for solving the problem]

[0008] This disclosure pertains to HMD (Head Mount) ed An information processing device for recognizing the hand of a user wearing the HMD from an image captured by a Display, the information processing device includes: detection means for detecting a hand from an image captured by the HMD; acquisition means for acquiring sensing data relating to the position of a device from a sensor mounted on a device worn on the user's hand; and processing calculation means for performing a comparison process to compare the movement of the hand recognized from the captured image with the movement of the device recognized from the sensing data, and determining that the hand detected from the captured image is the user's hand when the hand movement and the device movement match. [Effects of the Invention]

[0009] According to the present invention, the user's own hand can be recognized with simple processing and high accuracy from among the hands included in the field of view of the HMD. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram showing the configuration of a mixed reality system [Figure 2] Block diagram showing the hardware configuration of the HMD. [Figure 3] Block diagram showing the configuration of a small operating device. [Figure 4] A diagram illustrating the usage of a mixed reality system. [Figure 5] Flowchart of the object recognition process in the first embodiment [Figure 6] A diagram showing an example of the movement of a small operating device. [Figure 7] A diagram showing an example of hand position detected by a small control device and HMD. [Figure 8] Figure showing an example of hand movement in HMD images. [Figure 9] Flowchart of the object recognition process in the second embodiment [Figure 10] Flowchart of the object recognition process in the third embodiment [Figure 11] Flowchart of the object recognition process in the fourth embodiment [Figure 12] Flowchart of the object recognition process in the fifth embodiment [Figure 13] Flowchart of the object recognition process in the sixth embodiment [Figure 14] Flowchart of the object recognition process in the seventh embodiment [Modes for carrying out the invention]

[0011] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The embodiments described below are merely examples of means for realizing the present invention and may be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions. Furthermore, the embodiments can be combined as appropriate.

[0012] (First Embodiment) The configuration of the mixed reality system according to the first embodiment will be described with reference to FIGS. 1 to 4. FIG. 1 is a block diagram showing the overall configuration of the mixed reality system according to the first embodiment. FIG. 2 is a block diagram showing the hardware configuration of an HMD. FIG. 3 is a block diagram showing the configuration of a compact operation device. FIG. 4 is a diagram showing a usage mode of the mixed reality system.

[0013] A mixed reality system 1 includes an HMD (Head Mount ed Display) 100 and a compact operation device 300. The HMD 100 is a device to be worn on the head of a user, and is composed of a goggle apparatus 100A and an information processing apparatus 103. The compact operation device 300 is a device to be worn on the user's hand. FIG. 4 schematically shows an example of a state where the user wears the HMD 100 and the compact operation device 300.

[0014] A CPU 200 is a processor that performs overall control of the system by executing a program for operating the present system. Processing units 104 to 109 shown in FIG. 1 are implemented in software by the CPU 200 executing a program read from a ROM 202. Note that some of these processing units may be implemented by hardware independent of the CPU 200 (for example, an ASIC, an FPGA, or the like). A processor (GPU, DSP, or the like) for assisting arithmetic processing of the CPU 200 may be provided.

[0015] The goggle device 100A of the HMD100 is a head-mounted display device having an image capture unit 101 and an image display unit 102. As shown in the schematic diagram of Figure 4, the HMD100 of this embodiment uses a video see-through system in which the image capture unit 101 is positioned at the user's viewpoint, and the live video captured by the image capture unit 101 is displayed on the image display unit 102. In the case of the video see-through HMD100, it is common to treat the image capture unit 101 as the user's viewpoint position and posture. Note that although the HMD100 of this embodiment uses a video see-through system, this should be understood as merely an example. For example, it can also be applied to a virtual reality HMD that does not display live video on the image display unit 102. In that case, the image capture unit 101 can be used not as a see-through camera, but as a camera for capturing the hand, which is the object to be measured.

[0016] The HMD100's information processing unit 103 creates data for display on the image display unit 102 using images captured by the image acquisition unit 101. The information processing unit 103 is a small computer equipped with a processor (CPU 200) and memory (RAM 201, ROM 202) as shown in Figure 2. The information processing unit 103 may be built into the same enclosure as the goggle device 100A, or it may be configured in a separate enclosure from the goggle device 100A. If the information processing unit 103 and the goggle device 100A are configured in separate enclosures, the information processing unit 103 and the goggle device 100A are connected to each other via wired or wireless means so that they can communicate with each other.

[0017] The image capture unit 101 is fixed to the housing of the goggle device 100A and captures images of the real world at a predetermined frame rate (e.g., 30 frames / second). Images captured by the image capture unit 101 are stored in the captured image storage unit 104. The captured image storage unit 104 stores the images acquired from the image capture unit 101 in the RAM 201. The image capture unit 101 consists of two real cameras, with a real camera for the left eye and a real camera for the right eye positioned close to both eyes of the HMD wearer. Hereafter, the pair of left and right images obtained by the image capture unit 101 will be referred to as stereo camera images.

[0018] The image display unit 102 is fixed to the housing of the goggle device 100A and displays images generated by the information processing device 103. These images are, for example, mixed reality (MR) images created by combining live-action video captured by the image capture unit 101 with 3DCG digital content (virtual objects). The image display unit 102 has a structure that covers the field of view of the user wearing the goggle device 100A, providing the user with a highly immersive MR experience. This image display unit 102 is composed of, for example, an organic EL display or a liquid crystal display.

[0019] The hand detection unit 105 detects a hand from each of the captured images stored by the captured image storage unit 104 using image recognition and extracts contour points of the fingers. The three-dimensional position calculation unit 106 calculates the three-dimensional position of the contour points extracted by the hand detection unit 105. The x and y coordinates of the contour points are obtained with the lower left of the VRAM area of ​​the image captured by the image capture unit 101 of the HMD 100 set as (0,0). The z coordinate in the depth direction is obtained from the stereo camera image. This coordinate system is called the camera coordinate system or local coordinate system. The origin of the camera coordinate system does not have to be the lower left of the VRAM area and can be freely determined. In this embodiment, the hand detection unit 105 (or the hand detection unit 105 and the three-dimensional position calculation unit 106) constitutes a detection means for detecting a hand from captured images captured by the HMD 100.

[0020] For detecting hands from images, you can use classical machine learning algorithms such as support vector machines, or deep learning-based algorithms such as R-CNN, YOLO, SSD, or DCN. You can also use rule-based detection algorithms.

[0021] To correctly display the depth of a detected hand, one method involves extracting the hand region from a hand image captured by a stereo camera and calculating the hand's depth relative to the stereo camera. This method uses triangulation to determine the depth value for all corresponding points in the stereo image of the extracted hand contour (Patent Document 2). Patent Document 2 utilizes a stereo camera mounted on a video see-through type HMD, which is a display device for presenting augmented reality.

[0022] LeapMotion, from LeapMotion Inc., can measure the position and orientation of the hand, including the fingers. LeapMotion can detect the hand area from a stereo camera (image capture unit 101). The position and orientation of the fingers can also be estimated by using a separate depth sensor, so a depth sensor may be installed on the HMD 100.

[0023] In this embodiment, an arbitrary point (referred to as a reference point or representative point) is selected from the contour points extracted by the hand detection unit 105, and the three-dimensional position relative to that reference point is calculated. The reference point selected here is stored as the reference position during this process, and the reference three-dimensional position information is periodically stored in the RAM 201 in accordance with the movement of the fingers.

[0024] The relative position and orientation calculation unit 107 calculates the position and orientation of each captured image stored by the captured image storage unit 104. Position and orientation calculation can be performed using deep learning or by using existing publicly available libraries. The position and orientation of the captured image corresponds to the position and orientation of the HMD 100 in real space, that is, the position and line of sight of the user wearing the HMD 100. The position and orientation of the captured image are expressed in a global coordinate system.

[0025] The processing unit 109 performs a comparison calculation between the hand movement recognized based on the three-dimensional position information calculated from the captured image and the movement of the small operating device 300 recognized based on sensing data acquired from the sensor 303 mounted on the small operating device 300, which will be described later. Specifically, the processing unit 109 evaluates whether the relative movement amounts of the hand detected by the HMD 100 and the small operating device 300 match. If they match, the processing unit 109 determines that the hand detected by the HMD 100 and the hand to which the small operating device 300 is attached are the same hand, i.e., the hand of the user of the HMD 100. Generally, coordinates acquired by sensors have errors, so a certain error rate is allowed, for example, up to a 5 percent error rate is considered identical. The error rate can be freely determined.

[0026] When raw sensor data (unprocessed data) is sent from the small operating device 300 to the information processing device 103, the processing calculation unit 109 should convert the raw data into coordinate values ​​so that it can be compared with the values ​​calculated by the three-dimensional position calculation unit 106. Alternatively, the small operating device 300 may convert the raw data into coordinate values ​​first, and then send those coordinate values ​​to the information processing device 103.

[0027] The image drawing unit 108 draws (generates) an image to be displayed on the image display unit 102 of the HMD 100 based on the processing results of the processing calculation unit 109.

[0028] The proximity wireless communication unit 110 consists of, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller. The proximity wireless communication unit 110 outputs a modulated wireless signal from the antenna and demodulates the wireless signal received by the antenna, thereby realizing short-range wireless communication in accordance with the IEEE 802.15 standard (so-called Bluetooth®). In this embodiment, Bluetooth® Low Energy version 5.1, which has low power consumption, is used for Bluetooth® communication. The HMD 100 communicates with the small operating device 300 via the proximity wireless communication unit 110. It can exchange data with any external device.

[0029] RAM201 is volatile memory used as a working area for temporarily storing data and programs. ROM202 is non-volatile memory that non-temporarily stores data such as programs executed by CPU200, HMD100 configuration information, and configuration information needed for future use.

[0030] As shown in Figure 3, the miniature operating device 300 includes a control unit 301, an operating unit 302, a sensor 303, a working memory 304, a power control unit 305, and a communication unit 306. As shown in Figure 4, the miniature operating device 300 is a wearable device that can be attached to the user's hand or finger, and in this embodiment, a ring-shaped device is used. However, the form of the miniature operating device 300 is not limited to a hand-worn type; any form of device is acceptable as long as it can detect the position and posture of the user's own hand. For example, a wristwatch-type device, a hand-held type device, or a device that attaches to the finger may be used.

[0031] The control unit 301 controls the small operating device 300 according to the input signals and the program described later. Alternatively, instead of the control unit 301 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0032] The working memory 304 is used as a buffer memory for temporarily storing data acquired by the sensor 303, and as a work area for the control unit 301, etc.

[0033] Sensor 303 is a sensor used to measure the three-dimensional position, movement, and orientation (direction) of the user's hand wearing the small operating device 300, and is also called a position sensor, motion sensor, or orientation sensor. For example, an IMU (Inertial Measurement Unit) sensor can be used as sensor 303. An IMU sensor can calculate the change in orientation, relative direction, and position based on the angular velocity detected by the gyroscope and the acceleration detected by the accelerometer.

[0034] The control unit 302 is used to receive instructions from the user for the miniature operating device 300. The control unit 302 includes, for example, a power button for instructing the main power ON / OFF, including the supply of power to the control unit 301 of the miniature operating device 300, and a mode switching button for switching function modes. Furthermore, the control unit 302 includes a button for initiating communication with an external device such as the HMD 100 via the communication unit 306.

[0035] The power control unit 305 is a unit that supplies power for the operation of the small operating device 300.

[0036] The communication unit 306, like the proximity wireless communication unit 110 of the HMD 100, consists of, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller. The communication unit 306 implements short-range wireless communication in accordance with the IEEE 802.15 standard (so-called Bluetooth®). The small operating device 300 can exchange data with external devices such as the HMD 100 via the communication unit 306.

[0037] As shown in Figure 1, the proximity wireless communication unit 110 of the HMD 100 and the miniature operating device 300 are connected via Bluetooth® communication. For example, various sensing data acquired by the sensor 303 can be transmitted to the proximity wireless communication unit 110 of the HMD 100 via the communication unit 306 of the miniature operating device 300. In this embodiment, the proximity wireless communication unit 110 constitutes an acquisition means for acquiring sensing data related to the position and orientation of the miniature operating device 300 from the sensor 303 mounted on the miniature operating device 300. Furthermore, data communication between the HMD100 and the small operating device 300 is not limited to Bluetooth® communication; other communication methods may also be used.

[0038] Figure 5 is a flowchart of the operation target recognition process performed in the HMD 100 according to the first embodiment. The operation target recognition process is the process of recognizing the hand of the user wearing the HMD 100 (i.e., the hand of the person performing the gesture operation (operation target)) from among one or more hands (objects) included in the image captured by the HMD 100 (i.e., the field of view of the HMD 100). While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operation device 300 is periodically transmitted to the HMD 100, and the operation target recognition process shown in Figure 5 is repeatedly executed in a predetermined cycle in the HMD 100.

[0039] In step S500, the CPU 200 (processing unit 109) of the HMD 100 determines whether or not there has been a change in the position of the small operating device 300 based on sensing data acquired from the small operating device 300. Whether or not there has been a change in position can be determined, for example, by comparing multiple sets of sensing data (i.e., time-series data of position). Note that there is a certain error in the position coordinates measured by the sensor 303, so if the change in position is slight, it may be considered as "no change in position". If a change in the position of the small operating device 300 is detected in step S500, the CPU 200 assumes that the user of the HMD 100 is performing gesture operations using the hand wearing the small operating device 300, and proceeds to the processing in step S501. On the other hand, if no change in the position of the small operating device 300 is detected, the CPU 200 assumes that no gesture operations are being performed, and skips subsequent processing (recognition of the user's hand using the image on the HMD 100, gesture recognition, etc.). In this way, by preventing the execution of subsequent processing when the user's hand movements are not detected by the small operating device 300, unnecessary calculations can be reduced, power consumption can be lowered, and the risk of mistakenly recognizing another person's hand can be eliminated as much as possible.

[0040] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 retrieves the image captured by the image capture unit 101 from the captured image storage unit 104 and detects a hand from the captured image. If the captured image contains multiple hands, all of those hands are set as targets for determination to determine whether or not they belong to the user. Next, the CPU 200 (three-dimensional position calculation unit 106) calculates the three-dimensional position coordinates of the contour points of each hand targeted for determination. The captured image used here is an image taken at the same time as the sensing data referenced in step S500 (or the closest time if there is no image taken at exactly the same time).

[0041] If one or more hands to be judged are detected, the hands to be judged are selected in order in step S502, and the process in step S503 is executed for the selected hands. Since it is unknown which of the hands in the image belongs to the user, the judgment is made one by one. If no hands to be judged are detected, or if the process in step S503 has been performed for all hands to be judged, the judgment in step S502 is NO, and the process ends.

[0042] In step S503, the CPU 200 (processing unit 109) of the HMD 100 selects one reference contour point (reference point) from among the contour points of the hand to be judged. Next, the processing unit 109 converts the coordinate values ​​of the reference point from the camera coordinate system to the global coordinate system, taking into account the position and orientation of the captured image calculated by the relative position and orientation calculation unit 107. Then, the processing unit 109 evaluates whether the movement of the hand to be judged (i.e., the change in the position of the reference point) and the movement of the small operating device 300 (i.e., the change in the position of the small operating device 300) match. If it is determined that the movement of the hand to be judged and the movement of the small operating device 300 match (i.e., they are the movement of the same object), the CPU 200 proceeds to step S504. If it is determined that it will not proceed, the CPU 200 returns to step S502 and evaluates the next target for determination.

[0043] In step S504, the processing unit 109 recognizes that the hand to be judged is the hand of the user wearing the HMD 100 (the target of operation). Subsequently, the movement information of this target of operation is used for gesture recognition processing (not shown).

[0044] Figure 6 shows an example of the movement of the small operating device 300 frame by frame. The position of the small operating device 300 changes from P1(X1,Y1,Z1)→P2(X2,Y2,Z2)→P3(X3,Y3,Z3) in accordance with the movement of the hand. The position of the small operating device 300 for each frame is measured by the sensor 303. Since the small operating device 300 is attached (fixed) to the user's hand, the position of the small operating device 300 measured by the sensor 303 can be considered as the position of the user's hand. The table on the left of Figure 7 shows an example of time-series data of the hand position P (i.e., the temporal change in the hand position) measured and recorded by the sensor 303 of the small operating device 300.

[0045] Figure 8 shows an example of hand movement per frame included in an image captured by the image acquisition unit 101 of the HMD100. The hand position changes from Q1(x1,y1,z1) → Q2(x2,y2,z2) → Q3(x3,y3,z3). In the example in Figure 8, the contour point of the tip of the left index finger is selected as the reference point for indicating the hand position. The table on the right side of Figure 7 shows an example of time-series data of hand position Q (i.e., temporal change in hand position) detected and recorded from the image by the HMD100. The time-series data of hand position Q detected from the image is also called hand tracking information.

[0046] As shown in Figure 7, the information processing device 103 of the HMD 100 acquires information on the hand position P detected by the small operating device 300 and information on the hand position Q detected from the image for each corresponding time t1, t2, ... In step S503 of Figure 5, the processing calculation unit 109 compares the relative movement amount of the hand position P detected by the small operating device 300 and the relative movement amount of the hand position Q detected from the image for a predetermined period of time. The relative movement amount is the relative change in the hand position of the current frame with respect to the hand position of the previous frame. For example, the relative movement amount of position P1 (X1, Y1, Z1) at time t1 and position P2 (X2, Y2, Z2) at time t2 is expressed as (X2-X1, Y2-Y1, Z2-Z1). If the relative movement amounts over the predetermined period are the same or within a predetermined error range, the processing calculation unit 109 determines that the hand being judged is the hand of the user wearing the HMD 100. Here, the "predetermined period" used for comparison can be freely set. In this embodiment, it is set to approximately 3 to 10 frames. The "predetermined error range" should be set considering the measurement error of the sensor 303 and the error of the three-dimensional position calculated from the image.

[0047] By using relative movement for comparison, it becomes unnecessary to match the position detected by the small operating device 300 (the mounting position of the sensor 303) with the position of the reference point detected from the image, which has the advantage of greatly simplifying the process. For example, as shown in the examples in Figures 6 and 8, even if the mounting position of the sensor 303 is at the base of the index finger and the reference point is at the tip of the index finger, the relative movement of the two (see dashed arrows) is almost the same, making a simple comparison possible. If an absolute position comparison were to be performed, the measurement preparation would become complicated, requiring calibration to precisely match the reference point detected from the image with the mounting position of the sensor 303 on the small operating device 300.

[0048] As described above, in this embodiment, IMU information acquired from the sensor 303 of the small operating device 300 and hand tracking information acquired from the image captured by the image capture unit 101 of the HMD 100 are used. Then, among the hands included in the image, the small operating device 3 The system recognizes any hand movement that matches the hand movement detected by 00 as the hand of the user wearing the HMD100. This makes it possible to identify the user's own hand with simple processing and high accuracy, even if another person's hand is visible within the HMD100's field of view, thereby reducing errors in gesture control in MR. The method of this embodiment is highly convenient because it does not require learning the user's skeletal structure or movements as in conventional methods, nor does it require prior preparation such as alignment (calibration) between the HMD100 and the small operating device 300.

[0049] (Second embodiment) In the first embodiment, all hands detected by the HMD 100 were compared with the detection information of the miniature operating device 300. However, since it is necessary to continue comparing each target for determination for a predetermined period of time in order to determine whether it is a user's hand, this can take a lot of time if there are many hands in the field of view of the HMD 100. In the second embodiment, when one or more hands are detected from the captured image, the number of hands to be compared with the detection information of the miniature operating device 300 is narrowed down based on the distance from the HMD 100 to each hand. This reduces the number of candidate hands to be compared with the detection information of the miniature operating device 300, and is expected to shorten the comparison processing time. Below, detailed explanations of parts that are the same as in the first embodiment will be omitted, and the features of the second embodiment will be mainly described.

[0050] Figure 9 is a flowchart of the object recognition process performed in the HMD 100 according to the second embodiment. Processes identical to those in the first embodiment are given the same step numbers as in the flowchart of Figure 5. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the object recognition process shown in Figure 9 is repeatedly executed in a predetermined cycle in the HMD 100.

[0051] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small operating device 300, based on sensing data acquired from the small operating device 300. The processing from step S501 onwards is performed only if movement of the small operating device 300 is detected.

[0052] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 retrieves the image captured by the image capture unit 101 from the captured image storage unit 104 and detects a hand from the captured image. If the captured image contains multiple hands, all of those hands are set as targets for determination. In step S502, the hands to be determined are selected in order, and the processing in step S900 is executed for the selected hands.

[0053] In step S900, the CPU 200 (processing unit 109) of the HMD 100 calculates the distance D from the HMD 100 to the hand to be judged, and compares the distance D with the threshold ThD. The distance D can be obtained from the depth information acquired from the stereo image described in the first embodiment. Alternatively, if the HMD 100 is equipped with a depth sensor, the information measured by the depth sensor may be used. The threshold ThD should be set to a value slightly larger than the user's reach (distance within arm's reach) of the HMD 100, for example, a value of about 0.8m to 1.0m. The threshold ThD is set in the HMD 100 in advance.

[0054] If the distance D to the hand to be judged is greater than or equal to the threshold ThD (NO judgment in step S900), the CPU 200 determines that the hand to be judged is not the hand of the HMD 100 user but the hand of someone else. In this case, the comparison process (step S503) is not performed, and the process moves on to the next hand to be judged (step S502).

[0055] On the other hand, if the distance D to the hand to be judged is closer than the threshold ThD (YES judgment in step S900), the CPU 200 determines that the hand to be judged may be the hand of the user of the HMD 100 and proceeds to the comparison process (step S503). The subsequent processing is the same as that described in the first embodiment.

[0056] The configuration of this embodiment described above can achieve the same effects as the first embodiment. In addition, in this embodiment, the candidates to be compared are narrowed down based on the distance from the HMD 100 to the hand, so even if many hands are visible within the field of view of the HMD 100, processing costs can be reduced, resulting in shorter processing times and reduced power consumption.

[0057] (Third embodiment) In the first embodiment, all hands detected by the HMD 100 were compared with the detection information of the miniature operating device 300. However, since it is necessary to continue comparing each target for determination for a predetermined period of time in order to determine whether it is a user's hand, this can take a lot of time if there are many hands in the field of view of the HMD 100. In the third embodiment, when one or more hands are detected from the captured image, the hands to be compared with the detection information of the miniature operating device 300 are narrowed down based on the direction of movement of each hand. This reduces the number of candidate hands to be compared with the detection information of the miniature operating device 300, and is expected to shorten the comparison processing time. Below, detailed explanations of parts that are the same as in the first embodiment will be omitted, and the features of the third embodiment will be mainly described.

[0058] Figure 10 is a flowchart of the object recognition process performed in the HMD 100 according to the third embodiment. The same steps as in the flowchart of Figure 5 are used for processes that are the same as in the first embodiment. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the object recognition process in Figure 10 is repeatedly executed in a predetermined cycle in the HMD 100.

[0059] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small operating device 300, based on sensing data acquired from the small operating device 300. The processing from step S501 onwards is performed only if movement of the small operating device 300 is detected.

[0060] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 retrieves the image captured by the image capture unit 101 from the captured image storage unit 104 and detects a hand from the captured image. If the captured image contains multiple hands, all of those hands are set as targets for determination. In step S502, the hands to be determined are selected in order, and the processing in step S1000 is executed for the selected hands.

[0061] In step S1000, the CPU 200 (processing unit 109) of the HMD 100 determines whether the direction of movement of the hand being judged and the small operating device 300 are the same. The direction of movement can be calculated from the position information of any two frames. For example, the direction of movement can be the direction of a movement vector that starts at the three-dimensional position of the previous frame and ends at the three-dimensional position of the current frame. A specific example will be explained with reference to Figure 7. Assuming that the current frame is at time t2 and the previous frame is at time t1, the movement vector V1 of the hand being judged is obtained as {x2-x1, y2-y1, z2-z1}, and the movement vector V2 of the small operating device 300 is obtained as {X2-X1, Y2-Y1, Z2-Z1}. The processing unit 109 determines that the direction of movement of the hand being judged and the small operating device 300 are the same if the angle θ between vectors V1 and V2 is smaller than a predetermined threshold Thθ. The threshold Thθ should be set considering the measurement error of the sensor 303 and the error of the three-dimensional position calculated from the image. The threshold Thθ should be set in advance on the HMD100. Leave it set to that.

[0062] If the direction of movement of the hand being judged and the small operating device 300 are different (NO judgment in step S1000), the CPU 200 determines that the hand being judged is not the hand of the HMD 100 user but the hand of someone else. In this case, the comparison process (step S503) is not performed, and the process proceeds to the next target to be judged (step S502).

[0063] On the other hand, if the direction of movement of the hand being evaluated matches that of the small operating device 300 (YES determination in step S1000), the CPU 200 determines that the hand being evaluated may be the hand of the user of the HMD 100 and proceeds to the comparison process (step S503). The subsequent processing is the same as that described in the first embodiment.

[0064] The configuration of this embodiment described above can achieve the same effects as the first embodiment. In addition, since the candidates to be compared are narrowed down based on the movement direction of the hand to be judged and the small operating device 300, processing costs can be reduced even when many hands are visible in the field of view of the HMD 100, resulting in shorter processing times and reduced power consumption. In particular, this embodiment employs a method of simply calculating the movement direction from position information of two frames, which has the advantage of allowing for immediate narrowing down of candidates to be compared.

[0065] (Fourth embodiment) In the first embodiment, all hands detected by the HMD 100 were compared with the detection information of the miniature operating device 300. However, since it is necessary to continue comparing each target for determination for a predetermined period of time in order to determine whether it is the user's hand, this can take a lot of time if there are many hands in the field of view of the HMD 100. In the fourth embodiment, when one or more hands are detected from the captured image, the hands to be compared with the detection information of the miniature operating device 300 are narrowed down based on the extension direction of each hand. This reduces the number of candidate hands to be compared with the detection information of the miniature operating device 300, and is expected to shorten the comparison processing time. Below, detailed explanations of parts that are the same as in the first embodiment will be omitted, and the features of the fourth embodiment will be mainly described.

[0066] Figure 11 is a flowchart of the object recognition process performed in the HMD 100 according to the fourth embodiment. The same steps as in the flowchart of Figure 5 are used for processes that are the same as in the first embodiment. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the object recognition process in Figure 11 is repeatedly executed in a predetermined cycle in the HMD 100.

[0067] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small operating device 300, based on sensing data acquired from the small operating device 300. The processing from step S501 onwards is performed only if movement of the small operating device 300 is detected.

[0068] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 retrieves the image captured by the image capture unit 101 from the captured image storage unit 104 and detects a hand from the captured image. If the captured image contains multiple hands, all of those hands are set as targets for determination. In step S502, the hands to be determined are selected in order, and the processing in step S1100 is executed for the selected hands.

[0069] In step S1100, the CPU 200 (processing unit 109) of the HMD 100 determines whether the extension direction of the hand to be judged is within a predetermined range. The extension direction of the hand is from the arm to the fingers. This refers to the direction toward the end of the hand, and is represented by the direction of a vector starting from the wrist and ending at the fingertips. When the user of the HMD100 performs gesture operations with their right hand, the user's right hand is usually depicted as extending from bottom to top or from right to left within the HMD100's field of view. Similarly, when the user of the HMD100 performs gesture operations with their left hand, the user's left hand is usually depicted as extending from bottom to top or from left to right within the HMD100's field of view. Therefore, if a hand detected in the HMD100's captured image extends from the top of the field of view downwards, or extends from the opposite side of the hand performing the gesture operation, it is highly unlikely that the hand belongs to the HMD100 user. The angle of the user's hand extension that can take is set in advance as a "predetermined range" in the HMD100. For example, if we consider the upward direction in a captured image (the 12 o'clock direction) as 0 degrees, and the range from 0 to +180 degrees clockwise and 0 to -180 degrees counterclockwise, then a hand extending upwards from below will generally fall within the range of -45 to 45 degrees. Similarly, a hand extending from right to left will generally fall within the range of -90 to -45 degrees, and a hand extending from left to right will generally fall within the range of 45 to 90 degrees. Therefore, in this embodiment, for example, -90 to 90 degrees is set as a predetermined range, and any hand extending outside this range is determined to be someone else's hand. Note that the method of setting the predetermined range is not limited to this and can be set arbitrarily. For example, if it is known in advance whether the hand performing the gesture is the right or left hand, the predetermined range may be set even narrower. Alternatively, considering that tilting the user's head to the side causes the HMD100's field of view (captured image) to rotate, affecting the relative angle between the hand's extension direction and the captured image, a certain margin may be provided within a predetermined range.

[0070] If the extension direction of the hand being judged is outside the predetermined range (NO judgment in step S1100), the CPU 200 determines that the hand being judged is not the hand of the HMD 100 user but someone else's hand. In this case, the comparison process (step S503) is not performed, and the process moves on to the next hand to be judged (step S502).

[0071] On the other hand, if the extension direction of the hand being judged is within a predetermined range (YES judgment in step S1100), the CPU 200 determines that the hand being judged may be the hand of the user of the HMD 100 and proceeds to the comparison process (step S503). The subsequent processing is the same as that described in the first embodiment.

[0072] The configuration of this embodiment described above can achieve the same effects as the first embodiment. In addition, in this embodiment, the candidates to be compared are narrowed down based on the extension direction of the hand to be judged, so even when many hands are visible in the field of view of the HMD 100, processing costs are reduced, and effects such as shorter processing time and reduced power consumption are obtained.

[0073] (Fifth embodiment) In the first embodiment, the user's hand was determined by whether the hand movements detected by the HMD 100 matched the movements of the small operating device 300. In contrast, in the fifth embodiment, the user's hand is determined by image recognition using images captured by the image capture unit 101 of the HMD 100. Hereinafter, detailed explanations of parts identical to the first embodiment will be omitted, and the distinctive features of the fifth embodiment will be mainly described.

[0074] Figure 12 is a flowchart of the object recognition process performed in the HMD 100 according to the fifth embodiment. The same processes as in the first embodiment are given the same step numbers as in the flowchart of Figure 5. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the object recognition process in Figure 12 is repeatedly executed in a predetermined cycle in the HMD 100.

[0075] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small operating device 300, based on sensing data acquired from the small operating device 300. The processing from step S501 onwards is performed only if movement of the small operating device 300 is detected.

[0076] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 retrieves the image captured by the image capture unit 101 from the captured image storage unit 104 and detects a hand from the captured image. If the captured image contains multiple hands, all of those hands are set as targets for determination. In step S502, the hands to be determined are selected in order, and the processing in step S1200 is executed on the selected hands.

[0077] In step S1200, the CPU 200 (processing unit 109) of the HMD 100 determines by image recognition whether or not a small operating device 300 is attached to the hand being judged. As the image recognition algorithm, classical machine learning such as support vector machines may be used, deep learning-based algorithms such as R-CNN, YOLO, SSD, DCN may be used, or rule-based detection algorithms may be used.

[0078] If the small operating device 300 is not attached to the hand being judged (NO judgment in step S1200), the CPU 200 determines that the hand being judged is not the hand of the HMD 100 user but the hand of someone else. In this case, the process proceeds to the next target to be judged (step S502).

[0079] On the other hand, if the small operating device 300 is attached to the hand being evaluated (YES determination in step S1200), the CPU 200 determines that the hand being evaluated is the hand of the user of the HMD 100 and proceeds to step S504. The subsequent processing is the same as that described in the first embodiment.

[0080] The configuration of this embodiment described above can achieve the same effects as the first embodiment. In addition, this embodiment has the advantage that it is possible to immediately identify the user's hand because it is possible to determine from a single frame of captured image whether or not the small operating device 300 is attached to the hand to be judged.

[0081] (Sixth embodiment) In the first embodiment, the user's hand was identified by whether the hand movement detected by the HMD 100 matched the movement of the small operating device 300. In contrast, in the sixth embodiment, the user's hand is identified by whether the hand posture detected by the HMD 100 matches the posture of the small operating device 300. The posture of the small operating device 300 can be estimated from the angular velocity of the IMU information acquired by the sensor 303. Below, detailed explanations of parts identical to the first embodiment will be omitted, and the features of the sixth embodiment will be mainly described.

[0082] Figure 13 is a flowchart of the object recognition process performed in the HMD 100 according to the sixth embodiment. The same steps as in the flowchart of Figure 5 are used for processes that are the same as in the first embodiment. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the object recognition process shown in Figure 13 is repeatedly executed in a predetermined cycle in the HMD 100.

[0083] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small operating device 300, based on sensing data acquired from the small operating device 300. Step S500 proceeds only when movement of the small operating device 300 is detected. The process from step 1 onwards will be performed.

[0084] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 retrieves the image captured by the image capture unit 101 from the captured image storage unit 104 and detects a hand from the captured image. If the captured image contains multiple hands, all of those hands are set as targets for determination. In step S502, the hands to be determined are selected in order, and the processing in step S1300 is executed on the selected hands.

[0085] In step S1300, the CPU 200 (processing unit 109) of the HMD 100 evaluates whether the posture of the hand to be judged matches the posture of the small operating device 300. Specifically, the processing unit 109 calculates the posture angle of the small operating device 300 from the angular velocity of the IMU information acquired from the sensor 303. The processing unit 109 also calculates the posture angle of the hand to be judged by estimating the posture of the hand to be judged detected from the captured image. Any posture estimation algorithm is acceptable. For example, the three-dimensional coordinates of multiple feature points (joints, contour points, etc.) on the hand to be judged may be calculated, and the posture of the hand may be estimated based on the relative positional relationship of the multiple feature points. Alternatively, a deep learning-based algorithm may be used to estimate the posture from the hand image. Note that, considering the calculation error of the posture angle and the error of posture estimation, if the difference between the posture angle of the hand to be judged and the posture angle of the small operating device 300 is within a predetermined error range, it may be determined that the postures of the two match.

[0086] If the posture of the hand being judged differs from the posture of the small operating device 300 (NO judgment in step S1300), the CPU 200 determines that the hand being judged is not the hand of the HMD 100 user but someone else's hand. In this case, the process proceeds to the next item to be judged (step S502).

[0087] On the other hand, if the posture of the hand being judged matches the posture of the small operating device 300 (YES judgment in step S1300), the CPU 200 determines that the hand being judged is the hand of the user of the HMD 100 and proceeds to step S504. The subsequent processing is the same as that described in the first embodiment.

[0088] The configuration of this embodiment described above can achieve the same effects as the first embodiment. Furthermore, both the "comparison of movement" in the first embodiment and the "comparison of posture" in this embodiment may be performed, and if both movement and posture match, it may be determined that the hand belongs to the user. This is expected to further reduce misidentification (mistaking someone else's hand for the user's hand).

[0089] (Seventh Embodiment) In the first embodiment, all hands detected by the HMD 100 were compared with the movements of the small operating device 300. However, there may be cases where the user's hand is outside the field of view of the HMD 100 or in a blind spot of the image capture unit 101, making it impossible to properly acquire hand tracking information for the user's hand. Therefore, in the seventh embodiment, if the user's hand cannot be recognized from the captured image, the user is notified that the hand position is inappropriate and instructed to move the hand to a position where it can be detected by the HMD 100. Below, detailed explanations of parts identical to the first embodiment will be omitted, and the distinctive features of the seventh embodiment will be mainly described.

[0090] Figure 14 is a flowchart of the object recognition process performed in the HMD 100 according to the seventh embodiment. The same process as in the first embodiment is given the same step numbers as in the flowchart of Figure 5. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operating device 300 is periodically transmitted to the HMD 100, and the object recognition process in Figure 14 is repeated in a predetermined cycle in the HMD 100. It will be executed.

[0091] In step S500, the CPU 200 of the HMD 100 determines whether or not there has been a change in the position of the small operating device 300, based on sensing data acquired from the small operating device 300. The processing from step S501 onwards is performed only if movement of the small operating device 300 is detected.

[0092] In step S501, the CPU 200 (hand detection unit 105) of the HMD 100 retrieves the image captured by the image capture unit 101 from the captured image storage unit 104 and detects a hand from the captured image. If the captured image contains multiple hands, all of those hands are set as targets for determination. In step S502, the hands to be determined are selected in order, and the process in step S503 is executed for the selected hands.

[0093] If no hand to be judged is detected from the captured image, or if the user's hand is not recognized despite the processing in step S503 being performed for all targets to be judged, then a NO determination is made in step S502, and the process proceeds to step S1400. In step S1400, the CPU 200 (processing unit 109) of the HMD 100 displays a warning guidance on the image display unit 102 indicating that the user's hand is not visible in the HMD 100, thereby guiding the user of the HMD 100 to move their hand. The processing unit 109 may also notify the user via voice guidance that the hand position is inappropriate.

[0094] The configuration of this embodiment described above can achieve the same effects as the first embodiment. In addition, in this embodiment, if the user's hand position is inappropriate when performing gesture operations, it can be detected and the user can be prompted to correct the hand position. Therefore, the failure of gesture operations caused by inappropriate hand position can be reduced, and convenience can be improved.

[0095] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. The configurations described in the first to seventh embodiments may be combined with each other (as long as no technical inconsistencies arise).

[0096] Furthermore, the present invention also includes cases where a software program that realizes the functions of the above-described embodiments is supplied directly from a recording medium or via wired / wireless communication to a system or device having a computer capable of executing the program, and the program is executed. Therefore, the program code itself supplied to and installed on a computer in order to realize the functional processing of the present invention also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention. In this case, the form of the program is not limited as long as it has the functionality of a program, such as object code, a program executed by an interpreter, or script data supplied to an OS. The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory. Furthermore, as a method of supplying the program, a method in which a computer program forming the present invention is stored on a server on a computer network, and a connected client computer downloads and executes the computer program, is also conceivable. The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0097] The disclosures herein include the following configurations, methods, and programs: (Composition 1) HMD (Head Mount) ed An information processing device that recognizes the hand of the user wearing the HMD from an image captured by the Display, A detection means for detecting a hand from an image captured by the aforementioned HMD, An acquisition means for acquiring sensing data related to the position of a device from a sensor mounted on a device worn on the user's hand, A processing calculation means performs a comparison process to compare the movement of the hand detected from the captured image with the movement of the device recognized from the sensing data, and determines that the hand detected from the captured image is the user's hand when the hand movement and the device movement match. An information processing device having (Configuration 2) HMD (Head Mount) ed An information processing device that recognizes the hand of the user wearing the HMD from an image captured by the Display, A detection means for detecting a hand from an image captured by the aforementioned HMD, An acquisition means for acquiring sensing data related to the orientation of a device from a sensor mounted on a device worn on the user's hand, A processing calculation means performs a comparison process to compare the posture of the hand detected from the captured image with the posture of the device recognized from the sensing data, and determines that the hand detected from the captured image is the user's hand if the posture of the hand and the posture of the device match. An information processing device having (Composition 3) HMD (Head Mount) ed An information processing device that recognizes the hand of the user wearing the HMD from an image captured by the Display, A detection means for detecting a hand from an image captured by the aforementioned HMD, An acquisition means for acquiring sensing data relating to the position and orientation of a device from a sensor mounted on a device worn on the user's hand, A processing calculation means performs a comparison process to compare the movement and posture of the hand detected from the captured image with the movement and posture of the device recognized from the sensing data, and determines that the hand detected from the captured image is the user's hand when the hand movement and posture and the device movement and posture match. An information processing device having (Composition 4) When one or more hands are detected from the captured image, the processing calculation means narrows down the hands to be compared based on the distance from the HMD to each hand. An information processing device described in one of the configurations 1 to 3. (Composition 5) When one or more hands are detected from the captured image, the processing calculation means narrows down the hands to be compared based on the direction of movement of each hand. An information processing device described in any one of configurations 1 to 4. (Composition 6) When one or more hands are detected from the captured image, the processing calculation means narrows down the hands to be compared based on the extension direction of each hand. An information processing device described in any one of configurations 1 to 5. (Composition 7) HMD (Head Mount) ed An information processing device that recognizes the hand of the user wearing the HMD from an image captured by the Display, A detection means for detecting a hand from an image captured by the aforementioned HMD, A processing calculation means that determines by image recognition whether the device worn by the user on the hand is worn on the hand detected in the captured image, and if the device is worn on the hand detected in the captured image, determines that the hand detected in the captured image is the user's hand, An information processing device having (Composition 8) The processing calculation means notifies the user that the position of the hand is inappropriate if the user's hand is not recognized from the captured image. An information processing device described in any one of items 1 to 7 of the configuration. (Composition 9) When a change in the position of the device is detected by sensing data acquired from a sensor mounted on the device, the detection means and the processing calculation means are executed. An information processing device described in any one of configurations 1 to 8. (Composition 10) The processing calculation means, in the comparison process, compares the relative movement of the hand position with the relative movement of the device over a predetermined period, and determines that the hand movement and the device movement are in agreement if the relative movement of the hand position and the relative movement of the device are the same or within a predetermined error range. The information processing device described in configuration 1 or 3. (Composition 11) The aforementioned sensor is an IMU sensor. An information processing device as described in any one of items 1 to 10. (Composition 12) The device is a wearable device that is attached to the user's hand or finger. An information processing device as described in any one of items 1 to 11. (Composition 13) A goggle device that has an image capture unit and an image display unit, and is worn on the user's head, The information processing device described in one of the configurations 1 to 12, HMD (Head-Mounted Display) (Composition 14) A goggle device that has an image capture unit and an image display unit, and is worn on the user's head, The information processing device described in one of the configurations 1 to 12, A device worn on the user's hand and having sensors for measuring position and orientation, A system that has (Method 15) HMD (Head Mount) ed A recognition method for recognizing the hands of a user wearing the HMD from images captured by the Display, Information processing device, The steps include detecting a hand from the captured image taken by the HMD, The steps include acquiring sensing data regarding the position of the device from a sensor mounted on the device worn on the user's hand, For the hand detected from the captured image, a comparison process is performed to compare the movement of the hand recognized from the captured image with the movement of the device recognized from the sensing data, and if the hand movement and the device movement match, the hand detected from the captured image The step of determining that it is the user's hand, A recognition method for executing this. (Method 16) HMD (Head Mount) ed A recognition method for recognizing the hands of a user wearing the HMD from images captured by the Display, Information processing device, The steps include detecting a hand from the captured image taken by the HMD, The steps include acquiring sensing data regarding the orientation of a device from a sensor mounted on a device worn on the user's hand, A comparison process is performed to compare the posture of the hand detected from the captured image with the posture of the device recognized from the sensing data, and if the posture of the hand and the posture of the device match, it is determined that the hand detected from the captured image is the user's hand. A recognition method for executing this. (Method 17) HMD (Head Mount) ed A recognition method for recognizing the hands of a user wearing the HMD from images captured by the Display, Information processing device, The steps include detecting a hand from the captured image taken by the HMD, The steps include acquiring sensing data regarding the position and orientation of a device from a sensor mounted on a device worn on the user's hand, A comparison process is performed to compare the movement and posture of the hand detected from the captured image with the movement and posture of the device recognized from the sensing data, and if the movement and posture of the hand and the movement and posture of the device match, it is determined that the hand detected from the captured image is the user's hand. A recognition method for executing this. (Method 18) HMD (Head Mount) ed A recognition method for recognizing the hands of a user wearing the HMD from images captured by the Display, Information processing device, The steps include detecting a hand from the captured image taken by the HMD, The steps include: determining by image recognition whether the device worn by the user on the hand is worn on the hand detected in the captured image, and if the device is worn on the hand detected in the captured image, determining that the hand detected in the captured image is the user's hand; A recognition method for executing this. (Program 19) A program for causing an information processing device to execute each step of the recognition method described in any one of methods 15 to 18. [Explanation of Symbols]

[0098] 100 HMD 103 Information Processing Device 105 Hand detection unit 109 Processing Unit 110 Proximity Radio Communication Unit 300 small operating devices 303 Sensor

Claims

1. An information processing device that recognizes the hand of a user wearing an HMD (Head Mounted Display) from images captured by the HMD, A detection means for detecting a hand from an image captured by the aforementioned HMD, An acquisition means for acquiring sensing data related to the position of a device from a sensor mounted on a device worn on the user's hand, A processing calculation means performs a comparison process to compare the movement of the hand detected from the captured image with the movement of the device recognized from the sensing data, and determines that the hand detected from the captured image is the user's hand when the hand movement and the device movement match. An information processing device having

2. An information processing device that recognizes the hand of a user wearing an HMD (Head Mounted Display) from images captured by the HMD, A detection means for detecting a hand from an image captured by the aforementioned HMD, An acquisition means for acquiring sensing data related to the orientation of a device from a sensor mounted on a device worn on the user's hand, A processing calculation means performs a comparison process to compare the posture of the hand detected from the captured image with the posture of the device recognized from the sensing data, and determines that the hand detected from the captured image is the user's hand if the posture of the hand and the posture of the device match. An information processing device having

3. An information processing device that recognizes the hand of a user wearing an HMD (Head Mounted Display) from images captured by the HMD, A detection means for detecting a hand from an image captured by the aforementioned HMD, An acquisition means for acquiring sensing data relating to the position and orientation of a device from a sensor mounted on a device worn on the user's hand, For the hand detected from the aforementioned captured image, the movement and posture of the hand recognized from the captured image are compared with the movement and posture of the device recognized from the sensing data. A processing calculation means performs a comparison process and determines that the hand detected from the captured image is the user's hand when the hand movement and posture match those of the device. An information processing device having

4. When one or more hands are detected from the captured image, the processing calculation means narrows down the hands to be compared based on the distance from the HMD to each hand. The information processing apparatus according to any one of claims 1 to 3.

5. When one or more hands are detected from the captured image, the processing calculation means narrows down the hands to be compared based on the direction of movement of each hand. The information processing apparatus according to any one of claims 1 to 3.

6. When one or more hands are detected from the captured image, the processing calculation means narrows down the hands to be compared based on the extension direction of each hand. The information processing apparatus according to any one of claims 1 to 3.

7. The processing calculation means notifies the user that the position of the hand is inappropriate if the user's hand is not recognized from the captured image. The information processing apparatus according to any one of claims 1 to 3.

8. When a change in the position of the device is detected by sensing data acquired from a sensor mounted on the device, the detection means and the processing calculation means execute the necessary operations. The information processing apparatus according to any one of claims 1 to 3.

9. The processing calculation means, in the comparison process, compares the relative movement of the hand position with the relative movement of the device over a predetermined period, and determines that the hand movement and the device movement are in agreement if the relative movement of the hand position and the relative movement of the device are the same or within a predetermined error range. The information processing apparatus according to claim 1 or 3.

10. The aforementioned sensor is an IMU sensor. The information processing apparatus according to any one of claims 1 to 3.

11. The device is a wearable device that is attached to the user's hand or finger. The information processing apparatus according to any one of claims 1 to 3.

12. A goggle device that has an image capture unit and an image display unit, and is worn on the user's head, An information processing device according to any one of claims 1 to 3, HMD with

13. A goggle device that has an image capture unit and an image display unit, and is worn on the user's head, An information processing device according to any one of claims 1 to 3, A device worn on the user's hand and having sensors for measuring position and orientation, A system that has

14. A recognition method for recognizing the hand of a user wearing an HMD (Head-Mounted Display) from images captured by the HMD, Information processing device, The steps include detecting a hand from the captured image taken by the HMD, The steps include acquiring sensing data regarding the position of the device from a sensor mounted on the device worn on the user's hand, The steps include: performing a comparison process to compare the movement of the hand detected from the captured image with the movement of the device recognized from the sensing data, and determining that the hand detected from the captured image is the user's hand if the hand movement and the device movement match; A recognition method for executing this.

15. A recognition method for recognizing the hand of a user wearing an HMD (Head-Mounted Display) from images captured by the HMD, Information processing device, The steps include detecting a hand from the captured image taken by the HMD, The steps include acquiring sensing data regarding the orientation of a device from a sensor mounted on a device worn on the user's hand, A comparison process is performed to compare the posture of the hand detected from the captured image with the posture of the device recognized from the sensing data, and if the posture of the hand and the posture of the device match, it is determined that the hand detected from the captured image is the user's hand. A recognition method for executing this.

16. A recognition method for recognizing the hand of a user wearing an HMD (Head-Mounted Display) from images captured by the HMD, Information processing device, The steps include detecting a hand from the captured image taken by the HMD, The steps include acquiring sensing data regarding the position and orientation of a device from a sensor mounted on a device worn on the user's hand, The steps include: performing a comparison process to compare the movement and posture of the hand detected from the captured image with the movement and posture of the device recognized from the sensing data, and determining that the hand detected from the captured image is the user's hand if the hand movement and posture and the device movement and posture match; A recognition method for executing this.

17. A program for causing an information processing device to execute each step of the recognition method described in any one of claims 14 to 16.

Images