Information processing device, information processing method, program

Abstract

To provide a technique for implementing high operability for a user using gesture for operation.SOLUTION: An information processing apparatus includes: detection means which detects gesture of a first hand of a user; and processing means which switches the mode to a second operation mode, in a first operation mode in which first processing is executed by first operation using a second hand of the user, when the detection means detects specific gesture of the first hand, the second operation mode allowing second processing different from the first processing to be executed by the first operation using the second hand.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program. Background Art

[0002] There is a Mixed Reality (MR) technology that fuses a real space and a virtual space to allow an experiencer to perceive virtual objects. In the MR technology, a system synthesizes and presents computer graphics (CG) representing a virtual object onto a real landscape, and expresses contact between a real object and the virtual object.

[0003] In the MR technology, it is possible to operate a virtual object superimposed on a real landscape in accordance with a gesture by a user's hand. The user can move the CG object or the like by a gesture without using a controller. Further, the user can also perform character input, or draw lines, characters, or figures in a designated drawing area such as a whiteboard by a gesture.

[0004] Patent Document 1 discloses a method of displaying a GUI for operation and executing a function corresponding to a detected gesture. In Patent Document 1, the GUI for operation is displayed at a display position corresponding to the detection position of the gesture. In the GUI for operation, gestures and functions are associated with each other, and the function corresponding to the GUI for operation is executed in accordance with the detected gesture Prior Art Documents Patent Documents

[0005] Patent Document 1 Japanese Patent Laid-Open No. 2018-151851 Patent Document 2 Japanese Patent Laid-Open No. 2012-13514 Summary of Invention Problems to be Solved by the Invention

[0006] However, Patent Document 1 assumes that operations using gestures and a GUI are performed with one hand. As a result, there are usability issues (inconvenience, inefficiency, and cumbersomeness) due to performing gestures with one hand and operating the GUI with the same hand.

[0007] Therefore, the present invention aims to provide a technology that enables high operability for the user when using gestures for operation. [Means for solving the problem]

[0008] One aspect of the present invention is, An information processing device that generates data to be displayed on a head-mounted display, A detection means for detecting the user's first hand gesture, In a first operating mode in which a first process is executed by a first operation by the user's second hand, if the detection means detects a specific gesture of the first hand, the processing means switches to a second operating mode in which a second process different from the first process is executed by the first operation by the second hand. to have death, The processing means disables the switching of the operation mode by the first hand gesture when the head-mounted display is tilted downwards at a certain angle with respect to the horizontal. This is an information processing device characterized by the following features.

[0009] One aspect of the present invention is, An information processing method for generating data to be displayed on a head-mounted display, A detection step to detect the user's first hand gesture, a processing step of, in a first operation mode in which a first process is executed by a first operation performed by the user's second hand, switching to a second operation mode in which a second process different from the first process is executed by the first operation performed by the second hand, when a specific gesture of the first hand is detected in the detection step If the head-mounted display is tilted downwards at a certain angle relative to the horizontal, the first step of disabling the switching of the operation mode by the hand gesture is disabled. The information processing method is characterized by comprising [Advantageous Effects of Invention]

[0010] According to the present invention, high operability for a user can be achieved when gestures are used for operations. [Brief Description of Drawings]

[0011] [Figure 1] This is a block configuration diagram of the MR system according to the first embodiment. [Figure 2] This is a hardware configuration diagram of the MR system according to the first embodiment. [Figure 3] This is a block configuration diagram of the controller according to the second embodiment. [Figure 4] This is a diagram for explaining connection between the HMD and the controller according to the second embodiment. [Figure 5] This is a diagram for explaining operation mode switching according to the first embodiment. [Figure 6] This is a diagram for explaining rendering processing according to the first embodiment. [Figure 7] This is a flowchart of switching processing related to rendering according to the first embodiment. [Figure 8] This is a schematic flowchart of the switching processing according to the first embodiment. [Figure 9] This is a flowchart of processing of the MR system according to the second embodiment. [Figure 10] This is a diagram illustrating a relationship between a hand and a character group according to the third embodiment. [Figure 11] This is a diagram for explaining character input according to the third embodiment. [Figure 12]This is a diagram for explaining character input according to the fourth embodiment. [Figure 13] This is a diagram for explaining character erasure according to the fifth embodiment. [Figure 14] This is a diagram for explaining character string conversion according to the sixth embodiment. [Figure 15] This is a flowchart of processing of the MR system according to the third embodiment. [Figure 16] This is a flowchart of processing of the MR system according to the fourth embodiment. [Figure 17] This is a flowchart of processing of the MR system according to the fifth embodiment. [Figure 18] This is a flowchart of processing of the MR system according to the sixth embodiment. [Modes for Carrying Out the Invention]

[0012] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the embodiments described below are merely examples of implementation means of the present invention, and may be appropriately modified or changed according to the configuration of the apparatus to which the present invention is applied and various conditions. It is also possible to appropriately combine the respective embodiments.

[0013] <Embodiment 1> Figure 2 shows the hardware configuration of the MR system 1 (mixed reality system) according to the first embodiment. Figure 1 shows the block configuration of the MR system 1.

[0014] As shown in Figure 2, the MR system 1 includes an image capturing unit 101, a display unit 102, a communication unit 110, a CPU 200, a RAM 201, and a ROM 202.

[0015] The CPU 200 is the control unit that controls the entire MR system 1. The CPU 200 controls the entire MR system 1 by executing the application that operates the MR system 1. Specifically, the CPU 200 loads the application (program) stored in the ROM 202 into the RAM 201 and then executes the application. Each part shown in Figure 1 is realized by the CPU 200. However, one or more of the configurations shown in Figure 1 may be realized by hardware independent of the CPU 200.

[0016] As shown in Figure 1, the MR system 1 includes an HMD 100 and an information processing device 103.

[0017] The HMD100 is a head-mounted display device. The HMD100 is a video see-through type display device. The HMD100 has an image acquisition unit 101 and a display unit 102. Information regarding the settings of the HMD100 (setting information) is stored in the ROM 202.

[0018] In the HMD100, the image acquisition unit 101 is positioned in front of the eyes of the user wearing the HMD100. The display unit 102 displays the image (captured image) captured by the image acquisition unit 101. In the case of a video see-through type HMD, it is common to treat the position and orientation of the image acquisition unit 101 as the position and orientation of the user. Although the HMD100 in this embodiment is assumed to be a video see-through type HMD, this is merely an example. For example, this embodiment can also be applied to a virtual reality HMD that does not display captured images (images of real space) on the display unit 102. In this case, the image acquisition unit 101 is used, for example, as a camera to capture images of a hand, which is an object to be measured.

[0019] The image acquisition unit 101 captures images of the real world at a predetermined frame rate (e.g., 30 frames / second). The image acquisition unit 101 is fixed to the housing of the HMD 100. The image acquisition unit 101 has two real cameras. The real camera for the left eye and the real camera for the right eye are positioned close to the user's eyes. The pair of left and right images obtained by the image acquisition unit 101 can also be called a "stereo camera image".

[0020] The information processing device 103 generates data for display on the display unit 102 based on the captured image acquired by the image acquisition unit 101. The information processing device 103 may be included in the HMD 100 or may be hardware independent of the HMD 100. The information processing device 103 includes an captured image storage unit 104, a contour point extraction unit 105, a three-dimensional position calculation unit 106, a calculation unit 107, an image drawing unit 108, a processing calculation unit 109, and a communication unit 110.

[0021] The image storage unit 104 stores the captured images acquired by the image acquisition unit 101. The image storage unit 104 is implemented by RAM 201. Therefore, the captured images are stored in RAM 201.

[0022] The contour point extraction unit 105 extracts contour points of fingers from the captured image using image recognition technology.

[0023] The three-dimensional position calculation unit 106 calculates the three-dimensional positions (x, y, and z coordinates) of the contour points of the fingers extracted by the contour point extraction unit 105. The x and y coordinates are coordinates with the origin (0,0) being the lower left corner of the VRAM area of ​​the captured image. The coordinates of the origin do not have to be the lower left corner of the VRAM area and can be freely determined by the user. The z coordinate is a coordinate indicating the distance from the HMD 100 and can be obtained based on the captured image.

[0024] Furthermore, in order to correctly display the depth of the hand (value of the z coordinate), a method can be used in which the hand region is extracted from an image captured by a stereo camera, and the depth value of the hand relative to the stereo camera is calculated. This method can be used to determine the depth value by triangulation for all corresponding points in the image of the extracted hand contour, as described in Patent Document 2. Patent Document 2 utilizes a stereo camera mounted on a video see-through type HMD (display device for presenting augmented reality).

[0025] Furthermore, LeapMotion, Inc.'s LeapMotion can measure the position and orientation (position and orientation) of the hand, including the fingers. LeapMotion can detect the hand area from a stereo camera (image acquisition unit 101). Furthermore, by using a depth sensor (distance sensor)... Since the position and orientation of the fingers can also be estimated by this, a depth sensor may be installed on the HMD100.

[0026] In this embodiment, the three-dimensional position calculation unit 106 determines an arbitrary point from the contour points extracted by the contour point extraction unit 105 and calculates the three-dimensional position of the determined contour point. The position information of the determined contour point is stored in the RAM 201 as reference position information. In addition, new three-dimensional position information is periodically stored in the RAM 201 in accordance with the movement of the fingers.

[0027] The calculation unit 107 calculates the position and orientation of the hand in the captured image stored in the captured image storage unit 104. The hand position and orientation can be calculated using methods such as deep learning or by using existing publicly available libraries.

[0028] The image drawing unit 108 draws an image for display on the display unit 102 of the HMD 100 based on the processing results of the processing calculation unit 109.

[0029] The processing unit 109 compares the three-dimensional position (coordinate value) calculated by the three-dimensional position calculation unit 106 with the value (coordinate value) measured by an external sensor (such as the sensor 303 described later). The value measured by the sensor is the value obtained by Bluetooth® communication from the controller 300 (small operating device) described later. Specifically, the processing unit 109 determines whether the coordinate change of the hand detected by the HMD 100 matches the coordinate change of the controller 300. If the two coordinate changes match, the processing unit 109 determines that the hand detected by the HMD 100 and the hand to which the controller 300 is attached are the same hand (i.e., the user's hand of the HMD 100). Generally, there is an error in the coordinates obtained by sensors compared to the actual coordinates. Therefore, the processing unit 109 may determine that the two coordinate changes match if, for example, the difference between the two coordinate changes falls within a certain error rate (e.g., 5 percent). The error rate may be arbitrarily determined by the user.

[0030] Here, the processing unit 109 may perform calculations on the IMU information (inertia information) received from the controller 300 in order to obtain a value to compare with the three-dimensional position calculated by the three-dimensional position calculation unit 106. Alternatively, the control unit 301 of the controller 300 may perform calculations on the IMU information and transmit the resulting information to the HMD 100.

[0031] The communication unit 110 includes, for example, an antenna for wireless communication, a modulation / demodulation circuit (a circuit for processing wireless signals), and a communication controller. The communication unit 110 outputs a modulated wireless signal from the antenna, or demodulates a wireless signal received by the antenna. In this way, the communication unit 110 realizes short-range wireless communication in accordance with the IEEE 802.15 standard (so-called Bluetooth®). In this embodiment, Bluetooth Low Energy version 5.1, which has low power consumption, is used for Bluetooth communication.

[0032] Figures 5A and 5B illustrate the switching of operating modes in Embodiment 1. Hereinafter, one hand used to switch operating modes by gesture will be referred to as the "gesture hand." The other hand of the two hands will be referred to as the "processing hand." Furthermore, in Embodiment 1, the operation of the HMD100 is controlled by gesture, so gesture can be considered a "type of user operation."

[0033] In Figure 5A, the user is drawing characters using gestures (hand gestures) with their right hand as the hand that performs drawing operations (processing hand 501) on the drawing area 500. Here, the user uses their left hand as the hand that performs gestures to switch operation modes (gesture hand 502), thereby switching the processing hand The process being performed by 501 (execution process) can be switched to a different process. In Figure 5B, the gesture of the gesture hand 502 switches the process performed by the processing hand 501 from character drawing mode to character erasing mode. In order to switch the operation mode by gesture, the target operation mode (and the processes that can be executed in that operation mode) is pre-associated (assigned) to each gesture.

[0034] (Outline of the switching process) The following outlines the process by which the CPU 200 of the MR system 1 according to Embodiment 1 switches its operating mode, with reference to the flowchart in Figure 8.

[0035] Furthermore, the CPU 200, as a detection unit, can detect gestures of both the left and right hands from the captured image (the image acquired by the image acquisition unit 101). The CPU 200, as a processing unit, can switch the operating mode as described above, depending on whether the gesture is from the left or right hand.

[0036] In step S800, the CPU 200 determines whether a hand has been detected in the captured image (the image acquired by the image acquisition unit 101). The CPU 200 controls, for example, the contour point extraction unit 105 and the three-dimensional position calculation unit 106 to determine whether a hand has been detected in the captured image. If it is determined that a hand has been detected, the process proceeds to step S801. If it is determined that no hand has been detected, the process in step S800 is repeated.

[0037] In step S801, the CPU 200 determines whether the hand detected in step S800 is a gesture hand. If the detected hand is determined to be a gesture hand, the process proceeds to step S802. If the detected hand is determined not to be a gesture hand, the process returns to step S800.

[0038] For example, if the detected hand has an operating device attached, the CPU 200 determines that the detected hand is a gesture hand. Alternatively, the user may pre-set either the right or left hand as the gesture hand in the system settings menu screen. For example, if the left hand is set as the gesture hand, the CPU 200 determines in step S800 whether the detected hand is the left hand.

[0039] In step S802, the CPU 200 determines whether or not it has detected a gesture made by the hand (=gesture hand) detected in step S800. If it is determined that a gesture has been detected, the process proceeds to step S803. If it is determined that no gesture has been detected, the process in step S802 is repeated.

[0040] In step S803, the CPU 200 switches to the operating mode associated with the detected gesture (the operating mode that performs the processing associated with the gesture). The CPU 200 has previously associated gestures with operating modes and switches to the operating mode associated with the detected gesture. In Embodiment 1, different operating modes result in different processing being performed by the gesture of the processing hand.

[0041] (Example of drawing) Figures 6A to 6C illustrate an example of a user performing drawing using the HMD100 according to Embodiment 1.

[0042] In Figure 6A, the user is standing and drawing on the drawing area 600 with their right hand 601 (processing hand). The left hand 602, which is the gesture hand, is performing the same action as the right hand. While 601 is drawing, it does not need to move unless there is a need to switch operating modes. For this reason, in Figure 6A, the left hand 602 is in a lowered position.

[0043] Figure 6B shows the state where the user's left hand 602 is performing a gesture to switch the operating mode. At this time, in order to determine the gesture of the left hand 602, the left hand 602 must be included in the imaging range of the image acquisition unit 101 of the HMD 100. For this reason, the user raises their left hand 602 so that it is included in the imaging range of the image acquisition unit 101 of the HMD 100.

[0044] On the other hand, Figure 6C shows a state where drawing is performed after the drawing area 600 has been set on the desk while the user is seated. When the user is seated, even if there is no need to switch the operation mode by gesture, the left hand 602 is likely to be positioned over the drawing area 600. Therefore, there is a high possibility that the left hand 602 will be included in the imaging range of the image acquisition unit 101. Consequently, when the user is seated and drawing on a desk, it is necessary to restrict the switching of operation modes by gesture.

[0045] Furthermore, when a user is seated and performing tasks such as drawing on a desk, the HMD100 will be tilted downwards from the horizontal. Therefore, the HMD100 will be... HMD If the orientation of 100 is pointing downwards by a certain angle θ (or greater than a certain angle θ) from the horizontal, it can be determined that the user is seated and drawing on a desk.

[0046] Referring to the flowchart in Figure 7, we will explain the switching process executed by the CPU 200 of the MR system 1 (for example, the CPU 200 of the information processing device 103) when the user is drawing in the drawing area. In other words, the flowchart in Figure 7 is a flowchart that explains the flowchart in Figure 8 in more detail when drawing is being performed. Note that in order for the HMD 100 to detect gestures, both hands must be included in the drawing area in real space (an area with functions such as a whiteboard).

[0047] In step S700, the CPU 200 determines whether both hands are located in the drawing area based on the image captured by the image acquisition unit 101. If it is determined that both hands are not located in the drawing area, the process in step S700 is repeated. If it is determined that both hands are located in the drawing area, the process proceeds to step S701.

[0048] In step S701, the CPU 200 determines whether the user (HMD 100) is facing downwards. For example, the CPU 200 determines whether the HMD 100 is tilted at a certain angle θ in the negative direction (downward) relative to the horizontal direction (Figure 6 The system determines that the user is looking downwards if the tilt is greater than (see reference C). On the other hand, the CPU200 can determine that the user is performing drawing operations using a wall or the air as the drawing area if the tilt is less than a certain angle θ.

[0049] If it is determined that the user is looking downwards, the CPU 200 detects feature points from the captured image acquired by the image acquisition unit 101 and detects a plane in real space based on the feature points. Then, the process proceeds to step S702. On the other hand, if it is determined that the user is not looking downwards, the process proceeds to step S706.

[0050] In step S702, the CPU 200 determines whether the distance between the hands detected in step S700 and the plane (the plane detected in step S701) is shorter than a predetermined distance. If it is determined that the distance between the hands and the plane is shorter than the predetermined distance, the process proceeds to step S703. If it is determined that the distance between the hands and the plane is greater than or equal to the predetermined distance, the process proceeds to step S706.

[0051] In step S703, the CPU200 disables the switching of the operating mode by gestures of the gesture hand. This is to prevent the operating mode from being switched when the gesture hand is included in the imaging range and a gesture is detected, unless the user intends to switch the operating mode.

[0052] In step S704, the CPU 200 determines whether or not it has detected a predetermined gesture (a gesture to enable switching of operation modes by gesture). Here, the predetermined gesture is assumed to be pre-configured. If it is determined that the predetermined gesture has been detected, the process proceeds to step S705. If it is determined that the predetermined gesture has not been detected, the process in step S704 is repeated. Note that if a controller or the like is attached to the hand, "a predetermined operation on the controller" may be used instead of "a predetermined gesture".

[0053] In step S705, CPU200 enables the switching of the operation mode by gesture of the gesture hand (it undoes the disabling of the switching of the operation mode by gesture of the gesture hand).

[0054] In step S706, the CPU 200 determines, based on the captured image, whether or not a gesture by the gesture hand has been detected.

[0055] In step S707, the CPU 200 switches to the operating mode associated with the gesture detected in step S706. This causes the CPU 200 to switch the processing (execution process) to be performed in response to the operation of the processing hand. For this reason, if a certain gesture is performed with the gesture hand, and a certain operation is performed by the processing hand, the first process will be executed. On the other hand, if a different gesture is performed with the gesture hand, and the same operation is performed by the processing hand, a second process different from the first process will be executed.

[0056] In this case, if you want to frequently switch between processes performed by the processing hand (the hand that executes the process), it would be inefficient to perform both the execution and switching of processes using only the processing hand. On the other hand, in this embodiment, the operation mode is switched by a gesture of the other hand, which improves work efficiency (operability).

[0057] <Embodiment 2> Embodiment 2 describes a method for pre-setting a processing hand and a gesture hand using a controller 300 that is held in or worn on the user's hand.

[0058] Figure 3 shows the block configuration of the controller 300. The controller 300 is an operating device that can be worn on the fingers. However, the shape of the controller 300 does not need to be one that can be worn on the fingers; it can be any shape that can be held by the user's hand. The controller 300 includes a control unit 301, an operating unit 302, a sensor 303, a working memory 304, a power control unit 305, and a communication unit 306.

[0059] The control unit 301 controls the controller 300 (the entire device) according to the input signal or program. Alternatively, instead of the control unit 301 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0060] The operation unit 302 receives instructions from the user for the controller 300. For example, the operation unit 302 receives instructions from the user for the power button (power supply to the control unit 301 of the controller 300). supply This includes buttons for the user to control the main power ON / OFF, and mode switching buttons (buttons for switching operating modes). The operation unit 302 further includes a communication unit 306. It includes a button for initiating communication with external devices such as the HMD100.

[0061] Sensor 303 is a sensor for acquiring information such as IMU (Inertial Measurement Unit) information (hereinafter referred to as "sensor information"). Based on angular velocity (value detected by the gyro sensor) and acceleration (value detected by the accelerometer), sensor 303 can calculate the change in attitude, orientation (relative direction), and position of the controller 300.

[0062] The working memory 304 is used as a buffer memory (a memory for temporarily storing sensor information acquired by the sensor 303) and as a working area for the control unit 301, etc.

[0063] The power control unit 305 supplies power for the controller 300 to operate.

[0064] The communication unit 306, like the communication unit 110 of the HMD100, includes, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller. The communication unit 306 implements short-range wireless communication in accordance with the IEEE 802.15 standard.

[0065] Figure 4 shows the connection configuration between the HMD100 and the controller 300. The HMD100 and the controller 300 are connected via Bluetooth communication.

[0066] For example, sensor information acquired by sensor 303 is transmitted to the communication unit 110 of HMD 100 via the communication unit 306 of controller 300. Data communication between HMD 100 and controller 300 is not limited to Bluetooth; other communication methods can also be used.

[0067] Furthermore, the movement of the hand to which the controller 300 is attached can be determined from the movement of the controller 300 (sensor information). Therefore, the hand to which the controller 300 is attached does not necessarily have to be included in the imaging range of the image acquisition unit 101.

[0068] Here, the sensor information acquired by sensor 303 is sent to the communication unit 110 of HMD 100 via the communication unit 306. The HMD 100 can then detect the movement of the controller 300 based on the sensor information from sensor 303. Therefore, the HMD 100 can acquire more precise hand movements by using the sensor information from sensor 303, in addition to acquiring hand movements based on captured images. Furthermore, the HMD 100 can also acquire hand movements based on both captured images and sensor information. Consequently, when the controller 300 is attached, the hand to which the controller 300 is attached is used as a processing hand to enable fine control of the processing.

[0069] Referring to the flowchart in Figure 9, the processing performed by the CPU 200 of the MR system 1 according to Embodiment 2 will be explained. Below, referring to Figure 9, the process of setting the gesture hand (the hand that performs the gesture) and the processing hand (the hand that executes the processing) will be explained.

[0070] In step S900, the CPU 200 determines whether the controller 300 is attached to the user's hand. If it is determined that the controller 300 is attached to the user's hand, the process proceeds to step S901. If it is determined that the controller 300 is not attached to the user's hand, the process proceeds to step S902.

[0071] Here, the CPU 200 processes the image captured by the image capturing unit 101 into the image (image capture area). If both hands are visible in the image, the CPU 200 can determine from the captured image whether or not the controller 300 is attached to the hands. On the other hand, if only one hand is visible in the captured image, the CPU 200 can determine from the captured image whether or not the controller 300 is attached to the visible hand. Furthermore, if the HMD 100 has been communicating with the controller 300, the CPU 200 can detect that the controller 300 is attached to the user's hand even if at least one hand is not visible in the captured image.

[0072] In step S901, the CPU200 sets the hand without the controller 300 attached as the gesture hand. The CPU200 also sets the hand with the controller 300 attached as the processing hand.

[0073] In step S902, the CPU 200 sets one of the left or right hands as a gesture hand and the other hand as a processing hand, in response to user operations in the system menu. In other words, if the controller 300 is not installed, the CPU 200 sets either the left or right hand as a gesture hand in the same manner as in Embodiment 1.

[0074] In Embodiment 2, the hand to which the controller 300 is attached is treated as the processing hand. This allows the user to perform processing with the hand that is better able to accurately detect movement. As a result, it becomes easier to achieve the processing intended by the user.

[0075] After the processing hand and gesture hand are set, the flowchart shown in Figure 7 or Figure 8 of Embodiment 1 is performed. Furthermore, when using a controller 300 that needs to be grasped by the user, it is not easy for the user to perform fine processing with the hand holding the controller 300. For this reason, in such cases, the CPU 200 may set the hand not holding the controller 300 as the processing hand and the hand holding the controller 300 as the gesture hand.

[0076] <Embodiment 3> Embodiment 3 describes an MR system 1 that inputs characters using hand movements.

[0077] Figures 10A and 10B are schematic diagrams illustrating the relationship between a hand and a character group. In Embodiment 3, the user selects one of a character group 1000 containing multiple characters using the processing hand 501. Then, the user selects one character from the selected character group 1000 using the gesture hand 502. This inputs the character selected by the user. The multiple characters contained in each character group 1000 are pre-set.

[0078] For example, in Figure 10A, each of the character groups 1000 contains multiple Japanese hiragana characters. In Figure 10B, each of the character groups 1000 contains multiple alphabetic characters.

[0079] Here, the CPU 200 switches the selected character group 1000 when the calculation unit 107 detects the first gesture made by the processing hand 501 and also detects that the processing hand 501 has rotated. Note that hand rotation is a type of hand gesture.

[0080] Subsequently, if the calculation unit 107 detects a second gesture by the gesture hand 502, the CPU 200 selects a character from the selected character group 1000. The CPU 200 detects that the gesture hand 502 has rotated and switches the character to be selected. The CPU 200 inputs (confirms) a character if, for example, the calculation unit 107 detects a gesture other than the second gesture by the gesture hand 502. The first and second gestures must be set in advance. In the following explanation, both the first and second gestures will be described as pinching gestures (see Figures 10A and 10B). However, the first and second gestures may be different gestures from each other.

[0081] Referring to the flowchart in Figure 15, the processing performed by the CPU 200 of the MR system 1 according to Embodiment 3 will be explained.

[0082] In step S1500, the CPU 200 determines whether the processing hand 501 has performed the first gesture. If it is determined that the processing hand 501 has performed the first gesture, the process proceeds to step S1501. If it is determined that the processing hand 501 has not performed the first gesture, the process in step S1500 is repeated.

[0083] In step S1501, the CPU 200 switches its operating mode to character group selection mode. When the operating mode is switched to character group selection mode, the CPU 200 displays the character input area 1100 on the display unit 102 so that it is superimposed on the real space, as shown in (1) of Figure 11A and (1) of Figure 11B. The character input area 1100 is displayed at a predetermined distance above the higher of the two hands, for example, the processing hand 501 and the gesture hand 502. The CPU 200 also selects one of the pre-set character groups 1000 (for example, the first character group 1000).

[0084] In step S1502, the CPU 200 displays a group selection screen 1107 on the display unit 102, which is a screen for selecting a character group 1000, as shown in Figure 11A (2) and Figure 11B (2). As shown in Figures 11A and 11B, the selected character group 1000 or character is surrounded by a thick border.

[0085] In step S1503, the CPU 200 determines whether the processing hand 501 (the wrist of the processing hand 501) has rotated by a predetermined angle (threshold) or more (for example, whether the processing hand 501 has changed to a predetermined shape). If it is determined that the processing hand 501 has rotated by a predetermined angle or more, the process proceeds to step S1504. If it is determined that the processing hand 501 has not rotated by a predetermined angle or more, the process proceeds to step S1505.

[0086] In step S1504, the CPU 200 switches the character group 1000 to be selected according to the rotation direction of the processing hand 501 (the wrist of the processing hand 501), as shown in (3) of Figure 11A and (3) of Figure 11B. For example, if the processing hand 501 is rotating to the right, the CPU 200 selects the character group 1000 to the right of the currently selected character group 1000 on the group selection screen 1107.

[0087] In step S1505, the CPU 200 determines whether the gesture hand 502 has performed a second gesture. If it is determined that the gesture hand 502 has performed a second gesture, the process proceeds to step S1506. If it is determined that the gesture hand 502 has not performed a second gesture, the process proceeds to step S1503.

[0088] In step S1506, CPU200 processes the selected character group 1000. The operation mode is switched to character selection mode, which allows selecting a single character. At this time, the CPU 200 selects, for example, the first character of the selected character group 1000.

[0089] In step S1507, the CPU 200 displays a character selection screen 1108 (a screen in which multiple characters included in the selected character group 1000 are arranged in a line) on the display unit 102, as shown in (4) of Figure 11A and (4) of Figure 11B.

[0090] In step S1508, the CPU 200 determines whether the gesture hand 502 (the wrist of the gesture hand 502) has rotated by more than a predetermined angle (for example, whether the gesture hand 502 has changed to a predetermined shape). If it is determined that the gesture hand 502 has rotated by more than a predetermined angle, the process proceeds to step S1509. If it is determined that the gesture hand 502 has not rotated by more than a predetermined angle, the process proceeds to step S1510.

[0091] In step S1509, the CPU 200 switches the character to be selected according to the rotation direction of the gesture hand 502, as shown in (5) of Figure 11A and (5) of Figure 11B. For example, if the hand is rotating to the right, the CPU 200 selects the character to the right of the currently selected character on the character selection screen 1108.

[0092] In step S1510, the CPU 200 determines whether the gesture hand 502 has performed a second operation (a specific operation other than the second gesture). If it is determined that the gesture hand 502 has performed a second operation, the process proceeds to step S1511. If it is determined that the gesture hand 502 has not performed a second operation, the process proceeds to step S1508. Here, the second operation may be a gesture made by the gesture hand 502, or it may be an operation on a controller or the like attached to the gesture hand 502.

[0093] In step S1511, the CPU 200 inputs the currently selected character into the character input area 1100. In other words, the CPU 200 confirms the selection of the currently selected character.

[0094] In step S1512, CPU200 switches from character selection mode to character group selection mode.

[0095] In step S1513, the CPU 200 terminates the display of the character selection screen 1108 (hides the character selection screen 1108) as shown in (6) of Figure 11A and (6) of Figure 11B.

[0096] In step S1514, the CPU 200 determines whether the processing hand 501 has performed a first operation (an operation other than the first gesture). If it is determined that the processing hand 501 has performed a first operation, the process proceeds to step S1515. If it is determined that the processing hand 501 has not performed a first operation, the process proceeds to step S1501. Here, the first operation may be a gesture made by the processing hand 501, or it may be an operation on a controller or the like attached to the processing hand 501.

[0097] In step S1515, CPU200 exits the character group selection mode.

[0098] In step S1516, the CPU 200 terminates the display of the group selection screen 1107, as shown in (7) of Figure 11A and (7) of Figure 11B.

[0099] In the example shown in Figure 11A, when the processing hand 501 performs a first gesture (step S1500YES), the group selection screen 1107 is displayed as shown in (2) of Figure 11A (step S1502). Next, in response to the rotation of the processing hand 501 (step S1503YES), the selected character group 1000 is switched as shown in (3) of Figure 11A (step S1504). When the gesture hand 502 performs a second gesture (step S1505YES), the character selection screen 1108 is displayed as shown in (4) of Figure 11A (step S1507).

[0100] Subsequently, in accordance with the rotation of the gesture hand 502 (S1508YES), the selected character in the character selection screen 1108 is switched, as shown in (5) of Figure 11A (step S1509). When the gesture hand 502 performs a second operation (step S1510YES), the selected character is entered into the character input area 1100, as shown in (6) of Figure 11A (step S1511). Then, when the processing hand 501 performs a first operation (step S1514YES), the group selection screen 1107 is hidden, as shown in (7) of Figure 11A (step S1516).

[0101] According to Embodiment 3, the user can input characters without touching any operating components such as a keyboard or touch panel. This allows the user to input characters hygienically. Furthermore, since each hand is assigned a specific role, character input can be achieved without performing complex gestures with one hand.

[0102] <Embodiment 4> Embodiment 4 describes an MR system 1 that can switch to subgroups associated with character group 1000. Here, a subgroup is a group that includes characters related to each character included in character group 1000. For example, if character group 1000 includes multiple hiragana characters, the subgroup will include voiced versions of those multiple hiragana characters.

[0103] Figures 12A and 12B illustrate the processing according to Embodiment 4. For example, when selecting characters within character group 1000, if rotation of the processing hand 501 is detected, the system switches to the subgroup associated with the selected character group 1000. For example, Figure 12B shows the process of switching from character group 1000, which represents "E, F, G, H", to the subgroup, which represents "e, f, g, h", after rotation of the processing hand 501 is detected.

[0104] Embodiment 4 differs from the flowchart in Figure 15 in that steps S1610 and S1611 are added to the process in the flowchart in Figure 15 relating to Embodiment 3. For this reason, only the processes of steps S1610 and S1611 will be described. The process of step S1610 starts when it is determined in step S1508 that the gesture hand 502 has not rotated by more than a predetermined angle, or when the process of step S1509 is completed.

[0105] In step S1610, the CPU 200 determines whether the processing hand 501 has rotated by more than a predetermined angle. If it is determined that the processing hand 501 has rotated by more than a predetermined angle, the process proceeds to step S1611. If it is determined that the processing hand 501 has not rotated by more than a predetermined angle, the process proceeds to step S1510.

[0106] In step S1611, the CPU 200 switches the selected character group 1000 on the character selection screen 1108 to the subgroup associated with that character group 1000. Then, the CPU 200 selects the character in the subgroup that corresponds to the character currently selected in character group 1000.

[0107] According to Embodiment 4, it becomes possible to intuitively input voiced consonants in Japanese, or special characters such as lowercase letters and symbols in English.

[0108] <Embodiment 5> In Embodiment 5, the MR system 1 further erases characters.

[0109] Figures 13A and 13B illustrate the erasure of characters in Embodiment 5. When selecting characters within a character group 1000, if it is detected that the gesture hand 502 has rotated in a direction where no characters exist relative to the currently selected character, the characters are erased. During character erasure, a character erasure item 1302 is displayed, and characters continue to be erased. Character erasure ends when it is detected that the gesture hand 502 has rotated in a direction where characters exist relative to the character erasure item 1302. Character erasure may also be performed if any gesture by the gesture hand 502 is detected that indicates a direction in which characters do not exist relative to the currently selected character, among the directions in which characters are arranged on the character selection screen 1108. For example, if there are no characters to the left of the currently selected character, and a gesture by the gesture hand 502 pointing to the left with the index finger is detected, the characters may be erased.

[0110] Figure 13A shows how the gesture hand 502 is detected to have rotated to the left, and the character "no" is deleted from the character "kaimono" entered in the character input area 1100. Figure 13B shows how the gesture hand 502 is detected to have rotated to the left (a gesture indicating leftward movement), and the character "E" is deleted from the character "IMAGE" entered in the character input area 1100.

[0111] Referring to the flowchart in Figure 17, the processing performed by the CPU 200 of the MR system 1 according to Embodiment 5 will be explained. In the processing according to Embodiment 5, the processing of steps S1710 to S1712, as shown in the flowchart in Figure 17, is added to the processing in the flowchart in Figure 15. Therefore, only the processing of steps S1710 to S1712 will be explained below.

[0112] The process in step S1710 begins when it is determined in step S1508 that the gesture hand 502 has rotated by more than a predetermined angle.

[0113] In step S1710, the CPU 200 determines whether a character exists in the direction of rotation of the gesture hand 502 relative to the character selected on the character selection screen 1108. If it is determined that no character exists in the direction of rotation of the hand relative to the selected character, the process proceeds to step S1711. If it is determined that a character exists in the direction of rotation of the hand relative to the selected character, the process proceeds to step S1509.

[0114] In step S1711, the CPU 200 erases the last character entered in the character input area 1100. The CPU 200 also displays the character erasure item 1302.

[0115] In step S1712, if the CPU 200 detects that the gesture hand 502 has rotated by more than a predetermined angle, it determines whether or not a character exists in the direction of rotation of the gesture hand 502 relative to the character erasure item 1302. If it is determined that a character exists in the direction of rotation of the gesture hand 502, the process proceeds to step S1510. Otherwise, the process returns to step S1711.

[0116] For example, in state (2) of Figure 13B, when erasing characters in the character input area 1100, the gesture hand 502 is moved at a predetermined angle or greater in the direction in which the character exists (clockwise rotation direction). By rotating it, it changes to the state shown in (3) of Figure 13B, and the erasure of the characters is completed.

[0117] According to Embodiment 5, the user can perform not only character input but also character deletion using gestures of both the left and right hands.

[0118] <Embodiment 6> In Embodiment 6, the MR system 1 detects the shape and rotation of the user's hands and converts the string of characters (characters) entered in the character input area 1100.

[0119] Figures 14A and 14B illustrate the conversion of strings in Embodiment 6. Figure 14A shows an example of converting a string of hiragana characters to a string of kanji characters. Figure 14B shows an example of converting an English word to a synonym of that English word.

[0120] MR system 1 selects a string conversion candidate 1407 (the converted string) after detecting that the gesture hand 502 has performed a second gesture. MR system 1 changes the string to be converted when it detects that the processing hand 501 has performed a first gesture, and terminates the character conversion when it detects that the gesture hand 502 has performed a second operation.

[0121] Referring to the flowchart in Figure 18, the processing of the CPU 200 of the MR system 1 according to Embodiment 6 will be explained.

[0122] In step S1800, the CPU 200 determines whether the gesture hand 502 has performed a second gesture. If it is determined that a second gesture has been performed, the process proceeds to step S1801. If it is determined that a second gesture has not been performed, the process in step S1800 is repeated.

[0123] In step S1801, the CPU200 switches its operating mode to a character conversion mode that converts strings.

[0124] In step S1802, the CPU 200 displays (selects) the first (beginning) character in the character input area 1100 as the character to be converted 1408. At this time, the CPU 200 displays multiple conversion candidates 1407 for the character to be converted 1408, as shown in (2) of Figure 14A and (2) of Figure 14B. The CPU 200 also selects, for example, the top-leftmost conversion candidate 1407.

[0125] In step S1803, the CPU 200 determines whether or not the gesture hand 502 has rotated by a predetermined angle or more. If it is determined that the gesture hand 502 has rotated by the predetermined angle or more, the process proceeds to step S1804. If it is determined that the gesture hand 502 has not rotated by the predetermined angle or more, the process proceeds to step S1805.

[0126] In step S1804, the CPU 200 switches the selected conversion candidate 1407 to a character string present in a direction corresponding to the rotation direction of the gesture hand 502 in the left-right direction. For example, in (3) of FIG. 14A, by rotating the gesture hand 502 clockwise, the state where "kai" is selected as the conversion candidate 1407 for the character string "kai" is switched to the state where "kai" is selected. In (3) of FIG. 14B, by rotating the gesture hand 502 clockwise, the state where "PICTURE" is selected as the conversion candidate 1407 for the character string "IMAGE" is switched to the state where "PHOTOGRAPH" is selected.

[0127] In step S1805, the CPU 200 determines whether or not the processing hand 501 has rotated by a predetermined angle or more. If it is determined that the processing hand 501 has rotated by the predetermined angle or more, the process proceeds to step S1806. If it is determined that the processing hand 501 has not rotated by the predetermined angle or more, the process proceeds to step S1803.

[0128] In step S1806, as shown in (4) of FIG. 14A and (4) of FIG. 14B, the CPU 200 switches the selected conversion candidate 1407 in the up-down direction to a character string present in a direction corresponding to the rotation direction.

[0129] In step S1807, the CPU 200 determines whether the processing hand 501 has performed the first gesture. If it is determined that the first gesture has been performed, the process proceeds to step S1808. If it is determined that the first gesture has not been performed, the process proceeds to S1803.

[0130] In step S1808, CPU200 switches its operating mode to target switching mode, which switches the string 1408 to be converted.

[0131] In step S1809, the CPU 200 determines whether the processing hand 501 has rotated by more than a predetermined angle. If it is determined that the processing hand 501 has rotated by more than a predetermined angle, the process proceeds to step S1810. If it is determined that the processing hand 501 has not rotated by more than a predetermined angle, the process proceeds to step S1811. Note that the rotation of the gesture hand 502 may be determined instead of the rotation of the processing hand 501.

[0132] In step S1810, the CPU 200 switches the string 1408 to be converted according to the rotation direction of the processing hand 501, as shown in (5) of Figure 14A and (5) of Figure 14B.

[0133] In step S1811, the CPU 200 determines whether the processing hand 501 has performed a first operation (an operation other than the first gesture). If it is determined that a first operation has been performed, the process proceeds to step S1812. If it is determined that a first operation has not been performed, the process proceeds to step S1809.

[0134] In step S1812, the CPU 200 decides to perform a conversion on the currently selected string 1408. The CPU 200 switches its operating mode to character conversion mode.

[0135] In step S1813, the CPU 200 displays multiple conversion candidates 1407 for the string 1408 determined in step S1812.

[0136] In step S1814, the CPU 200 determines whether the gesture hand 502 has performed a second operation (an operation other than a specific gesture). If it is determined that a second operation has been performed, the process proceeds to step S1815. If it is determined that a second operation has not been performed, the process proceeds to step S1801.

[0137] In step S1815, CPU200 exits character conversion mode.

[0138] In step S1816, the CPU 200 terminates the display of the string conversion candidates 1407, as shown in (6) of Figure 14A and (6) of Figure 14B.

[0139] According to Embodiment 6, without performing any special operations, the characters entered in Embodiment 3 are entered during character input. It can be converted using a similar operation.

[0140] Instead of the "characters" described in Embodiments 3 to 5, "options" such as "marks" or "illustrations" may be used. Also, instead of "character groups," groups containing multiple options may be used.

[0141] Although it was explained that the processing of each flowchart in each embodiment is executed by the CPU 200 of the MR system 1, this CPU 200 may be included in the HMD 100 or in the information processing device 103. Also, the information processing device 103 may include the HMD 100. Furthermore, the processing that was executed by the processing hand through gestures may be executed by operations (such as pressing or sliding) on ​​a specific operating member by the processing hand.

[0142] Furthermore, in each embodiment, some of the gestures performed by the gesture hand may be performed by the processing hand. Conversely, some of the gestures performed by the processing hand may be performed by the gesture hand. In other words, in some steps of each flowchart, "gesture hand" may be read as "processing hand," and "processing hand" may be read as "gesture hand."

[0143] As described above, in each of the above embodiments, the function associated with the other hand (the process executed by the operation of the other hand) can be switched using a hand gesture detected from the image captured by the image capturing unit 101 of the HMD 100.

[0144] Furthermore, in the above, the statement "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" can be rephrased as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, the statement "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2" can be rephrased as "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, as long as no contradiction arises, the expression "greater than or equal to A" can be replaced with "A or greater than (higher; longer; more)" or rephrased as "greater than (higher; longer; more)." On the other hand, the expression "less than or equal to A" can be replaced with "A or less than (lower; shorter; fewer)" or rephrased as "less than (lower; shorter; fewer)." Furthermore, "larger than A (higher; longer; more)" can be rephrased as "greater than or equal to A," and "smaller than A (lower; shorter; fewer)" can be rephrased as "less than or equal to A."

[0145] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0146] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0147] The above-disclosed embodiments include the following configurations, methods, and programs. (Composition 1) A detection means for detecting the user's first hand gesture, In a first operating mode in which a first process is executed by a first operation by the user's second hand, if the detection means detects a specific gesture of the first hand, the processing means switches to a second operating mode in which a second process different from the first process is executed by the first operation by the second hand. An information processing device characterized by having the following features. (Configuration 2) The system further includes setting means for setting the user's left hand or right hand as the first hand in response to user operation. The information processing device according to configuration 1, characterized by the above. (Composition 3) If the controller is attached to the user's left or right hand, the system further includes setting means for setting the user's left hand or right hand as the first hand, depending on which hand the controller is attached to. The information processing device according to configuration 1, characterized by the above. (Composition 4) The processing means switches to an operation mode that performs execution processing associated with the first hand gesture in response to the first hand gesture detected by the detection means. The execution process is the process performed by the second operation performed by the second hand. An information processing device according to any one of configurations 1 to 3. (Composition 5) The processing means is The execution process is performed on a specific area of ​​real space displayed on the display device by the second hand operation, If the user's left and right hands are detected in the specified area, the operation mode is switched according to the gesture of the first hand detected by the detection means. The information processing apparatus according to configuration 4, characterized by the features described above. (Composition 6) The aforementioned display device is attached to the user's head, The processing means disables the switching of the operation mode by the first hand gesture when the display device is angled downwards at a certain angle with respect to the horizontal. The information processing apparatus according to configuration 5, characterized by the features described herein. (Composition 7) The processing means, after disabling the switching of the operation mode by the first hand gesture, enables the switching of the operation mode by the first hand gesture when a specific operation is performed. The information processing device according to configuration 6, characterized by the features described therein. (Composition 8) The detection means detects the first hand gesture and the second hand gesture, The processing means switches to the first operating mode when a specific gesture of the second hand is detected. The information processing device according to configuration 1, characterized by the above. (Composition 9) The first operating mode is an operating mode in which one selects one of several groups, each of which contains multiple options. The second operating mode is an operating mode in which one option is selected from the selected group. The information processing apparatus according to configuration 8, characterized by the above. (Composition 10) The device further comprises control means for controlling the display device to display a specific region where a choice is entered, superimposed on the real space including the first hand and the second hand, If one option is selected in the second operating mode, the processing means inputs the currently selected option into the specific area in response to the second operation. The information processing apparatus according to configuration 9, characterized by the features described therein. (Composition 11) The information processing apparatus according to configuration 10, characterized in that the control means controls the display device to display the specific area at a position a certain distance higher than the higher of the first hand position and the second hand position. (Composition 12) The control means controls the display device to display a plurality of options included in the selected group in the second operating mode, In the second operating mode, the processing means, when a gesture of the first hand or the second hand is detected that indicates a direction in which no options exist relative to the selected option among the directions in which the plurality of options are arranged, deletes at least one input option from the specific area. An information processing device according to configuration 10 or 11, characterized by the above. (Composition 13) The processing means is In the first operating mode, the group to be selected is switched depending on the rotation of the first hand or the second hand. In the second operating mode, the selected option is switched according to the rotation of the first hand or the second hand. An information processing apparatus according to any one of the configurations 9 to 12, characterized by the above. (Composition 14) The processing means is In the first operating mode, the group to be selected is switched depending on whether the first hand or the second hand has rotated more than a threshold. In the second operating mode, the selected option is switched depending on whether the first hand or the second hand has rotated more than the threshold. The information processing device according to configuration 13, characterized by the above. (Composition 15) At least one of the aforementioned groups is associated with a subgroup containing multiple options, The processing means, when selecting one option from a first group associated with a first subgroup, switches the first group to the first subgroup in response to a gesture of the first hand or the second hand. An information processing apparatus according to any one of the configurations 9 to 14, characterized by the above. (Composition 16) The first operating mode described above is an operating mode for converting strings, The second operating mode described above is an operating mode that switches the string to be converted. The information processing apparatus according to configuration 8, characterized by the above. (Composition 17) In the first operating mode, the processing means switches the converted string in accordance with at least one of the first hand gesture and the second hand gesture detected by the detection means. The information processing device according to configuration 16, characterized in that... (method) A detection step to detect the user's first hand gesture, In a first operating mode in which a first process is executed by a first operation by the user's second hand, if a specific gesture of the first hand is detected in the detection step, the system switches to a second operating mode in which a second process different from the first process is executed by the first operation by the second hand. An information processing method characterized by having the following features. (program) A program for causing a computer to function as one of the means of an information processing device described in any one of items 1 to 17 of the configuration. [Explanation of Symbols]

[0148] 1: MR system, 100: HMD, 103: Information processing device, 200: CPU

Claims

1. An information processing device that generates data to be displayed on a head-mounted display, A detection means for detecting the user's first hand gesture, In a first operating mode in which a first process is executed by a first operation by the user's second hand, if the detection means detects a specific gesture of the first hand, the processing means switches to a second operating mode in which a second process different from the first process is executed by the first operation by the second hand. It has, The processing means disables the switching of the operation mode by the first hand gesture when the head-mounted display is tilted downwards at a certain angle with respect to the horizontal. An information processing device characterized by the following:

2. The system further includes setting means for setting the user's left hand or right hand as the first hand in response to user operation. The information processing apparatus according to feature 1.

3. When the detection means detects the first hand gesture, the processing means switches to an operation mode that performs execution processing associated with the first hand gesture. The execution process is the process performed by the second operation performed by the hand. The information processing apparatus according to feature 1.

4. The processing means is The execution process is performed on a specific area of ​​the real space displayed on the head-mounted display by the second hand operation. If the user's left and right hands are detected in the aforementioned specific area, the operating mode is switched according to the gesture of the first hand detected by the detection means. The information processing apparatus according to feature 3.

5. The processing means, after disabling the switching of the operation mode by the first hand gesture, enables the switching of the operation mode by the first hand gesture when a specific operation is performed. The information processing apparatus according to feature 1.

6. The detection means detects the first hand gesture and the second hand gesture, The processing means switches to the first operating mode when a specific gesture of the second hand is detected. The information processing apparatus according to feature 1.

7. An information processing method for generating data to be displayed on a head-mounted display, A detection step to detect the user's first hand gesture, In a first operating mode in which a first process is executed by a first operation by the user's second hand, if a specific gesture of the first hand is detected in the detection step, the system switches to a second operating mode in which a second process different from the first process is executed by the first operation by the second hand. If the head-mounted display is tilted downwards at a certain angle relative to the horizontal, the first step of disabling the switching of the operation mode by the hand gesture is disabled. An information processing method characterized by having the following features.

8. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 6.

Images