Information processing apparatus, information processing method, and program

Abstract

To provide a technique for implementing high operability for a user using gesture for operation.SOLUTION: An information processing apparatus includes: detection means which detects gesture of a first hand of a user; and processing means which switches the mode to a second operation mode, in a first operation mode in which first processing is executed by first operation using a second hand of the user, when the detection means detects specific gesture of the first hand, the second operation mode allowing second processing different from the first processing to be executed by the first operation using the second hand.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is a technology called Mixed Reality (MR) that fuses real and virtual spaces and allows the user to sense virtual objects. In MR technology, the system synthesizes and displays computer graphics (CG) that represent virtual objects against real scenery, and expresses contact between real and virtual objects.

[0003] MR technology makes it possible to operate virtual objects superimposed on real scenery according to the user's hand gestures. Users can move CG objects by gestures without using a controller. Users can also use gestures to input text and draw lines, text, or shapes in a designated drawing area on a whiteboard, etc.

[0004] Patent Document 1 discloses a method of displaying an operation GUI and executing a function according to a detected gesture. In Patent Document 1, the operation GUI is displayed at a display position according to the detected position of the gesture. The operation GUI is associated with a gesture and a function, and a function corresponding to the operation GUI is executed according to the detected gesture. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-151851 A [Patent Document 2] JP 2012-13514 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, it is assumed that gestures and operations using a GUI are performed with one hand, which results in poor operability (inconvenience, poor efficiency, bothersomeness, etc.) due to performing gestures with one hand and operations using a GUI with the same hand.

[0007] Therefore, an object of the present invention is to provide a technique that realizes high operability for the user when gestures are used for operations. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: detection means for detecting a first hand gesture of a user; a processing means for switching to a second operation mode in which a second process different from the first process is executed by the first operation with the second hand of the user when a specific gesture of the first hand is detected by the detection means in a first operation mode in which a first process is executed by a first operation with the second hand of the user; The information processing device is characterized by having:

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: a detection step of detecting a first hand gesture of a user; a first operation mode in which a first process is executed by a first operation by the second hand of the user; a processing step of switching to a second operation mode in which a second process different from the first process is executed by the first operation with the second hand when a specific gesture of the first hand is detected in the detection step; The information processing method is characterized by having the following features. Effect of the Invention

[0010] According to the present invention, when gestures are used for operations, high operability can be achieved for the user. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of an MR system according to a first embodiment. [Diagram 2] FIG. 1 is a hardware configuration diagram of an MR system according to a first embodiment. [Diagram 3] FIG. 11 is a block diagram of a controller according to a second embodiment. [Figure 4] FIG. 11 is a diagram for explaining the connection between an HMD and a controller according to the second embodiment. [Diagram 5] FIG. 2 is a diagram for explaining switching of operation modes according to the first embodiment. [Figure 6] FIG. 4 is a diagram for explaining a drawing process according to the first embodiment. [Figure 7] 11 is a flowchart of a switching process related to drawing according to the first embodiment. [Figure 8] 4 is a flowchart showing an outline of a switching process according to the first embodiment. [Figure 9] 11 is a flowchart of processing of the MR system according to the second embodiment. [Figure 10] FIG. 11 is a diagram showing the relationship between hands and character groups according to the third embodiment. [Figure 11] FIG. 13 is a diagram illustrating character input according to the third embodiment. [Figure 12] FIG. 13 is a diagram illustrating character input according to the fourth embodiment. [Figure 13] FIG. 13 is a diagram illustrating erasing of characters according to the fifth embodiment. [Figure 14] FIG. 23 is a diagram for explaining character string conversion according to the sixth embodiment. [Figure 15] 11 is a flowchart of processing of the MR system according to the third embodiment. [Figure 16] 13 is a flowchart of processing of the MR system according to the fourth embodiment. [Figure 17]13 is a flowchart of processing of the MR system according to the fifth embodiment. [Figure 18] 13 is a flowchart of processing of the MR system according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is one example of a means for realizing the present invention, and may be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. In addition, each embodiment may be appropriately combined.

[0013] <Embodiment 1> 2 shows a hardware configuration of the MR system 1 (mixed reality system) according to the embodiment 1. FIG.

[0014] As shown in FIG. 2, the MR system 1 includes an image capturing unit 101, a display unit 102, a communication unit 110, a CPU 200, a RAM 201, and a ROM 202.

[0015] The CPU 200 is a control unit that controls the entire MR system 1. The CPU 200 controls the entire MR system 1 by executing an application for operating the MR system 1. Specifically, the CPU 200 loads an application (program) stored in the ROM 202 into the RAM 201 and then executes the application. Each unit shown in FIG. 1 is realized by the CPU 200. However, one or more components shown in FIG. 1 may be realized by hardware independent of the CPU 200.

[0016] As shown in FIG. 1, the MR system 1 includes an HMD 100 and an information processing device 103.

[0017] The HMD 100 is a head mounted display. The HMD 100 is a video see-through display. The HMD 100 includes an image capturing unit 101 and a display unit 102. Information relating to the settings of the HMD 100 (setting information) is stored in a ROM 202.

[0018] In the HMD 100, an image capturing unit 101 is disposed in front of the eyes of a user (user) wearing the HMD 100. The display unit 102 displays an image (captured image) captured by the image capturing unit 101 of the surroundings. In the case of a video see-through type HMD, a method of treating the position and orientation of the image capturing unit 101 as the position and orientation of the user is common. Note that the HMD 100 of this embodiment is a video see-through type HMD, but this is merely an example. For example, this embodiment can also be applied to an HMD for virtual reality that does not display a captured image (an image captured of real space) on the display unit 102. In this case, the image capturing unit 101 is used as a camera for capturing an image of a hand, which is a measurement target, for example.

[0019] The image capturing unit 101 captures images of a real space at a predetermined frame rate (e.g., 30 frames / second). The image capturing unit 101 is fixed to the housing of the HMD 100. The image capturing unit 101 has two real cameras. The real camera for the left eye and the real camera for the right eye are disposed at positions close to both eyes of the user. A pair of left and right images captured by the image capturing unit 101 can also be called "stereo camera images."

[0020] The information processing device 103 generates data to be displayed on the display unit 102, based on the captured image acquired by the image capturing unit 101. The information processing device 103 may be included in the HMD 100, or may be hardware independent of the HMD 100. The information processing device 103 has a captured image storage unit 104, a contour point extraction unit 105, a three-dimensional position calculation unit 106, a calculation unit 107, an image drawing unit 108, a processing calculation unit 109, and a communication unit 110.

[0021] The captured image storage unit 104 stores captured images acquired by imaging using the image imaging unit 101. The captured image storage unit 104 is realized by the RAM 201. Therefore, the captured images are stored in the RAM 201.

[0022] The contour point extraction unit 105 extracts contour points of the fingers from the captured image using image recognition technology.

[0023] The three-dimensional position calculation unit 106 calculates the three-dimensional positions (x coordinate, y coordinate, and z coordinate) of the contour points of the fingers extracted by the contour point extraction unit 105. The x coordinate and y coordinate are coordinates with the bottom left of the VRAM area of ​​the captured image as the origin (0,0). The coordinate of the origin does not have to be the bottom left of the VRAM area and may be freely determined by the user. The z coordinate is a coordinate indicating the distance from the HMD 100 and can be obtained based on the captured image.

[0024] In order to correctly display the depth of the hand (z coordinate value), a method can be used in which the area of ​​the hand is extracted from an image captured by a stereo camera and the depth value of the hand relative to the stereo camera is calculated. This method can use a method of calculating the depth value by triangulation for all corresponding points of the extracted hand outline in the captured image, as in Patent Document 2. Patent Document 2 uses a stereo camera mounted on a video see-through type HMD (a display device for presenting mixed reality).

[0025] LeapMotion, Inc.'s LeapMotion, can measure the position and orientation (position and orientation) of the hand, including the fingers. LeapMotion can detect the area of ​​the hand from a stereo camera (image capturing unit 101). It can also detect the area of ​​the hand by using a depth sensor (distance sensor). Since the position and orientation of the fingers can be estimated by the above, the HMD 100 may be provided with a depth sensor.

[0026] In this embodiment, the three-dimensional position calculation unit 106 determines one arbitrary point from the contour points extracted by the contour point extraction unit 105, and calculates the three-dimensional position of the determined contour point. Position information of the determined contour point is stored in the RAM 201 as reference position information. In addition, new three-dimensional position information is periodically stored in the RAM 201 in accordance with the movement of the finger.

[0027] The calculation unit 107 calculates the position and orientation of the hand in the captured image stored in the captured image storage unit 104. The calculation of the position and orientation of the hand can be performed using a method that uses deep learning or a method that uses an existing public library.

[0028] The image drawing unit 108 draws an image to be displayed on the display unit 102 of the HMD 100 based on the result of processing by the processing calculation unit 109 .

[0029] The processing and calculation unit 109 compares the three-dimensional position (coordinate value) calculated by the three-dimensional position calculation unit 106 with a value (coordinate value) measured by an external sensor (such as a sensor 303 described later). The value measured by the sensor is a value acquired by Bluetooth (registered trademark) communication from a controller 300 (small operating device) described later. Specifically, the processing and calculation unit 109 determines whether or not the coordinate change of the hand detected by the HMD 100 matches the coordinate change of the controller 300. If the two coordinate changes match, the processing and calculation unit 109 determines that the hand detected by the HMD 100 and the hand wearing the controller 300 are the same hand (i.e., the hand of the user of the HMD 100). Note that, generally, an error occurs in the coordinates acquired by the sensor from the actual coordinates. For this reason, the processing and calculation unit 109 may determine that the two coordinate changes match if, for example, the difference between the two coordinate changes is included within a certain error rate (for example, 5 percent). Note that the error rate may be arbitrarily determined by the user.

[0030] Here, the processing calculation unit 109 may perform calculation processing on IMU information (inertial information) received from the controller 300 in order to obtain a value to be compared with the three-dimensional position calculated by the three-dimensional position calculation unit 106. On the other hand, the control unit 301 of the controller 300 may transmit information obtained by performing calculation processing on the IMU information to the HMD 100.

[0031] The communication unit 110 has, for example, an antenna for wireless communication, a modulation / demodulation circuit (a circuit for processing wireless signals), and a communication controller. The communication unit 110 outputs a modulated wireless signal from the antenna, or demodulates a wireless signal received by the antenna. In this way, the communication unit 110 realizes short-distance wireless communication in accordance with the IEEE802.15 standard (so-called Bluetooth (registered trademark)). In this embodiment, the Bluetooth communication employs version 5.1 of Bluetooth Low Energy, which has low power consumption.

[0032] 5A and 5B are diagrams for explaining switching of operation modes in the first embodiment. Hereinafter, one hand for switching operation modes by gesture is called a "gesture hand." In addition, one of the two hands that is not the "gesture hand" is called a "processing hand." In addition, in the first embodiment, since the operation of the HMD 100 is controlled by gestures, the gestures can be said to be "a type of user operation."

[0033] In Fig. 5A, the user uses his right hand as a hand that performs drawing processing (processing hand 501) in a drawing area 500 to draw characters by gestures (hand gestures). Here, the user uses his left hand as a hand that performs a gesture for switching the operation mode (gesture hand 502), thereby The processing (execution processing) being performed by 501 can be switched to another processing. In Fig. 5B, the processing being performed by the processing hand 501 is switched from a character drawing mode to a character erasing mode by a gesture of the gesturing hand 502. Note that, in order to switch the operation mode by gesture, a destination operation mode (and a process executable in the operation mode) is linked (assigned) to each gesture in advance.

[0034] (Overview of the switching process) Hereinafter, an overview of the operation mode switching process performed by the CPU 200 of the MR system 1 according to the first embodiment will be described with reference to the flowchart of FIG.

[0035] Note that the CPU 200, as a detection unit, can detect gestures of the left and right hands from a captured image (an image acquired by the image capturing unit 101). The CPU 200, as a processing unit, can switch the operation mode as described above according to the gesture of the left or right hand.

[0036] In step S800, the CPU 200 determines whether or not a hand has been detected from the captured image (the image acquired by the image capturing unit 101). The CPU 200 controls, for example, the contour point extraction unit 105 and the three-dimensional position calculation unit 106 to determine whether or not a hand has been detected from the captured image. If it is determined that a hand has been detected, the process proceeds to step S801. If it is determined that a hand has not been detected, the process of step S800 is repeated.

[0037] In step S801, CPU 200 determines whether the hand detected in step S800 is a gesturing hand. If it is determined that the detected hand is a gesturing hand, the process proceeds to step S802. If it is determined that the detected hand is not a gesturing hand, the process returns to step S800.

[0038] For example, when an operation device or the like is attached to the detected hand, the CPU 200 determines that the detected hand is the gesturing hand. Also, the user may set the gesturing hand to either the right hand or the left hand in advance on a system setting menu screen or the like. For example, when the left hand is set as the gesturing hand, the CPU 200 determines whether the hand detected in step S800 is the left hand.

[0039] In step S802, the CPU 200 determines whether or not a gesture by the hand (gesture hand) detected in step S800 has been detected. If it is determined that a gesture has been detected, the process proceeds to step S803. If it is determined that a gesture has not been detected, the process of step S802 is repeated.

[0040] In step S803, the CPU 200 switches to an operation mode associated with the detected gesture (an operation mode for performing processing associated with the gesture). The CPU 200 associates a gesture with an operation mode in advance, and switches to the operation mode associated with the detected gesture. Note that in the first embodiment, if the operation mode is different, the processing executed according to the gesture of the processing hand is different.

[0041] (Drawing example) 6A to 6C are diagrams for explaining an example in which a user performs drawing using the HMD 100 according to the first embodiment.

[0042] In FIG. 6A, the user is standing and drawing in a drawing area 600 with his right hand 601 (processing hand). The left hand 602, which is the gesture hand, While 601 is drawing, there is no need to move unless there is a need to switch the operation mode. For this reason, in Fig. 6A, the left hand 602 is in a lowered state.

[0043] 6B shows a state in which the user's left hand 602 is making a gesture for switching the operation mode. At this time, in order to determine the gesture of the left hand 602, the left hand 602 needs to be included in the imaging range of the image capturing unit 101 of the HMD 100. For this reason, the user raises the left hand 602 so that it is included in the imaging range of the image capturing unit 101 of the HMD 100.

[0044] 6C shows a state in which the user is seated and then draws after drawing area 600 is set on the desk. When the user is seated, even if there is no need to switch the operation mode by a gesture, the left hand 602 is likely to be located on drawing area 600. Therefore, there is a high possibility that the left hand 602 will be included in the imaging range of image capturing unit 101. Therefore, when the user is seated and draws on the desk, it is necessary to restrict switching of the operation mode by a gesture.

[0045] When the user is sitting and drawing on a desk, the HMD 100 is tilted downward from the horizontal direction. Therefore, when the orientation of the MHD 100 is pointing downward from the horizontal direction by a certain angle θ (larger than the certain angle θ), the HMD 100 can determine that the user is sitting and drawing on the desk.

[0046] A switching process executed by the CPU 200 of the MR system 1 (for example, the CPU 200 of the information processing device 103) when the user is drawing in the drawing area will be described with reference to the flowchart of Fig. 7. That is, the flowchart of Fig. 7 is a flowchart for explaining in more detail the flowchart of Fig. 8 when the drawing process is performed. Note that, in order for the HMD 100 to detect a gesture, both hands must be included in the drawing area in the real space (an area in which a function such as a whiteboard is set).

[0047] In step S700, CPU 200 determines whether or not both hands are positioned in the drawing area based on the image of the drawing area captured by image capturing unit 101. If it is determined that both hands are not positioned in the drawing area, the process of step S700 is repeated. If it is determined that both hands are positioned in the drawing area, the process proceeds to step S701.

[0048] In step S701, the CPU 200 determines whether the user (HMD 100) is looking down. For example, the CPU 200 determines that the user is looking down when the HMD 100 is tilted in the negative direction (downward direction) from the horizontal direction by more than a certain angle θ (see FIG. 7C). On the other hand, when the tilt is smaller than the certain angle θ, the CPU 200 can determine that the user is performing drawing work with a wall, air, or the like as a drawing area.

[0049] If it is determined that the user is looking down, the CPU 200 detects feature points from the captured image acquired by the image capturing unit 101, and detects a plane in real space based on the feature points. Then, the process proceeds to step S702. On the other hand, if it is determined that the user is not looking down, the process proceeds to step S706.

[0050] In step S702, CPU 200 determines whether the distance between both hands detected in step S700 and the plane (the plane detected in step S701) is closer than a predetermined distance. If it is determined that the distance between both hands and the plane is closer than the predetermined distance, the process proceeds to step S703. If it is determined that the distance between both hands and the plane is greater than or equal to the predetermined distance, the process proceeds to step S706.

[0051] In step S703, the CPU 200 disables switching of the operation mode by the gesture of the gesturing hand. This is to prevent the operation mode from being switched when the gesture is detected due to the gesturing hand being included in the imaging range except when the user intends to switch the operation mode.

[0052] In step S704, CPU 200 determines whether or not a predetermined gesture (a gesture for enabling switching of the operation mode by gesture) has been detected. Here, it is assumed that the predetermined gesture is set in advance. If it is determined that the predetermined gesture has been detected, the process proceeds to step S705. If it is determined that the predetermined gesture has not been detected, the process of step S704 is repeated. Note that if a controller or the like is attached to the hand, a "predetermined operation on the controller" may be used instead of the "predetermined gesture."

[0053] In step S705, the CPU 200 enables switching of the operation mode by a gesture of the gesturing hand (cancels the disablement of switching of the operation mode by a gesture of the gesturing hand).

[0054] In step S706, the CPU 200 determines, based on the captured image, whether or not a gesture by a gesturing hand has been detected.

[0055] In step S707, the CPU 200 switches to an operation mode associated with the gesture detected in step S706. This causes the CPU 200 to switch the process (execution process) to be executed in response to the operation of the processing hand. Therefore, when a certain gesture is performed with the gesturing hand, a first process is executed when a certain operation is performed with the processing hand. On the other hand, when a different gesture is performed with the gesturing hand, a second process different from the first process is executed when the same certain operation is performed with the processing hand.

[0056] Here, when it is desired to frequently switch processes performed by a processing hand (the hand that executes the process), if the process is executed and the process is switched using only the processing hand, the work efficiency is poor. On the other hand, in this embodiment, the operation mode is switched by a gesture using the other hand that is not the processing hand, so the work efficiency (operability) is improved.

[0057] <Embodiment 2> In the second embodiment, a method for setting a processing hand and a gesture hand in advance using a controller 300 held in or worn by a user will be described.

[0058] 3 shows a block configuration of the controller 300. The controller 300 is an operation device shaped to be worn on a finger. However, the shape of the controller 300 does not need to be a shape that can be worn on a finger, and may be any shape that can be held by the user's hand. The controller 300 has a control unit 301, an operation unit 302, a sensor 303, a working memory 304, a power control unit 305, and a communication unit 306.

[0059] The control unit 301 controls the controller 300 (the entire device) in accordance with an input signal or program. Note that instead of the control unit 301 controlling the entire device, the entire device may be controlled by multiple pieces of hardware sharing the processing.

[0060] The operation unit 302 receives instructions from the user for the controller 300. The operation unit 302 includes, for example, a power button (main function including power sharing for the control unit 301 of the controller 300) and a power button (main function including power sharing for the control unit 301 of the controller 300). The operation unit 302 includes a button for starting communication with an external device such as the HMD 100 via the communication unit 306.

[0061] The sensor 303 is a sensor for acquiring information such as IMU (Inertial Measurement Unit) information (hereinafter referred to as "sensor information"), for example. The sensor 303 can calculate the change in attitude, orientation (relative direction), and position of the controller 300 based on the angular velocity (value detected by a gyro sensor) and acceleration (value detected by an accelerometer).

[0062] The working memory 304 is used as a buffer memory (memory for temporarily storing sensor information acquired by the sensor 303), a working area for the control unit 301, and the like.

[0063] The power supply control unit 305 supplies power for the controller 300 to operate.

[0064] The communication unit 306 includes, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller, similar to the communication unit 110 of the HMD 100. The communication unit 306 realizes short-range wireless communication in accordance with the IEEE802.15 standard.

[0065] 4 shows a connection configuration between the HMD 100 and the controller 300. The HMD 100 and the controller 300 are connected by communication using Bluetooth.

[0066] For example, sensor information acquired by the sensor 303 is transmitted to the communication unit 110 of the HMD 100 via the communication unit 306 of the controller 300. Data communication between the HMD 100 and the controller 300 is not limited to communication using Bluetooth, and other communication methods may be used.

[0067] The movement of the hand on which the controller 300 is worn can be determined from the movement (sensor information) of the controller 300. For this reason, the hand on which the controller 300 is worn does not necessarily need to be included in the imaging range of the image capturing section 101.

[0068] Here, the sensor information acquired by the sensor 303 is sent to the communication unit 110 of the HMD 100 via the communication unit 306. Then, the HMD 100 can detect the movement of the controller 300 by the sensor information of the sensor 303. Therefore, the HMD 100 can acquire more precise hand movements by using the sensor information of the sensor 303 other than the method of acquiring hand movements based on the captured image. The HMD 100 can also acquire hand movements based on both the captured image and the sensor information. Therefore, when the controller 300 is worn, the hand on which the controller 300 is worn is used as a processing hand so that fine control of processing is possible.

[0069] The process by the CPU 200 of the MR system 1 according to the second embodiment will be described with reference to the flowchart in Fig. 9. Below, the process of setting a gesturing hand (a hand that performs a gesture) and a processing hand (a hand that executes a process) will be described with reference to Fig. 9.

[0070] In step S900, the CPU 200 determines whether or not the user is wearing the controller 300 on his / her hand. If it is determined that the controller 300 is worn on the user's hand, the process proceeds to step S901. If it is determined that the controller 300 is not worn on the user's hand, the process proceeds to step S902.

[0071] Here, the CPU 200 performs the following on the image (imaging range) captured by the image capturing unit 101. When both hands are captured, it is possible to determine from the captured image whether or not the controller 300 is being worn on the hand. On the other hand, when only one hand is captured in the captured image, the CPU 200 determines from the captured image whether or not the controller 300 is being worn on the captured hand. Furthermore, when the HMD 100 has communicated with the controller 300, the CPU 200 can detect that the controller 300 is being worn on the user's hand even if at least one hand is not captured in the captured image.

[0072] In step S901, the CPU 200 sets the hand on which the controller 300 is not attached as a gesture hand, and also sets the hand on which the controller 300 is attached as a processing hand.

[0073] In step S902, the CPU 200 sets one of the left and right hands as a gesture hand and the other hand as a processing hand in response to a user operation on the system menu. In other words, when the controller 300 is not attached, the CPU 200 sets either the left or right hand as a gesture hand in the same manner as in the first embodiment.

[0074] In the second embodiment, the hand on which the controller 300 is worn is treated as the processing hand. This allows the user to perform processing with the hand whose movements are more likely to be detected accurately. This makes it easier to achieve the processing intended by the user.

[0075] After the processing hand and the gesture hand are set, the process of the flowchart shown in Fig. 7 or 8 of the first embodiment is performed. Furthermore, when the user uses a controller 300 that needs to be held by the hand, it is not easy for the user to perform detailed processing with the hand holding the controller 300. For this reason, in such a case, the CPU 200 may set the hand not holding the controller 300 as the processing hand, and the hand holding the controller 300 as the gesture hand.

[0076] <Embodiment 3> In the third embodiment, an MR system 1 in which characters are input by hand movements will be described.

[0077] 10A and 10B are schematic diagrams showing the relationship between hands and character groups. In the third embodiment, a user selects one of character groups 1000 including a plurality of characters using a processing hand 501. After that, the user selects one character from the selected character group 1000 with a gesture hand 502. This inputs the character selected by the user. Note that the plurality of characters included in each of the character groups 1000 are preset.

[0078] For example, in Figure 10A, each of the plurality of character groups 1000 includes a plurality of Hiragana characters of the Japanese language, and in Figure 10B, each of the plurality of character groups 1000 includes a plurality of alphabetic characters.

[0079] Here, when the calculation unit 107 detects a first gesture by the processing hand 501 and detects that the processing hand 501 has rotated, the CPU 200 switches the selected character group 1000. Note that hand rotation is also a type of hand gesture.

[0080] After that, when the calculation unit 107 detects a second gesture by the gesturing hand 502, the CPU 200 selects a character from the selected character group 1000. Then, the CPU 200 detects that the gesturing hand 502 has rotated, and switches the selected character. For example, when the calculation unit 107 detects a gesture other than the second gesture by the gesturing hand 502, the CPU 200 inputs (confirms) the character. The first gesture and the second gesture need to be set in advance. In the following, the first gesture and the second gesture will be described as both being gestures of pinching fingers (see FIG. 10A and FIG. 10B). However, the first gesture and the second gesture may be different from each other.

[0081] The process executed by the CPU 200 of the MR system 1 according to the third embodiment will be described with reference to the flowchart of FIG.

[0082] In step S1500, the CPU 200 determines whether or not the processing hand 501 has performed a first gesture. If it is determined that the processing hand 501 has performed the first gesture, the process proceeds to step S1501. If it is determined that the processing hand 501 has not performed the first gesture, the process of step S1500 is repeated.

[0083] In step S1501, CPU 200 switches the operation mode to a character group selection mode. When the operation mode is switched to the character group selection mode, CPU 200 displays character input area 1100 on display unit 102 so as to be superimposed on real space, as shown in (1) of FIG. 11A and (1) of FIG. 11B. Note that character input area 1100 is displayed at a position a predetermined distance higher than the higher of processing hand 501 and gesture hand 502, for example. CPU 200 also selects one preset character group 1000 (for example, the first character group 1000).

[0084] In step S1502, as shown in Fig. 11A (2) and Fig. 11B (2), CPU 200 displays group selection screen 1107, which is a screen for selecting character group 1000, on display unit 102. As shown in Fig. 11A and Fig. 11B, the selected character group 1000 or character is surrounded by a thick frame.

[0085] In step S1503, the CPU 200 determines whether or not the processing hand 501 (wrist of the processing hand 501) has rotated by a predetermined angle (threshold) or more (for example, whether or not the processing hand 501 has changed to a predetermined shape). If it is determined that the processing hand 501 has rotated by the predetermined angle or more, the process proceeds to step S1504. If it is determined that the processing hand 501 has not rotated by the predetermined angle or more, the process proceeds to step S1505.

[0086] In step S1504, the CPU 200 switches the selected character group 1000 depending on the rotation direction of the processing hand 501 (wrist of the processing hand 501), as shown in (3) of Figure 11A and (3) of Figure 11B. For example, if the processing hand 501 is rotating to the right, the CPU 200 selects the character group 1000 one character to the right of the currently selected character group 1000 on the group selection screen 1107.

[0087] In step S1505, the CPU 200 determines whether or not the gesturing hand 502 has performed the second gesture. If it is determined that the gesturing hand 502 has performed the second gesture, the process proceeds to step S1506. If it is determined that the gesturing hand 502 has not performed the second gesture, the process proceeds to step S1503.

[0088] In step S1506, CPU 200 selects the The operation mode is switched to a character selection mode in which one character is selected at a time. At this time, the CPU 200 selects, for example, the first character in the selected character group 1000.

[0089] In step S1507, CPU 200 displays character selection screen 1108 (a screen on which a plurality of characters included in selected character group 1000 are arranged in a line) on display unit 102, as shown in (4) of FIG. 11A and (4) of FIG. 11B.

[0090] In step S1508, CPU 200 determines whether gesturing hand 502 (wrist of gesturing hand 502) has rotated by a predetermined angle or more (for example, gesturing hand 502 has changed to a predetermined shape). If it is determined that gesturing hand 502 has rotated by a predetermined angle or more, proceed to step S1509. If it is determined that gesturing hand 502 has not rotated by a predetermined angle or more, proceed to step S1510.

[0091] 11A (5) and 11B (5), the CPU 200 switches the selected character depending on the rotation direction of the gesturing hand 502. For example, if the hand is rotated right, the CPU 200 selects the character one character to the right of the currently selected character on the character selection screen 1108.

[0092] In step S1510, the CPU 200 determines whether or not the gesturing hand 502 has performed a second operation (a specific operation other than the second gesture). If it is determined that the gesturing hand 502 has performed the second operation, the process proceeds to step S1511. If it is determined that the gesturing hand 502 has not performed the second operation, the process proceeds to step S1508. Here, the second operation may be a gesture by the gesturing hand 502, or may be an operation on a controller or the like attached to the gesturing hand 502.

[0093] In step S1511, the CPU 200 inputs the currently selected character into the character input area 1100. In other words, the CPU 200 finalizes the selection of the currently selected character.

[0094] In step S1512, CPU 200 switches from the character selection mode to the character group selection mode.

[0095] In step S1513, CPU 200 terminates the display of character selection screen 1108 (hides character selection screen 1108) as shown in (6) of FIG. 11A and (6) of FIG. 11B.

[0096] In step S1514, the CPU 200 determines whether or not the processing hand 501 has performed a first operation (an operation other than the first gesture). If it is determined that the processing hand 501 has performed the first operation, the process proceeds to step S1515. If it is determined that the processing hand 501 has not performed the first operation, the process proceeds to step S1501. Here, the first operation may be a gesture by the processing hand 501, or may be an operation on a controller attached to the processing hand 501.

[0097] In step S1515, CPU 200 ends the character group selection mode.

[0098] In step S1516, CPU 200 terminates the display of group selection screen 1107, as shown in (7) of FIG. 11A and (7) of FIG. 11B.

[0099] In the example shown in Fig. 11A, when the processing hand 501 performs a first gesture (step S1500 YES), the group selection screen 1107 is displayed as shown in Fig. 11A (2) (step S1502). Next, in response to the rotation of the processing hand 501 (step S1503 YES), the selected character group 1000 is switched as shown in Fig. 11A (3) (step S1504). When the gesturing hand 502 performs a second gesture (step S1505 YES), the character selection screen 1108 is displayed as shown in Fig. 11A (4) (step S1507).

[0100] Thereafter, in response to the rotation of the gesturing hand 502 (YES in S1508), the character selected in the character selection screen 1108 is switched (step S1509) as shown in (5) of FIG. 11A. When the gesturing hand 502 performs the second operation (YES in step S1510), the selected character is input to the character input area 1100 (step S1511) as shown in (6) of FIG. 11A. Then, when the processing hand 501 performs the first operation (YES in step S1514), the group selection screen 1107 is hidden (step S1516) as shown in (7) of FIG. 11A.

[0101] According to the third embodiment, the user can input characters without touching an operating member such as a keyboard or a touch panel. This allows the user to input characters in a hygienic manner. In addition, since each of the left and right hands is assigned a role, the user can input characters without performing complicated gestures with one hand.

[0102] <Embodiment 4> In the fourth embodiment, an MR system 1 capable of switching to a subgroup linked to a character group 1000 will be described. Here, a subgroup is a group including characters related to each of the characters included in the character group 1000. For example, if the character group 1000 includes multiple hiragana characters, the subgroup includes multiple voiced hiragana characters.

[0103] 12A and 12B are diagrams for explaining the process according to the fourth embodiment. For example, when selecting a character in the character group 1000, if it is detected that the processing hand 501 has rotated, the character group 1000 is switched to a subgroup linked to the selected character group 1000. For example, FIG. 12B shows a state in which the character group 1000 indicating "E, F, G, H" is switched to a subgroup indicating "e, f, g, h" upon detection of the rotation of the processing hand 501.

[0104] In the fourth embodiment, the only difference from the flowchart in Fig. 15 is that steps S1610 and S1611 are added to the process of the flowchart in Fig. 15 according to the third embodiment. Therefore, only the processes of steps S1610 and S1611 will be described. The process of step S1610 starts when it is determined in step S1508 that the gesturing hand 502 has not rotated by a predetermined angle or more, or when the process of step S1509 ends.

[0105] In step S1610, the CPU 200 determines whether the processing hand 501 has rotated by a predetermined angle or more. If it is determined that the processing hand 501 has rotated by the predetermined angle or more, the process proceeds to step S1611. If it is determined that the processing hand 501 has not rotated by the predetermined angle or more, the process proceeds to step S1510.

[0106] In step S1611, the CPU 200 switches the selected character group 1000 on the character selection screen 1108 to a subgroup linked to the character group 1000. Then, the CPU 200 selects a character in the subgroup to which the currently selected character in the character group 1000 corresponds.

[0107] According to the fourth embodiment, it becomes possible to intuitively input special characters such as voiced consonants in Japanese or lowercase letters and symbols in English.

[0108] <Embodiment 5> In the fifth embodiment, the MR system 1 further erases characters.

[0109] 13A and 13B are diagrams for explaining the erasure of characters in the fifth embodiment. When selecting a character in the character group 1000, if it is detected that the gesturing hand 502 has rotated in a direction in which no character exists as viewed from the currently selected character, the character is erased. During the erasure of the character, the character erasure item 1302 is displayed, and the character continues to be erased. If it is detected that the gesturing hand 502 has rotated in a direction in which a character exists as viewed from the character erasure item 1302, the erasure of the character ends. Note that the character may be erased when "any gesture of the gesturing hand 502 indicating a direction in which no character exists as viewed from the currently selected character among the directions in which characters are arranged on the character selection screen 1108" is detected. For example, if there is no character to the left of the currently selected character, and a gesture of the gesturing hand 502 pointing with the index finger to the left is detected, the character may be erased.

[0110] Fig. 13A shows how the left rotation of the gesturing hand 502 is detected, and the "no" is deleted from the character "kaimono" ("carrying something") input in the character input area 1100. Fig. 13B shows how the left rotation of the gesturing hand 502 (a gesture indicating a left direction) is detected, and the "E" is deleted from the character "IMAGE" ("image") input in the character input area 1100.

[0111] The process performed by the CPU 200 of the MR system 1 according to the fifth embodiment will be described with reference to the flowchart of Fig. 17. In the process according to the fifth embodiment, the processes of steps S1710 to S1712 as shown in the flowchart of Fig. 17 are added to the process of the flowchart of Fig. 15. Therefore, only the processes of steps S1710 to S1712 will be described below.

[0112] The process of step S1710 starts when it is determined in step S1508 that the gesturing hand 502 has rotated by a predetermined angle or more.

[0113] In step S1710, CPU 200 determines whether or not a character exists in the rotation direction of gesturing hand 502 from the character selected on character selection screen 1108. If it is determined that no character exists in the rotation direction of the hand from the selected character, the process proceeds to step S1711. If it is determined that a character exists in the rotation direction of the hand from the selected character, the process proceeds to step S1509.

[0114] In step S1711, the CPU 200 erases the last character input in the character input area 1100. The CPU 200 also displays the character deletion item 1302.

[0115] In step S1712, when CPU 200 newly detects that gesturing hand 502 has rotated by a predetermined angle or more, CPU 200 determines whether or not a character exists in the rotation direction of gesturing hand 502 as viewed from character erasure item 1302. If it is determined that a character exists in the rotation direction of gesturing hand 502, the process proceeds to step S1510. If not, the process returns to step S1711.

[0116] For example, in the state of (2) in FIG. 13B, when deleting a character in the character input area 1100, the gesture hand 502 is rotated in the direction in which the character exists (in the direction of right rotation) by a predetermined angle or more. By rotating the screen, the screen changes to the state shown in FIG. 13B (3), and the erasure of the characters is completed.

[0117] According to the fifth embodiment, the user can not only input characters but also delete characters by using gestures with the left and right hands.

[0118] <Embodiment 6> In the sixth embodiment, the MR system 1 detects the shape and rotation of both hands of the user and converts the character string (characters) input in the character input area 1100.

[0119] Figures 14A and 14B are diagrams for explaining character string conversion in embodiment 6. Figure 14A is a diagram showing an example of converting a hiragana character string into a kanji character string, and Figure 14B is a diagram showing an example of converting an English word into a synonym of the English word.

[0120] After detecting that the gesturing hand 502 has performed the second gesture, the MR system 1 selects a character string conversion candidate 1407 (converted character string). The MR system 1 changes the character string to be converted by detecting that the processing hand 501 has performed the first gesture, and ends the character conversion by detecting that the gesturing hand 502 has performed the second operation.

[0121] The process of the CPU 200 of the MR system 1 according to the sixth embodiment will be described with reference to the flowchart of FIG.

[0122] In step S1800, the CPU 200 determines whether or not the gesturing hand 502 has performed a second gesture. If it is determined that the second gesture has been performed, the process proceeds to step S1801. If it is determined that the second gesture has not been performed, the process of step S1800 is repeated.

[0123] In step S1801, CPU 200 switches the operation mode to a character conversion mode for converting a character string.

[0124] In step S1802, CPU 200 displays (selects) the first (leading) character string in character input area 1100 as character string 1408 to be converted. At this time, CPU 200 displays a plurality of conversion candidates 1407 for character string 1408 to be converted, as shown in (2) of Fig. 14A and (2) of Fig. 14B. CPU 200 also selects, for example, the top left conversion candidate 1407.

[0125] In step S1803, the CPU 200 determines whether the gesture hand 502 has rotated by a predetermined angle or more. If it is determined that the gesture hand 502 has rotated by a predetermined angle or more, the process proceeds to step S1804. If it is determined that the gesture hand 502 has not rotated by a predetermined angle or more, the process proceeds to step S1805.

[0126] In step S1804, the CPU 200 switches the selected conversion candidate 1407 to a character string existing in the direction corresponding to the rotation direction of the gesture hand 502 among the left - right directions. For example, in (3) of FIG. 14A, by rotating the gesture hand 502 clockwise, the conversion candidate 1407 of the character string "かい" is switched from the state where "回" is selected to the state where "階" is selected. In (3) of FIG. 14B, by rotating the gesture hand 502 clockwise, the conversion candidate 1407 of the character string "IMAGE" is switched from the state where "PICTURE" is selected to the state where "PHOTOGRAPH" is selected.

[0127] In step S1805, the CPU 200 determines whether the processing hand 501 has rotated by a predetermined angle or more. If it is determined that the processing hand 501 has rotated by a predetermined angle or more, the process proceeds to step S1806. If it is determined that the processing hand 501 has not rotated by a predetermined angle or more, the process proceeds to step S1803.

[0128] In step S1806, as shown in (4) of FIG. 14A and (4) of FIG. 14B, the CPU 200 switches the selected conversion candidate 1407 to a character string existing in the direction corresponding to the rotation direction among the up - down directions.

[0129] In step S1807, the CPU 200 determines whether the processing hand 501 has performed the first gesture. If it is determined that the first gesture has been performed, the process proceeds to step S1808. If it is determined that the first gesture has not been performed, the process proceeds to S1803.

[0130] In step S1808, CPU 200 switches the operation mode to a target switching mode in which character string 1408 to be converted is switched.

[0131] In step S1809, the CPU 200 determines whether or not the processing hand 501 has rotated by a predetermined angle or more. If it is determined that the processing hand 501 has rotated by the predetermined angle or more, the process proceeds to step S1810. If it is determined that the processing hand 501 has not rotated by the predetermined angle or more, the process proceeds to step S1811. Note that the rotation of the gesture hand 502 may be determined instead of the rotation of the processing hand 501.

[0132] In step S1810, the CPU 200 switches the character string 1408 to be converted depending on the rotation direction of the processing hand 501, as shown in (5) of FIG. 14A and (5) of FIG. 14B.

[0133] In step S1811, the CPU 200 determines whether or not the processing hand 501 has performed a first operation (an operation other than the first gesture). If it is determined that the first operation has been performed, the process proceeds to step S1812. If it is determined that the first operation has not been performed, the process proceeds to step S1809.

[0134] In step S1812, CPU 200 determines to perform conversion of currently selected character string 1408. CPU 200 switches the operation mode to the character conversion mode.

[0135] In step S1813, CPU 200 displays a plurality of conversion candidates 1407 for character string 1408 determined in step S1812.

[0136] In step S1814, the CPU 200 determines whether or not the gesturing hand 502 has performed a second operation (an operation other than a specific gesture). If it is determined that the second operation has been performed, the process proceeds to step S1815. If it is determined that the second operation has not been performed, the process proceeds to step S1801.

[0137] In step S1815, CPU 200 ends the character conversion mode.

[0138] In step S1816, CPU 200 ends the display of character string conversion candidates 1407, as shown in (6) of FIG. 14A and (6) of FIG. 14B.

[0139] According to the sixth embodiment, the characters input in the third embodiment can be input without any special operation. It can be converted by a similar operation.

[0140] Instead of the "characters" described in the third to fifth embodiments, "options" such as "marks" or "illustrations" may be used. Moreover, instead of the "character groups," groups including multiple options may be used.

[0141] In addition, although it has been described that the processing of each flowchart in each embodiment is executed by the CPU 200 of the MR system 1, this CPU 200 may be included in the HMD 100 or the information processing device 103. Furthermore, the information processing device 103 may include the HMD 100. Furthermore, the processing that has been executed by the processing hand through a gesture may be executed by the processing hand through an operation (such as pressing or sliding) on ​​a specific operating member.

[0142] Furthermore, in each embodiment, some of the gestures performed by the gesturing hand may be performed by the processing hand. Some of the gestures performed by the processing hand may be performed by the gesturing hand. That is, in some steps of each flowchart, the "gesturing hand" may be read as the "processing hand" and the "processing hand" may be read as the "gesturing hand."

[0143] As described above, in each of the above embodiments, a function associated with the other hand (a process executed by operating the other hand) can be switched using a hand gesture detected from an image captured by the image capturing unit 101 of the HMD 100.

[0144] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." For this reason, unless a contradiction occurs, the expression "equal to or greater than A" may be read as "A or greater (high; long; many)," or may be read as "greater than A (high; long; many)." On the other hand, the expression "equal to or less than A" may be read as "A or smaller (low; short; few)," or may be read as "smaller than A (low; short; few)." Furthermore, "bigger (higher; longer; more) than A" may be read as "A or greater," and "smaller (lower; shorter; fewer) than A" may be read as "A or less."

[0145] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0146] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0147] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) detection means for detecting a first hand gesture of a user; a processing means for switching to a second operation mode in which a second process different from the first process is executed by the first operation with the second hand of the user when a specific gesture of the first hand is detected by the detection means in a first operation mode in which a first process is executed by a first operation with the second hand of the user; 13. An information processing device comprising: (Configuration 2) The device further includes a setting means for setting the left hand or the right hand of the user as the first hand in response to a user operation. 2. The information processing device according to configuration 1. (Configuration 3) The present invention further includes a setting means for setting the left hand or the right hand of the user as the first hand depending on whether the controller is worn on the left hand or the right hand of the user when the controller is worn on the left hand or the right hand of the user. 2. The information processing device according to configuration 1. (Configuration 4) the processing means switches to an operation mode for performing an execution process associated with the first hand gesture in response to the first hand gesture detected by the detection means; The execution process is a process executed by the operation by the second hand. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The processing means includes: performing the execution process by operation with the second hand on a specific area of ​​a real space displayed on a display device; switching the operation mode in response to the first hand gesture detected by the detection means when the left and right hands of the user are detected in the specific area; 5. The information processing device according to configuration 4. (Configuration 6) The display device is mounted on a user's head, the processing means disables the switching of the operation mode by the first hand gesture when the display device is facing downward at a certain angle or less with respect to the horizontal direction. 6. The information processing device according to configuration 5. (Configuration 7) the processing means, when a specific operation is performed after disabling the switching of the operation mode by the first hand gesture, enables the switching of the operation mode by the first hand gesture. 7. The information processing device according to configuration 6. (Configuration 8) the detection means detects the first hand gesture and the second hand gesture; the processing means switches to the first operational mode when the second hand particular gesture is detected. 2. The information processing device according to configuration 1. (Configuration 9) the first operation mode being an operation mode for selecting one of a plurality of groups, each group including a plurality of options; the second operation mode being an operation mode for selecting one option in the selected group; 9. The information processing device according to configuration 8. (Configuration 10) The present invention further includes a control means for controlling a display device to display a specific area in which an option is input, the specific area being superimposed on a real space including the first hand and the second hand; When one option is selected in the second operation mode, the processing means inputs the currently selected option into the specific area in response to a second operation. 10. The information processing device according to configuration 9. (Configuration 11) The information processing device according to configuration 10, characterized in that the control means controls the display device to display the specific area at a position that is a certain distance higher than the higher of the first hand position and the second hand position. (Configuration 12) the control means controls the display device in the second operation mode to display a plurality of options included in a selected group side by side; When a gesture of the first hand or the second hand is detected in the second operation mode, the processing means erases at least one of the input options from the specific area, the gesture indicating a direction in which no options are present as viewed from a selected option, among directions in which the multiple options are arranged. 12. The information processing device according to configuration 10 or 11. (Configuration 13) The processing means includes: In the first operation mode, switching a group to be selected in response to a rotation of the first hand or the second hand; In the second operation mode, the option to be selected is switched in response to a rotation of the first hand or the second hand. 13. The information processing device according to any one of configurations 9 to 12. (Configuration 14) The processing means includes: in the first operation mode, switching a group to be selected in response to the first hand or the second hand being rotated by a larger amount than a threshold; In the second operation mode, the option to be selected is switched in response to the first hand or the second hand being rotated larger than the threshold value. 14. The information processing device according to configuration 13. (Configuration 15) At least one of the plurality of groups is associated with a subgroup including a plurality of options; the processing means switches the first group to the first subgroup in response to a gesture of the first hand or the second hand when selecting one option from a first group associated with a first subgroup; 15. The information processing device according to any one of configurations 9 to 14. (Configuration 16) the first operation mode is an operation mode for converting a character string; The second operation mode is an operation mode for switching a character string to be converted. 9. The information processing device according to configuration 8. (Configuration 17) In the first operation mode, the processing means switches the converted character string in response to at least one of the first hand gesture and the second hand gesture detected by the detection means. 17. The information processing device according to configuration 16. (method) a detection step of detecting a first hand gesture of a user; a processing step of switching to a second operation mode in which a second process different from the first process is executed by the first operation with the second hand of the user when a specific gesture of the first hand is detected in the detection step in a first operation mode in which a first process is executed by the first operation with the second hand of the user; 13. An information processing method comprising: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 17. [Explanation of symbols]

[0148] 1: MR system, 100: HMD, 103: information processing device, 200: CPU

Claims

1. An information processing device that generates data to be displayed on a head-mounted display, detection means for detecting a first hand gesture of a user; a processing means for switching to a second operation mode in which a second process different from the first process is executed in response to the first operation with the second hand when the detection means detects a specific gesture of the first hand in a first operation mode in which a first process is executed in response to a first operation with the second hand of the user; An information processing device comprising:

2. The device further includes a setting unit for setting the user's left hand or right hand as the first hand in response to a user operation.

2. The information processing apparatus according to claim 1, wherein:

3. The device further comprises a setting means for, when the controller is worn on the left or right hand of the user, setting the left or right hand of the user as the first hand depending on whether the controller is worn on the left or right hand of the user.

2. The information processing apparatus according to claim 1, wherein:

4. when the detection means detects the first hand gesture, the processing means switches to an operation mode in which an execution process associated with the first hand gesture is performed; The execution process is a process executed by an operation with the second hand.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. The processing means performing the execution process by operating the second hand on a specific area in real space displayed on the head-mounted display; when the left and right hands of the user are detected in the specific area, switching the operation mode in accordance with the gesture of the first hand detected by the detection means; 5. The information processing apparatus according to claim 4,

6. The processing means disables switching of the operation mode by the gesture of the first hand when the head-mounted display is pointing downward at a certain angle or more relative to the horizontal direction.

6. The information processing apparatus according to claim 5,

7. the processing means enables the switching of the operation mode by the first hand gesture when a specific operation is performed after disabling the switching of the operation mode by the first hand gesture; 7. The information processing apparatus according to claim 6,

8. the detecting means detects the first hand gesture and the second hand gesture; the processing means switches to the first operating mode when the specific gesture of the second hand is detected.

2. The information processing apparatus according to claim 1, wherein:

9. the first operation mode is an operation mode in which one of a plurality of groups, each of which includes a plurality of options, is selected; the second operation mode is an operation mode for selecting one option in the selected group; 9. The information processing apparatus according to claim 8,

10. a control unit that controls the head-mounted display to display a specific area where an option is to be input, the specific area being superimposed on a real space including the first hand and the second hand; When one option is selected in the second operation mode, the processing means inputs the currently selected option into the specific area in response to a second operation.

10. The information processing apparatus according to claim 9,

11. 11. The information processing device according to claim 10, wherein the control means controls the head-mounted display to display the specific area at a position that is a certain distance higher than the higher of the first hand position and the second hand position.

12. the control means controls the head-mounted display in the second operation mode to display a plurality of options included in a selected group side by side, and when a gesture of the first hand or the second hand is detected in the second operation mode, the processing means erases at least one of the input options from the specific area, the gesture indicating a direction in which no options are present as viewed from a selected option, among directions in which the multiple options are arranged.

12. The information processing device according to claim 10 or 11.

13. The processing means in the first operation mode, switching the group to be selected in response to a rotation of the first hand or a rotation of the second hand; In the second operation mode, the option to be selected is switched in response to a rotation of the first hand or a rotation of the second hand.

12. The information processing device according to claim 9, wherein the information processing device is a computer.

14. The processing means in the first operation mode, switching the group to be selected in response to the first hand or the second hand being rotated by a larger amount than a threshold; In the second operation mode, the option to be selected is switched in response to the first hand or the second hand being rotated by a larger amount than the threshold value.

14. The information processing apparatus according to claim 13,

15. At least one of the plurality of groups is associated with a subgroup including a plurality of options; the processing means, when selecting one option from a first group associated with a first subgroup, switches the first group to the first subgroup in response to a gesture of the first hand or the second hand; 12. The information processing device according to claim 9, wherein the information processing device is a computer.

16. the first operation mode is an operation mode for converting a character string; the second operation mode is an operation mode for switching character strings to be converted; 9. The information processing apparatus according to claim 8,

17. In the first operation mode, the processing means switches the converted character string in response to at least one of the first hand gesture and the second hand gesture detected by the detection means.

17. The information processing apparatus according to claim 16,

18. An information processing method for generating data to be displayed on a head-mounted display, comprising: a detecting step of detecting a first hand gesture of a user; a processing step of switching to a second operation mode in which a second process different from the first process is executed by the first operation with the second hand when a specific gesture of the first hand is detected in the detecting step in a first operation mode in which a first process is executed by a first operation with the second hand of the user; An information processing method comprising:

19. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 3.

Images