Information processing apparatus

Abstract

To provide a technology capable of reducing the risk that a virtual key is set at a position different from a correct position.SOLUTION: An information processing device includes: a virtual key setting unit which positions a first virtual key in virtual space; a gesture detection unit which detects a gesture performed by at least one finger; a fingertip detection unit which detects a fingertip position of a first finger different from the at least one finger; and a virtual key control unit which moves the first virtual key to a target position in the virtual space corresponding to the fingertip position of the first finger detected by the fingertip detection unit, when the gesture detection unit detects the gesture in a state that the first virtual key is located in the virtual space.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an information processing apparatus.

Background Art

[0002] Patent Document 1 discloses a technique for performing a key operation using an icon that is a virtual key. In this technique, when three fingertips are detected, an icon is assigned to each position of the three fingertips. Thereafter, when any one of the three fingertips moves a distance equal to or greater than a predetermined value, a key operation corresponding to the icon assigned to the fingertip that has moved a distance equal to or greater than the predetermined value is executed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique described in Patent Document 1, as a trigger for starting the assignment of a virtual key to a fingertip position, a gesture performed by a finger (for example, a gesture of overlapping the index finger and the middle finger) can be considered. In this mode, a virtual key is set at the fingertip position of the finger that has performed the gesture.

[0005] However, in this mode, since the gesture is performed by the finger to which the virtual key is set, there is a possibility that it becomes difficult to detect the fingertip position of the finger that has performed the gesture. For this reason, it becomes easy for the virtual key to be set at a position different from the correct position where the virtual key should be set.

[0006] An object of the present invention is to provide an information processing apparatus capable of reducing the setting of a virtual key at a position different from the correct position.

Means for Solving the Problems

[0007] An information processing apparatus according to one aspect includes a virtual key setting unit that positions a first virtual key in a virtual space, a gesture detection unit that detects a gesture performed by at least one finger, a fingertip detection unit that detects the fingertip position of a first finger different from the at least one finger, and a virtual key control unit that moves the first virtual key to a target position on the virtual space corresponding to the fingertip position of the first finger detected by the fingertip detection unit when the gesture detection unit detects the gesture in a situation where the first virtual key is positioned in the virtual space.

Effect of the Invention

[0008] According to one aspect, it is possible to reduce the situation where the virtual key is set at a position different from the correct position.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0010] A: First Embodiment A1: Information Processing System 100 FIG. 1 is a diagram showing the information processing system 100. The information processing system 100 includes an MR (Mixed Reality) glass 1 and a mobile device 2. The information processing system 100 displays the virtual keyboard 3 shown in FIG. 7 described later on the MR glass 1. The virtual keyboard 3 is located in the virtual space K2. The form of the virtual keyboard 3 is not limited to the form shown in FIG. 7 and can be changed as appropriate.

[0011] The user A wearing the MR glass 1 shown in FIG. 1 can simultaneously visually recognize the virtual keyboard 3 located in the virtual space K2 and the real object located in the real space K1. The information processing system 100 receives an operation on the virtual keyboard 3 from the user A. The information processing system 100 executes key operation processing according to the operation on the virtual keyboard 3.

[0012] The MR glasses 1 are a glasses-type display device. The MR glasses 1 are an example of a transmissive display device. A transmissive display device is a display device that transmits light and displays an image. The transmissive display device is not limited to the MR glasses 1, and for example, a goggle-shaped transmissive HMD (Head Mounted Display) may also be used.

[0013] The transmissive display device is an example of a display device such as XR (X Reality) glasses. XR glasses are also referred to as smart glasses. The display device such as XR glasses is not limited to the transmissive display device, and for example, a video see-through type display device may also be used. The video see-through type display device has a camera that generates an image of an object by imaging the object. The video see-through type display device displays an image obtained by superimposing an image of a virtual object on the image generated by the camera. The virtual object is, for example, the virtual keyboard 3. The video see-through type display device is, for example, a video see-through type HMD.

[0014] The MR glasses 1 include temples 91 and 92, a bridge 93, bodies 94 and 95, lenses 96L and 96R, and cameras 11L and 11R.

[0015] The body 94 includes a display panel for the left eye and an optical member for the left eye. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel.

[0016] The display panel for the left eye displays an image 4 representing the virtual keyboard 3 located in the virtual space K2. Figure 2 is a diagram showing an example of the image 4 representing the virtual keyboard 3. The virtual keyboard 3 is a virtual keyboard generated virtually. The virtual keyboard 3 does not exist in the real space K1. The virtual keyboard 3 has a plurality of virtual keys 31.

[0017] The display panel for the left eye in the body part 94 shown in FIG. 1 displays the image 4 by emitting light representing the image 4 of the virtual keyboard 3. The optical member for the left eye in the body part 94 is an optical member that guides the light emitted from the display panel for the left eye to the lens 96L.

[0018] The body part 95 includes a display panel for the right eye and an optical member for the right eye. The display panel for the right eye displays an image 4 representing the virtual keyboard 3 located in the virtual space K2. The display panel for the right eye displays the image 4 by emitting light representing the image 4 of the virtual keyboard 3. The optical member for the right eye is an optical member that guides the light emitted from the display panel for the right eye to the lens 96R.

[0019] The lenses 96L and 96R each have a half mirror. The half mirror of the lens 96L reflects the light guided by the optical member for the left eye to the left eye BL of the user A. The half mirror of the lens 96L guides the light representing the object located in the real space K1 to the left eye BL of the user A by transmitting the light representing the object located in the real space K1. The half mirror of the lens 96R reflects the light guided by the optical member for the right eye to the right eye BR of the user A. The half mirror of the lens 96R guides the light representing the object located in the real space K1 to the right eye BR of the user A by transmitting the light representing the object located in the real space K1. For this reason, the user A wearing the MR glasses 1 can visually recognize, for example, the right hand C of the user A and the virtual keyboard 3 as shown in FIG. 3.

[0020] The cameras 11L and 11R shown in FIG. 1 each image a subject. The subject is, for example, the left and right "hands" of the user A. "Hand" means the part of the body from the wrist to the fingertips of the user A. "Hand" includes the thumb, index finger, middle finger, ring finger, and little finger. In the following description, the "thumb" in the English-speaking world may be referred to as "finger". That is, "fingers" is a concept that includes the thumb, index finger, middle finger, ring finger, and little finger.

[0021] Cameras 11L and 11R are separated from each other by a first distance which is the baseline length. Therefore, when the subject is the user A's left and right hands, the information processing system 100 can identify the three-dimensional shapes of the left and right hands by using cameras 11L and 11R. The configuration for identifying the three-dimensional shapes of the left and right hands is not limited to cameras 11L and 11R. For example, the configuration for identifying the three-dimensional shapes of the left and right hands may be three or more cameras, or a depth sensor. Cameras 11L and 11R are an example of two or more sensors that measure physical quantities in order to identify the three-dimensional shapes of the left and right hands.

[0022] The three-dimensional shapes of the left and right hands indicate the fingertip positions of the ten fingers and the shapes of the palms of the left and right hands. Therefore, the information processing system 100 can detect a gesture performed by a finger of either the left or right hand based on a change in the three-dimensional shapes of the left and right hands.

[0023] A local coordinate system CS1 is defined in the MR glass 1. The local coordinate system CS1 is the coordinate system of the MR glass 1. The three-dimensional shapes of the left and right hands identified using cameras 11L and 11R are represented by the coordinates of the local coordinate system CS1. The local coordinate system CS1 is defined by an x1 axis, a y1 axis, and a z1 axis. The x1 axis, the y1 axis, and the z1 axis are orthogonal to each other. The origin of the local coordinate system CS1 is located at camera 11L. The z1 axis coincides with the imaging direction of camera 11L.

[0024] In FIG. 1, for simplicity of explanation, the origin of the local coordinate system CS1 is not located at camera 11L. The z1 axis does not coincide with the imaging direction of camera 11L. As shown in FIG. 1, the origin of the local coordinate system CS1 may not be located at camera 11L. The z1 axis may not coincide with the imaging direction of camera 11L. The origin of the local coordinate system CS1 may be located at camera 11R instead of camera 11L. When the origin of the local coordinate system CS1 is located at camera 11R, the z1 axis may coincide with the imaging direction of camera 11R.

[0025] The portable device 2 is a smartphone. The portable device 2 is not limited to a smartphone and may be, for example, a tablet or a notebook personal computer. The portable device 2 is also referred to as a terminal device. The portable device 2 is an example of an information processing device. The information processing device is not limited to the portable device 2 and may be, for example, a desktop personal computer that communicates directly or indirectly with the MR glasses 1, or a server that communicates directly or indirectly with the MR glasses 1.

[0026] The portable device 2 is connected to the MR glasses 1 by wire. The portable device 2 may be connected to the MR glasses 1 wirelessly. The portable device 2 controls the MR glasses 1. For example, the portable device 2 causes the MR glasses 1 to display an image 4 representing the virtual keyboard 3.

[0027] A2: MR glasses 1 FIG. 4 is a diagram showing an example of the MR glasses 1. The MR glasses 1 include a display device 12, an acceleration sensor 13, a gyro sensor 14, operation buttons 15, a communication device 16, a storage device 17, a processing device 18, and a bus 19, in addition to cameras 11L and 11R.

[0028] The bus 19 is a wiring for communicating information. The bus 19 connects the cameras 11L, the camera 11R, the display device 12, the acceleration sensor 13, the gyro sensor 14, the operation buttons 15, the communication device 16, the storage device 17, and the processing device 18 to each other. The bus 19 may be configured using a single bus or may be configured using different buses for each element such as a device.

[0029] The camera 11L generates image data D1 by imaging a subject. The image data D1 is data representing a moving image. The image data D1 includes a series of still image data. Each still image data of the image data D1 represents a still image that constitutes one frame of the moving image represented by the image data D1.

[0030] The camera 11R outputs image data D2 by imaging a subject. The image data D2 is data representing a moving image. The image data D2 includes a series of still image data. Each still image data of the image data D2 represents a still image that constitutes one frame of the moving image represented by the image data D2.

[0031] The display device 12 includes the lens 96L, the display panel for the left eye, the optical member for the left eye, the lens 96R, the display panel for the right eye, and the optical member for the right eye, which were described with reference to FIG. 1. The display device 12 transmits light representing an object located in the real space K1. The display device 12 further displays an image 4 representing the virtual keyboard 3 located in the virtual space K2. When the user A wears the MR glasses 1, the display device 12 is positioned in front of the left eye BL and the right eye BR of the user A. Therefore, the user A wearing the MR glasses 1 can visually recognize an object located in the real space K1 and an object located in the virtual space K2. For example, the user A can visually recognize the right hand C of the user A located in the real space K1 and the virtual keyboard 3 located in the virtual space K2.

[0032] The acceleration sensor 13 measures the acceleration of the MR glasses 1. The acceleration sensor 13 generates acceleration data E1 based on the measurement result of the acceleration of the MR glasses 1. The acceleration data E1 is data indicating the acceleration of the MR glasses 1. For example, the acceleration data E1 indicates the acceleration of the MR glasses 1 in the directions of the x1 axis, the y1 axis, and the z1 axis. Based on the acceleration data E1, the moving direction and the moving distance of the MR glasses 1 are specified.

[0033] The gyro sensor 14 measures the angular acceleration of the rotation of the MR glasses 1 about each of the x1 axis, the y1 axis, and the z1 axis. The gyro sensor 14 generates angular acceleration data E2 based on the measurement result of the angular acceleration of the MR glasses 1. The angular acceleration data E2 is data indicating the angular acceleration of the MR glasses 1. For example, the angular acceleration data E2 indicates the angular acceleration of the rotation of the MR glasses 1 about each of the x1 axis, the y1 axis, and the z1 axis. Based on the angular acceleration data E2, the rotation of the MR glasses 1 is specified.

[0034] The operation button 15 is a button operable by User A. The operation button 15 may have a plurality of buttons operable by User A. The operation button 15 generates operation data F1. The operation data F1 is data indicating the operation state of the operation button 15.

[0035] The communication device 16 communicates with the portable device 2 by wire. The communication device 16 may communicate with the portable device 2 wirelessly. The communication device 16 transmits comprehensive data G1 to the portable device 2. The comprehensive data G1 is data including the image data D1, the image data D2, the acceleration data E1, the angular acceleration data E2, and the operation data F1. The comprehensive data G1 may not include the operation data F1. The communication device 16 receives the image data H1 from the portable device 2. The image data H1 is data representing the image 4 of the virtual keyboard 3. The image data H1 is generated by the portable device 2 based on, for example, the image data D1, the image data D2, the acceleration data E1, and the angular acceleration data E2.

[0036] The storage device 17 is a recording medium readable by the processing device 18. The storage device 17 includes, for example, a non-volatile memory and a volatile memory. The non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, RAM (Random Access Memory). The storage device 17 stores the program PG1.

[0037] The processing device 18 includes one or more CPUs (Central Processing Unit). The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer.

[0038] The processing device 18 reads the program PG1 from the storage device 17. By executing the program PG1, the processing device 18 functions as an acquisition unit 181 and an operation control unit 182. At least one of the acquisition unit 181 and the operation control unit 182 may be constituted by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array).

[0039] The acquisition unit 181 acquires the image data D1, the image data D2, the acceleration data E1, the angular acceleration data E2, and the operation data F1.

[0040] The operation control unit 182 controls the MR glasses 1. For example, the operation control unit 182 receives the image data D1, the image data D2, the acceleration data E1, the angular acceleration data E2, and the operation data F1 from the acquisition unit 181. The operation control unit 182 generates comprehensive data G1 including the image data D1, the image data D2, the acceleration data E1, the angular acceleration data E2, and the operation data F1. The operation control unit 182 causes the communication device 16 to transmit the comprehensive data G1 to the portable device 2. When the communication device 16 receives the image data H1 from the portable device 2, the operation control unit 182 acquires the image data H1 from the communication device 16. The operation control unit 182 causes the display device 12 to display an image based on the image data H1. The image based on the image data H1 is, for example, an image 4 representing the virtual keyboard 3.

[0041] A3: Portable device 2 FIG. 5 is a diagram showing an example of the portable device 2. The portable device 2 includes an input device 21, a display device 22, a communication device 23, a storage device 24, a processing device 25, and a bus 26.

[0042] Bus 26 is a wiring for communicating information. Bus 26 connects an input device 21, a display device 22, a communication device 23, a storage device 24, and a processing device 25 to each other. Bus 26 may be configured using a single bus or may be configured using different buses for each element such as a device.

[0043] The input device 21 includes a touch panel. The input device 21 may include a plurality of operation keys in addition to the touch panel. The input device 21 may include a plurality of operation keys without including the touch panel. The input device 21 receives an operation performed by user A.

[0044] The display device 22 includes a display. The touch panel of the input device 21 is laminated on the display of the display device 22. The display device 22 displays various information.

[0045] The communication device 23 communicates with the MR glass 1 by wire. The communication device 23 may communicate with the MR glass 1 wirelessly. The communication device 23 receives the comprehensive data G1 from the MR glass 1. The communication device 23 transmits the image data H1 to the MR glass 1.

[0046] The storage device 24 is a recording medium readable by the processing device 25. The storage device 24 includes, for example, a non-volatile memory and a volatile memory. The storage device 24 stores the program PG2.

[0047] The processing device 25 includes one or more CPUs. The processing device 25 is another example of an information processing device. The processing device 25 reads the program PG2 from the storage device 24. By executing the program PG2, the processing device 25 functions as a hand data generation unit 251, a fingertip detection unit 252, a coordinate conversion unit 253, a virtual key setting unit 254, an image data generation unit 255, a gesture detection unit 256, a virtual key control unit 257, and an operation detection unit 258. At least one of the hand data generation unit 251, the fingertip detection unit 252, the coordinate conversion unit 253, the virtual key setting unit 254, the image data generation unit 255, the gesture detection unit 256, the virtual key control unit 257, and the operation detection unit 258 may be constituted by circuits such as a DSP, an ASIC, and an FPGA.

[0048] The hand data generation unit 251 acquires the comprehensive data G1 via the communication device 23. The hand data generation unit 251 reads the image data D1 and D2 from the comprehensive data G1. The image data D1 is the image data generated by the camera 11L of the MR glass 1. The image data D2 is the image data generated by the camera 11R of the MR glass 1. The hand data generation unit 251 generates hand data J1 based on the image data D1 and D2. The hand data J1 is data indicating the three-dimensional shapes of both hands of the user A in the coordinates of the local coordinate system CS1 defined in the MR glass 1.

[0049] For example, the hand data generation unit 251 generates the hand data J1 based on the still image data included in the image data D1 and the still image data included in the image data D2. The hand data generation unit 251 changes each of the still image data in the image data D1 for generating the hand data J1 and the still image data in the image data D2 for generating the hand data J1 to the latest still image data every time a predetermined time elapses. The hand data generation unit 251 generates the hand data J1 using the latest still image data every time the still image data is changed. The predetermined time is, for example, 0.2 seconds. The predetermined time is not limited to 0.2 seconds and may be a time shorter than 0.2 seconds or a time longer than 0.2 seconds.

[0050] The finger tip detection unit 252 detects the fingertips of each finger of the left and right hands of user A based on the hand data J1. The finger tip detection unit 252 generates finger tip data J2 based on the detection results of the fingertips of each finger. The finger tip data J2 is data indicating the finger tip positions of each finger by the coordinates of the local coordinate system CS1 defined in the MR glasses 1. The finger tip detection unit 252 generates the finger tip data J2 for each hand data J1.

[0051] The coordinate conversion unit 253 converts the coordinates of the local coordinate system CS1 defined in the MR glasses 1 into the coordinates of the world coordinate system CS2. The world coordinate system CS2 is a coordinate system defined in the real space K1.

[0052] FIG. 6 is a diagram showing an example of the relationship between the world coordinate system CS2 and the local coordinate system CS1. The world coordinate system CS2 is defined by the x2 axis, the y2 axis, and the z2 axis. The x2 axis, the y2 axis, and the z2 axis are mutually orthogonal.

[0053] The coordinate conversion unit 253 shown in FIG. 5 determines the positions of the x2 axis, the y2 axis, and the z2 axis of the world coordinate system CS2 in the real space K1 to the positions of the x1 axis, the y1 axis, and the z1 axis of the local coordinate system CS1 at the timing t0 in the real space K1. The timing t0 means the timing for defining the world coordinate system CS2 in the real space K1. The timing t0 is, for example, the timing at which the coordinate conversion unit 253 receives the "keyboard setting instruction" input by the user A to the input device 21. The "keyboard setting instruction" is an instruction for setting the virtual keyboard 3.

[0054] At the timing t0, the x1 axis, the y1 axis, and the z1 axis of the local coordinate system CS1 respectively coincide with the x2 axis, the y2 axis, and the z2 axis of the world coordinate system CS2. As the MR glasses 1 move after the timing t0, the x1 axis, the y1 axis, and the z1 axis of the local coordinate system CS1 respectively deviate from the x2 axis, the y2 axis, and the z2 axis of the world coordinate system CS2. As the MR glasses 1 rotate after the timing t0, the x1 axis, the y1 axis, and the z1 axis of the local coordinate system CS1 respectively deviate from the x2 axis, the y2 axis, and the z2 axis of the world coordinate system CS2.

[0055] The coordinate conversion unit 253 reads the acceleration data E1 from the comprehensive data G1. Based on the acceleration data E1, the coordinate conversion unit 253 identifies the movement of the MR glass 1 after the timing t0. As the movement of the MR glass 1, the coordinate conversion unit 253 identifies the movement direction of the MR glass 1 after the timing t0 and the movement distance of the MR glass 1 after the timing t0.

[0056] The coordinate conversion unit 253 reads the angular acceleration data E2 from the comprehensive data G1. Based on the angular acceleration data E2, the coordinate conversion unit 253 identifies the rotation of the MR glass 1 after the timing t0. As the rotation of the MR glass 1, the coordinate conversion unit 253 identifies the rotation direction of the MR glass 1 after the timing t0 and the rotation angle of the MR glass 1 after the timing t0.

[0057] Every time the coordinate conversion unit 253 receives the fingertip data J2 during the period after the timing t0, the coordinate conversion unit 253 identifies the movement of the MR glass 1 and the rotation of the MR glass 1. Every time the coordinate conversion unit 253 identifies the movement of the MR glass 1 and the rotation of the MR glass 1, based on the latest movement and rotation of the MR glass 1, the coordinate conversion unit 253 converts the fingertip position of each finger indicated by the fingertip data J2 from the coordinates of the local coordinate system CS1 to the coordinates of the world coordinate system CS2.

[0058] For example, based on the movement of the MR glass 1 during the period from timing t0 to the time of receiving the latest fingertip data J2, the coordinate conversion unit 253 identifies a translation matrix corresponding to the movement of the MR glass 1. Based on the rotation of the MR glass 1 during the period from timing t0 to the time of receiving the latest fingertip data J2, the coordinate conversion unit 253 identifies a rotation matrix corresponding to the rotation of the MR glass 1. By using the translation matrix and the rotation matrix, the coordinate conversion unit 253 converts the fingertip positions of each finger indicated by the latest fingertip data J2 from the coordinates of the local coordinate system CS1 to the coordinates of the world coordinate system CS2. The coordinate conversion unit 253 generates fingertip data J3. The fingertip data J3 is data indicating the fingertip positions of each finger indicated by the latest fingertip data J2 in the coordinates of the world coordinate system CS2. The coordinate conversion unit 253 generates the fingertip data J3 for each piece of fingertip data J2.

[0059] Further, each time the coordinate conversion unit 253 identifies the movement and rotation of the MR glass 1, based on the latest movement and rotation of the MR glass 1, the coordinate conversion unit 253 converts the coordinates of the local coordinate system CS1 of the camera 11L to the coordinates of the world coordinate system CS2. The coordinates of the local coordinate system CS1 of the camera 11L are predetermined. The coordinates of the local coordinate system CS1 of the camera 11L are, for example, the coordinates of the origin of the local coordinate system CS1.

[0060] Each time the coordinate conversion unit 253 identifies the movement and rotation of the MR glass 1, based on the latest movement and rotation of the MR glass 1, the coordinate conversion unit 253 converts the coordinates of the local coordinate system CS1 indicating the imaging direction of the camera 11L to the coordinates of the world coordinate system CS2. The imaging direction of the camera 11L is predetermined. The imaging direction of the camera 11L is, for example, the direction of the x1 axis of the local coordinate system CS1.

[0061] The coordinate conversion unit 253 generates camera data J4. The camera data J4 is data indicating the coordinates in the world coordinate system CS2 of camera 11L and the coordinates in the world coordinate system CS2 indicating the imaging direction of camera 11L. The camera data J4 is not limited to the above data, and may also be data indicating the coordinates in the world coordinate system CS2 of camera 11R and the coordinates in the world coordinate system CS2 indicating the imaging direction of camera 11R. The camera data J4 may be data indicating the coordinates in the world coordinate system CS2 of a point between camera 11L and camera 11R and the coordinates in the world coordinate system CS2 indicating the imaging direction of camera 11L or 11R. The camera data J4 is used, for example, to generate the image data H1 representing the image 4 of the virtual keyboard 3. The coordinate conversion unit 253 generates the camera data J4 each time the movement and rotation of the MR glass 1 are specified.

[0062] The virtual key setting unit 254 positions the virtual keyboard 3 in the virtual space K2. FIG. 7 is a diagram showing an example of the virtual keyboard 3 positioned in the virtual space K2. The virtual space K2 is a virtual three-dimensional space generated by the virtual key setting unit 254. The world coordinate system CS2 is defined in the virtual space K2. In the virtual space K2, the positions of the x2-axis, y2-axis, and z2-axis of the world coordinate system CS2 are fixed in advance.

[0063] The virtual key setting unit 254 shown in FIG. 5 positions the virtual keyboard 3 in the virtual space K2 according to the timing t1. The timing t1 means the timing for positioning the virtual keyboard 3 in the virtual space K2. The timing t1 is, for example, the timing when the virtual key setting unit 254 receives the "keyboard setting instruction" input by the user A to the input device 21.

[0064] Timing t1 is not limited to the timing described above. For example, timing t1 may be the timing when the gesture detection unit 256 described later detects the first gesture performed by user A. The first gesture is, for example, a gesture of bringing the fingertip of the index finger of user A's left hand into contact with the fingertip of the thumb of user A's left hand. The first gesture is not limited to the gesture described above. For example, the first gesture may be a gesture of bringing the fingertip of the middle finger of user A's left hand into contact with the fingertip of the thumb of user A's left hand.

[0065] The virtual key setting unit 254 determines the reference plane P1 on which the virtual keyboard 3 is arranged in the virtual space K2 at timing t1.

[0066] FIG. 8 is a diagram showing an example of the reference plane P1 in the virtual space K2. In FIG. 8, the right hand C of user A that does not exist in the virtual space K2 is shown by a dotted line for explanation. The coordinate position of the right hand C in the world coordinate system CS2 in the virtual space K2 shown in FIG. 8 coincides with the coordinate position of the right hand C in the world coordinate system CS2 in the real space K1.

[0067] The virtual key setting unit 254 shown in FIG. 5 determines the reference plane P1 based on, for example, the fingertip positions of a plurality of fingers of the right hand C. The number of fingertip positions required to determine the reference plane P1 is at least three.

[0068] The virtual key setting unit 254 first identifies the fingertip positions of three fingers out of the five fingers of the right hand C based on the latest fingertip data J3 at the time of timing t1. The virtual key setting unit 254 determines the plane including the fingertip positions of the three fingers as the reference plane P1.

[0069] If three points are determined in the virtual space K2, the plane including these three points is uniquely determined. When the reference plane P1 is a plane, the virtual key setting unit 254 determines the plane including the fingertip positions of the three fingers as the reference plane P1.

[0070] The reference plane P1 is not limited to a flat plane and may be, for example, a curved surface. When the reference plane P1 is a curved surface, the virtual key setting unit 254 determines the curved surface using a predetermined function. The predetermined function uniquely determines a curved surface according to the three-dimensional coordinates of three input points. The virtual key setting unit 254 inputs the fingertip positions of three fingers into the predetermined function. The virtual key setting unit 254 determines, as the reference plane P1, the curved surface determined by the predetermined function according to the fingertip positions of the three fingers. When the reference plane P1 is a curved surface, the virtual keyboard 3 may be a so-called ergonomic type keyboard.

[0071] The fingertip positions of the three fingers used to determine the reference plane P1 are, for example, the fingertip position of the thumb, the fingertip position of the ring finger, and the fingertip position of the little finger. The thumb, ring finger, and little finger are located at the left end of the right hand C, a position to the left of the right end of the right hand C, and the right end of the right hand C, respectively, when viewed from the back of the right hand C. Therefore, when the fingertip position of the thumb, the fingertip position of the ring finger, and the fingertip position of the little finger are used to determine the reference plane P1, the reference plane P1 is determined by the entire right hand C.

[0072] The virtual key setting unit 254 may also determine the reference plane P1 based on the fingertip positions of four or more fingers. In this case, the virtual key setting unit 254 first specifies the distance from each fingertip position to the reference plane P1 as an error. Subsequently, the virtual key setting unit 254 determines the reference plane P1 that minimizes the total error using, for example, the least squares method.

[0073] The virtual key setting unit 254 may determine the reference plane P1 without using the fingertip positions of multiple fingers. For example, the virtual key setting unit 254 may determine the reference plane P1 at an arbitrary position in the virtual space K2.

[0074] When the virtual key setting unit 254 determines the reference plane P1, it arranges the virtual keyboard 3 on the reference plane P1.

[0075] When the virtual key setting unit 254 arranges the virtual keyboard 3 on the reference plane P1, it identifies the fingertip position of the middle finger of the right hand C based on the latest fingertip data J3. The middle finger of the right hand C is an example of a reference finger. The reference finger is not limited to the middle finger of the right hand C, and may be, for example, the thumb of the right hand C, the index finger of the right hand C, the ring finger of the right hand C, or the little finger of the right hand C.

[0076] When the virtual key setting unit 254 identifies the fingertip position of the middle finger of the right hand C, it shifts the virtual keyboard 3 along the reference plane P1 to position the virtual key 31 representing "H" on the virtual keyboard 3 at the fingertip position of the middle finger of the right hand C. Hereinafter, the virtual key 31 representing "H" is referred to as the virtual key 31H. The virtual key 31H is an example of a first virtual key. The first virtual key is not limited to the virtual key 31H, and may be a virtual key 31 different from the virtual key 31H. The virtual key 31 different from the virtual key 31H is, for example, the virtual key 31 representing "J" or the virtual key 31 representing "K".

[0077] When the virtual key setting unit 254 positions the virtual key 31H at the fingertip position of the middle finger of the right hand C in the virtual space K2, it generates virtual space data J5. The virtual space data J5 is data representing the virtual space K2 in which the virtual keyboard 3 is arranged.

[0078] The image data generation unit 255 generates image data H1 based on the virtual space data J5 and the camera data J4. The camera data J4 indicates the coordinates of the world coordinate system CS2 of the camera 11L and the coordinates of the world coordinate system CS2 indicating the imaging direction of the camera 11L.

[0079] The image data generation unit 255 arranges the virtual camera N1 at the coordinate position of the world coordinate system CS2 of the camera 11L indicated by the camera data J4 in the virtual space K2 represented by the virtual space data J5. When the camera data J4 indicates the coordinate position of the world coordinate system CS2 of the camera 11R, the image data generation unit 255 arranges the virtual camera N1 at the coordinate position of the world coordinate system CS2 of the camera 11R indicated by the camera data J4 in the virtual space K2. When the camera data J4 indicates the coordinate position of a point between the camera 11L and the camera 11R in the world coordinate system CS2, the image data generation unit 255 arranges the virtual camera N1 at the coordinate position of the world coordinate system CS2 of the point indicated by the camera data J4 in the virtual space K2.

[0080] FIG. 9 is a diagram showing an example of the virtual camera N1 arranged in the virtual space K2. The imaging direction of the virtual camera N1 coincides with the imaging direction of the camera 11L indicated by the camera data J4. When the camera data J4 indicates the imaging direction of the camera 11R, the imaging direction of the virtual camera N1 coincides with the imaging direction of the camera 11R indicated by the camera data J4.

[0081] The image data generation unit 255 shown in FIG. 5 generates the captured image data obtained by the virtual camera N1 capturing the virtual space K2 as the image data H1. Therefore, the image data H1 represents an image corresponding to the line of sight of the user A wearing the MR glasses 1. An image corresponding to the line of sight of the user A wearing the MR glasses 1 is, for example, the image 4 representing the virtual keyboard 3.

[0082] The image data generation unit 255 causes the communication device 23 to transmit the image data H1 to the MR glasses 1. When the MR glasses 1 receive the image data H1, they display the image represented by the image data H1 (for example, the image 4 of the virtual keyboard 3). Therefore, the user A can operate the virtual keyboard 3 while looking at the virtual keyboard 3 displayed on the MR glasses 1.

[0083] As time elapses since the virtual keyboard 3 is placed in the virtual space K2, it becomes difficult for user A to recognize the distance between user A's finger and the virtual keyboard 3. For example, as time elapses, if the fingertip position C1 of user A moves away from the virtual keyboard 3 as shown in FIG. 10, user A will find it difficult to recognize the distance between user A's finger and the virtual keyboard 3.

[0084] The portable device 2 can reset the positional relationship between user A's finger and the virtual keyboard 3 so that user A can recognize the distance between user A's finger and the virtual keyboard 3 again. For example, when the portable device 2 detects a second gesture performed by the finger of user A's left hand, it moves the virtual key 31H to the target position M1 on the virtual space K2 corresponding to the fingertip position of the middle finger of user A's right hand C. That is, the portable device 2 moves the virtual key 31H to the target position M1 corresponding to the fingertip position of the finger that does not perform the second gesture. In this way, in the portable device 2, the finger performing the second gesture is different from the finger corresponding to the target position M1.

[0085] If the finger performing the second gesture is the same as the finger corresponding to the target position M1, depending on the content of the second gesture, it may be difficult to detect the fingertip position of the finger corresponding to the target position M1. Also, in order to make it easier to detect the fingertip position of the finger corresponding to the target position M1 when the finger performing the second gesture is the same as the finger corresponding to the target position M1, the content of the second gesture is restricted. That is, the degree of freedom of the second gesture becomes lower.

[0086] In the portable device 2, the finger performing the second gesture is different from the finger corresponding to the target position M1. Therefore, it is possible to reduce the difficulty of detecting the fingertip position of the finger corresponding to the target position M1 due to the second gesture, and it is possible to avoid the reduction of the degree of freedom of the second gesture.

[0087] FIG. 11 is a diagram showing an example of the movement of the virtual key 31H to the target position M1. In FIG. 11, the virtual keyboard 3 indicated by the dotted line and the virtual key 31H indicated by the dotted line represent the virtual keyboard 3 before the movement and the virtual key 31H before the movement. The virtual keyboard 3 indicated by the solid line and the virtual key 31H indicated by the solid line represent the virtual keyboard 3 after the movement and the virtual key 31H before the movement.

[0088] The gesture detection unit 256 shown in FIG. 5 detects a second gesture. The second gesture is a gesture for re-setting the positional relationship between the finger of user A and the virtual keyboard 3. The second gesture is, for example, a gesture in which the index finger of the left hand of user A touches the middle finger of the left hand of user A. The index finger of the left hand of user A and the middle finger of the left hand of user A are an example of at least one finger. The at least one finger is one finger, two fingers, three fingers, four fingers, five fingers, six fingers, seven fingers, eight fingers, or nine fingers out of the ten fingers of both hands of user A.

[0089] The second gesture is not limited to the above-described gesture. For example, the second gesture may be a gesture in which the fingertip of the little finger of the left hand of user A touches the fingertip of the thumb of the left hand of user A. The second gesture may be the same gesture as the first gesture.

[0090] The gesture detection unit 256 detects the second gesture based on a plurality of hand data J1 including the latest hand data J1. The hand data J1 shows the three-dimensional shape of both hands of user A. When the gesture detection unit 256 detects the second gesture, it generates detection data J6. The detection data J6 is data representing the detection of the second gesture. The gesture detection unit 256 provides the detection data J6 to the virtual key control unit 257.

[0091] The virtual key control unit 257 re - sets the positional relationship between the finger of user A and the virtual keyboard 3. When the gesture detection unit 256 detects a second gesture in a situation where the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the virtual key 31H to the target position M1 in the virtual space K2.

[0092] When the virtual key control unit 257 receives the detection data J6, it determines that the gesture detection unit 256 has detected a second gesture.

[0093] When the virtual key control unit 257 receives the detection data J6, it determines whether the virtual keyboard 3 is located in the virtual space K2 based on the virtual space data J5. For example, the virtual key control unit 257 determines whether the virtual keyboard 3 is located in the virtual space K2 by referring to the virtual space data J5. If the virtual space data J5 represents the virtual keyboard 3, the virtual key control unit 257 determines that the virtual keyboard 3 is located in the virtual space K2. If the virtual space data J5 does not represent the virtual keyboard 3, the virtual key control unit 257 determines that the virtual keyboard 3 is not located in the virtual space K2.

[0094] Note that the virtual space data J5 is generated when the virtual key setting unit 254 positions the virtual key 31H in the virtual space K2. Therefore, when the virtual space data J5 is generated, the virtual key control unit 257 may determine that the virtual keyboard 3 is located in the virtual space K2. When the virtual space data J5 is not generated, the virtual key control unit 257 may determine that the virtual keyboard 3 is not located in the virtual space K2.

[0095] When the virtual key control unit 257 determines that the virtual keyboard 3 is located in the virtual space K2, it determines the fingertip position of the middle finger of the right hand C indicated by the latest fingertip data J3 as the target position M1.

[0096] The middle finger of the right hand C is an example of a first finger that is different from "at least one finger". The first finger is not limited to the middle finger of the right hand C, and may be, for example, the thumb of the right hand C, the index finger of the right hand C, the ring finger of the right hand C, or the little finger of the right hand C. The first finger may also be the reference finger described above. The reference finger is the finger used to position the virtual keyboard 3 in the virtual space K2. In the configuration where the first finger is the reference finger, if the user A is aware of the positional relationship between the first finger and the reference finger (which is one finger used as both the first finger and the reference finger) and the virtual keyboard 3, the user A can recognize the positional relationship between the virtual keyboard 3 and the user's finger. Therefore, compared with the configuration where the first finger is different from the reference finger, it is easier for the user A to recognize the positional relationship between the position of the first virtual key and the user's finger. Note that the first finger may be different from the reference finger.

[0097] The fingertip position of the middle finger of the right hand C indicated by the latest fingertip data J3 is an example of "the target position on the virtual space K2 corresponding to the fingertip position of the first finger detected by the fingertip detection unit 252".

[0098] When the virtual key control unit 257 determines the target position M1, it controls the virtual key setting unit 254 to move the virtual key 31H to the target position M1 in the virtual space K2.

[0099] For example, the virtual key control unit 257 moves the virtual key 31H to the target position M1 by moving the virtual keyboard 3. For example, the virtual key control unit 257 provides the first movement instruction J7 to the virtual key setting unit 254, causing the virtual key setting unit 254 to move the virtual keyboard 3 and move the virtual key 31H to the target position M1. The first movement instruction J7 is an instruction to move the virtual key 31H to the target position M1 by moving the virtual keyboard 3. The virtual key setting unit 254 moves the virtual keyboard 3 in the virtual space K2 and moves the virtual key 31H to the target position M1 according to the first movement instruction J7.

[0100] Note that when the virtual keyboard 3 is not arranged in the virtual space K2, the virtual key control unit 257 does not move the virtual key 31H.

[0101] During the period from when the virtual key control unit 257 determines the target position M1 until it provides the first movement instruction J7, the virtual key setting unit 254 may be made to re-determine the reference plane P1 based on the latest fingertip data J3. For example, during the period from when the virtual key control unit 257 determines the target position M1 until it provides the first movement instruction J7, the virtual key control unit 257 provides an instruction to re-determine the reference plane P1 to the virtual key setting unit 254. The method for re-determining the reference plane P1 is the same as the method for determining the reference plane P1. In this case, the virtual key control unit 257 moves the virtual keyboard 3 on the re-determined reference plane P1, thereby moving the virtual key 31H to the target position M1.

[0102] When the virtual key setting unit 254 moves the virtual keyboard 3, it updates the virtual space data J5. The updated virtual space data J5 represents the virtual space K2 where the moved virtual keyboard 3 is located.

[0103] The operation detection unit 258 detects an operation on the virtual keyboard 3. For example, the operation detection unit 258 detects an operation on the virtual keyboard 3 based on the latest virtual space data J5 and the latest fingertip data J3. For example, when any of the plurality of fingertip positions indicated by the latest fingertip data J3 falls within any of the virtual keys 31 of the virtual keyboard 3, the operation detection unit 258 detects an operation on the virtual key 31 into which the fingertip position has entered. The operation detection unit 258 executes key operation processing based on the detection result of the operation on the virtual keyboard 3. The key operation processing is the processing associated with the operated virtual key 31. For example, when the virtual key 31H is operated, the operation detection unit 258 executes, as the key operation processing, the processing of inputting the character "H".

[0104] A4: Description of Operations The operation of the portable device 2 will be described. First, the operation of arranging the virtual keyboard 3 in the virtual space K2 will be described. Subsequently, the operation of moving the virtual keyboard 3 will be described.

[0105] FIG. 12 is a diagram for explaining an example of the operation of arranging the virtual keyboard 3 in the virtual space K2.

[0106] In step S101, the hand data generation unit 251 generates hand data J1 based on the image data D1 and D2. The hand data J1 is data indicating the three-dimensional shapes of both hands of user A in the coordinates of the local coordinate system CS1.

[0107] Subsequently, in step S102, the fingertip detection unit 252 generates fingertip data J2 based on the hand data J1. The fingertip data J2 is data indicating the fingertip positions of each finger of the left and right hands of user A in the coordinates of the local coordinate system CS1.

[0108] Subsequently, in step S103, the coordinate conversion unit 253 generates fingertip data J3 based on the fingertip data J2, the acceleration data E1, and the angular acceleration data E2. The fingertip data J3 is data indicating the fingertip positions of each finger indicated by the fingertip data J2 in the coordinates of the world coordinate system CS2.

[0109] Subsequently, in step S104, the coordinate conversion unit 253 generates camera data J4 based on the acceleration data E1 and the angular acceleration data E2. The camera data J4 is data indicating the coordinates of the world coordinate system CS2 of the camera 11L and the coordinates of the world coordinate system CS2 indicating the imaging direction of the camera 11L. Step S104 may be executed before step S103 is executed.

[0110] Subsequently, in step S105, the virtual key setting unit 254 positions the virtual keyboard 3 in the virtual space K2 according to the timing t1. The timing t1 is, for example, the timing at which the virtual key setting unit 254 receives a "keyboard setting instruction" input by the user A to the input device 21.

[0111] In step S105, the virtual key setting unit 254 determines the reference plane P1 in the virtual space K2 based on, for example, the fingertip positions of multiple fingers of the right hand C. After determining the reference plane P1, the virtual key setting unit 254 arranges the virtual keyboard 3 on the reference plane P1. When arranging the virtual keyboard 3 on the reference plane P1, the virtual key setting unit 254 identifies the fingertip position of the middle finger of the right hand C based on the latest fingertip data J3. After identifying the fingertip position of the middle finger of the right hand C, the virtual key setting unit 254 shifts the virtual keyboard 3 along the reference plane P1 to position the virtual key 31H representing "H" at the fingertip position of the middle finger of the right hand C.

[0112] FIG. 13 is a diagram for explaining an example of the operation of moving the virtual keyboard 3. When the gesture detection unit 256 detects a second gesture based on the hand data J1 in step S201, it generates detection data J6 in step S202. The second gesture is, for example, a gesture in which the index finger of the left hand of user A touches the middle finger of the left hand of user A. The detection data J6 is data representing the detection of the second gesture.

[0113] Subsequently, in step S203, the virtual key control unit 257 determines whether the virtual keyboard 3 is located in the virtual space K2 based on the virtual space data J5.

[0114] If the virtual key control unit 257 determines in step S203 that the virtual keyboard 3 is located in the virtual space K2, it determines the target position M1 based on the fingertip data J3 in step S204. The target position M1 is the fingertip position of the middle finger of the right hand C indicated by the latest fingertip data J3.

[0115] Subsequently, in step S205, the virtual key control unit 257 controls the virtual key setting unit 254 to move the virtual key 31H to the target position M1 in the virtual space K2. For this reason, user A wearing the MR glasses 1 obtains a feeling as if the virtual keyboard 3 is attracted to the fingertip of the middle finger of the right hand C.

[0116] If the gesture detection unit 256 does not detect the second gesture in step S201, the process shown in FIG. 13 ends. If the virtual keyboard 3 is not located in the virtual space K2 in step S203, the process shown in FIG. 13 ends.

[0117] A5: Summary of the First Embodiment The virtual key setting unit 254 positions the virtual key 31H in the virtual space K2. The gesture detection unit 256 detects a second gesture in which the index finger of the user A's left hand touches the middle finger of the user A's left hand. The fingertip detection unit 252 detects the fingertip positions of each finger of the user A's left and right hands. Therefore, the fingertip detection unit 252 detects at least the fingertip position of the middle finger of the user A's right hand C. When the gesture detection unit 256 detects the second gesture in a situation where the virtual key 31H is located in the virtual space K2, the virtual key control unit 257 moves the virtual key 31H to a target position M1 on the virtual space K2 corresponding to the fingertip position of the middle finger of the user A's right hand C detected by the fingertip detection unit 252.

[0118] The gesture detection unit 256 detects a gesture performed by a finger different from the middle finger of the right hand C corresponding to the movement destination of the virtual key 31H. Therefore, it is possible to avoid a situation where a gesture is performed by the middle finger of the right hand C. Thus, it is possible to solve the problem that it becomes difficult to detect the fingertip position of the middle finger of the user A's right hand C due to a gesture performed by the middle finger of the right hand C.

[0119] The virtual key 31H is an example of a first virtual key. The first virtual key is any one of the virtual keys 31 included in the virtual keyboard 3. The first virtual key is not limited to any one of the virtual keys 31 included in the virtual keyboard 3. For example, the first virtual key may be a virtual key not included in the virtual keyboard 3. A virtual key not included in the virtual keyboard 3 is, for example, a virtual PLAY key for video or a virtual stop key for video.

[0120] The middle finger of the right hand C corresponding to the destination of the virtual key 31H is an example of the first finger. The first finger is not limited to the middle finger of the right hand C. The first finger may be any one of the ten fingers of the left and right hands of the user A.

[0121] The index finger of the left hand of the user A performing the second gesture and the middle finger of the left hand of the user A are examples of at least one finger performing the second gesture. At least one finger performing the second gesture is not limited to the index finger of the left hand of the user A and the middle finger of the left hand of the user A. At least one finger performing the second gesture may be one or more fingers different from the first finger among the ten fingers of the left and right hands of the user A.

[0122] In the first embodiment, the combination of the finger performing the second gesture and the first finger is a combination of a finger of the left hand and a finger of the right hand C. The combination of a finger of the left hand and a finger of the right hand C means a combination of fingers of different hands. The combination of the finger performing the second gesture and the first finger is not limited to a combination of fingers of different hands. For example, the combination of the finger performing the second gesture and the first finger may be a combination of different fingers of the same hand. A combination of different fingers of the same hand is, for example, a combination of the thumb and the index finger of the right hand C, or a combination of the index finger and the middle finger of the left hand.

[0123] The virtual key setting unit 254 positions the virtual key 31H in the virtual space K2 by positioning the virtual keyboard 3 in the virtual space K2. When the gesture detection unit 256 detects the second gesture in a situation where the virtual keyboard 3 is positioned in the virtual space K2, the virtual key control unit 257 moves the virtual key 31H to the target position M1. Therefore, the virtual key included in the virtual keyboard 3 can be moved to the target position M1.

[0124] When the gesture detection unit 256 detects the second gesture in a situation where the virtual keyboard 3 is positioned in the virtual space K2, the virtual key control unit 257 moves the virtual key 31H to the target position M1 by moving the virtual keyboard 3. Therefore, the entire virtual keyboard 3 can be moved in response to the second gesture.

[0125] B: Modification Example The modification modes in the above embodiment are shown below. Two or more modes arbitrarily selected from the following modification modes may be appropriately combined within a range where they do not conflict with each other.

[0126] B1: First Modification Example In the first embodiment, instead of moving the first virtual key (for example, virtual key 31H) to the target position M1 by moving the virtual keyboard 3, the virtual key control unit 257 may move the first virtual key to the target position M1 without moving a virtual key 31 different from the first virtual key among the plurality of virtual keys 31.

[0127] For example, when the gesture detection unit 256 detects a second gesture in a situation where the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 provides a second movement instruction to the virtual key setting unit 254. The second movement instruction is an instruction to move the first virtual key to the target position M1 without moving a virtual key 31 different from the first virtual key (for example, virtual key 31H) among the plurality of virtual keys 31. The virtual key setting unit 254 moves the first virtual key to the target position M1 in the virtual space K2 without moving a virtual key 31 different from the first virtual key among the plurality of virtual keys 31 in response to the second movement instruction.

[0128] According to the first modification example, when the gesture detection unit 256 detects a second gesture in a situation where the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the first virtual key to the target position M1 without moving a virtual key 31 different from the first virtual key among the plurality of virtual keys 31. Therefore, when user A desires to operate only the first virtual key, it is possible to avoid moving a virtual key 31 that does not need to be moved. Thus, the process of moving a virtual key 31 that does not need to be moved can be reduced.

[0129] In the first modification example, the moved first virtual key may overlap with another virtual key 31. If the first virtual key overlaps with another virtual key 31, there is a possibility that the operation detection unit 258 may detect an operation on the other virtual key 31 even though user A operates the first virtual key. Therefore, when the first virtual key overlaps with another virtual key 31, the operation detection unit 258 may invalidate an operation on the virtual key 31 that overlaps with the first virtual key. In this case, it is possible to prevent the operation detection unit 258 from detecting an operation on the other virtual key 31 even though user A operates the first virtual key.

[0130] B2: Second Modification Example In the first embodiment and the first modification example, the first virtual key is fixed. For example, the virtual key 31H is fixed as the first virtual key.

[0131] However, in the first embodiment and the first modification example, the first virtual key may be changed. For example, when the gesture detection unit 256 detects a second gesture in a situation where the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 may determine, as the first virtual key, the virtual key closest to the target position M1 among the plurality of virtual keys 31.

[0132] In this case, when the gesture detection unit 256 detects a second gesture in a situation where the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the virtual key closest to the target position M1 among the plurality of virtual keys 31 to the target position M1.

[0133] For example, when the gesture detection unit 256 detects a second gesture in a situation where the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 first acquires the latest virtual space data J5.

[0134] Subsequently, the virtual key control unit 257 determines, as the first virtual key, the virtual key closest to the target position M1 among the plurality of virtual keys 31 by referring to the latest virtual space data J5.

[0135] Subsequently, the virtual key control unit 257 moves the first virtual key to the target position M1. For example, the virtual key control unit 257 provides a third movement instruction to the virtual key setting unit 254, causing the virtual key setting unit 254 to move the first virtual key to the target position M1. The third movement instruction is an instruction to move the virtual key 31 determined as the first virtual key to the target position M1 by moving the virtual keyboard 3. The virtual key setting unit 254 moves the virtual keyboard 3 in the virtual space K2 in response to the third movement instruction to move the first virtual key to the target position M1.

[0136] The virtual key control unit 257 may also move the first virtual key to the target position M1 by providing a fourth movement instruction to the virtual key setting unit 254. The fourth movement instruction is an instruction to move the virtual key determined as the first virtual key to the target position M1 without moving the virtual key 31 that is different from the virtual key determined as the first virtual key among the plurality of virtual keys 31. The virtual key setting unit 254 moves the virtual key determined as the first virtual key to the target position M1 in the virtual space K2 in response to the fourth movement instruction without moving the virtual key 31 that is different from the virtual key determined as the first virtual key among the plurality of virtual keys 31.

[0137] According to the second modification example, the virtual key control unit 257 determines the virtual key closest to the target position M1 among the plurality of virtual keys 31 as the first virtual key. The target position M1 is a position corresponding to the fingertip position of the first finger of user A (for example, the middle finger of the right hand C of user A). Therefore, user A can change the first virtual key by changing the position of the first finger.

[0138] In addition, when the gesture detection unit 256 detects the second gesture in a situation where the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the virtual key closest to the target position M1 among the plurality of virtual keys 31 to the target position M1. Therefore, the user A can move the virtual key to be operated to a position where it can be easily operated by the index finger by performing the second gesture with the index finger close to the virtual key to be operated.

[0139] B3: Third Modification Example In the first embodiment, the first modification example, and the second modification example, the user A positions the index finger of the right hand C at the location of the virtual key 31 to be operated by moving, for example, the index finger of the right hand C.

[0140] However, the movement range of the index finger of the right hand C is limited by the environment such as in a train and the posture of the user A.

[0141] Therefore, in the first embodiment, the first modification example, and the second modification example, the position of the virtual keyboard 3 may be changed based on the movement of, for example, "the left hand of the user A" which is different from "the right hand C of the user A".

[0142] In this case, since the virtual keyboard 3 can be moved by the left hand of the user A, the movement distance of the right hand C of the user A can be shortened compared to a configuration where the virtual keyboard 3 cannot be moved by the left hand of the user A.

[0143] FIG. 14 is a diagram showing a case where the right hand C moves to the virtual key 31 indicating "A" without moving the virtual keyboard 3 and a case where the left hand moves the virtual keyboard 3 and the right hand C moves to the virtual key 31 indicating "A". In FIG. 14, the movement distance L1 is the movement distance when only the right hand C moves to the virtual key 31 indicating "A" without moving the virtual keyboard 3. The movement distance L2 is the movement distance when the left hand moves the virtual keyboard 3 and the right hand C moves to the virtual key 31 indicating "A". As shown in FIG. 14, the movement distance L2 is shorter than the movement distance L1.

[0144] The right hand C is an example of the first hand. The first hand is not limited to the right hand C, and any hand having a first finger may be used. The left hand of user A is an example of the second hand. The second hand is not limited to the left hand of user A, and any hand different from the first hand may be used.

[0145] FIG. 15 is a diagram showing an example of the portable device 2A according to the third modification. Hereinafter, the differences between the portable device 2A and the portable device 2 shown in FIG. 5 will be mainly described.

[0146] In addition to the components that the portable device 2 has, the portable device 2A includes a motion detection unit 259 and a position change unit 260. The storage device 24 of the portable device 2A stores a program PG3 instead of the program PG2. The processing device 25 of the portable device 2A reads the program PG3 from the storage device 24. By executing the program PG3, the processing device 25 of the portable device 2A functions as a hand data generation unit 251, a fingertip detection unit 252, a coordinate conversion unit 253, a virtual key setting unit 254, an image data generation unit 255, a gesture detection unit 256, a virtual key control unit 257, an operation detection unit 258, a motion detection unit 259, and a position change unit 260. At least one of the hand data generation unit 251, the fingertip detection unit 252, the coordinate conversion unit 253, the virtual key setting unit 254, the image data generation unit 255, the gesture detection unit 256, the virtual key control unit 257, the operation detection unit 258, the motion detection unit 259, and the position change unit 260 may be constituted by a circuit such as a DSP, an ASIC, and an FPGA.

[0147] The motion detection unit 259 detects the movement of the left hand of User A. For example, the motion detection unit 259 detects the movement of the left hand of User A based on the hand data J1. The motion detection unit 259 detects, as the movement of the left hand of User A, the movement of the fingers of the left hand of User A performing the second gesture. The fingers of the left hand of User A performing the second gesture are, for example, the index finger of the left hand of User A and the middle finger of the left hand of User A. The movement of the left hand of User A is not limited to the movement of the fingers of the left hand of User A performing the second gesture, and may be the fingers of the left hand of User A that are different from the fingers of the left hand of User A performing the second gesture. The motion detection unit 259 detects, as the movement of the left hand of User A, the moving direction of the fingers of the left hand of User A performing the second gesture and the moving distance of the fingers of the left hand of User A performing the second gesture. The motion detection unit 259 generates motion detection data J8 based on the detection result of the movement of the left hand of User A. The motion detection data J8 is data indicating the moving direction of the fingers of the left hand of User A performing the second gesture and the moving distance of the fingers of the left hand of User A performing the second gesture.

[0148] The motion detection unit 259 may detect the movement of the left hand of User A in a situation where the coordinate position of the left hand of User A in the world coordinate system CS2 is a position in contact with the virtual keyboard 3. In other words, in a situation where User A wearing the MR glasses 1 overlaps a part of the left hand of User A with the virtual keyboard 3 displayed on the MR glasses 1, the motion detection unit 259 may detect the movement of the left hand of User A. In this case, the motion detection unit 259 determines, for example, based on the virtual space data J5 and the fingertip data J3, whether or not the coordinate position of the left hand of User A in the world coordinate system CS2 is a position in contact with the virtual keyboard 3.

[0149] The position changing unit 260 changes the position of the virtual keyboard 3 based on the movement of the left hand of user A detected by the movement detection unit 259. For example, in the virtual space K2, the position changing unit 260 moves the virtual keyboard 3 by the moving distance indicated by the movement detection data J8 in the moving direction indicated by the movement detection data J8. Taking an example, the position changing unit 260 changes the position of the virtual keyboard 3 by providing a keyboard movement instruction to the virtual key setting unit 254. The keyboard movement instruction is an instruction to move the virtual keyboard 3 by the moving distance indicated by the movement detection data J8 in the moving direction indicated by the movement detection data J8 in the virtual space K2. The virtual key setting unit 254 moves the virtual keyboard 3 by the moving distance indicated by the movement detection data J8 in the moving direction indicated by the movement detection data J8 in the virtual space K2 according to the keyboard movement instruction.

[0150] The image data generation unit 255 may scroll and display the virtual keyboard 3 on the MR glass 1 according to the movement of the virtual keyboard 3 executed by the position changing unit 260. For example, the image data generation unit 255 generates imaging image data obtained by imaging the movement of the virtual keyboard 3 executed by the position changing unit 260 with the virtual camera N1. The image data generation unit 255 generates image data H1 representing the scroll of the virtual keyboard 3 by editing the imaging image data. The image data generation unit 255 transmits the image data H1 from the communication device 23 to the MR glass 1, thereby scrolling and displaying the virtual keyboard 3 on the MR glass 1. The MR glass 1 scrolls and displays the virtual keyboard 3 based on the image data H1. The image data generation unit 255 is an example of a first display control unit. The MR glass 1 is an example of a display device.

[0151] According to the third modification example, the virtual keyboard 3 can be moved by the left hand of user A. Therefore, compared with a configuration in which the virtual keyboard 3 cannot be moved by the left hand of user A, the moving distance of the right hand C of user A can be shortened.

[0152] The second hand for moving the virtual keyboard 3 is the hand (for example, the left hand) having the finger that performs the second gesture. Therefore, user A can be made to recognize the second hand (for example, the left hand) having the finger that performs the second gesture as the hand related to the movement of the virtual keyboard 3, and can be made to recognize the first hand (for example, the right hand) different from the second hand as the hand for operating the virtual key 31. Thus, it becomes difficult for user A to confuse the hand related to the movement of the virtual keyboard 3 with the hand for operating the virtual key 31.

[0153] The motion detection unit 259 detects, as the motion of the second hand, the motion of the finger of the left hand of user A who performs the second gesture. Therefore, user A can move the virtual keyboard 3 with the finger that performs the second gesture.

[0154] When the image data generation unit 255 scrolls and displays the virtual keyboard 3 on the MR glass 1 in accordance with the movement of the virtual keyboard 3 executed by the position change unit 260, user A can be notified of the movement of the virtual keyboard 3 by the scroll display.

[0155] B4: Fourth modification In the first embodiment and the first to third modifications, the image data generation unit 255 may enlarge and display, on the MR glass 1, the virtual key closest to the target position M1 among the plurality of virtual keys 31.

[0156] FIG. 16 is a diagram showing an example of enlarging and displaying the virtual key closest to the target position M1 among the plurality of virtual keys 31. In FIG. 16, the virtual key 31H is the virtual key closest to the target position M1.

[0157] For example, the image data generation unit 255 determines, as the proximity virtual key, the virtual key among the plurality of virtual keys 31 that is closest to the target position M1 by referring to the latest virtual space data J5. The image data generation unit 255 generates captured image data obtained by imaging the virtual space K2 with the virtual camera N1. The image data generation unit 255 generates image data H1 representing an image in which only the proximity virtual key is enlarged by editing the captured image data. The image data generation unit 255 transmits the image data H1 from the communication device 23 to the MR glass 1, causing the MR glass 1 to enlarge and display the proximity virtual key. The MR glass 1 enlarges and displays the proximity virtual key based on the image data H1. The image data generation unit 255 is an example of a second display control unit. The MR glass 1 is an example of a display device.

[0158] According to the fourth modification example, the image data generation unit 255 causes the MR glass 1 to enlarge and display the virtual key among the plurality of virtual keys 31 that is closest to the target position M1. The virtual key closest to the target position M1 is likely to be the virtual key 31 to be operated on. Therefore, the virtual key 31 to be operated on can be enlarged and displayed. Thus, compared to a configuration in which the virtual key 31 to be operated on is not enlarged and displayed, it is easier for the user A to recognize the virtual key 31 to be operated on.

[0159] B5: Fifth Modification Example In the first embodiment and the first to fourth modification examples, the gesture detection unit 256 may detect the second gesture based on data different from the hand data J1.

[0160] Assume that the operation button 15 of the MR glass 1 is located on the body 94 of the MR glass 1 to receive an operation by a finger of the user A's left hand. Further, assume that the second gesture is a gesture of pressing the operation button 15. In this case, the gesture detection unit 256 detects the second gesture based on the operation data F1 included in the comprehensive data G1.

[0161] According to the fifth modification example, the second gesture can be detected without using the hand data J1.

[0162] B6: Sixth Modification Example In the first embodiment, the first to fifth modification examples, the fingertip detection unit 252 may detect the fingertip position based on the output of the position detection device attached to the fingertip.

[0163] According to the sixth modification example, the fingertip position can be detected without using the hand data J1.

[0164] B7: Seventh Modification Example Each element realized by the processing device 25 of the mobile device 2 or 2A may be realized by the processing device 18 of the MR glass 1. In this case, the MR glass 1 is an example of an information processing device.

[0165] According to the seventh modification example, the mobile device 2 can be omitted.

[0166] C: Others (1) Each function illustrated in FIG. 4, FIG. 5, or FIG. 15 is realized by an arbitrary combination of hardware and software. The method of realizing each function is not particularly limited. Each function may be realized using one physically or logically combined device, or may be realized using a device configured by directly or indirectly (for example, using wired, wireless, etc.) connecting two or more physically or logically separated devices. Each function may be realized by combining software with the above one device or the above plurality of devices.

[0167] (2) In this specification, the term "device" may be read as other terms such as a circuit, a device, or a unit.

[0168] (3) In each of the first embodiment and the first to seventh modification examples, the storage device 17 and the storage device 24 may be constituted by at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Further, the program may be transmitted from a network via a telecommunication line.

[0169] (4) Each of the first embodiment and the first to seventh modification examples may be applied to at least one of a system using LTE (Long Term Evolution), LTE-A (LTA-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (x is, for example, an integer or a decimal), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), other appropriate systems, and a next-generation system extended, modified, created, and defined based on these. Further, a plurality of systems may be combined (e.g., a combination of at least one of LTE and LTE-A and 5G, etc.) and applied.

[0170] (5) In each of the first embodiment and the first to seventh modification examples, the processing procedures, sequences, flowcharts, etc. exemplified may be rearranged as long as there is no contradiction. For example, regarding the methods described in this specification, the elements of various steps are presented in an exemplary order and are not limited to the specific order presented.

[0171] (6) In each of the first embodiment and the first to seventh modification examples, the input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or appended. The output information, etc. may be deleted. The input information, etc. may be transmitted to other devices.

[0172] (7) In each of the first embodiment and the first to seventh modification examples, the determination may be made based on a value represented by 1 bit (0 or 1), may be made based on a truth value (Boolean: true or false), or may be made based on a numerical comparison (e.g., comparison with a predetermined value).

[0173] (8) In each of the first embodiment and the first to seventh modified examples, the program exemplified should be broadly interpreted to mean instructions, instruction sets, codes, code segments, program codes, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether called a software, firmware, middleware, microcode, or a hardware description language, or by any other name. Also, software, or instructions, etc. may be transmitted and received via a transmission medium. For example, when software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cables, optical fiber cables, twisted pairs, and digital subscriber lines (DSL)) and wireless technologies (such as infrared rays, microwaves, etc.), at least one of these wired and wireless technologies is included within the definition of the transmission medium.

[0174] (9) Data and the like described in each of the first embodiment and the first to seventh modified examples may be represented using any of a variety of different technologies. For example, data, information, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that the terms described in this specification and the terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0175] (10) In each of the first embodiment and the first to seventh modified examples, the terms "system" and "network" are used interchangeably.

[0176] (11) In each of the first embodiment, the first to seventh modification examples, at least one of the MR glasses 1, the mobile device 2, and the mobile device 2A may be a mobile station. A mobile station may be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable term.

[0177] (12) A mobile station may be referred to as a transmitting device, a receiving device, a communication device, or the like. A mobile station may be a device mounted on a moving body, or the moving body itself. A moving body means a movable object. The moving speed of the moving body is arbitrary. The moving body can be stopped. The moving body includes, for example, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavator trucks, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, limousines, rickshaws, ships (ship and other watercraft), airplanes, rockets, artificial satellites, drones (registered trademark), multicopters, quadcopters, balloons, and things mounted on these, and is not limited thereto. The moving body may be a moving body that autonomously travels based on an operation command. The moving body may be a vehicle (for example, a car, an airplane, etc.), a moving body that moves unmanned (for example, a drone, an autonomous driving vehicle, etc.), or a robot (humanoid or unmanned). A mobile station also includes a device that does not necessarily move during a communication operation. For example, a mobile station may be an IoT (Internet of Things) device such as a sensor.

[0178] (13) In each of the first embodiment and the first to seventh modification examples, the term "determining" may encompass a wide variety of operations. "Determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up / searching / inquiring (e.g., looking up in a table, database, or another data structure), and considering something ascertained as "determined". Also, "determining" may include considering something received (e.g., receiving information), transmitted (e.g., transmitting information), input, output, accessed (e.g., accessing data in memory) as "judged" or "determined". Further, "determining" may include considering something resolved, selected, chosen, established, compared, etc. as "determined". That is, "determining" may include considering that some operation has been "determined". Also, "determining" may be read as "assuming", "expecting", "considering", etc.

[0179] (14) In each of the first embodiment and the first to seventh modification examples, the term "connected", or any variation thereof, means any direct or indirect connection or coupling between two or more elements, and can include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements can be physical, logical, or a combination thereof. For example, "connected" may be read as "accessed". As used in this disclosure, two elements can be considered to be "connected" or "coupled" to each other using at least one of one or more electric wires, cables, and printed electrical connections, and, as some non-limiting and non-exhaustive examples, electromagnetic energy having wavelengths in the radio frequency region, microwave region, and optical (both visible and invisible) region, etc.

[0180] (15) In each of the first embodiment and the first to seventh modification examples, the description "based on" does not mean "only based on" unless otherwise specified. In other words, the description "based on" means both "only based on" and "at least based on".

[0181] (16) Any reference to an element using designations such as "first" and "second" used in this specification does not generally limit the quantity or order of those elements. These designations can be used in this specification as a convenient way to distinguish between two or more elements. Therefore, a reference to a first and a second element does not mean that only two elements can be employed or that the first element must precede the second element in any form.

[0182] (17) In each of the first embodiment, the first to seventh modified examples, when terms such as "include", "including" and their modifications are used in this specification or the claims, these terms are intended to be inclusive, similar to the term "comprising". Further, the term "or" used in this specification or the claims is intended not to be an exclusive disjunction.

[0183] (18) Throughout this application, when articles are added by translation, such as a, an, and the in English, the present disclosure may include that the nouns following these articles are in the plural form.

[0184] (19) It is obvious to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and changed forms without departing from the spirit and scope of the present invention determined based on the description of the claims. Therefore, the description in this specification is for illustrative purposes and has no restrictive meaning for the present invention. Also, a plurality of aspects selected from the aspects exemplified in this specification may be combined.

[0185] D: Aspects grasped from the above-described embodiments or modified examples The following aspects are grasped from at least one of the above-described embodiments or modified examples.

[0186] D1: The first aspect The information processing apparatus according to the first aspect includes a virtual key setting unit, a gesture detection unit, a fingertip detection unit, and a virtual key control unit. The virtual key setting unit positions the first virtual key in the virtual space. The gesture detection unit detects a gesture performed by at least one finger. The fingertip detection unit detects the fingertip position of a first finger different from the at least one finger. When the gesture detection unit detects the gesture in a situation where the first virtual key is positioned in the virtual space, the virtual key control unit moves the first virtual key to a target position on the virtual space corresponding to the fingertip position of the first finger detected by the fingertip detection unit.

[0187] According to this aspect, it is possible to reduce the setting of the virtual key at a position different from the correct position.

[0188] D2: Second aspect In an example of the first aspect (second aspect), the first virtual key is any one of a plurality of virtual keys included in a virtual keyboard. The virtual key setting unit positions the first virtual key in the virtual space by positioning the virtual keyboard in the virtual space. When the gesture detection unit detects the gesture in a situation where the virtual keyboard is positioned in the virtual space, the virtual key control unit moves the first virtual key to the target position. According to this aspect, it is possible to move the virtual key included in the virtual keyboard to the target position.

[0189] D3: Third aspect In an example of the second aspect (third aspect), when the gesture detection unit detects the gesture in a situation where the virtual keyboard is positioned in the virtual space, the virtual key control unit moves the first virtual key to the target position without moving a virtual key different from the first virtual key among the plurality of virtual keys. According to this aspect, for example, when the user desires to operate only the first virtual key, it is possible to avoid moving a virtual key that does not need to be moved. Therefore, it is possible to reduce the process of moving a virtual key that does not need to be moved.

[0190] D4: Fourth aspect In an example of the second aspect (the fourth aspect), when the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space, the virtual key control unit moves the first virtual key to the target position by moving the virtual keyboard. According to this aspect, the entire virtual keyboard can be moved according to the gesture.

[0191] D5: The fifth aspect In an example of any one of the second aspect to the fourth aspect (the fifth aspect), when the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space, the virtual key control unit moves the virtual key closest to the target position among the plurality of virtual keys to the target position as the first virtual key. According to this aspect, the user can move the virtual key to be operated to a position where it is easy to operate with the first finger by making a gesture with the first finger close to the virtual key to be operated.

[0192] D6: The sixth aspect In an example of any one of the second aspect to the fifth aspect (the sixth aspect), it further includes a motion detection unit that detects the motion of a second hand different from the first hand having the first finger, and a position change unit that changes the position of the virtual keyboard based on the motion of the second hand detected by the motion detection unit. According to this aspect, the virtual keyboard can be moved by the second hand. Therefore, compared with a configuration in which the virtual keyboard cannot be moved by the second hand, the moving distance of the first hand required to operate the virtual key to be operated can be shortened.

[0193] D7: The seventh aspect In an example of the sixth aspect (the seventh aspect), it further includes a first display control unit that scrolls and displays the virtual keyboard on a display device in response to the change in the position of the virtual keyboard executed by the position change unit. According to this aspect, the user can be notified of the movement of the virtual keyboard by the scroll display.

[0194] D8: The eighth aspect In an example of the sixth or seventh aspect (eighth aspect), the second hand is the hand having the at least one finger. According to this aspect, the user can be made to recognize the second hand having the finger that makes a gesture as the hand related to the movement of the virtual keyboard, and the first hand different from the second hand as the hand that operates the virtual key. Therefore, it becomes difficult for the user to confuse the hand related to the movement of the virtual keyboard with the hand that operates the virtual key.

[0195] D9: Ninth aspect In an example of the eighth aspect (ninth aspect), the motion detection unit detects the movement of the at least one finger as the movement of the second hand. According to this aspect, the user can move the virtual keyboard with the finger that makes a gesture.

[0196] D10: Tenth aspect In an example of any one of the second to ninth aspects (tenth aspect), the display device further includes a second display control unit that enlarges and displays the virtual key closest to the target position among the plurality of virtual keys. The virtual key closest to the target position is likely to be the virtual key to be operated. Therefore, according to this aspect, the virtual key to be operated can be enlarged and displayed. Thus, compared with a configuration in which the virtual key to be operated is not enlarged and displayed, it is easier for the user to recognize the virtual key to be operated.

Explanation of reference numerals

[0197] 1…MR Glass, 2…Portable Device, 2A…Portable Device, 3…Virtual Keyboard, 4…Image, 11L…Camera, 11R…Camera, 12…Display Device, 13…Acceleration Sensor, 14…Gyro Sensor, 15…Operation Button, 16…Communication Device, 17…Memory Device, 18…Processing Device, 19…Bus, 21…Input Device, 22…Display Device, 23…Communication Device, 24…Memory Device, 25…Processing Device, 26…Bus, 31…Virtual Key, 91…Temple, 93…Bridge, 94…Torso, 95…Torso, 96L…Lens, 96R…Lens, 100…Information Processing System, 181…Acquisition Unit, 182…Operation Control Unit, 251…Hand Data Generation Unit, 252…Finger Tip Detection Unit, 253…Coordinate Conversion Unit, 254…Virtual Key Setting Unit, 255…Image Data Generation Unit, 256…Gesture Detection Unit, 257…Virtual Key Control Unit, 258…Operation Detection Unit, 259…Motion Detection Unit, 260…Position Change Unit.

Claims

1. A virtual key setting unit that positions a first virtual key in a virtual space; A gesture detection unit that detects a gesture performed by at least one finger; A fingertip detection unit that detects the fingertip position of a first finger different from the at least one finger; When the gesture detection unit detects the gesture in a situation where the first virtual key is positioned in the virtual space, a virtual key control unit that moves the first virtual key to a target position on the virtual space corresponding to the fingertip position of the first finger detected by the fingertip detection unit; comprising The first virtual key is any one of a plurality of virtual keys included in a virtual keyboard; The virtual key setting unit positions the first virtual key in the virtual space by positioning the virtual keyboard in the virtual space; When the gesture detection unit detects the gesture in a situation where the virtual keyboard is positioned in the virtual space, the virtual key control unit moves the first virtual key to the target position without moving a virtual key different from the first virtual key among the plurality of virtual keys; An information processing apparatus.

2. A virtual key setting unit that positions a first virtual key in a virtual space; A gesture detection unit that detects a gesture performed by at least one finger; A fingertip detection unit that detects the fingertip position of a first finger different from the at least one finger; When the gesture detection unit detects the gesture in a situation where the first virtual key is positioned in the virtual space, a virtual key control unit that moves the first virtual key to a target position on the virtual space corresponding to the fingertip position of the first finger detected by the fingertip detection unit; comprising The first virtual key is any one of a plurality of virtual keys included in a virtual keyboard; The virtual key setting unit positions the first virtual key in the virtual space by positioning the virtual keyboard in the virtual space; When the gesture detection unit detects the gesture in a situation where the virtual keyboard is positioned in the virtual space, the virtual key control unit moves, as the first virtual key, the virtual key closest to the target position among the plurality of virtual keys to the target position; An information processing apparatus.

3. When the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space, the virtual key control unit moves the first virtual key to the target position by moving the virtual keyboard. The information processing apparatus according to claim 2.

4. When the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space, the virtual key control unit moves, as the first virtual key, the virtual key closest to the target position among the plurality of virtual keys to the target position. The information processing apparatus according to claim 1.

5. A motion detection unit that detects a motion of a second hand different from the first hand having the first finger; A position changing unit that changes the position of the virtual keyboard based on the motion of the second hand detected by the motion detection unit, further comprising: The information processing apparatus according to any one of claims 1 to 4.

6. A first display control unit that scrolls and displays the virtual keyboard on the display device in response to the change in the position of the virtual keyboard executed by the position changing unit, further comprising: The information processing apparatus according to claim 5.

7. The second hand is a hand having the at least one finger. The information processing apparatus according to claim 5 or 6.

8. The motion detection unit detects the motion of the at least one finger as the motion of the second hand. The information processing apparatus according to claim 7.

9. A second display control unit that enlarges and displays, on the display device, the virtual key closest to the target position among the plurality of virtual keys, further comprising: The information processing apparatus according to any one of claims 1 to 8.

Images