Information processing device, information processing method, and program

Abstract

The present technology relates to an information processing device, an information processing method, and a program with which it is possible to perform input operations more suitably by using a virtual UI. An information processing device according to the present technology comprises: a tracking unit that tracks a portion of the body of a user; and a control unit that, on the basis of the result of tracking the portion, controls the position and orientation within a three-dimensional space of a virtual UI for receiving an input operation in at least a section of the tracked portion. The present technology can be applied, for example, to an HMD that receives an input operation performed by a user via a virtual keyboard.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable more suitable input operations to be performed using a virtual UI.

[0002] In recent years, head-mounted displays (HMDs) equipped with highly accurate finger recognition functions have appeared, and technologies for inputting characters and the like using virtual user interfaces (UIs) such as virtual keyboards have been actively developed. For example, Patent Literature 1 (JP-A-2005-102666) describes a technology for determining the position of a virtual keyboard based on the initial positions of at least three fingertips of a user.

[0003] International Publication No. 2020 / 039703

[0004] When performing input operations using a real keyboard, the user uses, for example, tactile stimuli from the keys as clues to grasp the relative positions of the fingers and the keyboard and finely adjust the positions of the fingers.When performing input operations using a virtual keyboard, the user cannot obtain tactile stimuli as clues, making it difficult for the user to grasp the size of the keyboard and the spacing between the keys, and making it difficult to finely adjust the positions of the fingers.

[0005] Therefore, it has been difficult to perform input operations using a virtual keyboard with the same accuracy or speed as when using a real keyboard.

[0006] The present technology has been developed in light of these circumstances, and enables input operations to be performed more conveniently using a virtual UI.

[0007] An information processing device according to one aspect of the present technology includes a tracking unit that tracks a part of a user's body, and a control unit that controls the position and orientation in three-dimensional space of a virtual UI that accepts input operations on at least a part of the tracked part based on the tracking results of the part.

[0008] An information processing method according to one aspect of the present technology includes an information processing device that tracks a part of a user's body and controls the position and orientation in three-dimensional space of a virtual UI that accepts input operations on at least a part of the tracked part based on the tracking results of the part.

[0009] A program according to one aspect of the present technology causes a computer to execute a process of tracking a part of a user's body and controlling the position and orientation in three-dimensional space of a virtual UI for accepting input operations on at least a part of the tracked part based on the tracking results of the part.

[0010] In one aspect of the present technology, a part of a user's body is tracked, and the position and orientation in three-dimensional space of a virtual UI for accepting input operations on at least a portion of the tracked part is controlled based on the tracking result of the part.

[0011] 1 is a diagram illustrating an HMD according to an embodiment of the present technology; FIG. 1 is a diagram illustrating an example of an input posture; FIG. 2 is a diagram illustrating an example of a virtual keyboard; FIG. 3 is a diagram illustrating an example of a keyboard surface; FIG. 4 is a diagram for explaining an offset; FIG. 5 is a diagram illustrating an example of a second reference plane; FIG. 6 is a diagram illustrating examples of a first reference plane, a second reference plane, and a keyboard surface during input; FIG. 7 is a block diagram illustrating an example of the functional configuration of an HMD; FIG. 8 is a diagram illustrating an example of a finger angle; FIG. 9 is a flowchart illustrating processing performed by an information processing unit; FIG. 10 is a flowchart illustrating virtual keyboard control processing; FIG. 11 is a diagram illustrating an example of a keyboard surface calculated based on an input degree; FIG. 12 is a diagram illustrating an example of weighted addition of the position and posture of the keyboard surface during standby and the position and posture of the keyboard surface during input; FIG. 13 is a flowchart illustrating virtual keyboard control processing; FIG. 14 is a block diagram illustrating an example of the configuration of hardware of a computer.

[0012] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Overview of the embodiment 2. Configuration and operation of the HMD 3. Modification

[0013] 1. Overview of the embodiment In recent years, HMDs equipped with highly accurate finger recognition functions have appeared, and there has been active development of technologies for inputting characters, etc. using a virtual keyboard. For example, Patent Literature 1 describes a technology for determining the position of a virtual keyboard based on the fingertip positions of at least three fingers of a user in an initial state.

[0014] When performing input operations using a real keyboard, the user grasps the positional relationship between the keyboard and their fingers and finely adjusts the position of their fingers using tactile stimuli such as the sensation of touching the surface or edge of the keys, the sensation of hitting the bottom of the keys, unevenness that serves as a clue to the home position, and support by the wrist.When performing input operations using a virtual keyboard, since the user cannot obtain tactile stimuli that serve as clues, it is difficult for the user to grasp the size of the keyboard and the spacing between the keys, making it difficult to finely adjust the position of their fingers.

[0015] Therefore, it has been difficult to perform input operations using a virtual keyboard with the same accuracy or speed as when using a real keyboard.

[0016] In one embodiment of the present technology, which has been conceived with a focus on the above points, a technology is proposed that enables more convenient input operations using a virtual keyboard. The present embodiment will be described in detail below.

[0017] FIG. 1 is a diagram showing an HMD 1 according to an embodiment of the present technology.

[0018] As shown in FIG. 1, a user U1 wears an HMD 1 on his / her head.

[0019] The HMD 1 is configured as a display device equipped with, for example, a camera and a non-transparent display. When wearing the HMD 1, the user U1 cannot see the outside world because the display and the housing block the user's view, but can see the image displayed on the display.

[0020] The display of the HMD 1 displays games, movies, web content, emails, etc. To enable the user U1 to input characters and other operations, the HMD 1 accepts input operations from the user U1 via a virtual keyboard. The virtual keyboard does not exist in real space but is virtually generated, for example, a keyboard with multiple keys arranged in a QWERTY layout. In this technology, when the user moves their hands, the virtual keyboard moves to follow the user's hands.

[0021] The HMD 1 uses a camera to track each part of the user U1's body and calculates the position and orientation of the virtual keyboard in the virtual space based on the position (orientation) of each part. The HMD 1 can identify the key pressed by the user U1 based on the positions of the user U1's fingers and the position of the virtual keyboard.

[0022] The flow of calculation of the position and orientation of the virtual keyboard by the HMD 1 will be described with reference to FIGS.

[0023] As shown in Fig. 2, when the user U1 is in an input posture, the HMD 1 displays a virtual keyboard on the display. The input posture indicates an intention to perform an input operation. In Fig. 2, the input posture is a posture in which both hands are extended forward as if resting fingers on the keyboard.

[0024] Note that when the user U1 performs a gesture other than the gesture of extending both hands forward, the posture of the user U1 may be determined to be the input posture. A captured image of a body part of the user U1 may be input to a learning model generated by machine learning or the like, and it may be determined whether the posture of the user U1 is the input posture.

[0025] FIG. 3 is a diagram showing an example of a virtual keyboard.

[0026] When user U1 extends both hands in front of him / her, as shown in Figure 3, for example, a virtual keyboard KL for the left hand is displayed in a position near user U1's left hand HL, and a virtual keyboard KR for the right hand is displayed in a position near user U1's right hand HR. The virtual keyboard KL is operated with the fingertips of each finger that is part of the left hand HL, and the virtual keyboard KR is operated with the fingertips of each finger that is part of the right hand HR. In addition, dots indicating the positions of the fingertips, bases, and wrists of each finger on the left hand HL and right hand HR are displayed. Note that it is not necessary to display the dots indicating the positions of the fingertips, bases, and wrists of each finger or the virtual keyboards KL and KR.

[0027] In FIG. 3, the left hand HL and the right hand HR are shown with dotted lines, which indicates that the left hand HL and the right hand HR are not displayed on the display of the HMD 1.

[0028] When performing input operations using a real integrated keyboard, the user grasps the relative positions of their left and right hands by resting their wrists on the keyboard or table to support their entire hands, or by receiving tactile stimuli when placing their fingertips on the keys. When performing input operations using a virtual keyboard, it is not possible to obtain clues such as support by the wrists or tactile stimuli on the fingertips, so it is preferable to display a split keyboard as the virtual keyboard, which does not require the user to grasp the relative positions of their left and right hands.

[0029] The input posture includes an input standby state in which the user is not moving his / her finger and is not performing an input operation, and an input state in which the user is moving his / her finger and performing an input operation. Hereinafter, the state when the user is in the input standby state will be referred to as "standby time," and the state when the user is in the input state will be referred to as "input time."

[0030] In standby mode, the HMD 1 defines a keyboard surface that indicates the area where the virtual keyboard is located.

[0031] FIG. 4 is a diagram showing an example of a keyboard surface.

[0032] As shown in Fig. 4, a keyboard surface KP1L for the left hand is defined based on a first reference plane PL1L including a point P1L indicating the position of the wrist of the left hand, a point P2L indicating the position of the tip of the index finger, and a point P3L indicating the position of the tip of the little finger. Specifically, the keyboard surface KP1L is defined so that the first reference plane PL1L and the keyboard surface KP1L have a predetermined positional relationship. For example, a certain range of area on a plane obtained by moving the first reference plane PL1L by a predetermined distance and tilting it by a predetermined angle is defined as the keyboard surface KP1L. The keyboard surface KP1L may be defined as an area on the same plane as the first reference plane PL1L.

[0033] Note that a plane including points indicating the positions of parts other than the wrist, the tip of the index finger, and the tip of the little finger may be set as the first reference plane. When performing input operations using a real keyboard, it is considered that at least the wrist, the tip of the index finger, and the tip of the little finger are stably placed on the keyboard or table during standby, so it is preferable to set as the first reference plane a plane including three points indicating the positions of the wrist, the tip of the index finger, and the tip of the little finger.

[0034] On the keyboard surface KP1L, for example, the "F" key K2 is located at the position of the index fingertip (point P2L), and the "A" key K1 is located at the position of the little fingertip (point P3L). By defining the positions of the keys on the virtual keyboard KL based on the positions of the index finger and little fingertips during standby, it is possible to display a virtual keyboard KL that fits the size and shape of the user's hand. Furthermore, by enlarging or reducing the size of the keyboard surface KP1L (the size of the virtual keyboard KL) or by deforming the keyboard surface KP1L (virtual keyboard KL) to match the width between the index finger and little finger during standby, it is possible to display a virtual keyboard KL that fits the size and shape of the user's hand.

[0035] As with the left hand, a keyboard surface KP1R for the right hand is defined based on a first reference plane PL1R that includes a point P1R indicating the position of the wrist of the right hand, a point P2R indicating the position of the tip of the index finger, and a point P3R indicating the position of the tip of the little finger. On the keyboard surface KP1R, for example, a "J" key K3 is located at the position of the tip of the index finger (point P2R), and for example, a ";" key K4 is located at the position of the tip of the little finger (point P3R).

[0036] As described above, during standby, the position and orientation of the keyboard surface is controlled based on the first reference plane. Specifically, the position and orientation of the keyboard surface is controlled so as to follow the first reference plane.

[0037] After defining the keyboard surface, the HMD 1 records an offset indicating the position and orientation relationship between the keyboard surface KP1L and a second reference plane PL2L, which is a plane corresponding to the back of the left hand (palm), as indicated by the white arrow in Fig. 5. The position and orientation relationship between the second reference plane PL2L and the keyboard surface KP1L is expressed, for example, by the relative position (distance) and relative orientation (angle) of the keyboard surface KP1L with respect to the second reference plane PL2L. In Fig. 5, the keyboard surface KP1L is defined as an area on the same plane as the first reference plane PL1L.

[0038] The position and orientation relationship indicated by the offset includes the position and orientation relationship between the second reference plane PL2L and the first reference plane PL1L, and the position and orientation relationship between the first reference plane PL1L and the keyboard surface KP1L. The offset also includes information indicating the size of the keyboard surface KP1L and information indicating the positions of the keys arranged on the keyboard surface KP1L.

[0039] Although not shown, an offset indicating the position and orientation relationship between a second reference plane PL2R, which is a plane corresponding to the back of the right hand (palm of the right hand), and the keyboard surface KP1R is also recorded. Even when the virtual keyboard is not displayed, the keyboard surface is defined and the offset is recorded.

[0040] FIG. 6 is a diagram showing an example of the second reference plane.

[0041] 6, the second reference plane PL2L corresponding to the back of the left hand is a plane that includes, for example, a point P11L indicating the position of the wrist of the left hand, a point P12L indicating the position of the base of the index finger, and a point P13L indicating the position of the base of the little finger, while the second reference plane PL2R corresponding to the back of the right hand is a plane that includes, for example, a point P11R indicating the position of the wrist of the right hand, a point P12R indicating the position of the base of the index finger, and a point P13R indicating the position of the base of the little finger.

[0042] As described above, during standby, the position and orientation of the keyboard surface is controlled based on the first reference plane, but during input, the position and orientation of the keyboard surface is controlled based on the second reference plane, not the first reference plane. Specifically, the position and orientation of the keyboard surface is controlled to follow the second reference plane.

[0043] FIG. 7 is a diagram showing an example of the first reference plane, the second reference plane, and the keyboard surface during input.

[0044] At the time of input, the HMD 1 calculates the keyboard surface KP1L so as to reproduce the position and orientation relationship between the second reference plane PL2L indicated by the offset recorded during standby and the keyboard surface KP1L, as shown by the outline arrow in Fig. 7. Specifically, the HMD 1 moves the second reference plane PL2L by the distance indicated by the offset recorded during standby, and calculates an area of ​​a predetermined range on the plane tilted by the angle indicated by the offset as the keyboard surface KP1L.

[0045] When inputting, the fingertips of the fingers move vigorously, causing the first reference plane PL1L to change vigorously. If the keyboard surface is controlled to follow the first reference plane PL1L, the keyboard surface will also change vigorously. Because the keyboard surface changes vigorously, it is difficult for the user to accurately press the desired key.

[0046] In the HMD 1 of the present technology, the position and orientation of the keyboard surface are controlled so as to follow the second reference plane PL2L corresponding to the back of the hand, which moves little even during input, making it possible to stabilize the keyboard surface. Therefore, even if the position of the fingers changes during input, the user can accurately press the desired keys, and the user can continue to perform input operations even if the position of the hand changes during input.

[0047] 2. Configuration and Operation of HMD Fig. 8 is a block diagram showing an example of the functional configuration of the HMD 1. The information processing unit 11 in Fig. 8 is realized, for example, by a CPU or the like provided in the HMD 1 executing a predetermined program.

[0048] As shown in FIG. 8, the information processing unit 11 includes a body position acquisition unit 21 , a state estimation unit 22 , a keyboard surface definition unit 23 , a recording unit 24 , a keyboard position calculation unit 25 , a drawing unit 26 , and an input acquisition unit 27 .

[0049] The body position acquisition unit 21 functions as a tracking unit that tracks parts of the user's body based on images captured by a camera provided in the HMD 1, and acquires the position and orientation of each part as a tracking result. The position and orientation of each part of the body are acquired as parameters of, for example, six degrees of freedom (6DoF). The body position acquisition unit 21 supplies body information indicating the position and orientation of each part of the body to the state estimation unit 22, the keyboard surface definition unit 23, the keyboard position calculation unit 25, and the input acquisition unit 27.

[0050] The state estimation unit 22 estimates the state of the user based on the physical information supplied from the body position acquisition unit 21. Specifically, the state estimation unit 22 estimates whether the user's posture is an input posture, whether the user is in an input state or an input standby state, and the like.

[0051] For example, the state estimation unit 22 estimates an input degree, which indicates numerically whether the user is in an input state or an input standby state, based on physical information. An input degree of 0 indicates that the user is in an input standby state, and an input degree of 1 indicates that the user is in an input state. The input degree is estimated based on the angular velocity and angular acceleration of the finger angles, the relative position and relative velocity of the fingertips of each finger with respect to the palm, etc.

[0052] FIG. 9 is a diagram showing an example of finger angles.

[0053] As shown in FIG. 9, for example, the angle of a line connecting point P21 indicating the position of the tip of the index finger and point P22 indicating the position of the base of the index finger, and the angle of a line connecting point P22 and point P23 indicating the position of the wrist, are defined as the finger angle of the index finger.

[0054] For example, when the finger angles of the index finger, middle finger, ring finger, and little finger are substantially the same, the state estimation unit 22 estimates the value of the input degree as 0 (input standby state).

[0055] The state estimation unit 22 can also estimate the user's state by inputting physical information into a learning model generated by machine learning or the like. The user can also indicate whether they are in an input state or an input standby state by making a gesture. For example, if the user moves their fingers vigorously or holds their hand out in front of them, the state estimation unit 22 estimates that the user is in an input state. Furthermore, for example, if the user's posture is such that their hand is held out in front of them vertically, a so-called forward-following posture, the state estimation unit 22 estimates that the user is in a standby state, and if the user holds their palm out in that posture, the state estimation unit 22 estimates that the user is in an input state.

[0056] The state estimation unit 22 controls the keyboard surface definition unit 23 and the keyboard position calculation unit 25 based on the estimation result of the user's state. For example, when the user is in an input standby state, the state estimation unit 22 causes the keyboard surface definition unit 23 to define a keyboard surface. Furthermore, when the user is in an input state, the state estimation unit 22 causes the keyboard position calculation unit 25 to calculate the position and orientation of the keyboard surface.

[0057] The keyboard surface definition unit 23 defines a keyboard surface based on the body information supplied from the body position acquisition unit 21, and causes the offset to be recorded in the recording unit 24.

[0058] The recording unit 24 records the offset supplied from the keyboard surface definition unit 23 .

[0059] The keyboard position calculation unit 25 acquires the offset from the recording unit 24, and calculates the position and orientation of the keyboard surface based on the offset and the body information supplied from the body position acquisition unit 21. The position and orientation of the keyboard surface is calculated as parameters with six degrees of freedom, for example.

[0060] The keyboard position calculation unit 25 generates keyboard data by arranging a keyboard surface of the size indicated by the offset at the calculated position and orientation, and arranging various keys on the keyboard surface at the key positions indicated by the offsets. The keyboard position calculation unit 25 supplies the keyboard data to the drawing unit 26 and the input acquisition unit 27. The keyboard surface definition unit 23 and the keyboard position calculation unit 25 function as a control unit that controls the position and orientation of the virtual keyboard.

[0061] The drawing unit 26 draws an image so that the virtual keyboard appears to be located at the position on the keyboard surface calculated by the keyboard position calculation unit 25, and displays the image on the display provided in the HMD 1. The drawing unit 26 may change the drawing method of the virtual keyboard based on the content of the input operation identified by the input acquisition unit 27. For example, the drawing unit 26 may draw keys pressed by a finger in a different color, or draw the pressed keys by sinking them into the bottom surface of the virtual keyboard.

[0062] It is not necessary to draw and display the virtual keyboard during both standby and input. Displaying the virtual keyboard has a higher priority during input than during standby.

[0063] The input acquisition unit 27 identifies the content of the input operation performed by the user based on the physical information supplied from the body position acquisition unit 21 and the keyboard data supplied from the keyboard position calculation unit 25. For example, when the position of a key on the virtual keyboard matches the position of the user's fingertip, the input acquisition unit 27 identifies that key as having been pressed.

[0064] Next, the processing performed by the information processing unit 11 having the above configuration will be described with reference to the flowchart of FIG.

[0065] In step S1, the body position acquisition unit 21 tracks the user's body parts, such as the hands, based on the captured image taken by the camera provided in the HMD 1, and acquires the position and orientation of the hands.

[0066] In step S2, the state estimation unit 22 determines whether the user's posture is an input posture based on the position and posture of the user's hand.

[0067] If it is determined in step S2 that the user's posture is not the input posture, the process returns to step S1, and hand tracking is repeated, for example, frame by frame, until the user's posture becomes the input posture.

[0068] On the other hand, if it is determined in step S2 that the user's posture is an input posture, in step S3, the information processing unit 11 performs virtual keyboard control processing. The virtual keyboard control processing calculates the position and posture of the keyboard surface and generates keyboard data. Details of the virtual keyboard control processing will be described later with reference to FIG. 11.

[0069] After the virtual keyboard control process is performed in step S3, the process returns to step S1, and the subsequent processes are repeated.

[0070] Next, the virtual keyboard control process performed in step S3 of FIG. 10 will be described with reference to the flowchart of FIG.

[0071] In step S11, the state estimation unit 22 determines whether the user is in an input standby state.

[0072] If it is determined in step S11 that the user is in an input standby state, in step S12, the keyboard surface definition unit 23 defines a keyboard surface based on the position and posture of the user's hands and records the offset. Thereafter, the keyboard position calculation unit 25 generates keyboard data by arranging various keys on the keyboard surface defined by the keyboard surface definition unit 23.

[0073] On the other hand, if it is determined in step S11 that the user is not in an input standby state, i.e., is in an input state, in step S13, the keyboard position calculation unit 25 calculates the position of the keyboard surface based on the offset recorded during standby and the position and orientation of the user's hands. Thereafter, the keyboard position calculation unit 25 arranges the keyboard surface according to the calculated positions and orientations, and generates keyboard data by arranging various keys on the keyboard surface.

[0074] After the keyboard data is generated in step S12 or step S13, the process returns to step S3 in FIG. 10, and the subsequent processes are carried out.

[0075] As described above, in the HMD 1 of the present technology, the hands of the user as body parts are tracked, and the position and orientation in three-dimensional space of the virtual keyboard for receiving input operations with the hands and fingers is controlled based on the results of hand tracking. Because the virtual keyboard moves to follow the hands, the user can perform input operations using the virtual keyboard with a feeling similar to that of using a real keyboard, without having to worry about the home position or finely correct the position of their fingers.

[0076] 3. Modifications Example of Calculating Keyboard Surface Position and Orientation Based on Input Degree The final keyboard surface position and orientation may be calculated by weighting and adding the keyboard surface position and orientation during standby and the keyboard surface position and orientation during input according to the input degree.

[0077] FIG. 12 is a diagram showing an example of the position and orientation of the keyboard surface calculated based on the input degree.

[0078] As shown on the left side of Fig. 12, when the input degree value is 0, the user is in an input standby state, and therefore a predetermined range of area on the same plane as the first reference plane PL1L is calculated as the keyboard surface KP1L. On the other hand, as shown on the right side of Fig. 12, when the input degree value is 1, the user is in an input state, and therefore a predetermined range on a plane different from the first reference plane PL1L, which is obtained by moving or tilting the second reference plane PL2L, is calculated as the keyboard surface KP1L.

[0079] For example, if the input degree value is 0.5, a surface obtained by blending the standby keyboard surface KP1L and the input keyboard surface KP1L is calculated as the final keyboard surface. Note that a keyboard surface obtained by blending the standby keyboard surface and the input keyboard surface may result in an indeterminate surface whose position and orientation are unstable, even though it does not partially follow the movement of the hand. Therefore, the input degree may be set to be estimated discretely between values ​​0 and 1, or, for example, the input degree may be set to be 1 even when the fingers move only slightly.

[0080] FIG. 13 is a diagram showing an example of weighted addition of the position and orientation of the keyboard surface during standby and the position and orientation of the keyboard surface during input.

[0081] First, as shown in #1 of Fig. 13, the position and orientation of the keyboard surface at the time of input is multiplied by the input degree as a weight, and then, as shown in #2 of Fig. 13, the position and orientation of the keyboard surface at the time of standby is multiplied by (1 - input degree) as a weight. Next, as shown in #3 of Fig. 13, the weighted position and orientation of the keyboard surface at the time of input and the position and orientation of the keyboard surface at the time of standby are added together to calculate the final position and orientation of the keyboard surface.

[0082] A virtual keyboard control process when calculating the keyboard position based on the input degree will be described with reference to the flowchart of FIG.

[0083] In step S21, the information processing unit 11 calculates the position and orientation of the keyboard surface based on the input degree. Specifically, the keyboard surface definition unit 23 defines the keyboard surface in standby mode in advance based on the positions of the user's fingers. Next, the keyboard position calculation unit 25 calculates the position and orientation of the keyboard surface during input based on the position and orientation of the user's hands. Finally, as described above, the keyboard position calculation unit 25 calculates the final position and orientation of the keyboard surface by weighting and adding the position and orientation of the keyboard surface during standby mode and the position and orientation of the keyboard surface during input according to the input degree.

[0084] The position and orientation of the keyboard surface during input here refers to the position and orientation calculated based on the current second reference plane, regardless of whether the user is in an input standby state or an input state.

[0085] In step S22, the keyboard position calculation unit 25 applies a stabilization filter, such as a low-pass filter, based on the amount of change in the position and orientation over time to the calculated position and orientation of the keyboard surface, to calculate the position and orientation to be reflected in the keyboard data.

[0086] In step S23, the drawing unit 26 draws an image so that the virtual keyboard appears to exist on the keyboard surface to which the stabilization filter has been applied, and displays the image on the display provided in the HMD 1.

[0087] If the position of the keyboard surface is sensitively controlled in response to finger movements or input degree, the keyboard surface may move violently, so it is preferable to apply a stabilization filter to stabilize the keyboard surface and then draw the virtual keyboard, as in the processing of steps S22 and S23. Note that in step S21, after the positions and orientations of the keyboard surface during standby and input are calculated (defined), a stabilization filter may be applied to the positions and orientations of each keyboard surface.

[0088] After the image is displayed on the display in step S23, the process returns to step S3 in FIG. 10, and the subsequent processes are carried out.

[0089] As described above, it is also possible to weight and add the position and orientation of the keyboard surface during standby and the position and orientation of the keyboard surface during input according to the input degree.

[0090] ・Application examples This technology can be applied not only to HMDs where the user cannot see the outside world, but also to HMDs with VST (Video See Through) functionality and AR (Augmented Reality) glasses.

[0091] - Example of a method for tracking hands and fingers Instead of tracking the user's hands and fingers using a camera installed in the HMD 1, the HMD 1 can also track the user's hands and fingers using another camera installed in a position where it can track the user's hands and fingers.

[0092] The HMD 1 can also estimate the position and posture of the user's hands and fingers based on the results of myoelectric measurements taken using an electromyographic band worn on the user's wrist.The HMD 1 can also track the user's hands and fingers using gloves worn by the user that are equipped with sensors that sense the posture of the hands and fingers.

[0093] Modified Virtual Keyboard The keyboard surface may be defined as a curved surface rather than a flat surface. It is possible to use a virtual keyboard whose shape and key layout are designed to fit the shape of the hand, such as an ergonomic keyboard. If the home position of the virtual keyboard is different from the home position of a normal keyboard, a plane including the positions of the fingers other than the index finger and little finger may be used as the first reference plane.

[0094] The virtual keyboard may be a keyboard other than a QWERTY keyboard. For example, the virtual keyboard may be a keyboard with only a numeric keypad. In this case, since the home position is such that the tip of the middle finger is placed on the "5" key, the keyboard surface may be defined using a plane including the positions of the tips of the index finger, middle finger, and ring finger as the first reference plane.

[0095] In addition to the virtual keyboard, the HMD 1 can also control the position and orientation of a virtual UI that appears in the air and is operated by pressing with a fingertip, based on the tracking results of the hand, etc. In addition to the virtual keyboard, the virtual UI can also include, for example, a scroll bar and piano keys.

[0096] The user can also select and indicate the finger whose fingertip position is to be referenced to determine the first reference plane.

[0097] When a user moves their hand to simultaneously press the Ctrl, Alt, and Del keys, if the virtual keyboard moves to follow their hand, the user may not be able to press the Ctrl, Alt, and Del keys.

[0098] Therefore, even if the positions of the Ctrl, Alt, and Del keys on the virtual keyboard do not match the positions of the fingertips of each finger, the input acquisition unit 27 can determine that the user has pressed the Ctrl, Alt, and Del keys based on the movements of each finger. For example, the input acquisition unit 27 inputs physical information into a learning model generated by machine learning or the like to determine that the user has pressed the Ctrl, Alt, and Del keys.

[0099] When the user is looking at the virtual keyboard, the HMD 1 can also control the virtual keyboard so that it is fixed in place without following the hands.

[0100] It is also possible to use an existing correction method for input errors, such as narrowing down the key that the user intended to press or estimating the word that the user intended to input based on the type of finger that the user moved, in which case the input acquisition unit 27 can more accurately identify the content of the user's input operation.

[0101] Some or all of the estimation of the user's state by the state estimation unit 22, the definition of the keyboard surface by the keyboard surface definition unit 23, and the calculation of the position and orientation of the keyboard surface by the keyboard position calculation unit 25 may be realized using a learning model generated by machine learning or the like.

[0102] Regarding the computer, the above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0103] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0104] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0105] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0106] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0107] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0108] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0109] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0110] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0111] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0112] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0113] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0114] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0115] (1) An information processing device comprising: a tracking unit that tracks a body part of a user; and a control unit that controls, based on a tracking result of the body part, a position and orientation in a three-dimensional space of a virtual UI that accepts an input operation using at least a part of the tracked body part. (2) The information processing device described in (1), wherein the tracking unit tracks a hand as the body part. (3) The information processing device described in (2), wherein the control unit controls the position and orientation of the virtual UI based on a first reference plane that includes the positions of the fingertips of the user's fingers when the user is in an input standby state in which the user is not performing the input operation. (4) The information processing device described in (3), wherein the control unit controls the position and orientation of the virtual UI based on a second reference plane that corresponds to the back of the hand when the user is in an input state in which the user is performing the input operation. (5) The information processing device described in (4), wherein the second reference plane is a plane that includes the positions of the bases of the fingers and the position of the wrist. (6) The information processing device according to (4) or (5), wherein the control unit records an offset indicating a position and orientation relationship between the second reference plane and the virtual UI when the user is in the input standby state. (7) The information processing device according to (6), wherein the control unit controls a position and orientation of the virtual UI so as to reproduce the position and orientation relationship between the second reference plane and the virtual UI indicated by the offset when the user is in the input state. (8) The information processing device according to any of (4) to (7), wherein the control unit controls a position and orientation of the virtual UI so as to establish a predetermined position and orientation relationship between the first reference plane and the virtual UI when the user is in the input standby state. (9) The information processing device according to any of (4) to (8), further comprising an estimation unit that estimates a state of the user. (10) The information processing device according to (9), wherein the estimation unit estimates an input degree that numerically indicates the state of the user.(11) The information processing device according to (10), wherein the control unit calculates a position and orientation to be reflected in the virtual UI by weighting and adding a position and orientation of the virtual UI based on the first reference plane and a position and orientation of the virtual UI based on the second reference plane according to the input degree. (12) The information processing device according to any of (2) to (11), wherein the virtual UI is a virtual keyboard on which a plurality of keys are arranged. (13) The information processing device according to (12), wherein the control unit arranges the keys at positions on the virtual keyboard based on a tracking result of the user's fingers when the device is in an input standby state in which the user is not performing the input operation. (14) The information processing device according to any of (1) to (13), wherein the control unit calculates a position and orientation to be reflected in the virtual UI by applying a filter based on a time-series change amount of the position and orientation of the virtual UI. (15) The information processing device according to any of (1) to (14), wherein the control unit controls a size of the virtual UI based on a tracking result of the user's body part when the device is in an input standby state in which the user is not performing the input operation. (16) An information processing method in which an information processing device tracks a body part of a user and controls the position and orientation in a three-dimensional space of a virtual UI for accepting input operations at at least a part of the tracked body part based on the tracking result of the body part. (17) A program for causing a computer to execute a process of tracking a body part of a user and controlling the position and orientation in a three-dimensional space of a virtual UI for accepting input operations at at least a part of the tracked body part based on the tracking result of the body part.

[0116] REFERENCE SIGNS LIST 1 HMD, 11 Information processing unit, 21 Body position acquisition unit, 22 State estimation unit, 23 Keyboard surface definition unit, 24 Recording unit, 25 Keyboard position calculation unit, 26 Drawing unit, 27 Input acquisition unit

Claims

1. An information processing device comprising: a tracking unit that tracks a part of a user's body; and a control unit that controls the position and orientation in three-dimensional space of a virtual UI for accepting input operations on at least a part of the tracked part based on the tracking results of the part.

2. The information processing device according to claim 1, wherein the tracking unit tracks a hand as the part of the body.

3. The information processing device according to claim 2, wherein the control unit controls the position and orientation of the virtual UI based on a first reference plane including the position of the fingertips when the user is in an input standby state in which the user is not performing the input operation.

4. The information processing device according to claim 3, wherein the control unit controls the position and orientation of the virtual UI based on a second reference plane corresponding to the back of the hand when the user is in an input state performing the input operation.

5. The information processing device according to claim 4, wherein the second reference plane is a plane including the position of the base of the fingers and the position of the wrist.

6. The information processing device according to claim 4, wherein the control unit records an offset indicating a position and orientation relationship between the second reference plane and the virtual UI when the user is in the input standby state.

7. The information processing device according to claim 6, wherein the control unit controls the position and orientation of the virtual UI so as to reproduce the position and orientation relationship between the second reference plane indicated by the offset and the virtual UI when the user is in the input state.

8. The information processing device according to claim 4, wherein the control unit controls the position and orientation of the virtual UI so that the first reference plane and the virtual UI have a predetermined position and orientation relationship when the user is in the input standby state.

9. The information processing device according to claim 4, further comprising an estimation unit that estimates the state of the user.

10. The information processing device according to claim 9, wherein the estimation unit estimates an input degree that indicates the user's state in a numerical value.

11. The information processing device according to claim 10, wherein the control unit calculates a position and orientation of the virtual UI based on the first reference plane and a position and orientation of the virtual UI based on the second reference plane, weighted according to the input degree, to add together, to calculate a position and orientation to be reflected in the virtual UI.

12. The information processing device according to claim 2, wherein the virtual UI is a virtual keyboard on which a plurality of keys are arranged.

13. The information processing device according to claim 12, wherein the control unit arranges the keys at positions on the virtual keyboard based on a tracking result of the user's fingers when the device is in an input standby state in which the user is not performing the input operation.

14. The information processing device according to claim 1, wherein the control unit calculates a position and orientation to be reflected in the virtual UI by applying a filter based on a time-series amount of change in the position and orientation of the virtual UI.

15. The information processing device according to claim 1, wherein the control unit controls a size of the virtual UI based on a tracking result of the body part when the user is in an input standby state in which the user is not performing the input operation.

16. An information processing method in which an information processing device tracks a part of a user's body and controls the position and orientation in three-dimensional space of a virtual UI for accepting input operations using at least a part of the tracked part based on the tracking results of the part.

17. A program for causing a computer to execute a process of tracking a part of a user's body and controlling the position and orientation in three-dimensional space of a virtual UI for accepting input operations via at least a part of the tracked part based on the tracking results of the part.

Images