Information processing device, information processing method, and program

Abstract

The present technology pertains to an information processing device, an information processing method, and a program that make it possible to perform an input operation more appropriately by using a virtual UI. An information processing apparatus according to the present technology includes a tracking unit that tracks a part of a body of a user, and a control unit that controls a position and an orientation of a virtual UI in a three-dimensional space based on a tracking result of the part, the virtual UI for receiving an input operation at at least a part of the tracked part. The present technology can be applied, for example, to an HMD that receives an input operation performed by a user via a virtual keyboard.

Description

Technical Field

[0001] This technology relates to information processing apparatus, information processing method, and program, and more specifically, to information processing apparatus, information processing method, and program that enable more appropriate input operations using a virtual UI. Background Technology

[0002] In recent years, head-mounted displays (HMDs) equipped with highly accurate finger recognition have emerged, and the development of technologies that use virtual user interfaces (UIs) such as virtual keyboards to perform input operations such as text input has become active. For example, Patent Document 1 discloses a technology for determining the position of a virtual keyboard based on the fingertip positions of at least three fingers of a user in an initial state.

[0003] Citation List

[0004] Patent documents

[0005] Patent Document 1: WO 2020 / 039703 A Summary of the Invention

[0006] The problem to be solved by the present invention

[0007] When using a physical keyboard for input, users rely on tactile stimuli, such as those from the keys, as cues to understand the positional relationship between their fingers and the keyboard, and to finely adjust their finger positions. When using a virtual keyboard, however, these tactile cues are absent. Therefore, users struggle to understand the keyboard's dimensions and key spacing, and find it difficult to finely adjust their finger positions.

[0008] Therefore, it is difficult to perform input operations using a virtual keyboard with the same accuracy or speed as using a real keyboard.

[0009] This technology was developed in view of such circumstances and enables more appropriate input operations using a virtual UI.

[0010] Solution to the problem

[0011] An information processing apparatus according to one aspect of the present technology includes: a tracking unit that tracks a part of a user's body; and a control unit that controls the position and orientation of a virtual UI in three-dimensional space based on the tracking results of the body part, the virtual UI being used to receive input operations at at least a portion of the tracked body part.

[0012] According to one aspect of the present technology, the information processing method includes: tracking a part of a user's body by an information processing device, and controlling the position and posture of a virtual UI in three-dimensional space based on the tracking result of the part, the virtual UI being used to receive input operations at at least a portion of the tracked part.

[0013] A procedure according to one aspect of the present technology causes a computer to perform the following processing: tracking a part of a user's body, and controlling the position and orientation of a virtual UI in three-dimensional space based on the tracking results of that part, the virtual UI being used to receive input operations at at least a portion of the tracked part.

[0014] In one aspect of this technology, a part of a user's body is tracked, and the position and orientation of a virtual UI in three-dimensional space are controlled based on the tracking results of that part. The virtual UI is used to receive input operations at at least a portion of the tracked part. Attached Figure Description

[0015] Figure 1 This is a view illustrating an HMD according to an embodiment of the present technology.

[0016] Figure 2 This is a view showing an example of the input pose.

[0017] Figure 3 This is a view showing an example of a virtual keyboard.

[0018] Figure 4 This is a view showing an example of a keyboard plane.

[0019] Figure 5 It is a view used to describe the offset.

[0020] Figure 6 This is a view showing an example of a second reference plane.

[0021] Figure 7 This is a view showing an example of the first reference plane, the second reference plane, and the keyboard plane during input time.

[0022] Figure 8 This is a block diagram illustrating an example of HMD's functional configuration.

[0023] Figure 9 This is a view showing an example of finger angles.

[0024] Figure 10 It is a flowchart used to describe the processes to be performed by the information processing unit.

[0025] Figure 11 It is a flowchart used to describe the virtual keyboard control process.

[0026] Figure 12 This is a view showing an example of a keyboard plane calculated based on input degree.

[0027] Figure 13 This is a view showing an example of the position and orientation of the keyboard plane during standby time and a weighted sum of the position and orientation of the keyboard plane during input time.

[0028] Figure 14 It is a flowchart used to describe the virtual keyboard control process.

[0029] Figure 15 This is a block diagram illustrating an example of a computer's hardware configuration. Detailed Implementation

[0030] The modes used to implement this technology will be described below. The descriptions will be given in the following order.

[0031] 1. Overview of the implementation method

[0032] 2. HMD Configuration and Operation

[0033] 3. Modification

[0034] <1. Overview of the Implementation Method>

[0035] In recent years, HMDs equipped with highly accurate finger recognition have emerged, and the development of technologies for performing input operations such as text using virtual keyboards has become active. For example, Patent Document 1 discloses a technology for determining the position of a virtual keyboard based on the fingertip positions of at least three fingers of a user in an initial state.

[0036] When using a physical keyboard for input, users rely on tactile cues such as the feel of touching the surface or edges of keys, the feel of the bottom of keys, the texture of the keys as a starting point, and wrist support to grasp the positional relationship between the keyboard and their fingers, and to finely adjust their finger positions. When using a virtual keyboard, these tactile cues are unavailable. Therefore, users struggle to understand the keyboard's dimensions and the spacing between keys, and find it difficult to finely adjust their finger positions.

[0037] Therefore, it is difficult to perform input operations using a virtual keyboard with the same accuracy or speed as using a real keyboard.

[0038] The implementation of this technology has been conceived with a focus on the above-mentioned points, and a technique that enables input operations to be performed more appropriately using a virtual keyboard has been proposed. This implementation will be described in detail below.

[0039] Figure 1This is a view illustrating an embodiment of HMD 1 according to the present technology.

[0040] like Figure 1 As shown, user U1 wears HMD 1 on their head.

[0041] HMD 1 is configured, for example, as a display device including a camera and a non-transmissive display. If user U1 wears HMD 1, the field of vision is obstructed by the display and the housing, and therefore user U1 cannot see the outside world, but can see the image displayed on the display.

[0042] Games, movies, web content, emails, etc., are displayed on the screen of HMD 1. To enable user U1 to perform input operations such as text, HMD 1 receives input from user U1 via a virtual keyboard. This virtual keyboard does not exist in real space and is a virtually generated keyboard, for example, a keyboard with multiple keys arranged in a QWERTY array. In this technology, if the user moves their hand, the virtual keyboard follows the user's hand movement.

[0043] HMD 1 uses a camera device to track every part of user U1's body and calculates the position and posture of the virtual keyboard in virtual space based on the position (posture) of each part. HMD 1 can specify the key pressed by user U1 based on the position of user U1's fingers and the position of the virtual keyboard.

[0044] Reference Figures 2 to 7 Describe the process by which HMD 1 calculates the position and orientation of the virtual keyboard.

[0045] like Figure 2 As shown, when user U1's pose is an input pose, HMD 1 displays a virtual keyboard on the screen. An input pose is a pose that indicates the intention to perform an input operation. Figure 2 In this context, the gesture of extending both hands forward (as if placing fingers on a keyboard) is set as the input gesture.

[0046] Note that postures other than extending both hands forward can also be identified as input postures for user U1. Whether a user U1's posture is an input posture can be determined by inputting captured images of user U1's body parts into a learning model generated through machine learning or similar methods.

[0047] Figure 3 This is a view showing an example of a virtual keyboard.

[0048] For example, Figure 3As shown, if user U1 extends both hands forward, a virtual keyboard KL for the left hand is displayed near the left hand HL, and a virtual keyboard KR for the right hand is displayed near the right hand HR. The virtual keyboard KL is operated by the fingertips of each finger, which is the part of the left hand HL, and the virtual keyboard KR is operated by the fingertips of each finger, which is the part of the right hand HR. Additionally, points indicating the positions of the fingertips, roots, and wrists of each finger in the left hand HL and right hand HR are displayed. Note that it is not always necessary to display the points indicating the positions of the fingertips, roots, and wrists of each finger, as well as the virtual keyboards KL and KR.

[0049] exist Figure 3 In the middle, the left HL and right HR are indicated by dashed lines, which indicates that the left HL and right HR are not displayed on the HMD 1 display.

[0050] When using a physical, integrated left-right keyboard for input, users support their hands by resting their wrists on the keyboard or table, or receive tactile stimulation by placing their fingertips on the keys, thus mastering the relative positions of their left and right hands. When using a virtual keyboard, cues such as wrist support and tactile stimulation of the fingertips are not available. Therefore, it is preferable to display a split-type keyboard, where the relative positions of the left and right hands do not require mastery, as a virtual keyboard.

[0051] Input posture includes an input standby state where the user does not move their finger and does not perform any input operation, and an input state where the user moves their finger and performs an input operation. In the following text, the time the user is in the input standby state will be referred to as standby time, and the time the user is in the input state will be referred to as input time.

[0052] During standby time, HMD 1 defines the keyboard plane of the area where the virtual keyboard is located.

[0053] Figure 4 This is a view showing an example of a keyboard plane.

[0054] like Figure 4As shown, a keyboard plane KP1L for the left hand is defined based on a first reference plane PL1L. This first reference plane includes a point P1L indicating the position of the left wrist, a point P2L indicating the position of the index fingertip, and a point P3L indicating the position of the little fingertip. Specifically, the keyboard plane KP1L is defined such that the first reference plane PL1L and the keyboard plane KP1L have a predetermined positional relationship. For example, a specific range on a plane obtained by moving the first reference plane PL1L a predetermined distance and tilting it by a predetermined angle is defined as the keyboard plane KP1L. The keyboard plane KP1L can be defined as a region on the same plane as the first reference plane PL1L.

[0055] Note that a plane including points indicating the positions of parts other than the wrist, the fingertips of the index finger, and the little finger can be set as the first reference plane. When performing input operations using a real keyboard, it is assumed that at least the fingertips of the wrist, index finger, and little finger are stably placed on the keyboard or desktop during standby time, and therefore, it is preferable to set the plane including three points indicating the positions of the fingertips of the wrist, index finger, and little finger as the first reference plane.

[0056] On the keyboard plane KP1L, for example, the key K2 for "F" is positioned at the tip of the index finger (point P2L), and the key K1 for "A" is positioned at the tip of the little finger (point P3L). By defining the positions of the keys on the virtual keyboard KL based on the positions of the fingertips of the index and little fingers during standby time, a virtual keyboard KL that matches the size and shape of the user's hand can be displayed. Furthermore, by enlarging or shrinking the size of the keyboard plane KP1L (the size of the virtual keyboard KL) or deforming the keyboard plane KP1L (the virtual keyboard KL) according to the width between the index and little fingers during standby time, a virtual keyboard KL that matches the size and shape of the user's hand can be displayed.

[0057] Similar to the left hand, the keyboard plane KP1R for the right hand is defined based on a first reference plane PL1R, which includes a point P1R indicating the position of the right wrist, a point P2R indicating the position of the index fingertip, and a point P3R indicating the position of the little fingertip. On the keyboard plane KP1R, for example, the "J" key K3 is arranged at the position of the index fingertip (point P2R), and the ";" key K4 is arranged at the position of the little fingertip (point P3R).

[0058] As described above, during standby time, the position and orientation of the keyboard plane are controlled based on a first reference plane. Specifically, the position and orientation of the keyboard plane are controlled to follow the first reference plane.

[0059] After defining the keyboard plane, HMD 1 records the offset of the position and orientation relationship between the second reference plane PL2L (which is the plane corresponding to the back of the left hand (the palm of the left hand)) and the keyboard plane KP1L, such as Figure 5 The arrow in the center indicates the position and orientation relationship between the second reference plane PL2L and the keyboard plane KP1L, which is represented, for example, by the relative position (distance) and relative orientation (angle) of the keyboard plane KP1L relative to the second reference plane PL2L. Figure 5 In this context, the keyboard plane KP1L is defined as a region on the same plane as the first reference plane PL1L.

[0060] The position and orientation relationships indicated by the offset include the position and orientation relationships between the second reference plane PL2L and the first reference plane PL1L, and between the first reference plane PL1L and the keyboard plane KP1L. Furthermore, the offset also includes information indicating the dimensions of the keyboard plane KP1L, and information indicating the positions of the keys arranged on the keyboard plane KP1L.

[0061] Note that, although not shown, the offset indicating the positional and orientational relationship between the second reference plane PL2R (which is the plane corresponding to the back of the right hand (the palm of the right hand)) and the keyboard plane KP1R is recorded in a similar manner. Even without displaying the virtual keyboard, the keyboard plane is defined and the offset is recorded.

[0062] Figure 6 This is a view showing an example of a second reference plane.

[0063] like Figure 6 As shown, the second reference plane PL2L corresponding to the back of the left hand is, for example, a plane including point P11L indicating the position of the left wrist, point P12L indicating the position of the base of the index finger, and point P13L indicating the position of the base of the little finger. Similarly, the second reference plane PL2R corresponding to the back of the right hand is, for example, a plane including point P11R indicating the position of the right wrist, point P12R indicating the position of the base of the index finger, and point P13R indicating the position of the base of the little finger.

[0064] As described above, during standby time, the position and orientation of the keyboard plane are controlled based on a first reference plane; however, during input time, the position and orientation of the keyboard plane are controlled based on a second reference plane (instead of the first reference plane). Specifically, the position and orientation of the keyboard plane are controlled to follow the second reference plane.

[0065] Figure 7 This is a view showing an example of the first reference plane, the second reference plane, and the keyboard plane during input time.

[0066] During input time, HMD 1 calculates the keyboard plane KP1L to reproduce the position and orientation relationship between the second reference plane PL2L and the keyboard plane KP1L, as indicated by the offset recorded during standby time. Figure 7 As indicated by the arrow in the center. Specifically, HMD 1 moves the second reference plane PL2L by the distance indicated by the offset recorded during standby time, and calculates the area within a predetermined range on the plane tilted at the angle indicated by the offset as the keyboard plane KP1L.

[0067] During input, the fingertips of each finger move rapidly, and therefore, the first reference plane PL1L also changes rapidly. If the keyboard plane is controlled to follow the first reference plane PL1L, the keyboard plane also changes rapidly. This rapid change in the keyboard plane makes it difficult for the user to accurately press the desired key.

[0068] In HMD 1 of this technology, the position and orientation of the keyboard plane are controlled to follow a second reference plane PL2L corresponding to the back of the hand, which has a small degree of movement even during input time, thereby stabilizing the keyboard plane. Therefore, even if the finger position changes during input time, the user can accurately press the desired key, and even if the hand position changes during input time, the user can continuously perform input operations.

[0069] <2. HMD Configuration and Operation>

[0070] Figure 8 This is a block diagram showing an example of the functional configuration of HMD 1. Figure 8 The information processing unit 11 in the HMD1 is implemented, for example, by executing a predetermined program by a CPU or the like set in the HMD1.

[0071] like Figure 8 As shown, the information processing unit 11 includes a body position acquisition unit 21, a state estimation unit 22, a keyboard plane limiting unit 23, a recording unit 24, a keyboard position calculation unit 25, a drawing unit 26, and an input acquisition unit 27.

[0072] The body position acquisition unit 21 serves as a tracking unit, tracking user body parts based on captured images obtained from cameras installed in HMD 1, and acquiring the position and pose of each body part as the tracking result. The position and pose of each body part are acquired, for example, as 6 degrees of freedom (6DoF) parameters. The body position acquisition unit 21 provides body information indicating the position and pose of each body part to the state estimation unit 22, the keyboard plane defining unit 23, the keyboard position calculation unit 25, and the input acquisition unit 27.

[0073] The state estimation unit 22 estimates the user's state based on the body information provided by the body position acquisition unit 21. Specifically, the state estimation unit 22 estimates whether the user's pose is an input pose, whether the user is in an input state or an input standby state, etc.

[0074] For example, the state estimation unit 22 estimates the input degree, which numerically indicates whether the user is in an input state or an input standby state, based on body information. An input degree of 0 indicates that the user is in an input standby state, and an input degree of 1 indicates that the user is in an input state. The input degree is estimated based on the angular velocity and angular acceleration of the finger angles, the relative position and relative velocity of each fingertip relative to the palm, etc.

[0075] Figure 9 This is a view showing an example of finger angles.

[0076] like Figure 9 As shown, for example, the angle of the following straight lines is defined as the finger angle of the index finger: the straight line connecting point P21, which indicates the position of the fingertip of the index finger, and point P22, which indicates the position of the base of the index finger, and the straight line connecting point P22 and point P23, which indicates the position of the wrist.

[0077] For example, when the finger angles of the index, middle, ring, and little fingers are roughly the same, the state estimation unit 22 estimates the value of the input degree as 0 (input standby state).

[0078] Note that the state estimation unit 22 can also estimate the user's state by inputting body information into a learning model generated by machine learning or the like. It can also indicate whether the user is in an input state or an input standby state by the user's gestures. For example, if the user moves their fingers rapidly or extends their hand forward, the state estimation unit 22 estimates that the user is in an input state. Furthermore, for example, if the user's posture is a so-called front-following posture with their hand extended vertically forward, the state estimation unit 22 estimates that the user is in a standby state; and if the user holds their hand down in this posture, the state estimation unit 22 estimates that the user is in an input state.

[0079] The state estimation unit 22 controls the keyboard plane defining unit 23 and the keyboard position calculation unit 25 based on the estimation result of the user's state. For example, when the user is in an input standby state, the state estimation unit 22 causes the keyboard plane defining unit 23 to define the keyboard plane. Furthermore, when the user is in an input state, the state estimation unit 22 causes the keyboard position calculation unit 25 to calculate the position and orientation of the keyboard plane.

[0080] The keyboard plane defining unit 23 defines the keyboard plane based on the body information provided by the body position acquisition unit 21, and causes the recording unit 24 to record the offset.

[0081] Recording unit 24 records the offset provided by keyboard plane limiting unit 23.

[0082] The keyboard position calculation unit 25 obtains the offset from the recording unit 24 and calculates the position and orientation of the keyboard plane based on the offset and the body information provided by the body position acquisition unit 21. For example, the position and orientation of the keyboard plane are calculated as six-degree-of-freedom parameters.

[0083] The keyboard position calculation unit 25 arranges a keyboard plane with an offset of an indicated dimension according to the calculated position and orientation, and arranges various keys on the keyboard plane at the positions offset from the indicated keys, thereby generating keyboard data. The keyboard position calculation unit 25 provides the keyboard data to the drawing unit 26 and the input acquisition unit 27. The keyboard plane defining unit 23 and the keyboard position calculation unit 25 serve as control units for controlling the position and orientation of the virtual keyboard.

[0084] The drawing unit 26 draws an image so that the virtual keyboard appears to exist at the position of the keyboard plane calculated by the keyboard position calculation unit 25, and displays the image on the monitor set in HMD 1. The drawing unit 26 can change the drawing method of the virtual keyboard based on the content of the input operation specified by the input acquisition unit 27. For example, the drawing unit 26 draws an image by changing the color of the key pressed by a finger, or by making the pressed key sink into the bottom side of the virtual keyboard.

[0085] Note that it's not always necessary to draw and display the virtual keyboard during both standby and input times. Displaying the virtual keyboard during input times takes higher priority than displaying it during standby.

[0086] The input acquisition unit 27 specifies the content of the input operation performed by the user based on the body information provided by the body position acquisition unit 21 and the keyboard data provided by the keyboard position calculation unit 25. For example, if the position of a key on the virtual keyboard matches the position of the user's fingertip, the input acquisition unit 27 specifies that the key has been pressed.

[0087] Next, refer to Figure 10 The flowchart describes the processing to be performed by the information processing unit 11 with the above configuration.

[0088] In step S1, the body position acquisition unit 21 tracks, for example, the hand, which is a part of the user's body, based on the captured image captured by the camera device set in HMD 1, and acquires the position and posture of the hand.

[0089] In step S2, the state estimation unit 22 determines whether the user's posture is the input posture based on the position and posture of the user's hand.

[0090] If it is determined in step S2 that the user's pose is not the input pose, the process returns to step S1, and for example, the hand tracking is repeated for each frame until the user's pose becomes the input pose.

[0091] On the other hand, if it is determined in step S2 that the user's posture is an input posture, the information processing unit 11 performs virtual keyboard control processing in step S3. Through virtual keyboard control processing, the position and posture of the keyboard plane are calculated, and keyboard data is generated. (This will be discussed later.) Figure 11 Describe the details of virtual keyboard control processing.

[0092] After the virtual keyboard control process is executed in step S3, the process returns to step S1 and the subsequent processes are repeated.

[0093] The following will refer to Figure 11 The flowchart description in the document should be... Figure 10 The virtual keyboard control processing is performed in step S3.

[0094] In step S11, the state estimation unit 22 determines whether the user is in an input standby state.

[0095] If, in step S11, it is determined that the user is in an input standby state, the keyboard plane defining unit 23 defines the keyboard plane based on the position and posture of the user's hands, and records the offset in step S12. Thereafter, the keyboard position calculation unit 25 generates keyboard data by arranging various keys on the keyboard plane defined by the keyboard plane defining unit 23.

[0096] On the other hand, if it is determined in step S11 that the user is not in an input standby state (i.e., in an input state), in step S13, the keyboard position calculation unit 25 calculates the position of the keyboard plane based on the offset recorded during the standby time and the position and posture of the user's hands. Thereafter, the keyboard position calculation unit 25 arranges the keyboard plane according to the calculated position and posture, and arranges various keys on the keyboard plane to generate keyboard data.

[0097] After keyboard data is generated in step S12 or S13, the process returns to... Figure 10 Step S3 in the process is performed, and subsequent processing is carried out.

[0098] As described above, in HMD 1 of this technology, the hand, which is a part of the user's body, is tracked, and the position and orientation of the virtual keyboard used to receive input operations using fingers are controlled in three-dimensional space based on the hand tracking results. The virtual keyboard moves in a manner that follows the hand, and therefore, the user can perform input operations using the virtual keyboard with a feeling close to that of using a real keyboard, without having to worry about the starting position or finely correct the position of the fingers.

[0099] <3. Revision>

[0100] Example of calculating the position and orientation of the keyboard plane based on input degree.

[0101] The final position and orientation of the keyboard plane can be calculated by weighted summing of the position and orientation of the keyboard plane during the idle time and the position and orientation of the keyboard plane during the input time.

[0102] Figure 12 This is a view showing an example of the position and orientation of the keyboard plane based on input degree calculations.

[0103] like Figure 12 As shown in the left part, when the input degree value is 0, the user is in input standby mode, and therefore, a predetermined range on the same plane as the first reference plane PL1L is calculated as the keyboard plane KP1L. On the other hand, as Figure 12 As shown in the right part, when the input degree is 1, the user is in an input state, and therefore, a predetermined range on a plane that is different from the first reference plane PL1L and on which the second reference plane PL2L is moved or tilted is calculated as the keyboard plane KP1L.

[0104] With an input degree value of, for example, 0.5, the final keyboard plane is calculated by mixing the keyboard plane KP1L during standby and the keyboard plane KP1L during input. Note that the keyboard plane obtained by mixing the keyboard plane during standby and the keyboard plane during input may become an uncertain plane with unstable position and orientation even if the keyboard plane does not partially follow the hand movement. Therefore, the input degree can be set to be estimated discretely between values ​​of 0 and 1, and for example, the input degree value can be set to 1 even if the finger moves slightly.

[0105] Figure 13 This is a view showing an example of the position and orientation of the keyboard plane during standby time and a weighted sum of the position and orientation of the keyboard plane during input time.

[0106] First, such as Figure 13As shown in #1, the input degree is multiplied by the position and orientation of the keyboard plane at the input time as a weight, and as... Figure 13 As shown in #2, (1 - input degree) is used as a weight to multiply the position and orientation of the keyboard plane during standby time. Next, as... Figure 13 As shown in #3, the weighted position and orientation of the keyboard plane during the input time are added to the weighted position and orientation of the keyboard plane during the standby time to calculate the final position and orientation of the keyboard plane.

[0107] Reference Figure 14 The flowchart describes the virtual keyboard control process when calculating the keyboard position based on the input degree.

[0108] In step S21, the information processing unit 11 calculates the position and orientation of the keyboard plane based on the input degree. Specifically, the keyboard plane defining unit 23 pre-defined the keyboard plane during the standby time based on the position of the user's fingers. Next, the keyboard position calculation unit 25 calculated the position and orientation of the keyboard plane during the input time based on the position and orientation of the user's hands. Finally, as described above, the keyboard position calculation unit 25 performed a weighted sum of the position and orientation of the keyboard plane during the standby time and the position and orientation of the keyboard plane during the input time based on the input degree to calculate the final position and orientation of the keyboard plane.

[0109] Note that the position and orientation of the keyboard plane during input time in this article are calculated based on the current second reference plane, and are independent of whether the user is in input standby or input state.

[0110] In step S22, the keyboard position calculation unit 25 applies a stabilizing filter (e.g., a low-pass filter) based on the time series changes in position and orientation to the calculated position and orientation of the keyboard plane in order to calculate the position and orientation to be reflected in the keyboard data.

[0111] In step S23, the drawing unit 26 draws an image so that the virtual keyboard appears to exist on the keyboard plane to which the stabilizing filter has been applied, and displays the image on the monitor set in HMD 1.

[0112] If the position of the keyboard plane is controlled sensitively to finger movement or input intensity, the keyboard plane may move drastically. Therefore, it is preferable to apply a stabilizing filter to stabilize the keyboard plane, and then draw an image of the virtual keyboard, as in the processing of steps S22 and S23. Note that the stabilizing filter can be applied to the position and orientation of each keyboard plane after calculating (limiting) the position and orientation of the keyboard plane under standby and input times in step S21.

[0113] After displaying the image on the monitor in step S23, the process returns to... Figure 10 Step S3 in the process is performed, and subsequent processing is carried out.

[0114] As mentioned above, a weighted sum of the keyboard plane position and orientation during standby time and during input time can also be performed based on the input degree.

[0115] Application Examples

[0116] This technology can be applied not only to HMDs where users cannot see the outside world, but also to HMDs with video see-through (VST) capabilities, augmented reality (AR) glasses, and more.

[0117] Examples of methods for tracking hands and fingers

[0118] Instead of using the camera device set in HMD 1 to track the user's hands and fingers, HMD 1 can use other camera devices installed in locations that can track the user's hands and fingers.

[0119] Furthermore, the HMD 1 can estimate the position and posture of a user's hand and fingers based on measurements of electromyographic potentials (EMGs) from an EMG band worn on the user's wrist. The HMD 1 can also track a user's hand and fingers using a glove fitted with sensors that detect hand and finger posture.

[0120] Modification of virtual keyboard

[0121] Instead of a flat surface, a curved surface can be defined as the keyboard plane. A virtual keyboard with a shape and key arrangement designed to match the shape of the hand, such as an ergonomic keyboard, can be used. In cases where the starting position of the virtual keyboard differs from that of a regular keyboard, a flat surface including the positions of the fingers other than the index and little fingers can be used as the first reference plane.

[0122] A virtual keyboard can be any keyboard other than a QWERTY keyboard. For example, a virtual keyboard could be a keyboard with only number keys. In this case, the starting position is such that the tip of the middle finger rests on the "5" key; for example, the keyboard plane can be defined by using a flat surface that includes the positions of the fingertips of the index, middle, and ring fingers as a first reference plane.

[0123] In addition to the virtual keyboard, the HMD 1 can also control the position and posture of a virtual UI that appears in the air and is to be pressed and operated by fingertips, based on hand tracking results. Besides the virtual keyboard, the virtual UI also includes, for example, scroll bars or a piano keyboard.

[0124] Users can also select and provide instructions on the position of their fingertips that will be involved to obtain a first reference plane.

[0125] If a user moves his or her hand to press the Ctrl, Alt, and Del keys simultaneously, the user may be unable to press the Ctrl, Alt, and Del keys if the virtual keyboard follows the hand movement.

[0126] Therefore, even if the positions of the Ctrl, Alt, and Del keys on the virtual keyboard do not match the positions of the fingertips, the input acquisition unit 27 can still recognize that the user has pressed the Ctrl, Alt, and Del keys based on the movement of each finger. For example, the input acquisition unit 27 inputs body information into a learning model generated by machine learning or the like, and specifies that the user has pressed the Ctrl, Alt, and Del keys.

[0127] When the user is looking at the virtual keyboard, the HMD 1 can also control the virtual keyboard to keep it in a fixed position and not follow the hand.

[0128] Existing methods for correcting input errors can be used, such as narrowing down the key the user intends to press based on the type of finger the user moves or estimating the word the user intends to type. In this case, the input acquisition unit 27 is able to more accurately specify the content of the user's input operation.

[0129] The following can be partially or fully implemented using a learning model generated by machine learning or the like: estimation of the user's state by the state estimation unit 22, limitation of the keyboard plane by the keyboard plane limitation unit 23, and calculation of the position and orientation of the keyboard plane by the keyboard position calculation unit 25.

[0130] computer

[0131] The above series of processes can be executed by hardware or by software. In the case of executing the series of processes by software, the program included in the software is installed from the program recording medium onto a computer, general-purpose personal computer, or similar device contained in dedicated hardware.

[0132] Figure 15 This is a block diagram illustrating an example of the hardware configuration of a computer performing the above series of processes according to a program.

[0133] The central processing unit (CPU) 501, read-only memory (ROM) 502 and random access memory (RAM) 503 are connected to each other via bus 504.

[0134] The input / output interface 505 is also connected to the bus 504. Input units 506, including a keyboard and mouse, and output units 507, including a display and speakers, are connected to the input / output interface 505. Furthermore, storage units 508, including hard disks and non-volatile memory, communication units 509, including network interfaces, and drivers 510 that drive removable media 511 are connected to the input / output interface 505.

[0135] In a computer configured as described above, for example, CPU 501 loads a program stored in memory unit 508 into RAM 503 via input / output interface 505 and bus 504, and executes the program to perform the series of processes described above.

[0136] For example, a program to be executed by CPU 501 is stored in removable medium 511 or provided via wired or wireless transmission media such as a local area network, the Internet or digital broadcasting, and then installed in storage unit 508.

[0137] A program to be executed by a computer may be a program that performs processing sequentially in the order described in this specification, or it may be a program that performs processing in parallel or at necessary moments, such as when a call is made.

[0138] Note that the effects described in this specification are illustrative and not limiting, and other effects may be provided.

[0139] The implementation of this technology is not limited to the above-described implementation, and various modifications can be made without departing from the scope of this technology.

[0140] For example, this technology can have a cloud computing configuration, in which one function is shared and processed collaboratively by multiple devices via a network.

[0141] Furthermore, each step described in the flowchart above can be performed by a single device, or by multiple devices in a shared manner.

[0142] Furthermore, in cases where a step includes multiple processes, the multiple processes included in a step can be performed by a single device, or they can be performed by multiple devices in a shared manner.

[0143] <Example of configuration combinations>

[0144] This technology can also be configured as follows.

[0145] (1) An information processing device, comprising:

[0146] A tracking unit that tracks parts of the user's body; and

[0147] A control unit controls the position and orientation of a virtual UI in three-dimensional space based on the tracking results of the part being tracked, the virtual UI being used to receive input operations at at least a portion of the part being tracked.

[0148] (2) The information processing device according to (1),

[0149] The tracking unit tracks the hand, which is the part in question.

[0150] (3) The information processing device according to (2),

[0151] When the user is in an input standby state where no input operation is performed, the control unit controls the position and posture of the virtual UI based on a first reference plane including the position of the fingertip.

[0152] (4) The information processing device according to (3),

[0153] When the user is in an input state performing the input operation, the control unit controls the position and posture of the virtual UI based on a second reference plane corresponding to the back of the hand.

[0154] (5) The information processing device according to (4),

[0155] The second reference plane is a plane that includes the position of the base of the finger and the position of the wrist.

[0156] (6) The information processing apparatus according to (4) or (5),

[0157] When the user is in the input standby state, the control unit records the offset indicating the position and orientation relationship between the second reference plane and the virtual UI.

[0158] (7) The information processing device according to (6),

[0159] When the user is in the input state, the control unit controls the position and orientation of the virtual UI to reproduce the position and orientation relationship between the second reference plane indicated by the offset and the virtual UI.

[0160] (8) The information processing apparatus according to any one of (4) to (7),

[0161] When the user is in the input standby state, the control unit controls the position and posture of the virtual UI so that the first reference plane and the virtual UI have a predetermined position and posture relationship.

[0162] (9) The information processing apparatus according to any one of (4) to (8) further comprises:

[0163] An estimation unit estimates the state of the user.

[0164] (10) The information processing apparatus according to (9),

[0165] The estimation unit estimates the input degree, which is a numerical indication of the user's state.

[0166] (11) The information processing apparatus according to (10),

[0167] The control unit calculates the position and pose to be reflected on the virtual UI by performing a weighted summation of the position and pose of the virtual UI based on the first reference plane and the position and pose of the virtual UI based on the second reference plane, according to the input degree.

[0168] (12) The information processing apparatus according to any one of (2) to (11),

[0169] The virtual UI is a virtual keyboard with multiple keys.

[0170] (13) The information processing apparatus according to (12),

[0171] The control unit arranges the keys at positions on the virtual keyboard based on the tracking results of the user's fingers in an input standby state when the user is not performing the input operation.

[0172] (14) The information processing apparatus according to any one of (1) to (13),

[0173] The control unit calculates the position and pose to be reflected on the virtual UI by applying a filter based on the time-series changes in the position and pose of the virtual UI.

[0174] (15) The information processing apparatus according to any one of (1) to (14),

[0175] The control unit controls the size of the virtual UI based on the tracking results of the part when the user is in an input standby state where the user is not performing the input operation.

[0176] (16) An information processing method, comprising the following operations performed by an information processing device:

[0177] Tracking parts of the user's body; and

[0178] The position and orientation of a virtual UI in three-dimensional space are controlled based on the tracking results of the said part, and the virtual UI is used to receive input operations at at least a portion of the tracked part.

[0179] (17) A program that causes a computer to perform the following processes:

[0180] Tracking parts of the user's body; and

[0181] The position and orientation of a virtual UI in three-dimensional space are controlled based on the tracking results of the said part, and the virtual UI is used to receive input operations at at least a portion of the tracked part.

[0182] List of reference numerals

[0183] 1 HMD

[0184] 11 Information Processing Unit

[0185] 21 Body position acquisition unit

[0186] 22 State estimation unit

[0187] 23 Keyboard plane limiting unit

[0188] 24 Recording Units

[0189] 25 Keyboard position calculation unit

[0190] 26 Drawing Units

[0191] 27 Input Acquisition Unit

Claims

1. An information processing apparatus, comprising: A tracking unit that tracks parts of the user's body; as well as A control unit controls the position and orientation of a virtual UI in three-dimensional space based on the tracking results of the part being tracked, the virtual UI being used to receive input operations at at least a portion of the part being tracked.

2. The information processing device according to claim 1, in, The tracking unit tracks the hand as the described part.

3. The information processing device according to claim 2, in, When the user is in an input standby state where no input operation is performed, the control unit controls the position and posture of the virtual UI based on a first reference plane including the position of the fingertip.

4. The information processing device according to claim 3, in, When the user is in an input state performing the input operation, the control unit controls the position and posture of the virtual UI based on a second reference plane corresponding to the back of the hand.

5. The information processing apparatus according to claim 4, in, The second reference plane is a plane that includes the position of the base of the finger and the position of the wrist.

6. The information processing apparatus according to claim 4, in, When the user is in the input standby state, the control unit records the offset indicating the position and orientation relationship between the second reference plane and the virtual UI.

7. The information processing apparatus according to claim 6, in, When the user is in the input state, the control unit controls the position and orientation of the virtual UI to reproduce the position and orientation relationship between the second reference plane indicated by the offset and the virtual UI.

8. The information processing apparatus according to claim 4, in, When the user is in the input standby state, the control unit controls the position and orientation of the virtual UI so that the first reference plane and the virtual UI have a predetermined position and orientation relationship.

9. The information processing apparatus according to claim 4, further comprising: An estimation unit estimates the state of the user.

10. The information processing apparatus according to claim 9, in, The estimation unit estimates the input degree, which is a numerical indication of the user's state.

11. The information processing apparatus according to claim 10, in, The control unit calculates the position and pose to be reflected on the virtual UI by performing a weighted summation of the position and pose of the virtual UI based on the first reference plane and the position and pose of the virtual UI based on the second reference plane, according to the input degree.

12. The information processing apparatus according to claim 2, in, The virtual UI is a virtual keyboard with multiple keys.

13. The information processing apparatus according to claim 12, in, The control unit arranges the keys at positions on the virtual keyboard based on the tracking results of the user's fingers in an input standby state when the user is not performing the input operation.

14. The information processing apparatus according to claim 1, in, The control unit calculates the position and pose to be reflected on the virtual UI by applying a filter based on the time-series changes in the position and pose of the virtual UI.

15. The information processing apparatus according to claim 1, in, The control unit controls the size of the virtual UI based on the tracking results of the part when the user is in an input standby state where the user is not performing the input operation.

16. An information processing method, comprising the following operations performed by an information processing device: Tracking parts of the user's body; and The position and orientation of a virtual UI in three-dimensional space are controlled based on the tracking results of the said part, and the virtual UI is used to receive input operations at at least a portion of the tracked part.

17. A program that causes a computer to perform the following processes: Tracking parts of the user's body; and The position and orientation of a virtual UI in three-dimensional space are controlled based on the tracking results of the said part, and the virtual UI is used to receive input operations at at least a portion of the tracked part.

Images