Virtual keyboard selection using multiple input modes

By displaying a virtual keyboard in an augmented reality environment and combining it with user hand movements and gaze input, the problems of inaccurate input and fatigue in virtual reality and augmented reality are solved, achieving a more efficient and comfortable user interaction experience.

CN119032332APending Publication Date: 2024-11-26CTRL-LABS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380033173.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-11
Filing Date
2023-04-10
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In existing virtual reality and augmented reality environments, inaccurate or slow user input leads to a poor user experience, and different input modes present ergonomic challenges such as fatigue and discomfort.

Method used

By displaying a virtual keyboard in an augmented reality environment, tracking the user's hand movements and gaze input, detecting hand movements that meet speed standards, and combining gaze input to parse character selection, the virtual keyboard is displayed outside the user's arm reach to reduce fatigue.

Benefits of technology

It improves the accuracy and speed of user input, reduces physical fatigue, provides a more natural and ergonomic interaction method, and improves the accuracy and time efficiency of traditional technologies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119032332A_ABST
    Figure CN119032332A_ABST
Patent Text Reader

Abstract

Aspects of the present disclosure relate to triggering virtual keyboard selection using multiple input modes. An interface manager may display an interface, such as a virtual keyboard, to a user in an artificial reality environment. Embodiments of an interface manager may track user eye gaze input and user hand input (e.g., hand motion or finger motion). The interface manager may parse the character selection on the virtual keyboard according to the tracked user gaze input based on detecting that the user hand motion meets the trigger criteria. For example, an interface manager may: detect that a tracked user's hand motion meets a trigger criterion at a given point in time; and parsing the selection from the virtual keyboard (e.g., the selection of the displayed character) according to the tracked user's gaze of the virtual keyboard at a given point in time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to triggering virtual keyboard selection using multiple input modes. Background Art

[0002] Artificial reality devices offer users the opportunity to experience mixed reality, augmented reality, and / or virtual reality. However, challenges remain with user interaction in these environments. Often, user input is inaccurate or slow, leading to user frustration. User input can take many forms, such as tracked motion, eye gaze, touch, and joysticks. This variety of input creates challenges in matching implemented functionality with user desired functionality. Additionally, the different input opportunities also create ergonomic challenges, as certain types of user input can cause fatigue overtime or can otherwise be uncomfortable for the user. Summary of the invention

[0003] According to the present invention, a method for triggering virtual keyboard selection using multiple input modes is provided, the method comprising: displaying a virtual keyboard in an augmented reality environment; tracking user hand movements and user gaze input relative to the virtual keyboard; detecting that the tracked user hand movements satisfy a speed standard; in response to detecting that the tracked user hand movements satisfy the speed standard, parsing character selection according to the user gaze input relative to the virtual keyboard at a time corresponding to detecting that the user hand movements satisfy the speed standard; wherein the parsing is not based on the hand position relative to the virtual keyboard; and using the parsed character selection to generate characters for a character sequence.

[0004] Optionally, the virtual keyboard is displayed at a distance greater than 0.5 meters from the user, at a distance beyond the user's arm's reach, or any combination thereof.

[0005] Optionally, the velocity criterion is met when a peak velocity of a user's hand movement or finger movement is detected.

[0006] Optionally, the method also includes analyzing the tracked user hand movement to determine the speed of the user hand movement or finger movement, wherein the tracked user hand movement is transformed relative to the user position so that the determined speed includes the speed relative to the user, and when the determined speed is the peak speed of the tracked user hand movement, the tracked user hand movement meets the speed standard.

[0007] Optionally, analyzing the tracked user hand movement to determine the speed for the user hand movement or finger movement includes filtering a first axis component of the tracked user hand movement and inverting a second axis component of the tracked user hand movement.

[0008] Optionally, the peak velocity of the user's hand movement or finger movement is detected by comparing a previously determined velocity with a currently determined velocity and detecting a negative velocity change based on the comparison.

[0009] Optionally, the character sequence is generated by: iteratively detecting that the user hand movement meets a speed criterion based on the tracking; and iteratively parsing character selection according to the user gaze input at a time corresponding to each instance of detecting that the user hand movement meets the speed criterion.

[0010] Optionally, the tracked user hand movement includes alternating movement between the user's hands, and the iterative detection of the user's hand movement satisfying the speed criterion and the iterative resolution of the character selection correspond to the alternating movement between the user's hands.

[0011] According to the present invention, a computer-readable storage medium is also provided, which stores multiple instructions. When these instructions are executed by a computing system, the computing system executes a process for triggering a virtual keyboard selection using multiple input modes, the process including: displaying a virtual keyboard in an augmented reality environment; tracking user hand movements and user gaze input relative to the virtual keyboard; detecting that the tracked user hand movements meet a speed standard; in response to detecting that the tracked user hand movements meet the speed standard, parsing a character selection based on the user gaze input relative to the virtual keyboard at a time corresponding to detecting that the user hand movements meet the speed standard, wherein the parsing is not based on the hand position relative to the virtual keyboard; and using the parsed character selection to generate characters for a character sequence.

[0012] Optionally, the virtual keyboard is displayed at a distance greater than 0.5 meters from the user, at a distance beyond the user's arm's reach, or any combination thereof.

[0013] Optionally, the velocity criterion is met when a peak velocity of a user's hand movement or finger movement is detected.

[0014] Optionally, the process also includes analyzing the tracked user hand movement to determine the speed of the user hand movement or finger movement, wherein the tracked user hand movement is transformed relative to the user position so that the determined speed includes the speed relative to the user, and when the determined speed is the peak speed of the tracked user hand movement, the tracked user hand movement meets the speed standard.

[0015] Optionally, the peak velocity of the user's hand movement or finger movement is detected by comparing a previously determined velocity with a currently determined velocity and detecting a negative velocity change based on the comparison.

[0016] Optionally, the character sequence is generated by: iteratively detecting that the user hand movement meets a speed criterion based on the tracking; and iteratively parsing character selection according to the user gaze input at a time corresponding to each instance of detecting that the user hand movement meets the speed criterion.

[0017] Optionally, the tracked user hand movement includes alternating movement between the user's hands, and the iterative detection of the user's hand movement satisfying the speed criterion and the iterative resolution of the character selection correspond to the alternating movement between the user's hands.

[0018] According to the present invention, a computing system for triggering virtual keyboard selection using multiple input modes is also provided, the computing system comprising: one or more processors; and one or more memories storing multiple instructions, which, when executed by the one or more processors, cause the computing system to execute a process, the process comprising: displaying a virtual keyboard in an augmented reality environment; tracking user hand movements and user gaze input relative to the virtual keyboard; detecting that the tracked user hand movements satisfy a speed standard; in response to detecting that the tracked user hand movements satisfy the speed standard, resolving character selections based on the user gaze input relative to the virtual keyboard at a time corresponding to detecting that the user hand movements satisfy the speed standard, wherein the resolution is not based on the hand position relative to the virtual keyboard; and generating characters for a character sequence using the resolved character selections.

[0019] Optionally, the virtual keyboard is displayed at a distance greater than 0.5 meters from the user, at a distance beyond the user's arm's reach, or any combination thereof.

[0020] Optionally, the velocity criterion is met when a peak velocity of a user's hand movement or finger movement is detected.

[0021] Optionally, the process also includes analyzing the tracked user hand movement to determine the speed of the user hand movement or finger movement, wherein the tracked user hand movement is transformed relative to the user position so that the determined speed includes the speed relative to the user, and when the determined speed is the peak speed of the tracked user hand movement, the tracked user hand movement meets the speed standard. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a block diagram showing an overview of various devices on which some embodiments of the present technology may operate.

[0023] Figure 2A is a line diagram illustrating a virtual reality headset that may be used in some embodiments of the present technology.

[0024] Figure 2B is a line diagram illustrating a mixed reality head mounted device that may be used in some embodiments of the present technology.

[0025] Figure 2C is a line diagram illustrating a controller that, in some implementations, a user may hold with one or both hands to interact with an artificial reality environment.

[0026] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0027] Figure 4 is a block diagram illustrating components that, in some implementations, may be used in a system employing the disclosed techniques.

[0028] Figure 5 is a schematic diagram showing the placement of a virtual keyboard relative to a user in an artificial reality environment.

[0029] Figure 6 is a schematic diagram illustrating user motion relative to a virtual keyboard that triggers character selection.

[0030] Figure 7 is a schematic diagram showing tracked user motion that triggers virtual keyboard character selection.

[0031] Figure 8 is a flow chart illustrating a process for triggering virtual keyboard selection using multiple input modes in some embodiments of the present technology.

[0032] Fig. 9 is a flow chart illustrating a process for detecting user motion that triggers virtual keyboard selection in some embodiments of the present technology.

[0033] The technology described herein may be better understood by referring to the following detailed description in conjunction with the drawings, in which like reference numbers indicate identical elements or elements with similar functions. DETAILED DESCRIPTION

[0034] Various aspects of the present disclosure relate to triggering a virtual keyboard selection using multiple input modes. An interface manager can display an interface such as a virtual keyboard to a user in an artificial reality environment. The virtual keyboard can be a three-dimensional volume or a two-dimensional display including alphanumeric characters arranged as elements. Various embodiments include an artificial reality system for displaying an artificial reality environment to a user and receiving multiple input modes from the user, and a "virtual keyboard" can be a keyboard provided by various artificial reality systems such as in augmented reality, virtual reality, mixed reality, etc.

[0035] Embodiments of the interface manager can track user eye gaze input and user hand input (e.g., hand motion or finger motion). For example, the interface manager can use one or more sensing devices (e.g., cameras, accelerometers, wearable sensors, etc.) to track eye gaze input to locate the user's gaze on the virtual keyboard. The interface manager can also use one or more sensing devices (e.g., cameras, accelerometers, wearable sensors, controllers, etc.) to track user hand input, such as hand motion (e.g., hand motion and / or finger motion), to detect when the user's hand motion meets the trigger criteria.

[0036] Embodiments of the interface manager may resolve character selections on a virtual keyboard based on detecting that a user's hand movement satisfies trigger criteria. For example, the interface manager may: a) detect that the tracked user hand movement satisfies trigger criteria at a given point in time; and b) resolve selections from the virtual keyboard (e.g., selections of displayed characters) based on the tracked user's gaze at the virtual keyboard at the given point in time. Multiple iterations of the tracked user hand movement satisfying the trigger criteria and resolving selections from the virtual keyboard based on the tracked user's gaze at the virtual keyboard may be performed to generate a character sequence (e.g., a word, a sentence, an alphanumeric character sequence, etc.). In some embodiments, the interface manager may track both hands of the user, and the multiple iterations may correspond to alternating hand movements between the user's hands.

[0037] In some embodiments, the trigger criteria for hand motion can detect a 'tap' motion of a user's hand / finger. For example, a tap motion can include forward motion (e.g., relative to the user), acceleration, and backward motion. In some embodiments, when the tracked user hand motion first moves forward and then reaches a peak velocity or minimum velocity (e.g., a velocity of 0 meters per second (m / s) to indicate that the motion has stopped), it can indicate that the user has performed a tap motion. The trigger criteria can be detecting that the tracked user's forward hand motion has reached a peak velocity, stopped, and / or detecting a retreat in motion (e.g., a backward velocity). When the trigger is detected (e.g., a tap is detected), the interface manager can parse the characters selected by the user's gaze on the virtual keyboard (e.g., the characters selected by the user's gaze when the tap was detected).

[0038] In some embodiments, the alternating motion between the user's hands may include an alternating clicking motion. For example, the user's right hand may perform a first click and trigger a first character selection based on the user's gaze during the first click, and then the user's left hand may perform a second click and trigger a second character selection based on the user's gaze during the second click. This alternating clicking motion may continue to trigger character selection until a character sequence is generated.

[0039] Embodiments of the disclosed technology may include an artificial reality system, or may be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with collected content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality, and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, a "cave" environment or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0040] As used herein, "virtual reality" or "VR" refers to an immersive experience in which the user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to systems in which a user views real-world images after they have passed through a computing system. For example, a tablet with a camera on the back can capture real-world images and then display them on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images as they pass through the system, such as by adding virtual objects. "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is generated in part by a computing system and in part from light reflected from objects in the real world. For example, an MR head-mounted device can be shaped as a pair of glasses with a transparent display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted device, thereby allowing the MR head-mounted device to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of the following: VR, AR, MR, or any combination or hybrid thereof.

[0041] Traditional XR input interfaces fail to provide an experience that meets user expectations, may cause fatigue and / or strain, and may not provide reliable / accurate results. For example, a traditional XR collider virtual keyboard can track user hand movements and identify when hand movements collide with virtual keyboard elements. However, due to limitations on where the virtual keyboard can be displayed, this traditional technology may cause hand / arm fatigue, neck fatigue, and / or eye fatigue, and may be inaccurate. Another XR keyboard can track user gaze to select each character from the virtual keyboard. However, this technology may cause eye strain and produce relatively low accuracy / reliability.

[0042] Embodiments of the disclosed technology are expected to overcome these deficiencies in conventional systems by combining input from multiple modes to support an improved user experience for interacting with an XR input interface. For example, both user gaze and user hand motion may be tracked, where the tracked user gaze may select characters from a virtual keyboard, and the tracked user hand motion may trigger character selection. For example, an interface manager may detect that the tracked user hand motion satisfies a triggering criterion (e.g., a click motion is detected) at a given point in time, and resolve character selections from a displayed virtual keyboard based on the tracked user gaze on the virtual keyboard at the given point in time.

[0043] Because the user's hands / fingers do not collide with the virtual keyboard to select characters, the virtual keyboard can be displayed at a distance optimized for visual convergence to minimize eye strain (e.g., at least 0.5 meters away and 0.8 meters to 1.5 meters away, etc.) and at a location that minimizes neck strain (e.g., within the user's line of sight given the user's gaze). In addition, using a clicking motion as a trigger for gaze-based character selection, a natural and ergonomic user mechanism is implemented that greatly improves the accuracy and time efficiency deficiencies of traditional gaze-based character selection techniques alone. In addition, the combination of gaze-based selection and hand motion triggering allows for a reduction in friction points (e.g., physical fatigue, stress locations, etc.) when the user "types" on the virtual keyboard, which can also result in higher speed and accuracy.

[0044] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1 is a block diagram showing an overview of various devices on which some embodiments of the disclosed technology can run. These devices may include hardware components of the computing system 100 that trigger virtual keyboard selection using multiple input modes. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a stand-alone head-mounted device that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted device and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head-mounted device and other processing operations are transferred to the core processing component. The following is combined with Figure 2A and Figure 2B An example head mounted device is described. In some implementations, the location data and environmental data may be collected solely by sensors incorporated into the head mounted device, while in other implementations, one or more of the plurality of non-head mounted device computing devices may include sensor components that may track environmental data or location data.

[0045] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among the computing devices 101 to 103).

[0046] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be communicated by a hardware controller that interprets the signals received from the input devices and transmits the information to the processors 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0047] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus, or a wireless connection, such as a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus. The processor 110 can communicate with a hardware controller for each device (e.g., for a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices are: a liquid crystal display (LCD) display screen, a light emitting diode (LED) display screen, a projection display, a holographic display, or an augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or network card, a video chip or video card, an audio chip or audio card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0048] In some embodiments, computing system 100 can use input from I / O device 140 (e.g., a camera, a depth sensor, an inertial motion unit (IMU) sensor, a global positioning system (GPS) unit, a laser radar (LiDAR) or other time-of-flight sensor, etc.) to identify and map the user's physical environment while tracking the user's position in the environment. This simultaneous localization and mapping (SLAM) system can generate a map (e.g., a topology, a grid, etc.) of an area (which can be a room, a building, an outdoor space, etc.) and / or obtain a map previously generated by computing system 100 or another computing system that has mapped the area. The SLAM system can track users in the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other position changes, etc.

[0049] The computing system 100 may include a communication device that can communicate wirelessly or wired with other local computing devices or network nodes. The communication device can communicate with another device or server over a network using, for example, the Transmission Control Protocol / Internet Protocol (TCP / IP). The computing system 100 can use the communication device to distribute operations across multiple network devices.

[0050] The processor 110 can access the memory 150, which can be included in one of the multiple computing devices of the computing system 100, or can be distributed on multiple computing devices of the computing system 100, or can be distributed on multiple other external devices. The memory includes one or more hardware devices for volatile storage or non-volatile storage, and can include both read-only memory and writable memory. For example, the memory may include one or more of the following: random access memory (RAM), various cache memories, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, and tape drive. The memory is not a propagation signal separated from the underlying hardware, so the memory is non-transient. The memory 150 may include a program memory 160, which stores programs and software, such as an operating system 162, an interface manager 164, and other applications 166. Memory 150 may also include data storage 170, which may include, for example, predictive models, threshold data, standard data, alphanumeric character data, configuration data, settings, user options or preferences, etc., which may be provided to program memory 160 or any element of computing system 100.

[0051] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that may be suitable for use with the technology include, but are not limited to: XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices, etc.

[0052] Figure 2A2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a band 210. Front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computing units 230. Position sensor 220, IMU 215, and computing unit 230 may be located inside HMD 200 and may not be visible to the user. In various embodiments, IMU 215, position sensor 220, and locator 225 may track the movement and position of HMD 200 in the real world and in an artificial reality environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locator 225 may emit infrared beams that produce light spots on real objects around HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position sensors, force sensors, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect the light points. The computing unit 230 in the HMD 200 may use the detected light points to extrapolate the position and movement of the HMD 200, and recognize the shape and position of real objects around the HMD 200.

[0053] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., micro-LED, laser (LASER)), etc.), some other display, or some combination thereof.

[0054] In some embodiments, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., through light emitted from the HMD 200), and the PC can use the HMD in conjunction with the outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0055] Figure 2B 2 is a line diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 Gigahertz (GHz) link) as indicated by link 256. In other embodiments, the mixed reality system 250 includes only a head mounted device without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a transparent display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as a light projector (e.g., lasers, LEDs, etc.), a camera, an eye tracking sensor, a micro-electromechanical system (MEMS) component, a network component, etc.

[0056] The projector can be coupled to a transparent display 258, for example, via an optical element to display media to the user. The optical element can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. for guiding light from the projector to the user's eyes. Image data can be transmitted from the core processing component 254 to the HMD 252 via a link 256. The controller in the HMD 252 can convert the image data into light pulses from the projector, which can be transmitted to the user's eyes via an optical element as output light. The output light can be mixed with the light passing through the display 258, allowing the output light to present virtual objects that appear to exist in the real world.

[0057] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example, with 3DoF or 6DoF, track parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects to appear stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0058] Figure 2C Controllers 270 (including controllers 276A and 276B) are shown, and in some embodiments, the user can hold controllers 270 with one or both hands to interact with the artificial reality environment presented by HMD 200 and / or HMD 250. Controllers 270 can communicate with the HMD directly or via an external device (e.g., core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit light points farther away. The HMD 200 or HMD 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., to track the controllers with 3DoF or 6DoF). The computing unit 230 or core processing component 254 in the HMD 200 can use this tracking to monitor the position and movement of the user's hands in combination with the IMU output and the position output. The controller may also include various buttons (eg, buttons 272A-272F) and / or joysticks (eg, joysticks 274A and 274B) that a user may actuate to provide input and interact with objects.

[0059] In various embodiments, HMD 200 or HMD 250 may also include additional subsystems (e.g., an eye tracking unit, an audio system, various network components, etc.) to monitor the user's interactions and indications of intent. For example, in some embodiments, instead of or in addition to a controller, one or more cameras included in HMD 200 or HMD 250 or from an external camera can monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources can illuminate one or both eyes of the user, and HMD 200 or HMD 250 can use an eye-facing camera to capture reflections of that light to determine eye position (e.g., based on a set of reflections around the user's cornea), thereby modeling the user's eyes and determining gaze direction.

[0060] Figure 3305B may be a HMD 200 or a HMD system 250. The client computing device 305 may be a logical connection to one or more remote computers (e.g., a server computing device) through a network 330 to operate in a network environment.

[0061] In some implementations, server 310 may be an edge server that receives multiple client requests and coordinates the fulfillment of these requests through other servers (e.g., servers 320A to 320C). Server computing device 310 and server computing device 320 may include a computing system, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, multiple server computing devices may each be a distributed computing environment that includes multiple computing devices located at the same physical location or located at geographically different physical locations.

[0062] Client computing device 305, and server computing devices 310 and 320 can all act as servers or clients of one or more other servers / client devices. Server 310 can be connected to database 315. Servers 320A to 320C can each be connected to corresponding databases 325A to 325C. As discussed above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share a database, or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can all be distributed computing environments that include multiple computing devices, can be located in their corresponding servers, or can be located at the same physical location or at geographically different physical locations.

[0063] The network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. The network 330 may be the Internet or some other public or private network. The client computing device 305 may be connected to the network 330 via a network interface, for example, by wired or wireless communication. Although the connections between the server 310 and the plurality of servers 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including the network 330 or a separate public or private network.

[0064] Figure 4 4 is a block diagram showing various components 400, which in some embodiments can be used in a system that employs the disclosed technology. Component 400 can be included in one device of computing system 100, or can be distributed across multiple devices of computing system 100. Component 400 includes hardware 410, middleware 420, and multiple dedicated components 430. As discussed above, a system that implements the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.) and a storage memory 418. In various embodiments, the storage memory 418 can be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, the storage memory 418 can be one or more hard drives or flash drives accessible via a system bus, or can be a cloud storage provider (e.g., located in storage device 315 or 325) or other network storage device accessible via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device (eg, client computing device 305 ), or on a server computing device (eg, server computing devices 310 or 320 ).

[0065] The middleware 420 may include components that coordinate resources between the hardware 410 and the plurality of specialized components 430. For example, the middleware 420 may include an operating system, services, a driver, a basic input output system (BIOS), a controller circuit, or other hardware or software systems.

[0066] Each dedicated component 430 may include the following software or hardware: the software or hardware is configured to perform operations for triggering virtual keyboard selection using multiple input modes. Each dedicated component 430 may include an input manager 434, a user tracker 436, a fragment generator 438, one or more prediction models 440, and components and application programming interfaces (APIs) (e.g., interface 432), which can be used to provide a user interface, transmit data, and control the multiple dedicated components. In some embodiments, component 400 may be located in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more of the multiple dedicated components 430. Although multiple dedicated components 430 are depicted as separate components, the multiple dedicated components 430 may be logical or other non-physical distinctions of functionality, and / or may be sub-modules or code blocks of one or more applications.

[0067] The input manager 434 can manage the display of a virtual interface (e.g., a virtual keyboard) and various input modes from the user to perform selections (e.g., character selections) from the virtual interface. For example, the input manager 434 can display a virtual keyboard within the user's line of sight. In some embodiments, the virtual keyboard can be displayed at a threshold distance from the user (or within a range of distances from the user). For example, the virtual keyboard can be displayed at the following locations: at least 0.5 meters away from the user, beyond the user's arm's length, any combination of these, or any other suitable distance. Various embodiments track the user's gaze, and the input manager 434 can control the display of the virtual keyboard so that the virtual keyboard continues to be within the user's line of sight given user movement (e.g., head turn, body turn, etc.).

[0068] Various embodiments of the input manager 434 can receive tracked user gaze input and tracked user hand input, and use these tracked inputs in combination to trigger one or more character selections from the displayed virtual keyboard. For example, the input manager 434 can compare the tracked user hand input (e.g., tracked hand movement and / or finger movement) with trigger criteria. When the tracked user hand input meets the trigger criteria, the input manager 434 can parse the virtual keyboard character selected by the tracked user gaze input when the tracked user hand input meets the trigger criteria.

[0069] For example, the tracked user hand input can be hand motion and / or finger motion. The input manager 434 can determine the speed of the hand / finger motion based on the tracking (e.g., by inferring the distance of motion over time). When the determined speed meets the speed standard, the input manager 434 can trigger the parsing of the character selected from the displayed virtual keyboard (e.g., based on the user gaze input). In some embodiments, the tracked hand / finger motion can be a click motion. An example user click motion can involve one or more fingers extending from the user's hand, where the user's hand / finger moves forward, accelerates, stops, and moves backward (relative to the user's body). Any other suitable motion can represent the user's click motion.

[0070] In some embodiments, the speed criterion includes detecting that the user's hand / finger has reached a minimum speed (e.g., about 0 m / s) after performing a forward motion (e.g., relative to the user). For example, when the user's hand / finger exceeds a first speed threshold, it can represent the forward motion in the click motion. When the determined speed drops below the minimum speed threshold (e.g., about 0 m / s) after the forward motion, it can represent the portion of the click motion when it reaches the target (e.g., when the hand / finger reaches the click target). In some embodiments, the speed criterion includes a first speed threshold and a minimum speed threshold. In this example, when the hand / finger forward speed exceeds the first speed threshold and at a future time point (e.g., 0.05 seconds, 0.1 seconds, any other fraction of a second, or any other suitable time point), the hand / finger motion being tracked meets the speed criterion. In this example, the time at which the speed criterion is met can correspond to the time at which the user's click motion reaches the click target.

[0071] In some embodiments, the speed criterion includes the detection of peak speed, such as detecting that the user's hand / finger has reached the peak speed after performing the forward motion. For example, when the user's hand / finger exceeds the first speed threshold, it can represent the forward motion in the click motion. When the determined speed reaches the peak speed after the forward motion, it can represent the part when the click motion reaches the target (for example, when the hand / finger reaches the click target). In some embodiments, the speed criterion includes the first speed threshold and the detection of the peak speed. In this example, when the hand / finger forward speed exceeds the first speed threshold and at a future time point (for example, 0.05 seconds, 0.1 seconds, any other fraction of a second, or any other suitable time point), the peak speed of the hand / finger forward speed is detected, the tracked hand / finger motion meets the speed criterion.

[0072] In some embodiments, the detection of peak velocity can be based on a comparison between the current velocity and the past velocity. For example, the determined previous time forward velocity of the user's hand / finger motion (e.g., calculated by processing the visual frames collected in the past) can be compared with the current time forward velocity of the user's hand / finger motion (e.g., calculated by processing the visual frames collected currently and one or more visual frames collected in the past). When the comparison indicates a speed drop, it can be determined that the peak velocity has been reached. In this example, the time when the speed standard is met can correspond to the time when the user's click motion reaches the click target. In some embodiments, the prediction model can be configured to predict the peak velocity of the tracked hand / finger motion. For example, the prediction model can predict the hand / finger motion peak velocity before the hand / finger motion occurs. The prediction of peak velocity can make the virtual keyboard more responsive. In some embodiments, the prediction model can be a machine learning model that is provided with training data that includes historical hand / finger motion data and a label for peak velocity (or the peak velocity label can be determined by the model).

[0073] Once the determined speed satisfies the speed criteria, the input manager 434 can resolve the characters selected by the gaze input tracked at the same point in time. For example, the input manager 434 can determine that the tracked user hand input (e.g., hand / finger movement) meets the trigger criteria (e.g., speed criteria) at a specific point in time, and the user gaze input at the specific point in time can correspond to a position on the virtual keyboard. The input manager 434 can resolve the character selection from the virtual keyboard based on the position on the virtual keyboard. For example, the position can be within the boundaries of the element of the virtual keyboard corresponding to the selected character, or the input manager 434 can perform one or more adjustments or predictions (e.g., using one or more prediction models 440) to resolve the selection between two, three, four or more virtual keyboard elements / characters.

[0074] In some embodiments, the speed criterion includes detecting a change in direction of the user's hand / finger movement. In one example, where forward motion (e.g., motion moving away from the user's torso in the direction of the user's gaze relative to the user's body) is represented as a positive speed value and backward motion (e.g., motion moving backward toward the user's torso relative to the user's body) is represented as a negative speed value, the speed criterion can be detecting a change in the sign (e.g., positive or negative) of the determined speed. When a change in direction of the determined speed is detected, it can represent the portion of the click motion immediately after the user's hand / finger reaches the click target (e.g., the retraction of the hand / finger after reaching the click target). In this example, the time at which the user's click motion reaches the click target can correspond to the time immediately before the determined speed is detected to meet the speed criterion.

[0075] In this example, once the input manager 434 detects that the determined speed meets the speed criteria, the input manager 434 can parse the characters selected by the user's gaze input at the time point immediately before the detection time. In some embodiments, when the time at which the determined speed meets the speed criteria is immediately after the time at which the user's hand / finger movement reaches the target, compensation can be performed when parsing the character selection based on the user's gaze input. For example, when a visual sensor (e.g., a camera) is used to track the user's hand movement input, one or more visual frames before the visual frame that triggers the detection of the determined speed meeting the speed criteria can be used to parse the selected characters. The input manager 434 can parse the character selection based on the user's gaze tracked at the time of one or more visual frames before the visual frame that triggers the detection. In some embodiments, the position of the selected keyboard key can be independent of the position at which the user makes a movement that meets the speed criteria-for example, the character selection position can be based entirely on the eye gaze direction. Any other suitable compensation can be implemented to parse the character selection based on the user's gaze input before or after the determined speed is detected to meet the speed criteria.

[0076] The user tracker 436 can track the user's body to obtain various modes of user input. For example, the user tracker 436 can track the user's eye gaze based on eye tracking, head tracking, and face tracking. One or more sensors of the XR system (e.g., cameras, wearable sensors such as one or more accelerometers, etc.) can sense the user's body state (e.g., head movement, eye movement, etc.) and the user tracker 436 can receive these sensed states to track the user's eye gaze. In another example, the user tracker 436 can also track the movement of one or both hands of the user. One or more sensors of the XR system (e.g., cameras, wearable sensors such as one or more accelerometers, handheld controllers, etc.) can sense the user's body state (e.g., hand movement, wrist movement, arm movement, and finger movement, etc.) and the user tracker 436 can receive these sensed states to track the user's hand movement. The tracked user gaze and the tracked user hand movement can be provided to the input manager 434 as the tracked user gaze input and the tracked user hand input.

[0077] The fragment generator 438 can generate a sequence of characters, such as a word or sentence, corresponding to the character selection parsed by the input manager 434. For example, the input manager 434 can parse the sequence of selected characters based on the tracked user hand input and the tracked user gaze input. In some embodiments, the fragment generator 438 can select from the outputs of one or more prediction models 440. For example, one or more prediction models 440 can predict the word being entered by the user based on an initial set of characters. The prediction may include multiple predicted words, each predicted word having a weighted probability. The fragment generator 438 can select from multiple word predictions using the user's historical usage data. In some embodiments, the user's historical usage data can be used to train a machine learning model to select from the word predictions of one or more prediction models 440.

[0078] One or more prediction models 440 may predict character selections or combinations of character selections. In some embodiments, one or more prediction models 440 may include a language model that stores character sequences and / or words in association with probabilities of use. A language model may be used to configure one or more prediction models 440 and / or to select among outputs generated by one or more prediction models 440. One or more prediction models 440 may include any other suitable character prediction model known to those of ordinary skill in the art.

[0079] The one or more prediction models 440 may be machine learning models configured to receive tracked user gaze input and predict an element from a virtual interface (e.g., an alphanumeric character from a virtual keyboard) that a user intends to select using the tracked user gaze. The one or more models may be trained using training data that includes example instances of tracked user gaze input and labels of selected alphanumeric characters.

[0080] Various embodiments display a virtual keyboard that supports gaze-based character selection at a distance from the user. Character input in an XR environment can be achieved through various techniques, such as: key selection of a virtual keyboard using a collider (e.g., detecting that the tracked user hand motion overlaps a two-dimensional area or a three-dimensional volume of a virtual key), gaze-based character selection, and other suitable techniques. However, the distance and placement of the displayed virtual keyboard can improve the user experience.

[0081] Figure 5 5 is a schematic diagram showing placement of a virtual keyboard relative to a user in an artificial reality environment. Schematic diagram 500 includes user 502, XR system 504, virtual keyboard 506, distance 508, user 510, XR system 512, virtual keyboard 514, distance 516, user 518, XR system 520, virtual keyboard 522, and distance 524. XR system 504 displays virtual keyboard 506 to user 502 at distance 508, XR system 512 displays virtual keyboard 514 to user 510 at distance 516, and XR system 520 displays virtual keyboard 522 to user 518 at distance 524.

[0082] The display of virtual keyboard 506 can combine several advantages that cannot be achieved by the display of virtual keyboards 514 and 522. For example, distance 508 can achieve a convergence distance to user 502 that reduces eye fatigue and enhances visual clarity. For example, distance 508 can be about 1 meter, or between 0.8 meters and 1.5 meters. Distances 516 and 524 that are less than distance 508 cannot achieve the benefits of the convergence distance.

[0083] XR systems 512 and 520 are also configured to receive input from users 510 and 518 based on hand / finger interactions between user 510 and virtual keyboard 514 and user 518 and virtual keyboard 522 (e.g., virtual keyboards 514 and 522 are collider keyboards). Due to this input approach, virtual keyboards 514 and 522 are displayed within arm's reach of users 510 and 518, requiring distances 516 and 518 to be less than the optimal distance for convergence / reduced eyestrain.

[0084] Additionally, the virtual keyboard 514 is displayed near the vertical midpoint of the user 510, thereby enabling more comfortable hand / finger interaction with the virtual keyboard 514. However, this requires the user 510 to bend at the neck to see the virtual keyboard 514, thereby causing neck strain over time. The display position of the virtual keyboard 522 reduces neck strain for the user 518 because the virtual keyboard is displayed at shoulder height. However, over time, hand / finger interaction with the virtual keyboard 522 will cause arm fatigue.

[0085] The virtual keyboard 506 is displayed in a vertical position that does not require the user 504 to bend her neck, thereby reducing neck strain issues. In addition, the virtual keyboard 506 does not require overlap between the user's 502 hands / fingers and the virtual keys of the virtual keyboard 506. Therefore, arm fatigue is also reduced or eliminated. The XR system 504 is configured to receive a combination of user hand input and user gaze input relative to the virtual keyboard 506, which supports the selection of characters from the virtual keyboard 506 using techniques that improve the user experience.

[0086] Figure 6 600 includes a tracked user hand 602, a virtual keyboard 604, and a tracked user gaze 606. In some implementations, diagram 600 illustrates a technique for XR system 504 to support character selection by user 502 at virtual keyboard 506.

[0087] The tracked user hand 602 can be tracked by the XR system using any suitable sensor, such as: one or more cameras, wearable sensors, handheld controllers, and any other suitable sensors. The virtual keyboard 604 can be displayed in a location that the tracked user hand 602 cannot reach. The tracked user gaze 606 can be tracked relative to the virtual keyboard 604 so that the position on the virtual keyboard corresponding to the tracked user gaze can be resolved. Instead of making character selections based on the position on the virtual keyboard 604 targeted by the tracked user hand 602, character selections can be resolved based on the position on the virtual keyboard 604 targeted by the tracked user gaze 606.

[0088] The tracked user hand 602 can perform a trigger action (e.g., a push motion) that triggers character selection based on the tracked user gaze 606. For example, the tracked user hand 602 depicts the user hand in an orientation to perform a push motion, such as extending one finger from the user's hand with the remaining fingers bent backward. Any other suitable hand / finger orientation can be implemented. The user's hand can then perform a push motion including forward motion, acceleration, stop, and backward motion.

[0089] In some embodiments, when the user's hand slows down or stops, it can indicate a triggering moment for triggering the following operations: parsing the character selection from the virtual keyboard 604 according to the tracked user gaze 606. The perceived movement of the tracked user hand 602 can be compared to the trigger criteria, and the trigger criteria can be configured to detect the target moment when the tracked user hand 602 reaches the target portion of the push motion. The tracked user gaze 606 relative to the virtual keyboard 604 at the target moment can be used to parse the character selection from the virtual keyboard. In some embodiments, the algorithm / model can parse the character selection according to the tracked user gaze 606 at the target moment.

[0090] In some embodiments, the tracked user hand 602 can perform a push motion using both hands alternately, and each time a target moment is detected, the user gaze tracked at the target moment can be used to resolve the character selection. Thus, the user can use the alternating motion of the tracked user hand 602 and the tracked user gaze 606 to enter a character sequence.

[0091] Figure 7 Schematic diagram showing the tracked user motion that triggers the selection of virtual keyboard characters. Schematic diagram 700 includes diagram 702, graph 704, graph 706, point 708, and point 710. Diagram 702 shows the tracked user movement, including the tracked movement of the user's hands. Graphs 704 and 706 correspond to the trigger criteria metric / value for each tracked user's hand. For example, graph 704 may correspond to the tracked user's left hand motion, and point 708 corresponds to the time point when the tracked user's left hand motion meets the trigger criteria (e.g., when the plotted metric value meets the threshold).

[0092] As time goes by, the left hand of the user performs several pushing motions, and point 708 is relevant to the moment when the left hand motion of the user being tracked meets the triggering criterion. At these moments, the character from the virtual keyboard can be parsed according to the user's gaze input for the virtual keyboard being tracked. Similarly, graphics 706 can correspond to the user's right hand motion being tracked, and point 710 corresponds to the time point when the user's right hand motion being tracked meets the triggering criterion. In some embodiments, the algorithm / model can be based on the user's gaze at the tracking of points 708 and / or 710 places to parse the character selection. As shown in graphics 704, point 708, graphics 706 and point 710, the hand motion that triggers the character selection according to the user's gaze at the tracking can alternate between the left hand and the right hand of the user, and a single hand can continuously perform multiple hand motions, any combination of these hand motions, or can realize any other suitable hand motion combination.

[0093] Those skilled in the art will understand that the above described Figures 1 to 7 The components shown in the flowcharts and in each of the flowcharts discussed below may be changed in various ways. For example, the order of the logic may be rearranged, multiple sub-steps may be performed in parallel, the logic shown may be omitted, other logic may be included, etc. In some embodiments, one or more of the multiple components described above may perform one or more of the multiple processes described below.

[0094] Figure 8 800 is a flow chart illustrating a process for triggering virtual keyboard selection using multiple input modes in some embodiments of the present technology. In some embodiments, process 800 may be performed by an application or operating system of an XR system to receive input from a user. The input received from the user may configure the application, the operating system, any other element of the XR system, or be transmitted to any other suitable computing device.

[0095] At block 802, process 800 may display a virtual keyboard to a user. For example, the XR system may include a display component that displays the virtual keyboard as a two-dimensional display content or a three-dimensional volume. The virtual keyboard may be configured as a pattern of virtual keys corresponding to characters. The virtual keyboard may be displayed at a distance from the user that is greater than 0.5 m, between 0.8 m and 1.5 m, beyond arm's length from the user, any combination thereof, or any other suitable distance.

[0096] At block 804, process 800 may track user gaze and user hand movement to determine user gaze input and user hand input. For example, one or more sensors (e.g., a camera, an accelerometer, a wearable sensor, a handheld controller, a component of an inertial measurement unit (IMU), etc.) may track the user's body, including the user's head movement, eye movement, arm movement, hand movement, finger movement, torso movement, and any other suitable movement. The sensed movement may be tracked to generate tracked user gaze input and tracked user hand / finger input (e.g., user hand movement).

[0097] At block 806, process 800 may determine whether the tracked user hand movement meets the trigger criteria. For example, the tracked user hand movement may be compared to the trigger criteria. In some implementations, the trigger criteria may be a speed criterion. For further disclosure regarding the tracked user hand movement meeting the trigger criteria, see Fig. 9 When the tracked user hand movement meets the trigger criteria, the process 800 can proceed to block 808. When the tracked user hand movement does not meet the trigger criteria, the process 800 can proceed to block 810.

[0098] At block 808, process 800 may resolve a character selection from the displayed virtual keyboard based on the user gaze input. For example, the position of the tracked user gaze input relative to the displayed virtual keyboard may be determined. The determined position may be within the boundaries of the element of the virtual keyboard corresponding to the selected character, or one or more adjustments or predictions may be performed (e.g., using one or more prediction models) to resolve the selection between two, three, four, or more characters. In some embodiments, the position may be based only on the gaze direction at the time of the click gesture (i.e., not based on the location where the user made the click gesture).

[0099] In some embodiments, when the tracked user hand input meets the trigger criteria, the selected character can be parsed according to the tracked user gaze input at the detection time (or within a threshold time from the detection time) when the tracked user hand input meets the trigger criteria. For example, the trigger criteria can be a speed criterion, and when the tracked hand movement or finger movement is detected to meet the speed criterion, the character parsing according to the tracked gaze input can be triggered. The character selection can be parsed only according to the gaze input tracked when the hand movement or finger movement is detected to meet the speed criterion, or according to the gaze input tracked at the adjustment time (e.g., a predetermined time adjustment in the past or future) from the detection time. For example, a user can look at a key and start making a click gesture, and look at her next expected key before the click gesture is completed. The predetermined adjustment time can be a quantity determined for the user, which is generally used for the length of time the user looks at the expected key before the trigger criteria, or the predetermined adjustment time can be a quantity specific to the current user, which is determined by the loading process of the user entering a known phrase, or based on the user's use of the system over time and the association of the input correction made by the user with the time value. In some embodiments, the adjustment of time can be determined by a trained machine learning model. For example, the machine learning model can receive training data including historical hand / finger motion data over time and labels of expected detection times and / or target keys. The machine learning model can learn trends using hand / finger motion and intent of multiple users to determine effective time adjustments.

[0100] At block 810, process 800 may determine whether end criteria are met. For example, the user may signal the end of the input session using the virtual keyboard (e.g., by selecting a close element, performing a predetermined gesture, or any other suitable manner), or an idle timer may expire without receiving user input (e.g., without a tracked user hand movement that satisfies the trigger condition). When the end criteria are met, process 800 may proceed to block 812. When the end criteria are not met, process 800 may loop back to block 804, where process 800 tracks user gaze input and user hand movement to resolve additional character selections.

[0101] At block 812, process 800 can end display of the virtual keyboard. For example, the virtual keyboard can stop being displayed, and the XR system can transition to another interface, display mode, or operating mode. Implementations of process 800 can be performed to generate a character sequence corresponding to the user input.

[0102] Fig. 9900 is a flow chart illustrating a process for detecting a user motion that triggers a virtual keyboard selection in some embodiments of the present technology. In some embodiments, process 900 may be performed by an application or operating system of an XR system to receive input from a user. The input received from the user may configure the application, the operating system, any other element of the XR system, or be transmitted to any other suitable computing device. Process 900 may be performed in conjunction with Figure 8 The process 800 is combined to trigger character selection from a virtual keyboard.

[0103] At box 902, process 900 can determine the current speed of the user's hand movement. For example, the tracked user hand input can correspond to hand movement and / or finger movement, and the speed of the hand / finger movement can be determined by inferring the movement distance over time. In some embodiments, one or more cameras are used to track the user's hand movement, and multiple visual frames collected can be processed to locate the user's hand / fingers in each frame. By processing consecutive visual frames, the distance moved by the user's hand / fingers over the time span of these frames can be inferred to determine the speed of the user's hand movement.

[0104] In some embodiments, the tracked user hand motion is transformed relative to the user position (e.g., the tracked user body position / gaze) so that the determined velocity includes a velocity relative to the user. The determined velocity of the user hand motion can also be filtered and / or adjusted. For example, a first axis component (e.g., an x-axis component) of the tracked user hand motion can be filtered. In another example, a second axis component (e.g., a y-axis component) of the tracked user hand motion can be reversed.

[0105] At block 904, process 900 may compare the current speed of the user's hand movement to a past speed of the user's hand movement. For example, a previous speed of the user's hand movement (e.g., calculated using past visual frames) may be compared to the current speed of the user's hand movement, and the comparison may be stored. At block 906, process 900 may determine whether a change in speed is detected that satisfies a triggering criterion. For example, the triggering criterion may include a speed change criterion.

[0106] In some embodiments, the speed change criterion may be detection of a decrease in speed that indicates that the peak speed has been reached. For example, when a comparison between the current forward speed and the past forward speed shows a decrease in speed, the comparison indicates that the peak speed has been reached. In some embodiments, the speed criterion is met when the stored comparison indicates a decrease in the determined speed.

[0107] In some embodiments, the speed change criterion can be the detection of a change in sign (e.g., direction of motion) indicating that a change in speed direction has been detected. In one example, where forward motion (e.g., motion away from the user's torso in the user's gaze direction relative to the user's body) is represented as a positive speed value and backward motion (e.g., motion moving backward toward the user's torso relative to the user's body) is represented as a negative speed value, the speed criterion can be the detection of a change in sign (e.g., positive or negative). In some embodiments, the speed criterion is met when the stored comparison indicates a change in sign of the determined speed.

[0108] When the speed change meets the trigger criteria, process 900 can proceed to block 908. When the speed change does not meet the trigger criteria, process 900 can proceed to block 910. At block 908, process 900 can trigger character selection using the displayed virtual keyboard based on the user gaze input. For example, when the trigger criteria are met, the process 900 can trigger Figure 8 Frame 808.

[0109] At block 910, process 900 may move to the next instance of user hand motion tracking. In some implementations, the user hand motion may be tracked using the acquired visual frames. The next instance of user hand motion tracking may be the next visual frame (or next batch of visual frames) that may be used to determine an updated current velocity of the user hand motion. Any other suitable tracking and next tracking instance may be implemented.

[0110] In this specification, references to "embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described with respect to that embodiment is included in at least one embodiment of the present disclosure. The appearances of these phrases in different places in the specification do not necessarily all refer to the same embodiment, nor are they necessarily separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be present in some embodiments but not in other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not in other embodiments.

[0111] As used herein, being higher than a threshold value means that the value of the item being compared is higher than other values ​​specified, the item being compared is located in an item with a maximum value, a certain specified number, or the item being compared has a value within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of the item being compared is lower than other values ​​specified, the item being compared is located in an item with a minimum value, a certain specified number, or the item being compared has a value within a specified minimum percentage value. As used herein, being within a threshold value means that the value of the item being compared is between two other values ​​specified, the item being compared is located in an item with a middle specified number, or the item being compared has a value within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting fast connection" may be understood to mean selecting a connection with a value corresponding to the connection speed with which it is higher than a threshold value that is assigned.

[0112] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items, such as A and A; B, B and C; A, A, B, C and C; etc.

[0113] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except as limited by the appended claims.

[0114] Any patent, patent application and other references mentioned above are incorporated herein by reference. If necessary, various aspects may be modified to provide further embodiments using the systems, functions and concepts of the various references mentioned above. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.

Claims

1. A method for triggering a virtual keyboard selection using multiple input modes, the method comprising: Display a virtual keyboard in an augmented reality environment; tracking user hand movements and user gaze input relative to the virtual keyboard; detecting that the tracked hand movement of the user satisfies a speed criterion; In response to detecting that the tracked user hand movement satisfies the velocity criterion, resolving a character selection based on the user gaze input relative to the virtual keyboard at a time corresponding to detecting that the user hand movement satisfies the velocity criterion, wherein the parsing is not based on hand position relative to the virtual keyboard; as well as Generate characters for a character sequence using the parsed character selection.

2. The method according to claim 1, wherein: The virtual keyboard is displayed at a distance greater than 0.5 meters from the user, at a distance beyond the user's arm's reach, or any combination thereof; and / or The speed standard is met when the peak speed of the user's hand movement or finger movement is detected.

3. The method according to claim 2, further comprising: Analyze the tracked user hand movement to determine the speed of the user hand movement or finger movement, wherein the tracked user hand movement is transformed relative to the user position so that the determined speed includes the speed relative to the user, and when the determined speed is the peak speed of the tracked user hand movement, the tracked user hand movement meets the speed standard.

4. The method according to claim 3, wherein: Analyzing the tracked user hand movement to determine the speed of the user hand movement or finger movement includes: filtering out a first axis component of the tracked user hand movement and inverting a second axis component of the tracked user hand movement; and / or Therein, the peak velocity of the user's hand movement or finger movement is detected by comparing a previously determined velocity with a currently determined velocity and detecting a negative change in velocity based on the comparison.

5. The method according to claim 1, wherein: The character sequence is generated in the following way: Iteratively detecting, based on the tracking, that the user's hand movement satisfies a speed criterion; and At a time corresponding to each instance in which the user hand movement is detected to satisfy the speed criterion, character selections are iteratively resolved based on the user gaze input.

6. The method according to claim 5, wherein: The tracked user hand motion includes alternating motion between the user's hands, and the iterative detection of the user hand motion satisfying the speed criterion and the iterative parsing of the character selection correspond to the alternating motion between the user's hands.

7. A computer-readable storage medium storing a plurality of instructions, which, when executed by a computing system, cause the computing system to perform a process for triggering a virtual keyboard selection using a plurality of input modes, the process comprising: Display a virtual keyboard in an augmented reality environment; tracking user hand movements and user gaze input relative to the virtual keyboard; detecting that the tracked hand movement of the user satisfies a speed criterion; In response to detecting that the tracked user hand movement satisfies the velocity criterion, resolving a character selection based on the user gaze input relative to the virtual keyboard at a time corresponding to detecting that the user hand movement satisfies the velocity criterion, wherein the parsing is not based on hand position relative to the virtual keyboard; as well as Generate characters for a character sequence using the parsed character selection.

8. The computer-readable storage medium according to claim 7, wherein: The virtual keyboard is displayed at a distance greater than 0.5 meters from the user, at a distance beyond the user's arm's reach, or any combination thereof; and / or The speed standard is met when the peak speed of the user's hand movement or finger movement is detected.

9. The computer-readable storage medium of claim 8, wherein: The process also includes: Analyze the tracked user hand movement to determine the speed of the user hand movement or finger movement, wherein the tracked user hand movement is transformed relative to the user position so that the determined speed includes the speed relative to the user, and when the determined speed is the peak speed of the tracked user hand movement, the tracked user hand movement meets the speed standard.

10. The computer-readable storage medium of claim 9, wherein: Analyzing the tracked user hand movement to determine the speed of the user hand movement or finger movement includes: filtering out a first axis component of the tracked user hand movement and inverting a second axis component of the tracked user hand movement; and / or Therein, the peak velocity of the user's hand movement or finger movement is detected by comparing a previously determined velocity with a currently determined velocity and detecting a negative change in velocity based on the comparison.

11. The computer-readable storage medium according to claim 7, wherein: The character sequence is generated in the following way: Iteratively detecting, based on the tracking, that the user's hand movement satisfies a speed criterion; as well as At a time corresponding to each instance in which the user hand movement is detected to satisfy the speed criterion, character selections are iteratively resolved based on the user gaze input.

12. The computer-readable storage medium of claim 11, wherein: The tracked user hand motion includes alternating motion between the user's hands, and the iterative detection of the user hand motion satisfying the speed criterion and the iterative parsing of the character selection correspond to the alternating motion between the user's hands.

13. A computing system for triggering virtual keyboard selection using multiple input modes, the computing system comprising: one or more processors; as well as one or more memories storing a plurality of instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: Display a virtual keyboard in an augmented reality environment; tracking user hand movements and user gaze input relative to the virtual keyboard; detecting that the tracked hand movement of the user satisfies a speed criterion; In response to detecting that the tracked user hand movement satisfies the velocity criterion, resolving a character selection based on the user gaze input relative to the virtual keyboard at a time corresponding to detecting that the user hand movement satisfies the velocity criterion, wherein the resolving is not based on a hand position relative to the virtual keyboard; and Generate characters for a character sequence using the parsed character selection.

14. The system according to claim 13, wherein: The virtual keyboard is displayed at a distance greater than 0.5 meters from the user, at a distance beyond the user's arm's reach, or any combination thereof; and / or The speed standard is met when the peak speed of the user's hand movement or finger movement is detected.

15. The system of claim 14, wherein: The process also includes: Analyze the tracked user hand movement to determine the speed of the user hand movement or finger movement, wherein the tracked user hand movement is transformed relative to the user position so that the determined speed includes the speed relative to the user, and when the determined speed is the peak speed of the tracked user hand movement, the tracked user hand movement meets the speed standard.