Artificial reality input using multiple modalities

Through the multi-modal input method, combining gaze and touchpad input, dynamically move virtual keyboard indicators and using predictive text models, the user fatigue and inaccurate input caused by traditional virtual keyboards are solved, achieving a more efficient and natural virtual Keyboard interaction.

CN119013641APending Publication Date: 2024-11-22CTRL-LABS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380032867.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-08
Filing Date
2023-04-09
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Traditional artificial reality input interfaces cannot provide an experience that meets users' expectations, resulting in user fatigue and inaccurate input, especially the design of virtual keyboards has problems with hand and eye strain.

Method used

The multimodal input method is adopted, combining gaze input and touchpad input, dynamically move and parse sliding segments through indicators on the virtual keyboard, and character candidate selection is generated using the predicted text model to achieve more natural and efficient character selection.

Benefits of technology

Improves the speed and accuracy of character selection, reduces user fatigue, supports natural virtual keyboard interaction, reduces hardware costs, and improves input reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013641A_ABST
    Figure CN119013641A_ABST
Patent Text Reader

Abstract

Aspects of the present disclosure relate to an interface for receiving input using multiple modalities in an artificial reality environment. The interface may be a virtual keyboard displayed in an artificial reality environment, the virtual keyboard including a plurality of characters arranged as a plurality of elements. Various embodiments include an artificial reality device / system for displaying an artificial reality environment and receiving user input in a first modality, and a controller device for receiving user input in an additional input modality. For example, an artificial reality system may be configured to receive a user gaze input as a first input modality, and a controller device may be configured to receive an input in a second modality, such as a touch input received at a touchpad. An interface manager may process inputs in multiple modalities to control indicators on a virtual interface. The interface manager may also parse character selections in the virtual interface according to the input.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a virtual interface for receiving input using multiple modalities in an artificial reality environment. Background Art

[0002] Artificial reality devices provide users with the opportunity to experience artificial reality, augmented reality, and / or virtual reality. However, user interaction in these environments continues to present challenges. Often, user input is inaccurate or slow, leading to user frustration. User input can take many forms, such as tracked motion, eye gaze, touch, and joysticks. These various inputs present challenges in matching the implemented functionality with the user's intended functionality. In addition, different input opportunities also present ergonomic challenges, as certain types of user input may cause fatigue over time or may otherwise make the user uncomfortable. Summary of the invention

[0003] The invention relates to a method for receiving input in an artificial reality environment using multiple modalities according to claim 1, a computing system for receiving input in an artificial reality environment using multiple modalities according to claim 7, a computer readable storage medium according to claim 13, and a software product according to claim 15. Advantageous embodiments may comprise the features of the dependent claims.

[0004] Therefore, according to a first aspect of the present disclosure, a method for receiving input in an artificial reality environment using multiple modalities is provided, wherein the method includes: displaying a virtual keyboard in the artificial reality environment; receiving user gaze input related to the virtual keyboard; placing an indicator on the virtual keyboard based on the received user gaze input; receiving user hand input as a sliding segment on the surface of a controller device, the sliding segment having a starting point and an ending point, wherein the starting point of the sliding segment is mapped to the placed indicator; dynamically moving the indicator on the virtual keyboard based on the relative positions of the starting point and the ending point on the surface of the controller device; and resolving the sliding segment into two or more selections in the virtual keyboard based on the relative positions of the starting point and the ending point.

[0005] In some embodiments, the starting point may be received at the controller device at a first time, and the indicator may be placed according to a user gaze input proximate to the first time.

[0006] In some embodiments, there may be no mapping of areas of the surface of the controller device to portions of the virtual keyboard prior to receiving user hand input controlling movement of the indicator.

[0007] In some embodiments, the controller device may include a trackpad and the user hand input may include touch input.

[0008] In some embodiments, the method may further include detecting receipt of an edge touch input at an edge of the trackpad, wherein the edge touch input is approximately stationary for a threshold duration; and dynamically moving the indicator along a vector corresponding to the edge touch input within the duration.

[0009] In some embodiments, the vector may be defined based at least on the edge location of the touch pad at which the edge touch input is received.

[0010] In some embodiments, edge touch input may be detected when a touch input is received at a predetermined edge region of the touch panel.

[0011] In some embodiments, the initiation of touch input at the trackpad may initiate ad-hoc mapping that maps the start point of the slide segment to an indicator placed on the virtual keyboard.

[0012] In some embodiments, parsing the sliding segment based on the relative positions of the start point and the end point may include: using a predictive text model to generate one or more matching probabilities for candidate selections of alphanumeric characters, wherein the sliding segment may be parsed using the generated one or more matching probabilities.

[0013] In addition, according to a second aspect of the present disclosure, a computing system for receiving input in an artificial reality environment using multiple modalities is provided, wherein the computing system includes: one or more processors; and one or more memories storing instructions, which, when executed by the one or more processors, cause the computing system to execute a process, the process including: displaying a virtual keyboard in the artificial reality environment; receiving user gaze input related to the virtual keyboard; parsing the user gaze input to determine a first selection in the virtual keyboard, and placing a displayed indicator on the virtual keyboard at the first selection; receiving user hand input having a starting point and an ending point via a controller device, wherein the starting point is mapped to an indicator placed at the first selection; dynamically moving the indicator on the virtual keyboard based on the relative positions of the starting point and the ending point; and parsing the user hand input into two or more selections in the virtual keyboard based on the first selection and the relative positions of the starting point and the ending point.

[0014] In some embodiments, the starting point may be received at the controller device at a first time, and the first selection may be resolved from a user gaze input proximate to the first time.

[0015] In some embodiments, there may be no mapping of areas of the surface of the controller device to portions of the virtual keyboard prior to receiving user hand input controlling movement of the indicator.

[0016] In some embodiments, the controller device may include a trackpad and the user hand input may include touch input.

[0017] In some embodiments, the computing system or the process may further include: detecting receipt of an edge touch input at an edge of the touch pad, wherein the edge touch input is approximately stationary for a threshold duration; and dynamically moving the indicator along a vector corresponding to the edge touch input within the duration.

[0018] In some embodiments, the vector may be defined based at least on the edge location of the touch pad at which the edge touch input is received.

[0019] In some embodiments, edge touch input may be detected when a touch input is received at a predetermined edge region of the touch panel.

[0020] In some embodiments, the initiation of touch input at the trackpad can initiate a point-to-point mapping that maps the starting point of the slide segment to an indicator placed at the first selection.

[0021] In some embodiments, parsing the sliding segment based on the first selection and the relative positions of the start point and the end point may include: using a predictive text model to generate one or more matching probabilities for candidate selections of alphanumeric characters, wherein the sliding segment may be parsed using the generated one or more matching probabilities.

[0022] In addition, according to a third aspect of the present disclosure, a computer-readable storage medium is provided, which stores instructions, which, when executed by a computing system or preferably by a computing system as above, causes the computing system to execute a process for receiving input in an artificial reality environment using multiple modalities, wherein the process includes: displaying a virtual keyboard in the artificial reality environment; receiving user gaze input related to the virtual keyboard; placing an indicator on the virtual keyboard based on the received user gaze input; receiving user hand input having a starting point and an end point via a controller device, wherein the starting point is mapped to an indicator placed at a first selection; dynamically moving the indicator on the virtual keyboard based on the relative positions of the starting point and the end point; and parsing the user hand input into two or more selections in the virtual keyboard based on the relative positions of the starting point and the end point.

[0023] In some embodiments, the starting point may be received at the controller device at a first time, and the indicator may be placed according to a user gaze input proximate to the first time.

[0024] In addition, according to a fourth aspect of the present disclosure, a software product is provided, which includes instructions, which, when executed by a computing system or preferably by a computing system as described above, cause the computing system to execute a process for receiving input in an artificial reality environment using multiple modalities, wherein the process includes: displaying a virtual keyboard in the artificial reality environment; receiving user gaze input related to the virtual keyboard; placing an indicator on the virtual keyboard based on the received user gaze input; receiving user hand input having a starting point and an ending point via a controller device, wherein the starting point is mapped to an indicator placed at a first selection; dynamically moving the indicator on the virtual keyboard based on the relative positions of the starting point and the ending point; and parsing the user hand input into two or more selections in the virtual keyboard based on the relative positions of the starting point and the ending point.

[0025] It will be understood that any features described herein that are suitable for incorporation into one or more aspects or embodiments of the present disclosure are intended to be generalizable to any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure may be understood by those skilled in the art based on the specification, claims and drawings of the present disclosure. The foregoing general description and the following detailed description are exemplary and illustrative only, and are not intended to limit the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a block diagram showing an overview of a device on which some embodiments of the present technology may operate.

[0027] Figure 2A is a line diagram illustrating a virtual reality headset that may be used in some embodiments of the present technology.

[0028] Figure 2B is a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.

[0029] Figure 2C is a line diagram illustrating a plurality of controllers that, in some implementations, a user may hold in one or both hands to interact with an artificial reality environment.

[0030] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0031] Figure 4 is a block diagram illustrating various components that may be used in a system employing the disclosed technology in some implementations.

[0032] Figure 5is a flow chart illustrating a process for receiving input in an artificial reality environment using multiple modalities, used in some embodiments of the present technology.

[0033] Figure 6 is a flow chart illustrating a process for receiving trackpad edge input in an artificial reality environment used in some embodiments of the present technology.

[0034] Figure 7 is a schematic diagram showing a virtual keyboard and a received sliding segment.

[0035] Figure 8 is a schematic diagram showing visual alignment according to different mappings between a controller device and a virtual keyboard.

[0036] Fig. 9 is a schematic diagram showing a virtual interface for receiving a sliding pattern.

[0037] Fig.10 is a schematic diagram showing a virtual interface for setting a slide pattern.

[0038] The various techniques described herein may be better understood by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numbers refer to the same elements or functionally similar elements. DETAILED DESCRIPTION

[0039] Various aspects of the present disclosure relate to an interface for receiving input using multiple modalities in an artificial reality environment. For example, the interface can be a virtual keyboard displayed in an artificial reality environment, the virtual keyboard including multiple alphanumeric characters arranged as multiple elements. Various embodiments include an artificial reality device / system for displaying an artificial reality environment and receiving user input in one modality, and a controller device for receiving user input in an additional input modality. For example, the artificial reality device and / or system can be configured to receive gaze input as a first input modality. The controller device can be configured to receive input in a second modality, such as touch input received at a trackpad. The interface manager can process input in multiple modalities to dynamically move a cursor on a virtual interface. The interface manager can also parse multiple character selections, words, or one or more other suitable alphanumeric character combinations based on the input.

[0040] The interface manager initiates a mapping between the user's hand input and the cursor position on the virtual keyboard based on the user's gaze input. For example, the interface manager can display / dynamically move the cursor based on the user's input in multiple modes. In the case where the controller device includes a touchpad, the interface manager can, before detecting the user's touch on the touchpad: a) dynamically move the cursor based on the user's gaze input; b) display the cursor at a default position until a touch input is detected; c) do not display the cursor until a touch input is detected; or d) a combination thereof.

[0041] When the interface manager detects a touchpad input, the interface manager can initiate a mapping between the location of the initial touch input on the touchpad and the initial placement of the cursor on the virtual keyboard. The interface manager can place / display the cursor at a location on the virtual keyboard based on the user's gaze when the touchpad input is detected. The interface manager can then dynamically move the cursor from the initial location (based on the touch input). In some embodiments, the element selector can resolve a first element (e.g., alphanumeric character) selection based on a user gaze input that is temporally proximate to the initial touch input.

[0042] In some embodiments, the touchpad is configured to receive touch input as a sliding segment. The sliding segment may include a starting point and an ending point. In one example, the interface manager may map the starting point of the first sliding segment to the placed indicator (in some embodiments, at the first selection resolved using the user gaze input). For example, the start of the touch input at the touchpad initiates the mapping between the touchpad and the indicator. The user's sliding input is sensed by the touchpad as a touch input (e.g., a touch between two or more sensed points), and the interface manager dynamically moves the cursor from the initial placement of the cursor based on the touch input. In some embodiments, the interface manager terminates the mapping between the touch input at the touchpad and the indicator at the virtual keyboard when the touch input at the touchpad is detected to end.

[0043] The interface manager can also detect when a touch pad input is received at an edge of the touch pad. For example, an edge boundary can be defined (e.g., predefined) for the touch pad, and the interface manager can detect when the touch input is at the edge boundary. Because the user cannot continue to swipe in a direction other than the edge of the touch pad, the interface manager can dynamically move the indicator according to a vector (e.g., in a constant motion) when the edge input is detected. For example, the vector can be determined based on the position of the edge input relative to the touch pad, the touch input close to the edge input (e.g., the swipe direction before the edge), or a combination of these.

[0044] Embodiments of the disclosed technology may include an artificial reality system, or may be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and the artificial reality or extra reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with collected content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave" environment projection system or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0045] As used herein, "virtual reality" or "VR" refers to an immersive experience in which a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views real-world images after they have passed through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display the images on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images as they pass through the system (e.g., by adding virtual objects). "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is partially generated by a computing system and partially constitutes light reflected by objects in the real world. For example, an MR head-mounted viewer can be shaped as a pair of glasses with a see-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, thereby allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of the following: VR, AR, MR, or any combination or hybrid thereof.

[0046] Traditional XR input interfaces fail to provide an experience that meets user expectations, may cause fatigue and / or strain, and fail to achieve reliable / accurate results. For example, a traditional XR collider virtual keyboard can track user hand movements and identify when hand movements collide with virtual keyboard elements. However, this traditional technology can cause hand / arm fatigue, provide poor input speed, and do not support use cases in public places. Another traditional XR keyboard can track user gaze to select individual characters from the virtual keyboard. However, this traditional technology can cause eye strain, also provide poor input speed, and produce relatively low accuracy / reliability.

[0047] It is contemplated that various embodiments of the disclosed technology overcome these deficiencies in conventional systems by combining inputs from multiple modalities to support an improved user experience for interacting with an XR input interface. For example, user gaze input can be used to place an indicator on a virtual keyboard, and can be used to initiate a mapping between a controller device (e.g., a trackpad) and an indicator on the virtual keyboard. The mapping can be used to control the indicator on the virtual keyboard based on hand input received at the controller device (e.g., touch / slide input at a trackpad). The hand input can then be parsed to generate one or more element combinations (e.g., words, numbers, or other character combinations).

[0048] Using user gaze input to place indicators on the virtual keyboard and using hand input to select elements on the virtual keyboard based on the placed indicators reduces the risk of eye strain. This is at least because hand input (e.g., trackpad input) can cause less user fatigue than gaze input when interacting with the virtual keyboard. In addition, these combined input modalities also improve the speed of character selection and the accuracy / reliability of character selection. Slide-based character selection can allow users to "type" on the virtual keyboard with a natural and ergonomic feel, which can result in improved speed and accuracy.

[0049] In addition, using the user's gaze to place an indicator on the virtual keyboard improves the natural feel of the virtual interface without the need to integrate the display device with the touchpad input device and eliminates the delay caused by requiring the user to lift and view the keyboard displayed on the touchpad input device. Embodiments that resolve the first character selection based on the user's gaze input further support these improvements.

[0050] Prior to interacting with a controller device (e.g., interacting with a trackpad via touch), the user's relationship to the XR display is typically based on where the user is looking (e.g., user gaze). Mapping trackpad touches to user gaze provides a natural transition from gaze-based XR interaction to trackpad XR interaction (e.g., or hybrid gaze and trackpad XR interaction). This is more advantageous than placing an indicator at a predetermined location when initiating touch input, as the predetermined location may result in a divergence point that would feel unnatural when the user's gaze is elsewhere. Furthermore, the benefit of this more natural input is achieved using hardware that is easier to interact with and less expensive to manufacture than a trackpad device with a display.

[0051] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1is a block diagram showing an overview of devices on which some embodiments of the disclosed technology may run. These devices may include hardware components of a computing system 100 that receive input using multiple modalities in an augmented reality environment. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted view device that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted view device and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head-mounted view device and other processing operations are offloaded to the core processing component. The following is combined with Figure 2A and Figure 2B An example head mounted view device is described. In some implementations, location data and environmental data may be collected solely by sensors incorporated into the head mounted view device, while in other implementations, one or more of the plurality of non-head mounted view computing devices may include sensor components that may track environmental data or location data.

[0052] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among the computing devices 101 to 103).

[0053] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be communicated by a hardware controller that interprets the signals received from the input devices and transmits the information to the processors 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0054] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus, or a wireless connection, such as a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus. The processor 110 can communicate with a hardware controller for each device (e.g., for a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices are: a liquid crystal display (LCD) display screen, a light emitting diode (LED) display screen, a projection display, a holographic display, or an augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or network card, a video chip or video card, an audio chip or audio card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0055] In some embodiments, input from I / O devices 140 (e.g., cameras, depth sensors, inertial motion unit (IMU) sensors, GPS units, LiDAR or other time-of-flight sensors, etc.) can be used by computing system 100 to identify and map the user's physical environment and track the user's position within the environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., topology, grid, etc.) of an area (which can be a room, building, outdoor space, etc.) and / or obtain a map previously generated by computing system 100 or another computing system that has mapped the area. The SLAM system can track users within the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other position changes, etc.

[0056] The computing system 100 may include a communication device that can communicate wirelessly or wired with other local computing devices or network nodes. The communication device can communicate with another device or server over a network, for example, using a Transmission Control Protocol / Internet Protocol (TCP / IP) protocol. The computing system 100 can use the communication device to distribute operations across multiple network devices.

[0057] Processor 110 may access memory 150, which may be included in one of the computing devices of computing system 100, or may be distributed across the computing devices of computing system 100, or across multiple other external devices. Memory includes one or more hardware devices for volatile or non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include one or more of the following: random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, tape drive, etc. Memory is not a propagation signal that is separated from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160 that stores programs and software, such as operating system 162, interface manager 164, and other application programs 166. The memory 150 may also include a data storage 170, which may include, for example, language model data that may be provided to the program memory 160 or any element in the computing system 100, data representing stored sliding patterns associated with characters, historical data, training data, mapping data, configuration data, settings, user options or preferences, etc.

[0058] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with the technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices, etc.

[0059] Figure 2A2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a band 210. Front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computing units 230. Position sensor 220, IMU 215, and computing unit 230 may be located inside HMD 200 and may not be visible to the user. In various embodiments, IMU 215, position sensor 220, and locator 225 may track the movement and position of HMD 200 in the real world and in an artificial reality environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locator 225 may emit infrared beams that produce light spots on real objects around HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position, force, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect light points. The computing unit 230 in the HMD 200 may use the detected light points to infer the position and movement of the HMD 200, and to recognize the shape and position of real objects around the HMD 200.

[0060] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., micro-LED, laser (LASER)), some other display, or some combination thereof.

[0061] In some embodiments, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., via light emitted from the HMD 200), and the PC can use the HMD in conjunction with the outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0062] Figure 2B 2 is a line diagram of a mixed reality HMD system 250 that includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 Gigahertz (GHz) link) as indicated by link 256. In other embodiments, the mixed reality system 250 includes only a head mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a see-through display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as a light projector (e.g., lasers, LEDs, etc.), a camera, an eye tracking sensor, a micro-electromechanical system (MEMS) component, a network component, etc.

[0063] The projector can be coupled to the see-through display 258, for example, via an optical element, to display media to the user. The optical element can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. for directing light from the projector to the user's eyes. The image data can be transmitted from the core processing component 254 to the HMD 252 via a link 256. The controller in the HMD 252 can convert the image data into a plurality of light pulses from the projector, which can be transmitted to the user's eyes via the optical element as output light. The output light can be mixed with the light passing through the display 258, thereby allowing the output light to present the following virtual objects: These virtual objects appear to exist in the real world.

[0064] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example, with 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects to appear stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0065] Figure 2C A plurality of controllers 270 (including controllers 276A and 276B) are shown, which in some embodiments can be held in one or both hands by a user to interact with an artificial reality environment presented by HMD 200 and / or HMD 250. Controllers 270 can communicate with the HMD directly or via an external device (e.g., core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit light points farther away. The HMD 200 or HMD 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., tracking the controllers with 3DoF or 6DoF). The computing unit 230 or core processing component 254 in the HMD 200 can combine the IMU output and the position output and use this tracking to monitor the position and movement of the user's hands. The controller may also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A-274B) that a user may actuate to provide input and interact with objects.

[0066] In various embodiments, HMD 200 or HMD 250 may also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc., to monitor indications of user interaction and intent. For example, in some embodiments, instead of or in addition to a controller, one or more cameras included in HMD 200 or HMD 250 or from an external camera can monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources can illuminate one or both eyes of the user, and HMD 200 or HMD 250 can use eye-facing cameras to collect reflections of light to determine eye position (e.g., based on a set of reflections around the user's cornea), model the user's eyes and determine gaze direction.

[0067] Figure 3 305B may be a HMD 200 or a HMD system 250. The plurality of client computing devices 305 may be operated in a network environment using logical connections to one or more remote computers (e.g., server computing devices) through a network 330.

[0068] In some implementations, server 310 may be an edge server that receives multiple client requests and coordinates the fulfillment of these requests through other servers (e.g., servers 320A to 320C). Server computing devices 310 and 320 may include a computing system, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, each of these server computing devices may be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.

[0069] Client computing device 305, and server computing devices 310 and 320 can all act as servers or clients of one or more other servers / client devices. Server 310 can be connected to database 315. Servers 320A to 320C can each be connected to corresponding databases 325A to 325C. As discussed above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share a database, or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can all be distributed computing environments that include multiple computing devices, can be located in their corresponding servers, or can be located at the same physical location or at geographically different physical locations.

[0070] The network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. The network 330 may be the Internet or some other public or private network. The client computing device 305 may be connected to the network 330 via a network interface, for example, by wired or wireless communication. Although the connections between the server 310 and the plurality of servers 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including the network 330 or a separate public or private network.

[0071] Figure 44 is a block diagram showing a plurality of components 400 that can be used in a system that employs the disclosed technology in some embodiments. Each component 400 can be included in a device of the computing system 100, or can be distributed on multiple devices of the computing system 100. Component 400 includes hardware 410, middleware 420, and multiple dedicated components 430. As discussed above, a system implementing the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.) and a storage memory 418. In various embodiments, the storage memory 418 can be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, the storage memory 418 can be one or more hard drives or flash drives accessible via a system bus, or can be a cloud storage provider (e.g., in memory 315 or 325) or other network memory accessible via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device (eg, client computing device 305 ), or on a server computing device (eg, server computing devices 310 or 320 ).

[0072] Middleware 420 may include components that communicate resources between hardware 410 and a plurality of specialized components 430. For example, middleware 420 may include an operating system, services, drivers, a basic input output system (BIOS), controller circuits, or other hardware or software systems.

[0073] Multiple dedicated components 430 may include software or hardware that is configured to perform operations for receiving input using multiple modalities in an XR environment. Multiple dedicated components 430 may include: an interface controller 434; an element selector 436; one or more prediction models 438; and components and application programming interfaces (APIs) (e.g., interface 432) that can be used to provide a user interface, transmit data, and control the dedicated components. In some embodiments, each component 400 may be in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more dedicated components 430. Although each dedicated component 430 is depicted as a separate component, these dedicated components may be logical or other non-physical distinctions of functionality, and / or may be sub-modules or code blocks of one or more applications.

[0074] Interface controller 434 can control an interface displayed in the XR environment. For example, the interface can be a virtual keyboard, and interface controller 434 can use input from multiple modalities to control placement / movement of indicators at the virtual keyboard. Interface controller 434 receives user input in a combination of multiple input modalities, such as a combination of gaze input and trackpad input, a combination of gaze input and joystick input, a combination of gaze input and tracked hand input, or any other suitable combination or combinations of multiple modalities.

[0075] One or more sensors may be used to track gaze input, such as one or more cameras, accelerometers, other wearable sensors, or other suitable sensors. In some embodiments, inputs from multiple cameras, wearable sensors, and / or other sensors may be combined to track the user's head state. The tracked head state may include head movement, the direction the head is pointing, eye movement, the direction the eyes are looking at (e.g., eye gaze), and other suitable head states and head movements. One or more machine learning models may be configured to track the user's head state and / or eye state, such as a computer vision model including a convolutional neural network or any other suitable model. In some embodiments, the model may include a three-dimensional (3D) model of one or both eyes of the user, and the machine learning model may obtain image data depicting the user's eyes (e.g., showing flashes around the cornea), and the machine learning model uses the image data to estimate the direction of the eyes (relative to the user's head) in the 3D model. Therefore, one or more machine learning models may be trained to receive input from a camera (e.g., a series of images) and / or input from other sensors, and output the user's head state / eye gaze.

[0076] The tracked hand input may be tracked using one or more sensors, such as one or more cameras, accelerometers, or other suitable sensors. In some embodiments, inputs from multiple cameras, wearable sensors, and / or other sensors may be combined to track the user's physical state. The tracked physical state may include hand movements, finger movements, arm movements (e.g., forearm movements, wrist movements, etc.), and other suitable physical states or body movements. One or more machine learning models may be configured to track the user's physical state, such as a computer vision model including a convolutional neural network or any other suitable model. The one or more machine learning models may be trained to receive input from a camera (e.g., a series of images) and / or input from other sensors, and output the user's physical state / tracked hand movements and hand positions.

[0077] Touchpad input and joystick input may be received using a suitable controller device, such as a touchpad configured to detect / sense a user's touch, or a joystick having a movable component or a component that can sense force. The interface controller 434 may receive input from these different modalities and place / move a cursor on the virtual keyboard based on the input.

[0078] The interface controller 434 can place a cursor on the virtual keyboard according to the user gaze input modality. For example, the interface controller 434 can parse a specific position on the displayed virtual keyboard according to the user gaze input tracked at a given point in time (e.g., the projected gaze ray), place the cursor at the parsed position, and display the placed cursor on the virtual keyboard to the user. For example, the parsed position on the virtual keyboard according to the user gaze input can be the point where the projected gaze ray intersects the displayed virtual keyboard (e.g., a two-dimensional coordinate on the virtual keyboard), or any other suitable technique can be used to parse the position. In some embodiments, the element selector 436 can parse the first selected alphanumeric character in the virtual keyboard according to the tracked user gaze input, and the interface controller 434 can place the cursor at / near the selected alphanumeric character.

[0079] In some embodiments, the interface controller 434 places a cursor in response to an initial touch input received at the touchpad of the controller device (or the initiation of any other suitable hand input). For example, the initial touch input can be part of a sliding pattern received at the touchpad. The start of the sliding pattern initiates a mapping between the placed cursor and the touchpad position receiving the initial touch. The interface controller 434 can then dynamically move the cursor from the initial placement position based on the sliding pattern received at the touchpad. For example, the interface controller 434 can then receive a user's sliding touch input as a touch input between two or more coordinates, and can dynamically move the placed cursor based on the sliding touch input. The interface controller 434 can display to the user the dynamic movement of the cursor on the virtual keyboard.

[0080] The element selector 436 resolves the selected element in the virtual interface (e.g., a virtual keyboard) based on the received input. In some embodiments, the element selector 436 can resolve alphanumeric characters, words, and / or character combinations based on the received input in a modality associated with the user's hand input. For example, a trackpad can receive user touch input in the form of a slide, a joystick can receive direction and timing input to resolve a slide pattern, a tracked user's hand without the aid of the device can perform movement (e.g., through the movement of a thumb on the side of a user's index finger) to form a slide pattern, etc.

[0081] A sliding segment can be a touch input representing movement between two points (e.g., a starting point and an ending point). Several sliding segments can be combined to form a user's sliding motion. For example, the ending point of a previous sliding segment can be the starting point of a subsequent sliding segment. Based on the mapping between the position of the touch input on the touch pad and the position of the indicator on the virtual keyboard, the path of the first sliding segment can include (e.g., included on the path between the starting point and the ending point) two or more alphanumeric characters, the path of the next sliding segment can also include two or more alphanumeric characters, and additional sliding segments can continue until the user's sliding motion ends (e.g., the user lifts his finger from the touch pad, allows the joystick to remain in the center position for a threshold time, or lifts his thumb to not touch his index finger). The entire sliding action can form a path including several alphanumeric characters. The element selector 436 can use one or more prediction models 438 (based on the mapping of the touch pad to the indicator) to parse the path of the sliding motion into a combination of alphanumeric characters, such as words, numbers, or other suitable combinations of multiple characters.

[0082] In some embodiments, the element selector 436 can use the user gaze input to parse the initial alphanumeric character selection in the virtual keyboard. One or more prediction models 438 (e.g., a machine learning model, algorithm, or other suitable model) can be configured to receive the user's tracked user gaze input and use the tracked user gaze to predict the element that the user intends to select. The element selector 436 can then parse additional alphanumeric characters, words, and / or character combinations based on the received input in a modality associated with the user's hand input (e.g., a touchpad input corresponding to a sliding motion). For example, the element selector 436 can parse a character combination based on a first selection (e.g., selected using gaze input) and one or more sliding segments that select the remaining characters.

[0083] One or more prediction models 438 can predict element selections or combinations of element selections, such as selections of alphanumeric characters and / or selections of combinations of alphanumeric characters. One or more prediction models 438 can be configured to receive a user's tracked user gaze input and use the tracked user gaze to predict an element in the virtual interface (e.g., an alphanumeric character in a virtual keyboard) that the user intends to select. The model can be trained with training data that includes example instances of tracked user gaze input and labels of selected alphanumeric characters.

[0084] One or more prediction models 438 may be configured to receive a sliding segment (relative to the position of an element on the virtual keyboard), or a combination of sliding segments, and predict an alphanumeric character, two alphanumeric characters, or a combination of alphanumeric characters for the sliding pattern. For example, one or more prediction models 438 may be used to match one or more current sliding segments to known sliding segments associated with a predetermined character and / or a series of characters. In another example, one or more prediction models 438 may include a language model that stores character sequences and / or words associated with usage probabilities. A language model may be used to configure one or more prediction models 438 and / or to select among outputs generated by one or more prediction models 438. One or more prediction models 438 may include any other suitable character prediction model known to a person of ordinary skill in the art.

[0085] In some embodiments, user gaze input is used to select a first character from the virtual keyboard, while hand input (e.g., touch input) is used to select subsequent characters. When one or more prediction models 438 predict one or more characters of a slide segment, the input may include a touch input representing a slide on the virtual keyboard and the initially selected character. For example, one or more prediction models 438 may predict subsequent character selections based on the slide input given a first character selection.

[0086] It will be understood by those skilled in the art that the above Figures 1 to 4 The components shown in the flowcharts and the components shown in each of the flowcharts discussed below can be changed in various ways. For example, the logical order can be rearranged, multiple sub-steps can be performed in parallel, the logic shown can be omitted, other logic can be included, etc. In some embodiments, one or more of the above components can perform one or more of the processes described below.

[0087] Figure 5 is a flow chart illustrating a process 500 for receiving input in an XR environment using multiple modalities, used in some embodiments. In some embodiments, process 500 may be performed by an application or operating system of an XR system to receive input from a user. The input received from the user may configure the application, operating system, any other element of the XR system, or may be transmitted to any other suitable computing device.

[0088] At block 502, process 500 may display a virtual keyboard in an XR environment. For example, an XR device may include a display component that displays a virtual keyboard to a user. The virtual keyboard may include a plurality of alphanumeric characters arranged as a plurality of elements. The virtual keyboard may be displayed at a distance in front of the user in the XR environment, for example, to appear three feet away from the user.

[0089] At block 504, process 500 can receive user gaze input related to the virtual keyboard. Input from one or more sensors (e.g., a camera, an accelerometer, other wearable sensors, and LiDAR or other time-of-flight systems, etc.) can be used to track the user's gaze (which can be a combination of eye position and head position) relative to displayed elements (e.g., alphanumeric characters) of the virtual keyboard.

[0090] In box 506, process 500 can parse the user gaze input to place an indicator on the virtual keyboard. For example, the indicator can be placed on the virtual keyboard at a two-dimensional coordinate corresponding to the user gaze input. In some embodiments, the user gaze input can be projected onto the virtual keyboard (e.g., as a vector projection of the user gaze that intersects the displayed virtual keyboard according to the tracking), and the indicator can be placed at the projected position. In some embodiments, the initial touch input received at the controller device (or other hand input tracked by the controller or hand) triggers the placement of the indicator. The gaze input for placing the indicator can represent the gaze of the user at the time of the initial touch input, or at a time close to the initial touch input (e.g., a threshold time amount before or after the initial touch input). The placed indicator can be displayed at the virtual keyboard.

[0091] In some embodiments, at box 506, process 500 can parse the user gaze input to determine the first selection in the virtual keyboard, and place an indicator on the virtual keyboard at the first selection. For example, the user gaze input can be parsed to select an element in the virtual keyboard. In some embodiments, the initial touch input received at the controller device (or other hand input tracked by the controller or hand) triggers the parsing of the first selection. The gaze input for parsing the first selection can represent the gaze of the user at the time of the initial touch input or at a time close to the initial touch input (e.g., a threshold amount of time before or after the initial touch input). Each embodiment places an indicator at the selected element in the virtual keyboard. For example, the indicator can be a cursor placed at or near the selected element (e.g., an alphanumeric character) (e.g., a threshold distance from the selected element).

[0092] At box 508, process 500 can receive a user hand input as a sliding segment on the surface of a controller device, the sliding segment having a starting point and an ending point. For example, the controller device may include a touch pad, and the user hand input may be a touch input sensed on the surface of the touch pad. The touch input may include a sliding segment having a starting point and an ending point. In some cases, the starting point may be the initial touch point or the ending point of the previous segment, and the ending point may be the current location of the user input (e.g., when the user continues a straight or curved input), a change in the input direction, or a point at which the user stops providing input (e.g., lifts his finger off the input device). Some embodiments include a sliding pattern that includes several sliding segments in combination.

[0093] When the user hand input (e.g., touch input) is initially sensed, the starting point of the sliding segment can be mapped to the placed indicator (e.g., triggered by the start of the user hand input). For example, the relative position of the initial touch input at the trackpad can be mapped to the relative position of the indicator on the virtual keyboard.

[0094] At block 510, process 500 may dynamically move an indicator on a virtual keyboard based on the relative positions of a starting point and an ending point on a surface of a controller device. For example, the indicator may be dynamically moved from an initial position of the indicator based on movement of a sensed user touch input from the starting point along a relative route of motion of the user input. The relative positions of the starting point and the ending point may represent a touch input that moves over time, and the indicator may be dynamically moved based on the touch input. The dynamic movement of the indicator may be displayed to the user on the virtual keyboard.

[0095] At box 512, process 500 can resolve the sliding segment into two or more selections in the virtual keyboard based on the relative positions of the starting point and the ending point. For example, one or more prediction models can receive a touch input representing a sliding segment on the virtual keyboard and output a prediction of one or more characters selected by the sliding segment. In some embodiments, key points in the sliding pattern (e.g., changes in direction, starting and ending points of the sliding segment, points where the user changes the sliding speed, etc.) can be used to identify which corresponding points on the virtual keyboard indicate the expected input characters. In some embodiments, box 512 can be performed on the entire sliding input after box 514 (e.g., for the entire group of sliding segments). For example, the pattern defined by the group of sliding segments can be provided to a machine learning model, or can be used to match with patterns of known words to resolve the user's intended input.

[0096] In some implementations, process 500 can resolve the sliding segment into two or more selections in the virtual keyboard based on the first selection (e.g., selected based on user gaze input) and the relative positions of the start point and the end point at block 512. For example, one or more prediction models can receive a touch input representing a sliding segment on a virtual keyboard and the first selection (e.g., a character) and output a prediction of one or more characters selected by the sliding segment.

[0097] At box 514, process 500 can determine whether hand input is still being received. For example, process 500 can determine whether an input signal is received from a touch pad of a controller device (yes at box 514), or whether the user has lifted his finger from the touch pad (no at box 514). In other embodiments, other signals can be used to identify the end of the sliding input, such as the joystick remaining in the center position for a threshold time (e.g., 0.1 seconds or 0.5 seconds), or the user lifting his thumb to not contact another finger. If touch input is still being received, process 500 can return to box 508 to receive touch input. Process 500 can continue to boxes 510 and 512, so that the indicator can be dynamically moved according to the touch input (e.g., continuous sliding input), and additional selections in the virtual keyboard can be parsed (e.g., alphanumeric character selections for continuous sliding can be parsed).

[0098] At block 514, when process 500 determines that no more touchpad input is being received, process 500 may proceed to block 516. At block 516, process 500 may terminate the touch input session. For example, during the touch input session, a displayed cursor may be dynamically moved based on the touch input. After the touch input session is terminated, the cursor may be: a) removed from the virtual keyboard, b) placed at a default position, c) dynamically moved based on input in another modality (e.g., user gaze input).

[0099] In some implementations, at one or more blocks 512 and 514, the combination of character selections can generate character sequences corresponding to the user's input, such as words, numbers, sentences, and any other suitable character combinations. For example, the predictive model can generate these character sequences using characters parsed from the user input, by matching input patterns to known word patterns, etc. The process 500 can translate input from the user in multiple modalities into alphanumeric characters so that the user can "type" in the XR environment.

[0100] In some implementations, when providing input in some modalities, such as touch input, a user may encounter limitations that affect the user's ability to interact with the virtual keyboard. Figure 6600 is a flow chart illustrating a process for receiving trackpad edge input in an XR environment used in some embodiments of the present technology. In some embodiments, process 600 can be performed by an application or operating system of an XR system to receive input from a user. The input received from the user can configure the application, operating system, any other element of the XR system, or can be transmitted to any other suitable computing device.

[0101] Various embodiments of process 600 may be used in conjunction with Figure 5 Process 500 in the embodiment of the present invention is combined with operation to receive touch input from a user. In some embodiments, process 600 can be performed in conjunction with a controller device having a touchpad or joystick for receiving touch input from a user. In other cases, process 600 can be performed with a user inputting directional input (e.g., moving his thumb while it is in contact with a side of his index finger) through hand tracking.

[0102] At block 602, process 600 can determine whether an edge input is detected. For example, an edge input can be a touch input received at an edge of a trackpad, a joystick input at a maximum range of motion of a joystick, or a user extending their thumb to the edge of the range of motion of the thumb while the thumb is in contact with their index finger. For example, a boundary can be defined at the edge of a trackpad, and an edge input can be detected when a touch input is received within the boundary. When an edge input is detected, process 600 proceeds to block 604.

[0103] At block 604, process 600 may determine a vector of a slide and / or indicator movement during an edge input. For example, a vector may be defined based on an edge input position relative to a touchpad, a joystick input relative to a center position, or a thumb tip position relative to an initial contact position with an index finger. For example, an edge input at the upper right corner of a touchpad may correspond to a defined vector that is tilted upward and to the right, an edge input to the right of the center of the touchpad may correspond to a defined vector that is pointed to the right (e.g., not tilted upward or downward), and an edge input at the lower right corner of the touchpad may correspond to a defined vector that is tilted downward and to the right. In another example, a vector may be determined based on a touch input representing a touch motion of a user before reaching an edge boundary. In another example, an edge input may be an input from a joystick, and a vector may be defined based on an angle of the joystick at a maximum range of motion. In yet another example, an edge input may be a movement of one or more of the tracked fingers at the edge of a range of motion (e.g., a range of motion when a thumb is in contact with an index finger), wherein a vector may be defined based on the position of the one or more fingers of the user relative to the center of the range of motion. In some implementations, a vector can be defined to simulate pointer movement based on an input (eg, a touch input, a joystick input, a tracked finger input) that immediately precedes the detected edge input in time.

[0104] At block 606, process 600 can dynamically move the indicator along the vector. For example, when the user maintains edge input (e.g., keeps his thumb in contact with the edge of the trackpad), the indicator can dynamically move along the path defined by the determined vector in a constant motion.

[0105] In some embodiments, the dynamically moving indicator represents a slide for selecting a character from the virtual keyboard. For example, an edge input received at the touchpad does not represent a slide by the user because the edge input itself is static and does not reflect indicator movement. Therefore, the touch input can be enhanced to represent dynamic indicator movement so that the touch input represents indicator movement on the virtual keyboard. In this example, the enhanced touch input can correspond to the indicator movement displayed to the user and thus reflect the character selection desired by the user.

[0106] After block 606, process 600 may return to block 602. The indicator may dynamically move along the vector (or another determined vector if the edge input changes) until the edge input is no longer detected at block 602. When the edge input is no longer detected, process 600 proceeds to block 608.

[0107] At block 608, process 600 may dynamically move the indicator based on the touch input received from the user. For example, when edge input is no longer detected, the user may provide touch input on the trackpad in the form of one or more sliding segments, and the indicator may be dynamically moved based on the one or more sliding segments (e.g., the touch input moves between the start point and the end point of the sliding segment). In another example, when edge input is not detected, the user may no longer provide touch input on the trackpad. In this example, the cursor may no longer be displayed, may be displayed at the last position of the cursor, may be moved to a default position, or may be controlled by another input modality (e.g., user gaze).

[0108] Other embodiments may provide alternatives Figure 6 edge touch function in, or with Figure 6 Additional functionality combined with the edge touch functionality in . For example, when the user's touch input approaches the edge of the trackpad, an indicator can be displayed at the virtual keyboard with an effect (e.g., fade, squeeze, or color change, etc.). In another example, the gaze input tracked during the sliding touch input can be used to enhance the touch input. When the touch input and the gaze input point to the same / similar virtual keyboard location, the tracked gaze input can increase the speed at which the indicator moves on the virtual keyboard. When the edge of the trackpad is reached (e.g., an edge input is detected), the tracked gaze input can also be used to continue the movement of the indicator to a certain position of the virtual keyboard. For example, similar to Figure 6 A function in can determine a vector for edge input, and when the user's tracked gaze on the virtual keyboard is consistent with the vector, the indicator can move along the vector. Figure 8 Additional details regarding edge inputs are provided.Process 600 can repeat as the user continues to provide swipe inputs.

[0109] Figure 7 is a schematic diagram showing a virtual keyboard and a received sliding segment. Figure 7Schematic diagram 700, virtual keyboard 702, indicator 704 and sliding segment 706 are depicted. Virtual keyboard 702 can be an object displayed in an artificial reality environment, for example, an object displayed in front of a user by an artificial reality device when text input is specified. Schematic diagram 700 shows that indicator 704 is initially placed at alphanumeric character element "i" on virtual keyboard 702 based on the user's gaze input. Subsequent touchpad input is used to dynamically move indicator 704. The gaze input and the placed indicator 704 represent a mapping between the placement of indicator 704 on virtual keyboard 702 and the initial touch position on the touchpad. The sliding segment 706 and the movement of indicator 704 on virtual keyboard 702 are caused by touch input received at the touchpad. The touch input dynamically moves indicator 704 based on the initial mapping.

[0110] Schematic diagram 700 may represent a virtual keyboard displayed to a user when the user provides input in different modalities. For example, an indicator 704 may be displayed to the user positioned at an alphanumeric character element "i" in response to the user's gaze input and initial touch on the trackpad. A dynamic movement and sliding pattern of indicator 704 may then be displayed to the user (e.g., starting with sliding segment 706 and continuing to add sliding segments in response to the user's touch input on the trackpad). A virtual keyboard may be displayed to the user at a distance in front of the user's face in an XR environment to support gaze cursor placement.

[0111] Figure 8 is a schematic diagram showing visual alignment according to different mappings between a controller device and a virtual keyboard. Figure 8 Controller devices 802 and 804 are depicted, as well as visual alignments 806 and 808. Embodiments include a point-to-point mapping between relative touch locations on the trackpad and pointer placement on the virtual keyboard. Due to the point-to-point nature of the mapping, some initial user touches may result in challenging mappings.

[0112] For example, controller device 802 depicts an initial touch 810 on the left side of the device, while controller device 804 depicts an initial touch 812 in the middle of the device, and in both cases the user is looking at the "z" key on the corresponding virtual keyboard. The corresponding visual alignments 806 and 808 (not necessarily displayed to the user, but can be provided in the display of the controller device or as a virtual object overlaid by an associated artificial reality device) rely on the placement of indicators at the virtual keyboard (placed according to user gaze (and in some embodiments, resolved character selection)) and the resulting mapping generated (e.g., a mapping between indicator placement on the virtual keyboard and the initial touch location on controller devices 802 and 804). For example, both visual alignments 806 and 808 show the indicator placement of the "z" key that the user was focused on when the user touched the trackpad mapped under initial user touches 810 and 812, respectively.

[0113] Controller device 802 and initial touch 810 located on the left side of the device present less challenges to interacting with the virtual keyboard. A portion of the virtual keyboard depicted by visual alignment 806 is not reachable on the trackpad (e.g., the characters "p" and "l"). Controller device 804 and initial touch 812 located in the middle of the device present greater challenges to interacting with the virtual keyboard. A large portion of the virtual keyboard depicted by visual alignment 808 is not reachable on the trackpad (e.g., almost half of the characters in the virtual keyboard).

[0114] Different techniques can be used to mitigate these challenges. For example, a user can hold the controller device in landscape mode (e.g., instructions can be provided to hold the controller device in landscape mode) to provide more trackpad space for the virtual keyboard. In another example, edge input functionality can be implemented so that indicator movement can continue even when reaching the edge of the trackpad (e.g., reference to FIG. 1 ). Figure 6 ), touch input at the edge of the touchpad may cause the pointer to move according to the defined vector. This edge input technique can be used to reach virtual keyboard characters that would not be reachable without the technique. For example, you can use Figure 6 The edge input functionality described in is used to reach the inaccessible portion of the virtual keyboard shown in visual alignment 808.

[0115] In some embodiments, by changing the relative mapping between the touch position on the touchpad and the indicator position on the virtual keyboard, the edge input can realign the mapping. For example, the edge input function causes the cursor movement on the virtual keyboard that does not correspond to the movement of the user's touch on the touchpad. The relative mapping of the position of the edge of the touchpad that receives the user's edge input is dynamically changed by the indicator movement on the virtual keyboard. Consider the virtual keyboard shown in visual alignment 808. The user can provide edge input at the edge of the controller device 804 until the indicator reaches the character "p" on the virtual keyboard. The edge input can convert the mapping between the touchpad of the controller device 804 and the virtual keyboard until the right edge of the touchpad reaches the character "p" of the virtual keyboard. This realignment achieves higher accessibility of the virtual keyboard characters and alleviates the problems caused by the original mapping shown by visual alignment 808.

[0116] Other embodiments may provide additional functionality in place of or in combination with the mapping realignment functionality. For example, (once a mapping is established by initiating touch input) the displayed virtual keyboard may be supplemented with an overlay of the trackpad mapping and / or an overlay of the borders of the trackpad mapping. The overlay may provide feedback to the user so that mitigation techniques may be used to remedy problematic mappings and / or support natural sliding motions.

[0117] In another example, when the user's touch produces a problematic mapping, feedback (e.g., tactile feedback, sound, flashing light, displayed indicator, or other suitable feedback) can be provided by the controller device to the user (or by the AR device to the user). For example, a portion of the virtual keyboard that is unreachable according to the initial touchpad mapping can be determined, and when the portion meets a condition (e.g., a percentage threshold), feedback can be provided to the user. The feedback can encourage the user to initiate a touch session at a favorable location. In one example, the user can be prompted to enter a touch session at a desired location. Figure 8 An initial touch generated by the mapping represented by visual alignment 808 may trigger tactile feedback to the user.

[0118] Embodiments may also change the manner in which the indicator moves on the virtual keyboard in response to a touch input on the touchpad. For example, the motion ratio implemented may be mapped 1:1 (e.g., a touch input that performs a full horizontal traversal of the touchpad may correspond to a full horizontal traversal of the virtual keyboard). In another example, the motion ratio may be 1:2 (e.g., a touch input that performs a half horizontal traversal of the touchpad may correspond to a full horizontal traversal of the virtual keyboard), 2:1, 1:1.5, or any other suitable motion ratio. In some embodiments, the motion ratio may be dynamic based on a mapping generated by an initial touch by the user. For example, a default motion ratio (e.g., 1:1) may be used for a good mapping, however, when a problematic mapping is initiated (e.g., a mapping that uses the default ratio to make a large portion of the virtual keyboard inaccessible), the default ratio may be changed (e.g., to 1:2 or 1:1.5).

[0119] In another example, the movement of the indicator on the virtual keyboard according to the touch input can be controlled linearly. Considering the example of the 1:1 motion ratio above, the linear control causes the touch input that performs a complete horizontal traversal of the trackpad to move the indicator in the complete horizontal traversal of the virtual keyboard. In some embodiments, the touch speed (or acceleration) can be used to enhance the indicator movement. For example, the indicator movement can increase the speed based on the touch input speed (e.g., the determined finger speed).

[0120] In some embodiments, user gaze input can be used for virtual keyboard character selection, and touch input is used to trigger the selection. For example, a touch sensed on a controller device (e.g., a trackpad) can trigger element selection, so that a given virtual keyboard character pointed to by the user's gaze at the time of the touch input is selected. The technology implements user gaze as a mechanism for pointing to a virtual keyboard element, and touch input as a mechanism for triggering selection of the pointed element. This technology can be used to select a series of characters to generate words, numbers, and character sequences, etc.

[0121] Embodiments may also use the gaze input modality and the hand input modality to receive other types of input, such as patterns. Fig. 9 is a schematic diagram showing a virtual interface for receiving a pattern. Fig. 9Schematic diagram 900 and interface display 902 are depicted. Interface display 902 can be displayed to the user in an XR environment, for example, at a distance in front of the user's face. Interface display 902 includes several elements (e.g., points) arranged in a shape. The user's gaze can be tracked (e.g., using a camera, accelerometer, wearable sensor, or other suitable sensor) to identify the initial target element. Then, hand input (e.g., touch input at a touchpad, joystick input, tracked finger input) can be received to initiate selection of the target element. For example, when the initial hand input is detected, the element pointed to by the user's gaze can be selected. In some embodiments, the detected touch input can also place an indicator (e.g., a cursor) at the selected element. For example, the indicator can be placed on the virtual interface 902 and dynamically moved by the user's hand input.

[0122] The detection of hand input and the placement of the cursor realize the mapping between the indicator and the hand input. For example, the hand input can be a touch input on a touch pad, and the mapping can map the relative position of the touch input on the touch pad to the relative position of the placed indicator at the virtual interface 902. Then, the indicator can be dynamically moved by the hand input. The hand input of the user can be a sliding input, which (given the relative mapping of the hand input to the indicator position) generates a path including two or more elements in each element on the interface display 902. The sliding input can be parsed with the initially selected element to generate a pattern. For example, the sliding path can include a series of element selections that form a pattern. In some embodiments, the pattern can be a lock pattern of a device or other suitable security protocols for accessing a device or application. In some embodiments, the hand input that triggers the initial element selection, causes the indicator to be placed, and generates the path that forms the pattern can be a joystick input, a tracked finger input, or any other suitable hand input.

[0123] Fig.10 is a schematic diagram showing a virtual interface for setting a pattern. Fig.10 A schematic diagram 1000, a setting pattern interface 1002, and a confirmation pattern interface 1004 are depicted. For example, user gaze input and user hand input may be used to select a pattern selected by Fig. 9 The interface display 902 depicts the elements in FIG. The selection can be displayed to the user by setting the pattern interface 1002. Similar to reference Fig. 9In the described functionality, hand input can be used to move indicators and select multiple parts (e.g., buttons) on the set pattern interface 1002. The user can save the element selection or retry the selection (if the selection is incorrect or not what the user expected). Once the pattern is set, the pattern can be displayed to the user through the confirm pattern interface 1004. If the user is satisfied with the pattern, the user can enter the pattern. Otherwise, the user can clear the pattern and retry the pattern input.

[0124] References in this specification to "multiple embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present disclosure. The appearances of these phrases in different places in the specification do not necessarily all refer to the same embodiment, nor are they necessarily separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be rendered by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements for some embodiments but not for other embodiments.

[0125] As used herein, being higher than a threshold value means that the value of a comparison item is higher than other values ​​specified, the comparison item is located in a certain specified number of items with a maximum value, or the comparison item has a value within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of a comparison item is lower than other values ​​specified, the comparison item is located in a certain specified number of items with a minimum value, or the comparison item has a value within a specified minimum percentage value. As used herein, being within a threshold value means that the value of a comparison item is between two other values ​​specified, the comparison item is located in a middle specified number of items, or the comparison item has a value within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting a fast connection" may be understood to mean selecting a connection with a value corresponding to a connection speed assigned thereto that is higher than a threshold value.

[0126] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of the following: A, B, C or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items (e.g., A and A; B, B and C; A, A, B, C and C, etc.).

[0127] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The above-mentioned specific features and actions are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except in the appended claims.

[0128] If necessary, various aspects may be modified to employ the systems, functions, and concepts of the various references described above to provide yet further embodiments. If statements or subject matter in a document incorporated by reference conflict with statements or subject matter of the present application, the present application shall prevail.

Claims

1. A method for receiving input in an artificial reality environment using multiple modalities, the method comprising: Displaying a virtual keyboard in an artificial reality environment; receiving user gaze input associated with the virtual keyboard; placing an indicator on the virtual keyboard according to the received user gaze input; receiving a user hand input as a sliding segment on a surface of a controller device, the sliding segment having a starting point and an ending point, wherein the starting point of the sliding segment is mapped to the placed indicator; dynamically moving the indicator on the virtual keyboard based on the relative positions of the starting point and the ending point on the surface of the controller device; and The sliding segment is parsed into two or more selections in the virtual keyboard according to the relative positions of the starting point and the ending point.

2. The method according to claim 1, wherein: The origination point is received at the controller device at a first time, and the indicator is placed in response to a user gaze input proximate to the first time; and / or Wherein, prior to receiving the user hand input controlling movement of the indicator, there is no mapping of areas of the surface of the controller device to portions of the virtual keyboard.

3. The method according to claim 1 or 2, wherein: The controller device comprises a touch pad and the user hand input comprises a touch input.

4. The method according to claim 3, further comprising: detecting receipt of an edge touch input at an edge of the touch panel, wherein the edge touch input is approximately stationary for a threshold duration; and dynamically moving the indicator along a vector corresponding to the edge touch input for the duration; and / or The vector is defined at least according to the edge position of the touch panel receiving the edge touch input; and / or The edge touch input is detected when a touch input is received in a predetermined edge area of ​​the touch panel.

5. The method according to any one of claims 3 to 4, wherein: The initiation of the touch input at the trackpad initiates a point-to-point mapping that maps the starting point of the slide segment to the indicator placed on the virtual keyboard.

6. A method according to any one of the preceding claims, wherein: Parsing the sliding segment based on the relative positions of the start point and the end point includes: using a predictive text model to generate one or more matching probabilities for candidate selections of alphanumeric characters, wherein the sliding segment is parsed using the generated one or more matching probabilities.

7. A computing system for receiving input in an artificial reality environment using multiple modalities, the computing system comprising: one or more processors; as well as one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: Displaying a virtual keyboard in an artificial reality environment; receiving user gaze input associated with the virtual keyboard; parsing the user gaze input to determine a first selection in the virtual keyboard and placing the displayed indicator on the virtual keyboard at the first selection; receiving, via a controller device, a user hand input having a start point and an end point, wherein the start point is mapped to the indicator placed at the first selection; dynamically moving the indicator on the virtual keyboard according to the relative positions of the starting point and the ending point; and The user hand input is parsed into two or more selections in the virtual keyboard according to the first selection and the relative positions of the start point and the end point.

8. The computing system of claim 7, wherein: The starting point is received at the controller device at a first time, and the first selection is resolved based on a user gaze input proximate to the first time; and / or Wherein, prior to receiving the user hand input controlling movement of the indicator, there is no mapping of areas of the surface of the controller device to portions of the virtual keyboard.

9. The computing system according to claim 7 or 8, wherein: The controller device comprises a touch pad and the user hand input comprises a touch input.

10. The computing system of claim 9, wherein: The process also includes: detecting receipt of an edge touch input at an edge of the touch panel, wherein the edge touch input is approximately stationary for a threshold duration; and dynamically moving the indicator along a vector corresponding to the edge touch input for the duration; and / or The vector is defined at least according to the edge position of the touch panel receiving the edge touch input; and / or The edge touch input is detected when a touch input is received in a predetermined edge area of ​​the touch panel.

11. The computing system according to any one of claims 9 to 10, wherein: Initiation of the touch input at the trackpad initiates a point-to-point mapping that maps the starting point of the slide segment to the indicator placed at the first selection.

12. The computing system according to any one of claims 7 to 11, wherein: Parsing the sliding segment based on the first selection and the relative positions of the start point and the end point includes: using a predictive text model to generate one or more matching probabilities for candidate selections of alphanumeric characters, wherein the sliding segment is parsed using the generated one or more matching probabilities.

13. A computer-readable storage medium storing instructions which, when executed by a computing system or preferably by a computing system according to any one of claims 7 to 12, cause the computing system to perform a process for receiving input in an artificial reality environment using multiple modalities, the process comprising: Displaying a virtual keyboard in an artificial reality environment; receiving user gaze input associated with the virtual keyboard; placing an indicator on the virtual keyboard according to the received user gaze input; receiving, via a controller device, a user hand input having a start point and an end point, wherein the start point is mapped to the indicator placed at the first selection; dynamically moving the indicator on the virtual keyboard according to the relative positions of the starting point and the ending point; and The user hand input is interpreted as two or more selections in the virtual keyboard according to the relative positions of the start point and the end point.

14. The computer-readable storage medium of claim 13, wherein: The origination point is received at the controller device at a first time, and the indicator is positioned based on a user gaze input proximate to the first time.

15. A software product comprising instructions which, when executed by a computing system or preferably by a computing system according to any one of claims 7 to 12, cause the computing system to perform a process for receiving input in an artificial reality environment using multiple modalities, the process comprising: Displaying a virtual keyboard in an artificial reality environment; receiving user gaze input associated with the virtual keyboard; placing an indicator on the virtual keyboard according to the received user gaze input; receiving, via a controller device, a user hand input having a start point and an end point, wherein the start point is mapped to the indicator placed at the first selection; dynamically moving the indicator on the virtual keyboard according to the relative positions of the starting point and the ending point; and The user hand input is interpreted as two or more selections in the virtual keyboard according to the relative positions of the start point and the end point.