Virtual keyboard based on AR

Abstract

A gesture-based text input user interface for an augmented reality (AR) device is provided. The AR system detects a start text input gesture made by a user of the AR system, generates a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys, and provides the virtual keyboard user interface to the user. The AR system determines a user's selection of one or more selected virtual keys of the plurality of virtual keys using the one or more camera devices, and generates input text data based on the one or more selected virtual keys. The AR system provides input text data to a user using a display of the AR system.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. application serial number 17 / 804,818, filed on May 31, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to user interfaces, and more particularly to user interfaces used in augmented reality and virtual reality. Background Art

[0004] A head-mounted device can be implemented with a transparent or translucent display through which the user of the head-mounted device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or translucent display, and can also see objects (e.g., virtual objects such as renderings of 2D or 3D graphic models, images, videos, text, etc.) generated for display as part of the surrounding environment and / or superimposed on the surrounding environment. This is generally referred to as "augmented reality" or "AR". The head-mounted device can also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is generally referred to as "virtual reality" or "VR". As used herein, unless the context indicates otherwise, the term "AR" refers to either or both of augmented reality and virtual reality as traditionally understood.

[0005] A user of a head mounted device can access and use computer software applications to perform various tasks or participate in recreational activities. Performing tasks or participating in recreational activities may include entering text. To enter text, the user interacts with a text input user interface provided by the head mounted device. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of a head mounted device according to some examples.

[0008] Figure 2 According to some examples Figure 1 Additional views of the head-mounted device.

[0009] Figure 3 is a diagrammatic representation of a machine in the form of a computing device according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.

[0010] Figure 4A is a sequence diagram of gesture-based keyboard processing for an AR system according to some examples.

[0011] Figure 4B Gestures for directing an AR system are shown according to some examples.

[0012] Figure 4C A virtual keyboard user interface for an AR system is shown according to some examples.

[0013] Figure 5A is a sequence diagram of hand tracking keyboard processing for an AR system according to some examples.

[0014] Figure 5B Another gesture for directing an AR system is shown according to some examples.

[0015] Figure 5C Another virtual keyboard user interface for an AR system is shown according to some examples.

[0016] Fig. 6A is a sequence diagram of another hand tracking keyboard process for an AR system according to some examples.

[0017] Figure 6B Another gesture for directing an AR system is shown according to some examples.

[0018] Figure 6C Another virtual keyboard user interface for an AR system is shown according to some examples.

[0019] Figure 7 is a block diagram illustrating a software architecture within which the present disclosure may be implemented according to some examples.

[0020] Figure 8 is a block diagram showing a networked system including details of a head-mounted AR system according to some examples.

[0021] Fig. 9 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network according to some examples. DETAILED DESCRIPTION

[0022] AR systems, such as AR devices worn by users, are limited when it comes to available user input modalities. Indicating user intent and invoking an action or application is more complex for users of AR systems when compared to other mobile devices such as mobile phones. When using a mobile phone, a user can go to the home screen and tap on a specific icon to launch an application. However, due to the lack of physical input devices such as a touch screen or keyboard, such interactions are not easily performed on AR systems. Typically, users can indicate their intent by pressing a limited number of hardware buttons or using a small touchpad. Therefore, it is desirable to have the following input modality that allows a wider range of inputs that can be used by the user to indicate input intent through user input.

[0023] According to some examples, an input modality that may be used with an AR system is hand tracking combined with direct manipulation of virtual objects (DMVO), wherein a user is provided with a user interface displayed to the user in an AR overlay with a 2D or 3D rendering. The rendering is a graphical model in 2D or 3D, wherein virtual objects located in the model correspond to interactive elements of the user interface. In this manner, the user perceives virtual objects as objects within an overlay in the user's field of view of a real-world scene when wearing the AR system, or perceives virtual objects as objects within a virtual world as seen by the user when wearing the AR system. To allow the user to manipulate virtual objects, the AR system detects the user's hands and tracks the movement, position, and / or location of the hands to determine the user's interaction with the virtual objects.

[0024] In another example, gestures that do not involve DMVO provide another input modality suitable for use with an AR system, such as an AR device worn by a user. The gesture is made by a user who moves and positions parts of the user's body, and those parts of the user's body can be detected by the AR system when the user is wearing the AR system. The detectable parts of the user's body may include parts of the user's upper body, arms, hands, and fingers. The components of the gesture may include the movement of the user's arms and hands, the position of the user's arms and hands in space, and the positioning of the user holding his upper body, arms, hands, and fingers. Gestures are useful in providing AR experiences to users because gesture input provides a way to provide user input to the AR system during the AR experience without allowing the user to divert their attention from the AR experience. As an example, in an AR experience that is an operating manual for a machine, a user can simultaneously view a machine in a real-world scene through the lens of the AR system, view the AR overlay on the real-world scene view of the machine, and provide user input to the AR system.

[0025] By combining hand tracking DMVO and gesture input modalities, an improved text input user interface is provided to a user of an AR system. In some examples, the user makes a gesture to open a virtual keyboard. The virtual keyboard includes virtual keys that the user manipulates using a text input hand (e.g., his left hand). The AR system determines the user's selection of one or more selected virtual keys from a plurality of virtual keys using a DMVO method, and generates input text data based on the one or more selected virtual keys. The AR system provides the input text data to the user using a display of the AR system.

[0026] In another example, the user is provided with an infinite light cursor, which the user manipulates using his text input hand. The user manipulates the infinite light cursor to select one or more virtual keys of the virtual keyboard by intersecting the infinite light cursor with the one or more virtual keys.

[0027] Other technical features will be readily apparent to those skilled in the art from the following drawings, descriptions and claims.

[0028] Figure 1 According to some examples, a head mounted device (e.g., Figure 1 100). The glasses 100 may include a frame 102 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a bridge 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be disposed in the respective left optical element holder 104 and the right optical element holder 106. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0029] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0030] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 802 discussed below.

[0031] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0032] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0033] In some examples, left camera 114 and right camera 116 provide video frame data for use by glasses 100 to extract 3D information from a real scene.

[0034] The glasses 100 may also include a touchpad 126 mounted to or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is typically arranged vertically, approximately parallel to the temple of the user in some examples. As used herein, typically vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. Additional user input can be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide a device by which the glasses 100 can receive input from the user of the glasses 100.

[0035] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured in left and right optical element holders 104, 106, respectively.

[0036] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0037] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 superimposed with a view of the real-world scene seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 superimposed with a view of the real-world scene seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides an optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's view of the real-world scene, including displaying a user interface to the user of the glasses 100.

[0038] However, it should be understood that other display technologies or configurations may be utilized within the optical engine to display images to a user in the user's field of view. For example, instead of providing a projector 204 and a waveguide, an LCD, LED or other display panel or surface may be provided.

[0039] In use, the user of the glasses 100 will be presented with information, content, and various user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Figure 8 Interact with the glasses 100 through voice input or touch input on a client device 826 shown in FIG. 1 and / or hand movements, position, and orientation detected by the glasses 100 .

[0040] Figure 3 is a diagrammatic representation of a machine 300 (e.g., a computing device) in which instructions 310 (e.g., software, programs, applications, applet, app, or other executable code) may be executed that cause the machine 300 to perform any one or more of the methodologies discussed herein. The machine 300 may be used as Figure 1 The computer 120 of the glasses 100. For example, the instructions 310 can cause the machine 300 to perform any one or more of the methodologies described herein. The instructions 310 transform a general purpose, unprogrammed machine 300 into a specific machine 300 that is programmed to perform the functions described and illustrated in the manner described. The machine 300 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 300 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, the instructions 310 specifying actions to be taken by the machine 300. In addition, while a single machine 300 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute the instructions 310 to perform any one or more of the methodologies discussed herein.

[0041] Machine 300 may include processor 302, memory 304, and I / O component 306, which may be configured to communicate with each other via bus 344. In some examples, processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 308 and processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but machine 300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0042] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via the bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. During execution of the instructions 310 by the machine 300, the instructions 310 may also reside, in whole or in part, within the main memory 314, within the static memory 316, within the machine-readable medium 320, within the storage unit 318, within one or more of the processors 302 (e.g., within a processor's cache memory), or within any suitable combination thereof.

[0043] I / O components 306 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is to be understood that I / O components 306 may include Figure 3306. In various examples, the I / O component 306 may include an output component 328 and an input component 332. The output component 328 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component 332 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides a touch position and / or force or touch gesture, or other tactile input component), an audio input component (e.g., a microphone), etc.

[0044] In another example, the I / O component 306 may include: a biometric component 334, a motion component 336, an environmental component 338, or a positioning component 340, as well as various other components. For example, the biometric component 334 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 336 may include an inertial measurement unit (IMU), an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 338 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals associated with the surrounding physical environment. Positioning component 340 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can provide altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0045] A variety of technologies may be used to implement communications. I / O components 306 also include communication components 342 that are operable to couple machine 300 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication components 342 may include a network interface component or another suitable device that interfaces with network 322. In other examples, communication components 342 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Components (e.g. Low power consumption), Device 324 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0046] In addition, the communication component 342 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Hypercode, UCC RSS-2D barcode and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 342, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0047] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 310) when executed by processor 302 cause various operations to implement the disclosed examples.

[0048] Instructions 310 may be transmitted or received via a network interface device (e.g., a network interface component included in communication component 342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 322. Similarly, instructions 310 may be transmitted or received to device 324 via coupling 326 (e.g., a peer-to-peer coupling) using a transmission medium.

[0049] Figure 4A is a sequence diagram of a gesture-based keyboard process 400 for an AR system (e.g., glasses 100) according to some examples, Figure 4B is an illustration of a start / stop text input gesture 444, and Figure 4C 4 is an illustration of a virtual keyboard user interface 450. During gesture-based keyboard processing 400, the AR system implements the virtual keyboard user interface 450 with a gesture text input application 426 using a gesture recognition method and a DMVO method.

[0050] During the gesture-based keyboard process 400, in operation 430, one or more cameras 420 of the AR system generate real-world scene video frame data 432 of a real-world scene from the perspective of a user of the AR system. The one or more cameras 420 transmit the real-world scene video frame data 432 to the tracking service 424. The real-world scene video frame data 432 includes tracking video frame data of detectable parts of the user's body (including parts of the user's upper torso, arms, hands, and fingers). The tracking video frame data includes: video frame data of the movement of parts of the user's upper torso, arms, and hands when the user makes gestures or moves his hands and fingers to interact with the virtual keyboard user interface 450; video frame data of the position of the user's arms and hands in space when the user makes gestures or moves his hands and fingers to interact with the virtual keyboard user interface 450; and video frame data of the user maintaining the positioning of his upper torso, arms, hands, and fingers when the user makes gestures or moves his hands and fingers to interact with the virtual keyboard user interface 450.

[0051] In operation 440, the tracking service 424 scans, detects, and tracks landmarks in the real-world scene about parts of the user's upper body, arms, and hands. In some examples, the tracking service 424 receives real-world scene video frame data 432 from one or more cameras 420, and extracts features of the user's upper body, arms, and hands from the tracking video frame data included in the real-world scene video frame data 432. The tracking service 424 generates current tracking data 464 based on the extracted features. The current tracking data 464 includes landmark data, which includes landmark identification, location in the real-world scene, and classification information of one or more landmarks associated with the user's upper body, arms, and hands. The tracking service 424 transmits the current tracking data 464 to the gesture recognition service 422. In addition, the tracking service 424 enables the current tracking data 464 to be used for applications executed on the AR system, such as the gesture text input application 426.

[0052] In operation 438, gesture recognition service 422 receives current tracking data 464 from tracking service 424, and generates current detected gesture data 436 based on current tracking data 464. In some examples, gesture recognition service 422 generates one or more current skeletal models of the user's upper body, arms, hands, and fingers based on landmark data of landmarks included in current tracking data 464. Gesture recognition service 422 compares one or more current skeletal models with previously generated gesture skeletal models. Gesture recognition service 422 determines a detected gesture based on comparison of one or more current skeletal models with gesture skeletal models, and generates current detected gesture data 436 based on the detected gesture. In another example, gesture recognition service 422 generates one or more current skeletal models based on landmark data. Gesture recognition service 422 determines a detected gesture based on classifying the current skeletal model using an artificial intelligence method and a gesture model previously generated using a machine learning method. Gesture recognition service 422 generates current detected gesture data 436 based on the detected gesture.

[0053] In some examples, one or more cameras 420 , tracking service 424 , and gesture recognition service 422 operate continuously so that currently detected gesture data 436 and currently detected gesture data 436 are available to applications executing on the AR system as needed.

[0054] In operation 402, gesture text input application 426 detects a start text input gesture, such as start / stop text input gesture 444, based on current detected gesture data 436 received from gesture recognition service 422. The start text input gesture is an instruction by a user to begin text input into text scene object 442 of an AR experience provided by the AR system to the user.

[0055] In operation 404, in response to detecting the start text input gesture, the gesture text input application 426 generates a virtual keyboard user interface 450 including a virtual keyboard 446. The virtual keyboard 446 includes a plurality of virtual objects that constitute interactive virtual keys 452 of the virtual keyboard 446. The virtual keys 452 are geometric virtual objects that have corresponding positions in a user interface geometric model or volume that corresponds to the volume of space occupied by the virtual keyboard user interface 450 in the real world scene. As an example, the width (X) and height (Y) of the user interface geometric model are defined by the field of view from the perspective of the user of the AR system, and the depth (Z) is defined by a physical length of 100 cm with the user's eye location as the origin. The virtual keyboard 446 is assigned a depth position of 50 cm in the user interface geometric model for the user's eye location, which allows the user to reach the virtual keyboard 446 with his hand while partially extending his arm.

[0056] The gestural text input application 426 generates rendering data 434 of the virtual keyboard user interface 450 and transmits the rendering data 434 to the optical engine 428 of the AR system. In operation 406, the optical engine 428 provides the virtual keyboard user interface 450 to the user in the display of the AR system based on the rendering data 434.

[0057] In operation 408, gesture text input application 426 detects holding of an input text gesture (e.g., input text gesture 448) by the user using their free hand 458 based on current detected gesture data 436 received from gesture recognition service 422. For example, gesture text input application 426 determines that the current detected gesture identified in current detected gesture data 436 is the same as input text gesture 448.

[0058] When the user maintains the input text gesture 448, the user moves his text input hand 460 in a continuous motion 454 across the virtual keys to input the expected text, such as a word. While performing the continuous motion 454, the user will pass through the virtual keys representing the characters included in the expected text, as well as additional virtual keys representing the characters not included in the expected text. When one or more cameras 420, tracking service 424, and gesture recognition service 422 operate continuously, the current detection gesture data 436 includes continuous motion gesture data of the continuous motion 454 generated when the user maintains the input text gesture 448. In operation 410, the gesture text input application 426 receives the current detection gesture data 436 and collects the continuous motion gesture data included in the current detection gesture data 436.

[0059] In operation 412, gesture text input application 426 detects a release of input text gesture 448 by the user based on current detected gesture data 436 received from gesture recognition service 422. For example, gesture text input application 426 determines that the currently detected gesture identified in current detected gesture data 436 is not an input text gesture 448 held by the user using their free hand 458.

[0060] In operation 414, gesture text input application 426 generates input text data 456 based on the collected continuous motion gesture data of continuous motion 454. In some examples, gesture text input application 426 maps the collected continuous motion gesture data to text data using artificial intelligence methods and a continuous motion gesture model previously generated using machine learning methods. Gesture text input application 426 generates current detection gesture data 436 based on the mapped text data.

[0061] The gesture text input application 426 communicates the input text data 456 to the text scene object 442. In operation 416, the text scene object 442 provides the input text data 456 to the user in a display of the AR system.

[0062] In operation 418, gesture text input application 426 detects an end text input gesture, such as but not limited to start / stop text input gesture 444, based on current detected gesture data 436 received from gesture recognition service 422, and gesture text input application 426 closes virtual keyboard user interface 450 and terminates. In some examples, the end text input gesture can be any gesture, such as an upward swipe gesture, a downward swipe gesture, a left swipe gesture, a right swipe gesture, a fist, raising a hand in a "stop gesture", etc.

[0063] In some examples, in operation 418, gesture text input application 426 executes loop 462 until gesture text input application 426 detects that the user makes an end text input gesture. Loop 462 includes operation 408 (detecting that the user makes and holds input text gesture 448), operation 410 (collecting continuous motion gesture data), operation 412 (detecting that the user releases the input text gesture), and operation 414 (generating input text data 456 and transmitting input text data 456 to text scene object 442). In this way, the user can enter multiple words or text into text scene object 442.

[0064] In some examples, during operation 410, when the gesture text input application 426 collects continuous motion gesture data, the gesture text input application 426 generates estimated text in the typeahead search mode based on the partial set of the collected continuous motion gesture data. For example, when the user performs a continuous motion with his text input hand 460 while holding the input text gesture 448 with his free hand 458, the gesture text input application 426 determines the partial set of continuous motion gesture data before the user releases the input text gesture 448 detected in operation 418. The gesture text input application 426 uses an artificial intelligence method and a continuous motion gesture model previously generated using a machine learning method to map the partial set of continuous motion gesture data to text data. The gesture text input application 426 generates input text data 456 based on the mapped text data, and transmits the input text data 456 to the text scene object 442. In operation 416, the text scene object 442 provides the input text data 456 to the user in the display of the AR system. If the user determines that the estimated text is the expected text, the user releases the input text gesture 448.

[0065] In some examples, virtual keyboard user interface 450 remains within the field of view of the user of the AR system at a fixed depth (z distance) from the perspective of the user using the AR system.

[0066] In some examples, virtual keyboard 446 includes a virtual key that can be selected by a user to close virtual keyboard 446 and terminate gesture-based keyboard processing 400, such as a "Cancel" or "Exit" virtual key.

[0067] In some examples, gesture text input application 426 is an application used by the AR system to provide an AR experience to the user. The AR system uses gesture-based keyboard processing 400 to provide an input modality for the user to enter input text data 456 into a text scene object 442 of the AR experience.

[0068] In some examples, the AR system's gesture text input application 426 performs the functionality of the gesture recognition service 422 and the tracking service 424 by leveraging various APIs and system libraries.

[0069] Figure 5A is a sequence diagram of a hand tracking keyboard process 500 of an AR system (eg, glasses 100) according to some examples, Figure 5B is an illustration of a start / end text input gesture 548, and Figure 5C 5 is an illustration of a virtual keyboard user interface 538. During the hand tracking keyboard process 500, the AR system utilizes gesture recognition methods and DMVO methods to implement a virtual keyboard user interface 538 for the hand tracking text input application 522.

[0070] During the hand tracking keyboard process 500, in operation 526, one or more cameras 516 of the AR system generate real-world scene video frame data 528 of the real-world scene from the perspective of the user of the AR system. The one or more cameras 516 transmit the real-world scene video frame data 528 to the tracking service 520. The real-world scene video frame data 528 includes tracking video frame data of detectable parts of the user's body (including parts of the user's upper torso, arms, hands, and fingers). The tracking video frame data includes: video frame data of the movement of parts of the user's upper torso, arms, and hands when the user makes gestures or moves his hands and fingers to interact with the virtual keyboard user interface 538; video frame data of the position of the user's arms and hands in space when the user makes gestures or moves his hands and fingers to interact with the virtual keyboard user interface 538; and video frame data of the user maintaining the positioning of his upper torso, arms, hands, and fingers when the user makes gestures or moves his hands and fingers to interact with the virtual keyboard user interface 538.

[0071] In operation 536, the tracking service 520 scans, detects, and tracks landmarks in the real-world scene about portions of the user's upper body, arms, and hands. In some examples, the tracking service 520 receives real-world scene video frame data 528 from one or more cameras 516, and extracts features of the user's upper body, arms, and hands from the tracking video frame data included in the real-world scene video frame data 528. The tracking service 520 generates current tracking data 550 based on the extracted features. The tracking data includes landmark data, which includes landmark identification, location in the real-world scene, and classification information identifying one or more landmarks associated with the movement of the user's upper body, arms, and hands. The tracking service 520 transmits the current tracking data 550 to the gesture recognition service 518. In addition, the tracking service 520 enables the current tracking data 550 to be used in applications executed on the AR system, such as a hand tracking text input application 522.

[0072] In operation 534, the gesture recognition service 518 receives current tracking data 550 from the tracking service 520, and generates current detected gesture data 532 based on the current tracking data 550. In some examples, the gesture recognition service 518 generates one or more current skeletal models of the user's upper body, arms, hands, and fingers based on the landmark data of the landmarks included in the current tracking data 550. The gesture recognition service 518 compares the one or more current skeletal models with the previously generated gesture skeletal models. The gesture recognition service 518 determines the detected gesture based on the comparison of the one or more current skeletal models with the gesture skeletal models, and generates the current detected gesture data 532 based on the detected gesture. In another example, the gesture recognition service 518 generates one or more current skeletal models based on the landmark data. The gesture recognition service 518 determines the detected gesture based on classifying the current skeletal model using an artificial intelligence method and a gesture model previously generated using a machine learning method. The gesture recognition service 518 generates the current detected gesture data 532 based on the detected gesture.

[0073] In some examples, one or more cameras 516 , tracking service 520 , and gesture recognition service 518 operate continuously so that current detected gesture data 532 and current tracking data 550 can be used as needed for any application being executed by the AR system.

[0074] During execution of the hand tracking text input application 522 by the AR system, in operation 502, the hand tracking text input application 522 detects a start text input gesture, such as a start / end text input gesture 548, based on current detected gesture data 532 received from the gesture recognition service 518. The start text input gesture is an instruction by the user to begin text input into a text scene object 554 of an AR experience provided by the AR system to the user.

[0075] In response to detecting an input text gesture, in operation 504, the hand tracking text input application 522 generates a virtual keyboard user interface 538 including a virtual keyboard 540. The virtual keyboard 540 includes a plurality of virtual objects that constitute interactive virtual keys 544 of the virtual keyboard 540. The virtual keys 544 are geometric virtual objects that have corresponding positions in a user interface geometric model or volume that corresponds to the volume of space occupied by the virtual keyboard user interface 538 in the real world scene. As an example, the width (X) and height (Y) of the user interface geometric model are defined by the field of view from the perspective of the user of the AR system, and the depth (Z) is defined by a physical length of 100 cm with the user's eye location as the origin. The virtual keyboard 540 is assigned a depth position of 50 cm in the user interface geometric model for the user's eye location, which allows the user to reach the virtual keyboard 540 with his hand while partially extending his arm.

[0076] In operation 504, the hand tracking text input application 522 generates rendering data 530 of a virtual keyboard user interface 538 and transmits the rendering data 530 to the optical engine 524 of the AR system. In operation 506, the optical engine 524 provides the virtual keyboard user interface 538 to the user in the display of the AR system based on the rendering data 530.

[0077] In operation 508, the hand tracking text input application 522 receives current tracking data 550 from the tracking service 520. The hand tracking text input application 522 detects landmarks associated with the user's text input hand 542 (e.g., the index finger tip of the user's text input hand 542) based on the current tracking data 550. The hand tracking text input application 522 generates landmark colliders 546 in the virtual keyboard user interface 538 based on the landmarks. To select a virtual key from the virtual keys 544, the user moves their text input hand 542 to move the landmark collider 546 within the virtual keyboard user interface 538. To move the landmark collider 546 over the virtual key 544 without selecting the virtual key, the user retracts their text input hand 542 from the position of the virtual keyboard 540 in the user interface geometry model, thereby clearing the virtual key 544. To select a virtual key, the user positions his text input hand 542 over the virtual key and extends his text input hand 542 to "press" or "poke" the virtual key until the landmark collider 546 collides with the selected virtual key. The hand tracking text input application 522 determines the user's selection of one or more selected virtual keys by detecting a collision between the landmark collider 546 and the one or more selected virtual keys.

[0078] In operation 510, the hand tracking text input application 522 generates input text data 556 based on the selected virtual key. In some examples, each virtual key in the virtual keys 544 is associated with a character. The hand tracking text input application 522 converts the associated character into the input text data 556.

[0079] The hand tracking text input application 522 transmits the input text data 556 to the text scene object 554. In operation 512, the text scene object 554 provides the input text data 556 to the user in the display of the AR system.

[0080] In operation 514, the hand tracking text input application 522 detects an end text input gesture, such as, but not limited to, a start / end text input gesture 548, based on the current detected gesture data 532 received from the gesture recognition service 518. In response to detecting the end text input gesture, the hand tracking text input application 522 closes the virtual keyboard user interface 538 and terminates. In some examples, the end text input gesture can be any gesture, such as an upward swipe gesture, a downward swipe gesture, a left swipe gesture, a right swipe gesture, a fist, raising a hand in a "stop gesture," and the like.

[0081] In some examples, the hand tracking text input application 522 executes loop 552 until an end text input gesture, such as start / end text input gesture 548, is detected in operation 514. Loop 552 includes operation 508 (determining the selected virtual key) and operation 510 (generating input text data 556 and transmitting 556 to text scene object 554). In this way, the hand tracking text input application 522 allows the user to enter multiple characters into the text scene object 554.

[0082] In some examples, the virtual keyboard 540 includes a virtual key that can be selected by the user to close the virtual keyboard 540 and terminate the hand tracking keyboard process 500, such as a "Cancel" or "Exit" virtual key.

[0083] In some examples, one or more cameras 516 , tracking service 520 , and gesture recognition service 518 are executed continuously by the AR system so that current detected gesture data 532 and current tracking data 550 are available as needed to any application being executed by the AR system.

[0084] In some examples, virtual keyboard user interface 538 remains within the field of view of the user of the AR system at a fixed depth (z distance) from the perspective of the user using the AR system.

[0085] In some examples, hand tracking text input application 522 is an application provided by the AR system to the user during the AR experience provided to the user. The AR system provides an input modality to the user using hand tracking keyboard process 500 to enter input text data 556 into a text scene object 554 of the AR experience.

[0086] In some examples, the hand tracking text input application 522 of the AR system performs the functionality of the gesture recognition service 518 and the tracking service 520 by leveraging various APIs and system libraries.

[0087] Fig. 6A is a sequence diagram of a hand tracking keyboard process 600 of an AR system (eg, glasses 100 ) according to some examples, Figure 6B is an illustration of a start / end text input gesture 656, and Figure 6C 6 is an illustration of a virtual keyboard user interface 640. During the hand tracking keyboard process 600, the AR system utilizes gesture recognition methods and DMVO methods to implement a virtual keyboard user interface 640 for the hand tracking text input application 624.

[0088] During the hand tracking keyboard process 600, in operation 628, one or more cameras 618 of the AR system generate real-world scene video frame data 630 of the real-world scene from the perspective of the user of the AR system. The one or more cameras 618 transmit the real-world scene video frame data 630 to the tracking service 622. Included in the real-world scene video frame data 630 are tracking video frame data of detectable parts of the user's body (including parts of the user's upper torso, arms, hands, and fingers). The tracking video frame data includes: video frame data of the movement of parts of the user's upper torso, arms, and hands when the user makes gestures or moves his hands and fingers to interact with the virtual keyboard user interface 640; video frame data of the position of the user's arms and hands in space when the user makes gestures or moves his hands and fingers to interact with the virtual keyboard user interface 640; and video frame data of the user maintaining the positioning of his upper torso, arms, hands, and fingers when the user makes gestures or moves his hands and fingers to interact with the virtual keyboard user interface 640.

[0089] In operation 638, the tracking service 622 scans, detects, and tracks landmarks in the real-world scene about portions of the user's upper body, arms, and hands. In some examples, the tracking service 622 receives real-world scene video frame data 630 from one or more cameras 618, and extracts features of the user's upper body, arms, and hands from the tracking video frame data included in the real-world scene video frame data 630. The tracking service 622 generates current tracking data 658 based on the extracted features. The tracking data includes landmark data, which includes landmark identification, location in the real-world scene, and classification information identifying one or more landmarks associated with the movement of the user's upper body, arms, and hands. The tracking service 622 transmits the current tracking data 658 to the gesture recognition service 620. In addition, the tracking service 622 enables the current tracking data 658 to be used for applications such as hand tracking text input applications 624 that are being executed on the AR system.

[0090] In operation 636, the gesture recognition service 620 receives current tracking data 658 from the tracking service 622, and generates current detected gesture data 634 based on the current tracking data 658. In some examples, the gesture recognition service 620 generates one or more current skeletal models of the user's upper body, arms, hands, and fingers based on the landmark data of the landmarks included in the current tracking data 658. The gesture recognition service 620 compares the one or more current skeletal models with the previously generated gesture skeletal models. The gesture recognition service 620 determines the detected gesture based on the comparison of the one or more current skeletal models with the gesture skeletal models, and generates the current detected gesture data 634 based on the detected gesture. In another example, the gesture recognition service 620 generates one or more current skeletal models based on the landmark data. The gesture recognition service 620 determines the detected gesture based on classifying the current skeletal model using an artificial intelligence method and a gesture model previously generated using a machine learning method. The gesture recognition service 620 generates the current detected gesture data 634 based on the detected gesture.

[0091] In some examples, one or more cameras 618 , tracking service 622 , and gesture recognition service 620 operate continuously so that current detected gesture data 634 and current tracking data 658 are available as needed for any application being executed by the AR system.

[0092] During execution of the hand tracking text input application 624 by the AR system, in operation 602, the hand tracking text input application 624 detects a start text input gesture, such as a start / end text input gesture 656, based on the current detected gesture data 634 received from the gesture recognition service 620. The start text input gesture is an instruction by the user to begin text input into a text scene object 662 of an AR experience provided by the AR system to the user.

[0093] In response to detecting an input text gesture, in operation 604, the hand tracking text input application 624 generates a virtual keyboard user interface 640 including a virtual keyboard 642. The virtual keyboard 642 includes a plurality of virtual objects that constitute interactive virtual keys 648 of the virtual keyboard 642. The virtual keys 648 are geometric virtual objects that have corresponding positions in a user interface geometric model or volume that corresponds to the volume of space occupied by the virtual keyboard user interface 640 in the real world scene. As an example, the width (X) and height (Y) of the user interface geometric model are defined by the field of view from the perspective of the user of the AR system, and the depth (Z) is defined by a physical length of 100 cm with the user's eye location as the origin. The virtual keyboard 642 is assigned a depth position of 50 cm in the user interface geometric model for the user's eye location, which allows the user to reach the virtual keyboard 642 with his hand while partially extending his arm.

[0094] In addition to the virtual keyboard 642, the hand tracking text input application 624 generates an infinite light ray cursor 652 associated with the user's text input hand 646 (e.g., the user's left hand) in the virtual keyboard user interface 640. The hand tracking text input application 624 determines a landmark 650 associated with the text input hand 646, such as the tip of the index finger of the text input hand, based on the current tracking data 658 received from the tracking service 622. The hand tracking text input application 624 sets the origin of the infinite light ray cursor 652 at the user's eye location. During operation, the hand tracking text input application 624 projects the infinite light ray cursor 652 through the landmark 650 associated with the user's text input hand 646. When the user moves his text input hand 646, the infinite light ray cursor 652 intersects one or more of the virtual keys 648 to enable the user to control the infinite light ray cursor 652 to the virtual key that the user wants to select, one or more of which is aligned with the user's eye location and the landmark 650.

[0095] As part of operation 604, the hand tracking text input application 624 generates rendering data 632 of a virtual keyboard user interface 640 and transmits the rendering data 632 to an optical engine 626 of the AR system. In operation 606, the optical engine 626 provides the virtual keyboard user interface 640 to the user in a display of the AR system based on the rendering data 632.

[0096] The user moves his text input hand 646 to move the landmark 650 within the virtual keyboard user interface 640 to select a virtual key from the virtual keys 648 by manipulating the infinite ray cursor 652 to intersect with a virtual key in the virtual keys 648. The user makes a selection key gesture, such as gesture 654, with the free hand 644, such as his right hand, to instruct the hand tracking text input application 624 to select the virtual key currently intersected by the infinite ray cursor 652. In operation 608, the hand tracking text input application 624 detects the selected key gesture 654 based on the current detection gesture data 634 received from the gesture recognition service 620.

[0097] In operation 610, hand tracking text input application 624 determines a user selection of a selected virtual key. In some examples, hand tracking text input application 624 detects an intersection between infinite ray cursor 652 and a virtual key in virtual keys 648. Hand tracking text input application 624 determines the intersecting virtual key as the selected virtual key.

[0098] In operation 612, the hand tracking text input application 624 generates input text data 664 based on the selected virtual key. In some examples, each virtual key in the virtual keys 648 is associated with a character. The hand tracking text input application 624 converts the associated character into input text data 664.

[0099] The hand tracking text input application 624 transmits the input text data 664 to the text scene object 662. In operation 614, the text scene object 662 provides the input text data 664 to the user in the display of the AR system.

[0100] In operation 616, the hand tracking text input application 624 detects an end text input gesture, such as, but not limited to, a start / end text input gesture 656, based on the current detected gesture data 634 received from the gesture recognition service 620. In response to detecting the end text input gesture, the hand tracking text input application 624 closes the virtual keyboard user interface 640 and terminates. In some examples, the end text input gesture can be any gesture, such as an upward swipe gesture, a downward swipe gesture, a left swipe gesture, a right swipe gesture, a fist, raising a hand in a "stop gesture," and the like.

[0101] In some examples, the hand tracking text input application 624 executes loop 660 until an end text input gesture is detected, such as the start / end text input gesture 656. Loop 660 includes operation 608 (detecting an input text gesture), operation 610 (determining the selected virtual key), and operation 612 (generating input text data 664 and transmitting 664 to the text scene object 662). In this way, the hand tracking text input application 624 allows the user to enter multiple characters into the text scene object 662.

[0102] In some examples, the infinite ray cursor 652 is not used. Instead, in operation 610, the hand tracking text input application 624 receives current tracking data 658 from the tracking service 622. The hand tracking text input application 624 detects a landmark 650 associated with the user's text input hand 646 (e.g., the index finger tip of the user's text input hand 646) based on the current tracking data 658. The hand tracking text input application 624 generates a landmark collider in the virtual keyboard user interface 640 based on the landmark 650. To select a virtual key from the virtual keys 648, the user moves their text input hand 646 to move the landmark collider within the virtual keyboard user interface 640. To move the landmark collider over the virtual key 648 without selecting the virtual key, the user retracts their text input hand 646 from the position of the virtual keyboard 642 in the user interface geometry model, thereby clearing the virtual key 648. To select a virtual key, the user positions his text input hand 646 over the virtual key and extends his text input hand 646 to "press" or "poke" the virtual key until the landmark collider collides with the selected virtual key. The hand tracking text input application 624 determines the user's selection of the selected virtual key by detecting a collision between the landmark collider and the selected virtual key.

[0103] In some examples, the virtual keyboard user interface 640 includes a user-selectable virtual object, such as a "Done," "Cancel," or "Exit" button or virtual key, that the user selects to terminate the hand tracking keyboard process 600. The user-selectable virtual object can be included as a virtual key in the virtual keyboard 642, or included in the text scene object 662, or included in the AR experience as a user-selectable button, etc.

[0104] In some examples, one or more cameras 618 , tracking service 622 , and gesture recognition service 620 are executed continuously by the AR system so that current detected gesture data 634 and current tracking data 658 are available as needed to any application being executed by the AR system.

[0105] In some examples, virtual keyboard user interface 640 remains within the field of view of the user of the AR system at a fixed depth (z distance) from the perspective of the user using the AR system.

[0106] In some examples, hand tracking text input application 624 is an application provided by the AR system to the user during an AR experience provided to the user. The AR system uses hand tracking keyboard process 600 to provide an input modality to the user to enter input text data 456 into a text scene object 442 of the AR experience.

[0107] In some examples, the hand tracking text input application 624 of the AR system performs the functionality of the gesture recognition service 620 and the tracking service 622 by utilizing various APIs and system libraries.

[0108] Figure 7 700 is a block diagram illustrating a software architecture 704 that may be installed on any one or more of the devices described herein. The software architecture 704 is supported by hardware, such as a machine 702, including a processor 720, a memory 726, and an I / O component 738. In this example, the software architecture 704 may be conceptualized as a stack of layers in which each layer provides specific functionality. The software architecture 704 includes layers such as an operating system 712, a library 708, a framework 710, and an application 706. In operation, the application 706 invokes an API call 750 through the software stack and receives a message 752 in response to the API call 750.

[0109] The operating system 712 manages hardware resources and provides public services. The operating system 712 includes, for example, a kernel 714, services 716, and drivers 722. The kernel 714 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 714 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 716 can provide other public services to other software layers. Drivers 722 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 722 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0110] The library 708 provides a low-level common infrastructure used by the application 706. The library 708 may include a system library 718 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 708 may include an API library 724, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting two-dimensional (2D) and three-dimensional (3D) graphics content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 708 may also include various other libraries 728 to provide many other APIs to the application 706 .

[0111] The framework 710 provides a high-level common infrastructure used by the applications 706. For example, the framework 710 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 710 may provide a wide range of other APIs that may be used by the applications 706, some of which may be specific to a particular operating system or platform.

[0112] In some examples, applications 706 may include a home application 736, a contacts application 730, a browser application 732, a book reader application 734, a location application 742, a media application 744, a messaging application 746, a game application 748, and a variety of other applications such as third-party applications 740. Application 706 is a program that executes the functions defined in the program. Various programming languages ​​can be used to create one or more of the applications 706 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party application 740 (e.g., a third-party application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on a mobile operating system or another mobile operating system. In this example, third-party applications 740 can call API calls 750 provided by the operating system 712 to facilitate the functions described herein.

[0113] Figure 8 830 is a block diagram showing a networked system 800 including details of the glasses 100 according to some examples. The networked system 800 includes the glasses 100, a client device 826, and a server system 832. The client device 826 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 836 and / or a high-speed wireless connection 834. The client device 826 is connected to the server system 832 via a network 830. The network 830 may include any combination of wired and wireless connections. The server system 832 may be one or more computing devices that are part of a service or network computing system. The client device 826 and any elements of the server system 832 and the network 830 may use Figure 7 and Figure 3 The details of the software architecture 704 or machine 300 described respectively are implemented.

[0114] The glasses 100 include a data processor 802, a display 810, one or more cameras 808, and additional input / output elements 816. The input / output elements 816 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 802. Figure 7 and Figure 3 Examples of input / output elements 816 are further discussed. For example, input / output elements 816 may include any I / O component 306, including output component 328, motion component 336, etc. Figure 2 Examples of display 810 are discussed in . In the particular examples described herein, display 810 includes displays for the user's left eye and right eye.

[0115] Data processor 802 includes image processor 806 (eg, video processor), GPU and display driver 838, tracking module 840, interface 812, low power circuitry 804, and high speed circuitry 820. The components of data processor 802 are interconnected by bus 842.

[0116] The interface 812 refers to any source of user commands provided to the data processor 802. In one or more examples, the interface 812 is a physical button that, when pressed, sends a user input signal from the interface 812 to the low-power processor 814. The low-power processor 814 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 814 can process pressing such a button within a first time period as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 812 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the camera 808. In other examples, the interface 812 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 826.

[0117] The image processor 806 includes circuitry for receiving signals from the camera 808 and processing those signals from the camera 808 into a format suitable for storage in the memory 824 or for transmission to the client device 826. In one or more examples, the image processor 806 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from the camera 808, and volatile memory used by the microprocessor in operation.

[0118] The low power circuit system 804 includes a low power processor 814 and a low power wireless circuit system 818. These elements of the low power circuit system 804 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 814 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 814 can accept user input signals from the interface 812. The low power processor 814 can also be configured to receive input signals or command communications from a client device 826 via a low power wireless connection 836. The low power wireless circuit system 818 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 818. In other examples, other low power communication systems may be used.

[0119] High-speed circuit system 820 includes a high-speed processor 822, a memory 824, and a high-speed wireless circuit system 828. High-speed processor 822 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 802. High-speed processor 822 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 834 using high-speed wireless circuit system 828. In some examples, high-speed processor 822 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Figure 7 The high-speed processor 822, which executes the software architecture of the data processor 802, manages data transmission with the high-speed wireless circuit system 828, in addition to any other duties. In some examples, the high-speed wireless circuit system 828 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 828 can implement other high-speed communication standards.

[0120] The memory 824 includes any storage device capable of storing camera data generated by the camera 808 and the image processor 806. Although the memory 824 is shown as being integrated with the high-speed circuitry 820, in other examples, the memory 824 may be a separate, independent element of the data processor 802. In some such examples, electrical wiring may provide a connection from the image processor 806 or the low-power processor 814 to the memory 824 through a chip including the high-speed processor 822. In other examples, the high-speed processor 822 may manage the addressing of the memory 824 so that the low-power processor 814 will direct the high-speed processor 822 whenever a read or write operation involving the memory 824 is desired.

[0121] The tracking module 840 estimates the pose of the glasses 100. For example, the tracking module 840 uses image data and associated inertial data from the camera 808 and the positioning component 340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene environment). The tracking module 840 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world scene environment. The tracking module 840 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 810.

[0122] The GPU and display driver 838 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 810 when the glasses 100 are operating in a traditional augmented reality mode. In this mode, the GPU and display driver 838 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world scene environment.

[0123] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 826 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 706, such as the messaging application 746.

[0124] Fig. 9 is a block diagram illustrating an example messaging system 900 for exchanging data (e.g., messages and associated content) over a network. The messaging system 900 includes multiple instances of a client device 826 that host several applications including a messaging client 902 and other applications 904. The messaging client 902 is communicatively coupled to other instances of the messaging client 902 (e.g., hosted on respective other client devices 826), a messaging server system 906, and a third-party server 908 via a network 830 (e.g., the Internet). The messaging client 902 may also communicate with the locally hosted application 904 using an application program interface (API).

[0125] The messaging client 902 is able to communicate and exchange data with other messaging clients 902 and with a messaging server system 906 via the network 830. The data exchanged between the messaging clients 902 and between the messaging clients 902 and the messaging server system 906 includes functions (e.g., commands for invoking functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0126] The messaging server system 906 provides server-side functionality to a particular messaging client 902 via the network 830. Although some functions of the messaging system 900 are described herein as being performed by the messaging client 902 or by the messaging server system 906, the location of some functions within the messaging client 902 or within the messaging server system 906 may be a design choice. For example, it may be technically preferable to initially deploy some technologies and functions within the messaging server system 906, but then migrate the technologies and functions to the messaging client 902 where the client device 826 has sufficient processing power.

[0127] The messaging server system 906 supports various services and operations provided to the messaging client 902. Such operations include transmitting data to the messaging client 902, receiving data from the messaging client 902, and processing data generated by the messaging client 902. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 900 is invoked and controlled by functions available via the user interface (UI) of the messaging client 902.

[0128] Turning now specifically to the messaging server system 906, an application program interface (API) server 910 is coupled to and provides a programming interface to an application server 914. The application server 914 is communicatively coupled to a database server 916, which facilitates access to a database 920 that stores data associated with messages processed by the application server 914. Similarly, a web server 924 is coupled to and provides a web-based interface to the application server 914. To this end, the web server 924 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0129] The application program interface (API) server 910 receives and transmits message data (e.g., commands and message payloads) between the client device 826 and the application server 914. Specifically, the application program interface (API) server 910 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 902 to invoke the functions of the application server 914. The application program interface (API) server 910 exposes various functions supported by the application server 914, including: account registration; login functionality; sending messages from a particular messaging client 902 to another messaging client 902 via the application server 914, sending media files (e.g., images or videos) from the messaging client 902 to the messaging server 912, and possible access by another messaging client 902; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 826; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 902).

[0130] The application server 914 hosts several server applications and subsystems, including, for example, a messaging server 912, an image processing server 918, and a social network server 922. The messaging server 912 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 902. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 902. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 912.

[0131] The application server 914 also includes an image processing server 918 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 912.

[0132] The social network server 922 supports various social networking functions and services and makes these functions and services available to the messaging server 912. To this end, the social network server 922 maintains and accesses an entity graph within the database 920. Examples of functions and services supported by the social network server 922 include identifying other users in the messaging system 900 with whom a particular user has relationships or that the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0133] The messaging client 902 may notify the user of the client device 826 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 902 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 902 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0134] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0135] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0136] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0137] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed for use with other components, and are usually part of a program that performs a specific function of a related function. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. For example, software configures one or more specific processors accordingly to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein specific one or more processors are examples of hardware. For example, some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some of the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0138] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0139] "Machine storage media" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media," "device storage media," and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media," "computer storage media," and "device storage media" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media."

[0140] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. A processor may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes called "cores") that can execute instructions simultaneously.

[0141] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0142] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method comprising: performing, by one or more processors of an augmented reality (AR) system: detecting, using one or more cameras of the AR system, a start text input gesture by a user of the AR system; In response to detecting the start text input gesture, generating a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys; providing the virtual keyboard user interface to the user using a display of the AR system; determining, using one or more cameras of the AR system, a selection by the user of one or more selected virtual keys of the plurality of virtual keys; generating input text data based on the one or more selected virtual keys; as well as The input text data is provided to the user using a display of the AR system.

2. The computer-implemented method of claim 1 , further comprising: detecting, using the one or more cameras, an end text input gesture by the user; as well as The virtual keyboard user interface is closed.

3. The computer-implemented method of claim 1 , wherein: Determining the user's selection of the one or more selected virtual keys further comprises: detecting landmarks associated with a text input hand of the user based on current tracking data received from a tracking service; generating a landmark collider based on the landmark; and Selection of the one or more selected virtual keys by the user is determined by detecting a collision between the landmark collider and the one or more selected virtual keys.

4. The computer-implemented method of claim 1 , wherein: Detecting, using one or more cameras of the AR system, a start text input gesture by a user of the AR system further comprises: generating tracking data using the one or more cameras; generating one or more skeletal models based on the tracking data; and The start text input gesture is determined based on classifying the one or more skeletal models using an artificial intelligence method and a gesture model previously generated using a machine learning method.

5. The computer-implemented method of claim 1 , wherein: Detecting, using one or more cameras of the AR system, a start text input gesture by a user of the AR system further comprises: generating tracking data using the one or more cameras; generating one or more skeletal models based on the tracking data; comparing the one or more skeletal models to previously generated posed skeletal models; and A start text input gesture is determined based on a comparison of the one or more skeletal models to the gesture skeletal model.

6. The computer-implemented method of claim 1 , wherein: The AR system includes a head-mounted device.

7. A computing device for an AR system, comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising: detecting, using one or more cameras of the AR system, a start text input gesture by a user of the AR system; In response to detecting the start text input gesture, generating a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys; providing the virtual keyboard user interface to the user using a display of the AR system; determining, using the one or more cameras, selection by the user of one or more selected virtual keys of the plurality of virtual keys; generating input text data based on the one or more selected virtual keys; as well as The input text data is provided to the user using a display of the AR system.

8. The computing device of the AR system according to claim 7, wherein: The instructions also cause the computing device to perform operations including: detecting, using the one or more cameras, an ending text input gesture by the user; and The virtual keyboard user interface is closed.

9. The computing device of the AR system according to claim 7, wherein: The instructions causing the computing device to perform an operation of determining the user's selection of the one or more selected virtual keys further cause the computer to perform the following operations: detecting landmarks associated with a text input hand of the user based on current tracking data received from a tracking service; generating a landmark collision symbol based on the landmark; as well as Selection of the one or more selected virtual keys by the user is determined by detecting a collision between the landmark collider and the one or more selected virtual keys.

10. The computing device of the AR system according to claim 7, wherein: Detecting, using one or more cameras of the AR system, a start text input gesture by a user of the AR system further comprises: generating tracking data using the one or more cameras; generating one or more skeletal models based on the tracking data; and The start text input gesture is determined based on classifying the one or more skeletal models using an artificial intelligence method and a gesture model previously generated using a machine learning method.

11. The computing device of the AR system according to claim 7, wherein: Detecting, using one or more cameras of the AR system, a start text input gesture by a user of the AR system further comprises: generating tracking data using the one or more cameras; generating one or more skeletal models based on the tracking data; comparing the one or more skeletal models to previously generated posed skeletal models; and A start text input gesture is determined based on a comparison of the one or more skeletal models to the gesture skeletal model.

12. The computing device of the AR system according to claim 7, wherein: The AR system includes a head-mounted device.

13. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer of an AR system, cause the computer to perform operations comprising: detecting, using one or more cameras of the AR system, a start text input gesture by a user of the AR system; In response to detecting the start text input gesture, generating a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys; providing the virtual keyboard user interface to the user using a display of the AR system; determining, using one or more cameras of the AR system, a selection by the user of one or more selected virtual keys of the plurality of virtual keys; generating input text data based on the one or more selected virtual keys; as well as The input text data is provided to the user using a display of the AR system.

14. The non-transitory computer readable storage medium of claim 13, wherein: The instructions also cause the computer to perform operations including: detecting, using one or more cameras of the AR system, an ending text input gesture by the user; and The virtual keyboard user interface is closed.

15. The non-transitory computer readable storage medium of claim 13, wherein: The instructions causing the computer to perform an operation of determining the user's selection of the one or more selected virtual keys further cause the computer to perform the following operations: detecting landmarks associated with a text input hand of the user based on current tracking data received from a tracking service; generating a landmark collision symbol based on the landmark; as well as Selection of the one or more selected virtual keys by the user is determined by detecting a collision between the landmark collider and the one or more selected virtual keys.

16. The non-transitory computer readable storage medium of claim 13, wherein: The instructions causing the computer to perform an operation of detecting, using one or more cameras of the AR system, a start text input gesture by a user of the AR system further cause the computer to perform the following operations: generating tracking data using the one or more cameras; generating one or more skeletal models based on the tracking data; as well as The start text input gesture is determined based on classifying the one or more skeletal models using an artificial intelligence method and a gesture model previously generated using a machine learning method.

17. The non-transitory computer readable storage medium of claim 13, wherein: The instructions causing the computer to perform an operation of detecting, using one or more cameras of the AR system, a start text input gesture by a user of the AR system further cause the computer to perform the following operations: generating tracking data using the one or more cameras; generating one or more skeletal models based on the tracking data; comparing the one or more skeletal models to previously generated posed skeletal models; as well as A start text input gesture is determined based on a comparison of the one or more skeletal models to the gesture skeletal model.

18. The non-transitory computer readable storage medium of claim 13, wherein: The AR system includes a head-mounted device.

Images