Visual and audio wake-up commands

Abstract

A gesture-based wake-up process for an AR system is described herein. The AR system places a hand tracking input pipeline of the AR system in a suspend mode. A camera component of the hand tracking input pipeline detects a possible visual wake-up command being issued by a user of the AR system. Based on detecting a possible visual wake-up command, the AR system wakes up the hand tracking input pipeline and places the camera components in a full mode of operation. If the AR system verifies the possible visual wake-up command as an actual wake-up command using the hand tracking input pipeline, the AR system initiates execution of the AR application.

Description

[0001] Priority claim

[0002] This application claims the benefit of U.S. patent application Ser. No. 17 / 948,004, filed on Sept. 19, 2022, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to user interfaces, and more particularly to user interfaces used in augmented reality and virtual reality. Background Art

[0004] A head-mounted device can be implemented with a transparent or translucent display through which the user of the head-mounted device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or translucent display, and can also see objects (e.g., virtual objects such as renderings of 2D or 3D graphic models, images, videos, text, etc.) generated for display as part of the surrounding environment and / or superimposed on the surrounding environment. This is generally referred to as "augmented reality" or "AR". The head-mounted device can also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is generally referred to as "virtual reality" or "VR". As used herein, unless the context indicates otherwise, the term "AR" refers to either or both of augmented reality and virtual reality as traditionally understood.

[0005] In order to save power and extend the usage time, the AR system often suspends itself, such as entering a sleep mode. Therefore, it is expected that the AR system provides a mechanism for the user of the AR system to unsuspend or wake up the AR system. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of a head mounted device according to some examples.

[0008] Figure 2 According to some examples Figure 1 Another view of the headset.

[0009] Figure 3 is a diagrammatic representation of a machine in the form of a computing device according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.

[0010] Figure 4Ais a collaboration diagram of a hand tracking input pipeline for an AR system according to some examples.

[0011] Figure 4B is a diagram of a data structure according to some examples.

[0012] Figure 4C is an illustration of another data structure according to some examples.

[0013] Figure 5 is an activity diagram of the visual system wakeup and suspension process of an AR system according to some examples.

[0014] Figure 6 is an activity diagram of the audio system wakeup and suspend process for an AR system according to some examples.

[0015] Figure 7 is a block diagram illustrating a software architecture within which the present disclosure may be implemented according to some examples.

[0016] Figure 8 is a block diagram showing a networked system including details of a head-mounted AR system according to some examples.

[0017] Fig. 9 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network according to some examples. DETAILED DESCRIPTION

[0018] AR systems are limited when it comes to available user input modalities. Indicating user intent and invoking an action or application is more complex for users of AR systems when compared to other mobile devices such as mobile phones. When using a mobile phone, a user can go to the home screen and tap on a specific icon to launch an application. However, due to the lack of physical input devices such as a touch screen or keyboard, such interactions are not easily performed on AR systems. Typically, users can indicate their intent by pressing a limited number of hardware buttons or using a small touchpad. Therefore, it is desirable to have an input modality that allows a larger range of inputs that can be used by users to indicate their intent through user input.

[0019] In some examples, the input modality utilized by the AR system is the recognition of gestures made by the user that do not involve direct manipulation (DMVO) of virtual objects. The gesture is made by the user moving and positioning parts of the user's body, and these parts of the user's body can be detected by the AR system when the user is wearing the AR system. The detectable parts of the user's body may include parts of the user's upper body, arms, hands, and fingers. The components of the gesture may include the movement of the user's arms and hands, the position of the user's arms and hands in space, and the user maintaining the positioning of his upper body, arms, hands, and fingers. Gestures are useful in providing AR experiences to users because gestures provide a way to provide user input to the AR system during the AR experience without causing the user to divert their attention from the AR experience. As an example, in an AR experience that is an operating manual for a machine, the user can simultaneously view a machine in a real-world scene through the lens of the AR system, view the AR overlay on the real-world scene view of the machine, and provide user input to the AR system.

[0020] AR systems have limited power and thermal budgets. To save power, an AR system can put itself into a suspended mode and enter a low-power state when not in use. It is expected that the user can signal the AR system to exit the suspended mode so that the user can interact with the AR system. Such a signal can be a gesture similar to other gestures that form the AR system's hand tracking interaction language. However, recognizing gestures will typically require power for computing resources that may not be available in the suspended mode.

[0021] In some examples, a camera component of the AR system recognizes a conservative approximation of a possible visual wake-up command for use as a wake-up command, and wakes up the AR system when the visual wake-up command is recognized, thereby placing the AR system in a fully operational mode. In the fully operational mode, the AR system verifies that the possible visual wake-up command is indeed a visual wake-up command, and the AR system initiates an interactive session.

[0022] In some examples, when the AR system is in a suspended mode, the AR system identifies possible visual wake-up commands for use as wake-up commands. The AR system includes a camera component that is part of a hand tracking input pipeline and has a trained onboard visual command classification capability that recognizes conservative approximations of visual wake-up commands with high recall but low precision. When in suspended mode, the AR system uses the onboard visual command classification of the camera component to identify possible visual wake-up commands based on video frame data generated by a camera of the camera component. When the onboard visual command classification of the camera component identifies a possible visual wake-up command, the AR system wakes up the remaining components of the hand tracking input pipeline. The hand tracking input pipeline is temporarily awakened and uses a complete high-precision, high-recall hand tracking model to evaluate possible visual wake-up commands, which recognizes visual wake-up commands with the same accuracy as recognizing other gestures. If the hand tracking input pipeline does not recognize a possible visual wake-up command as a visual wake-up command, the AR system returns to suspend mode, or if the onboard visual command classification item of the camera device component temporarily recognizes that a possible visual wake-up command is actually a visual wake-up command, the AR system initiates an interactive user session (at full power).

[0023] In some examples, the AR system also recognizes and processes audio data that may include an audio wake-up command.

[0024] Other technical features will be readily apparent to those skilled in the art from the following drawings, descriptions and claims.

[0025] Figure 1 is a head mounted AR system according to some examples (e.g., Figure 1 100). The glasses 100 may include a frame 102, which is made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a crossbar 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0026] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0027] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 802 discussed below.

[0028] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0029] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0030] In some examples, left camera 114 and right camera 116 provide video frame data for use by glasses 100 to extract 3D information from a real scene.

[0031] The glasses 100 may also include a touchpad 126 mounted to or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is typically arranged vertically, approximately parallel to the temple of the user in some examples. As used herein, typically vertically aligned means that the touchpad is more vertical than horizontal, but may be more vertical than this. Additional user input can be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide a device by which the glasses 100 can receive input from a user of the glasses 100.

[0032] Figure 2 The eyeglasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured in left and right optical element holders 104, 106, respectively.

[0033] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0034] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 superimposed with a view of the real-world scene seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 superimposed with a view of the real-world scene seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides an optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's view of the real-world scene, including displaying a user interface to the user of the glasses 100.

[0035] However, it will be appreciated that other display technologies or configurations may be utilized within the optical engine to display images to a user in the user's field of view. For example, instead of the projector 204 and waveguide, an LCD, LED or other display panel or surface may be provided.

[0036] In use, the user of the glasses 100 will be presented with information, content, and various user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Figure 8 Interact with the glasses 100 through voice input or touch input on a client device 826 shown in FIG. 1 and / or hand movements, positions, and locations recognized by the glasses 100.

[0037] Figure 3 is a diagrammatic representation of a machine 300 (e.g., a computing device) in which instructions 310 (e.g., software, programs, applications, applet, app, or other executable code) may be executed that cause the machine 300 to perform any one or more of the methodologies discussed herein. The machine 300 may be used as Figure 1 Computer 120 of glasses 100. For example, instruction 310 can cause machine 300 to perform any one or more of the methods described herein. Instruction 310 transforms a general unprogrammed machine 300 into a specific machine 300 that is programmed to perform the functions described and shown in the described manner. Machine 300 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 300 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 300 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular phones, smart phones, mobile devices, head-mounted devices (e.g., smart watches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 310 specifying actions to be taken by machine 300. Further, while a single machine 300 is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute instructions 310 to perform any one or more of the methodologies discussed herein.

[0038] Machine 300 may include processor 302, memory 304, and I / O component 306, which may be configured to communicate with each other via bus 344. In some examples, processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 308 and processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but machine 300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0039] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via the bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. During execution of the instructions 310 by the machine 300, the instructions 310 may also reside, in whole or in part, within the main memory 314, within the static memory 316, within the machine-readable medium 320, within the storage unit 318, within one or more of the processors 302 (e.g., within a processor's cache memory), or within any suitable combination thereof.

[0040] I / O components 306 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It will be appreciated that I / O components 306 may include Figure 3306. In various examples, the I / O components 306 may include output components 328 and input components 332. The output components 328 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 332 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0041] In another example, the I / O component 306 may include: a biometric component 334, a motion component 336, an environmental component 338, or a positioning component 340, as well as various other components. For example, the biometric component 334 includes a component for recognizing expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 336 may include an inertial measurement unit (IMU), an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental components 338 include, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect concentrations of hazardous gases for safety or measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals associated with the surrounding physical environment. Positioning components 340 include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect air pressure that can provide altitude), orientation sensor components (e.g., magnetometers), etc.

[0042] A variety of technologies may be used to achieve communication. I / O components 306 also include communication components 342 that are operable to couple machine 300 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication components 342 may include a network interface component or another suitable device that interfaces with network 322. In other examples, communication components 342 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low energy consumption), Device 324 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0043] In addition, the communication component 342 can detect the identifier or include a component that is operable to detect the identifier. For example, the communication component 342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Supercode, UCC RSS-2D barcode and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 342, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0044] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 310) when executed by processor 302 cause various operations to implement the disclosed examples.

[0045] Instructions 310 may be transmitted or received via a network interface device (e.g., a network interface component included in communication component 342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 322. Similarly, instructions 310 may be transmitted or received to device 324 via coupling 326 (e.g., a peer-to-peer coupling) using a transmission medium.

[0046] Figure 4A is a collaboration diagram of a hand tracking input pipeline 458 of an AR system (e.g., glasses 100) according to some examples, and Figure 4B and Figure 4C is a diagram of a data structure. Camera component 402 uses one or more cameras (eg, Figure 1 The camera device 114 and 116 of the AR system generates real-world scene frame data 426 of the real-world scene from the perspective of the user of the AR system. The camera device component 402 transmits the real-world scene frame data 426 to the skeleton model inference component 404. The real-world scene frame data 426 includes tracking video frame data of detectable parts of the user's body (including parts of the user's upper body, arms, hands, and fingers). The tracking video frame data includes: video frame data of the movement of parts of the user's upper body, arms, and hands when the user makes gestures or moves their hands and fingers to interact with the real-world scene; video frame data of the position of the user's arms and hands in space when the user makes gestures or moves their hands and fingers to interact with the real-world scene; and video frame data of the user maintaining the positioning of his upper body, arms, hands, and fingers when the user makes gestures or moves their hands and fingers to interact with the real-world scene. In some examples, the camera device component 402 includes a visual wake-up command classification item 456, which can be used to identify gestures intended to be visual wake-up commands made by a user using the AR system. In some examples, visual wake-up command classifier 456 identifies a conservative approximation of a visual wake-up command issued by a user with high recall but low precision. In some examples, visual wake-up command classifier 456 is a neural network trained using machine learning methods to recognize visual wake-up commands.

[0047] The skeleton model inference component 404 scans, identifies and tracks the landmarks on the upper body, arms and hands of the user in the real world scene. In some examples, the skeleton model inference component 404 receives the real world scene frame data 426 from the camera device component 402, and extracts the features of the upper body, arms and hands of the user from the tracking video frame data included in the real world scene frame data 426. The skeleton model inference component 404 generates the skeleton model data 432 based on the extracted features. The skeleton model data 432 includes landmark data, which includes landmark identification, location in the real world scene, and classification information of one or more landmarks associated with the upper body, arms and hands of the user. The skeleton model inference component 404 transmits the skeleton model data 432 to the hand classification item inference component 406. In addition, the skeleton model inference component 404 makes the skeleton model data 432 available to applications (e.g., AR DMVO application component 422) executed on the AR system.

[0048] The camera device component 402 transmits the real-world scene frame data 426 to the rough hand positioning inference component 412. The rough hand positioning inference component 412 generates coordinate transformation data 430 based on the real-world scene frame data 426. The coordinate transformation data 430 includes a continuously updated transformation of the coordinate system of the skeletal model from the skeletal model data 432 and the coordinate system of the user coordinate system of the AR system. In some examples, the object rough hand positioning inference component 412 receives the real-world scene video real-world scene frame data 426 of the real-world scene, and extracts features of objects (including the user's upper body, arms, and hands) in the real-world scene from the real-world scene video frame data. The object rough hand positioning inference component 412 generates coordinate transformation data 430 based on the extracted features. The rough hand positioning inference component 412 transmits the coordinate transformation data 430 to the AR DMVO application component 422.

[0049] The hand classification term inference component 406 receives the skeletal model data 432 from the skeletal model inference component 404, and generates the hand classification term probability data 428 based on the skeletal model data 432. The posture is specified by the hand tracking input pipeline 458 according to the combination of hand classification terms. The hand classification terms are in turn composed of the combination and relationship of the landmarks included in the skeletal model data 432. Because the hand tracking input pipeline 458 extracts the hand classification terms from the skeletal model data 432 through the hand tracking input pipeline 458 in a different layer than assembling the hand movements into postures, the designer of the AR system can create new postures constructed from existing hand classification terms that constitute already known postures without having to retrain the machine learning component of the hand tracking input pipeline 458. In some examples, the hand classification term inference component 406 compares one or more skeletal models included in the skeletal model data 432 with previously generated hand classification term models, and generates one or more hand classification term probabilities based on the comparison. One or more hand classification item probability indications can identify the probability of a specified hand classification item based on the skeleton model data 432. The hand classification item inference component 406 generates hand classification item probability data 428 based on the one or more hand classification item probabilities. In another example, the hand classification item inference component 406 determines one or more hand classification item probabilities based on classifying the skeleton model using an artificial intelligence method and a hand classification item model previously generated using a machine learning method. The hand classification item inference component 406 transmits the hand classification item probability data 428 to the posture inference component 408 and the posture text input recognition component 410.

[0050] The gesture inference component 408 receives the hand classification term probability data 428 and determines the audio data 466 based on the hand classification term probability data 428. In some examples, the gesture inference component 408 compares the hand classification term identified in the hand classification term probability data 428 with the gesture identification data identifying a particular gesture. A gesture identification consists of one or more hand classification terms corresponding to a particular gesture. A gesture identification is defined using a grammar whose symbols correspond to the hand classification terms. For example, the posture identifier of a posture is "LEFT_PALMAR_FINGERS EXTENDED_RIGHT PALMAR_FINGERS_EXTENDED", where: "LEFT" is a symbol corresponding to the hand classification item indicating that the user's left hand has been recognized; "PALMAR" is a symbol corresponding to the hand classification item indicating that the palm of the user's hand has been recognized and modifies "LEFT" to indicate that the user's left palm has been recognized; "FINGERS" is a symbol corresponding to the hand classification item indicating that the user's fingers have been recognized; and "EXTENDED" is a symbol corresponding to the hand classification item indicating that the user's fingers are extended and modifies "FINGERS". In another example, the posture identifier is a single tag such as a number that identifies the posture based on the component hand classification item of the posture. The posture identifier identifies the posture in the context of a physical description of the posture. The posture inference component 408 transmits 466 to the system framework component 416.

[0051] The gesture text input recognition component 410 receives the hand classification term probability data 428 and generates the symbol data 414 based on the hand classification term probability data 428. In some examples, the gesture inference component 408 compares the hand classification term identified in the hand classification term probability data 428 with the symbol data identifying specific characters, words, and commands. For example, the symbol data for the gesture is the character "V" as a gesture of a sign language symbol in American Sign Language (ASL). A single hand classification item for a gesture may be "LEFT" for the left hand, "PALMAR" for the palm of the left hand, "INDEXFINGER" for the index finger, "EXTENDED" for the modified "INDEXFINGER", "MIDDLEFINGER" for the middle finger, "EXTENTED" for the modified "MIDDLEFINGER", "RINGFINGER" for the ring finger, "CURLED" for the modified "RINGFINGER", "LITTLEFINGER" for the little finger, "CURLED" for the modified "LITTLEFINGER", "THUMB" for the thumb, and "CURLED" for the modified "THUMB".

[0052] In some examples, the entire word may also be identified by the gesture text input recognition component 410 based on the hand classification terms indicated by the hand classification term probability data 428. In another example, a command may be identified by the gesture text input recognition component 410 based on the hand classification terms indicated by the hand classification term probability data 428, such as a command corresponding to a specified set of keystrokes in an input system having a keyboard.

[0053] The gesture inference component 408 and the gesture text input recognition component 410 transmit the audio data 466 and the symbol data 414, respectively, to the system framework component 416. The system framework component 416 receives the audio data 466 and the symbol data 414 (collectively and individually referred to as "input event data") and generates undirected input event data 452 or directional input event data 454 based in part on the input event data. Undirected input events belonging to the undirected category of input events are routed to operating system level components such as the system user interface component 418. Directional input events belonging to the directional category of input events are routed to target components such as the AR interactive application component 420.

[0054] In some examples of processing input data received from gesture inference component 408 and gesture text input recognition component 410 that can be classified as undirected input event data 452, system framework component 416 classifies the input data as undirected input event data 452 based on the input data and component registration data described below. System framework component 416 routes the input data as undirected input event data 452 to system user interface component 418 based on classifying the input data as undirected input event data 452.

[0055] The system user interface component 418 receives the undirected input event data 452 and determines a target component based on an indication or selection of a virtual object associated with the target component by the user when making a gesture corresponding to the undirected input event data 452. In some examples, the system user interface component 418 determines the position of the user's hand in the real-world scene when the gesture is made. The system user interface component 418 determines a set of virtual objects currently provided to the user by the AR system in the AR experience. The system user interface component 418 determines a virtual object whose apparent position in the real-world scene is related to the position of the user's hand in the real-world scene when the gesture is made. The system user interface component 418 determines the target AR application component by looking up the AR application component associated with the virtual object in an internal data structure of the AR system, and determines the AR application component as the target AR application component.

[0056] The system user interface component 418 registers with the system framework component 416 the target AR application component to which the directional input event data 454 is to be routed. The system framework component 416 registers the component registration data, such as Figure 4BThe component registration data 442 is stored in a data store to be accessed during operation of the system framework component 416. The component registration data 442 includes a component ID field 434 that identifies the target AR application component. The component registration data 442 also includes a registered language field 440 that identifies the language model associated with the target AR application component. The language model is a forward-looking language-specific model that can be used to help predict the next input symbol when the hand tracking input pipeline 458 is used to input text as a single character or word. The component registration data 442 also includes one or more registered gesture fields 436 and / or registered symbol fields 438, which indicate gestures and symbols that will be routed to the registered AR application component. As shown, the component ID field 434 includes the AR application component identifier "TEXTENTRY"; the registered language field 440 identifies the language associated with the registered AR application component, namely "ENGLISH"; the registered posture field 436 includes the posture identifier routed to the registered target AR application component, namely "LEFT_PALMAR_FINGERS EXTENDED_RIGHT_PALMAR_FINGERS_EXTENDED", and the registered symbol field 438 identifies a set of symbols routed to the registered AR application component (i.e., "[*]" representing all symbols).

[0057] As another example of component registration data, Figure 4C The component registration data 444 includes: a component ID field 446, which includes the AR application component identifier "EMAIL"; a registered language field 440, which identifies the language associated with the registered AR application component, namely "ASL", and a registered symbol field 448, which identifies a set of symbols (i.e., the word "EMAIL") that are routed to the registered AR application component.

[0058] Refer again Figure 4Aand a system framework component 416 that processes input data received from the gesture inference component 408 and the gesture text input recognition component 410 that can be classified as non-directional input event data 452, the system framework component 416 classifying the input data received from the gesture inference component 408 and the gesture text input recognition component 410 as non-directional input event data 452 or directional input event data 454 based on the input data and the component registration data. In some examples, when processing the symbol data 414, the system framework component 416 searches a registered symbol field of the component registration data, such as the registered symbol field 438 of the component registration data 442, for a registered symbol that matches the symbol data. When the system framework component 416 determines a match, the system framework component 416 determines that the symbol data is the directional input event data 454. The system framework component 416 also determines a target AR application component based on the target AR application component identified in the component ID field (e.g., component ID field 434) of the component registration data that includes the matching registration symbol. In a similar manner, when processing audio data 466, system framework component 416 searches a registered gesture field of component registration data, such as registered gesture field 436 of component registration data 442, for a registered gesture that matches the gesture input data. When system framework component 416 determines a match, system framework component 416 determines that the gesture input data is directional input event data 454, and also determines a target AR application component to which the directional input event data 454 is to be routed. In the event that system framework component 416 determines that symbol data and / or gesture input data for the input data is not found in the component registration data, 416 determines that the input data is to be classified as undirectional input event data 452 and is to be routed to system user interface component 418.

[0059] In another example of processing directional input event data 454, an AR application component (e.g., AR interactive application component 420) registers itself with the system framework component 416. To do so, the AR application component registers component registration data (e.g., Figure 4B The component registration data 442 is transmitted to the system framework component 416. The system framework component 416 receives the component registration data and stores the component registration data in a data store for use in routing the directional input event data 454 to the AR application component.

[0060] In another example of processing directional input event data 454, the AR system determines based on hinting that the directional input event data 454 is to be routed to the AR application component. For example, if the AR system executes the current AR application component in a single application mode state, the current AR application component is hinted as the AR application component to which the directional input event data 454 is to be routed.

[0061] In some examples, the system framework component 416 transmits feedback data 424 to the hand classification term inference component 406 and the gesture inference component 408 to improve the accuracy of the inferences made by the hand classification term inference component 406 and the gesture inference component 408. In some examples, the system framework component 416 generates the feedback data 424 based on the user context data (e.g., component registration data that registers the AR application component and data about the hand classification terms that make up the registered gestures and the hand classification terms that make up the gestures associated with the registered symbols). The component registration data includes information about the gestures and symbols in the audio data 466 and the symbol data 414 that are routed to the AR application component as part of the directional input event data 454, as well as the language of the symbols. In addition, the system framework component 416 includes information about the composition of a particular gesture including the hand classification terms associated with the gestures and symbols.

[0062] In another example of processing feedback data 424, the system framework component 416 transmits a prompt as part of the feedback data 424 to the hand classification term inference component 406, the gesture inference component 408, and the gesture text input recognition component 410. The system framework component 416 generates the prompt based on a language model associated with the AR application component, for example, by a language specified in the registered language field 450 in the component registration data 444. The gesture text input recognition component 410 determines a possible next symbol N based on the previous characters N-1, N-2, etc. and the language model. In some examples, the system framework component 416 generates the prompt based on the language model, which is a hidden Markov model that predicts what the next symbol N is based on one or more of the previous characters N-1, N-2, etc. In another example, the gesture text input recognition component 410 uses an AI method to generate the next symbol N based on a language model generated using a machine learning method. The system framework component 416 generates the prompt based on the next symbol N. In some examples, the system framework component 416 determines the next gesture associated with the next symbol N by mapping the next symbol N to the next gesture based on a lookup table that associates symbols with gestures. The system framework component 416 decomposes the next gesture into a set of one or more next hand classification terms. The system framework component 416 transmits the next gesture as part of the feedback data 424 to the gesture inference component 408, and transmits the set of next hand classification terms as part of the feedback data 424 to the hand classification term inference component 406.

[0063] AR application components executed by the AR system, such as the AR DMVO application component 422, the system user interface component 418, and the AR interactive application component 420, are consumers of data (e.g., coordinate transformation data 430, skeletal model data 432, audio data 466, and symbolic data 414) generated by the hand tracking input pipeline 458. The AR system executes the AR DMVO application component 422 to provide a user interface to a user of the AR system using direct manipulation of visual objects within a 2D or 3D user interface. The AR system executes the system user interface component 418 to provide a system-level user interface such as a command console, etc. to a user of the AR system using gestures as an input modality. The AR system executes the AR interactive application component 420 to provide a user interface such as an AR experience to a user of the AR system using gestures as an input modality.

[0064] In some examples, the camera component 402, the skeletal model inference component 404, and the coarse hand position inference component 412 communicate using an automatically synchronized shared memory buffer. In addition, the skeletal model inference component 404 and the coarse hand position inference component 412 publish the skeletal model data 432 and the coordinate transformation data 430, respectively, on a memory buffer that can be accessed by components and applications (e.g., the AR DMVO application component 422) outside the hand tracking input pipeline 458.

[0065] In many examples, the hand classification term inference component 406, the gesture inference component 408, and the gesture text input recognition component 410 transmit the hand classification term probability data 428, the audio data 466, and the symbol data 414, respectively, via an inter-process communication method.

[0066] In some examples, the AR system includes an audio input pipeline 468. The audio input pipeline 468 includes a microphone component 460 that captures audio input data 464 of a real-world scene and transmits the audio input data 464 to an audio service component 462. The audio service component 462 receives the audio input data 464 and generates audio data 466 based on the audio input data 464. The audio data 466 includes recognized audio events in the real-world scene (including words spoken by a user of the AR system and sounds made by the user, such as the user clapping or snapping fingers), which are intended to be audio commands to the AR system by the user. In some examples, the audio service component 462 generates audio data 466 by mapping the audio input data 464 to the audio data 466 based on an audio event model created using an artificial intelligence method and a machine learning method.

[0067] In some examples, microphone component 460 includes an onboard audio wake-up command classifier 470 that can be used to identify possible audio wake-up commands issued by a user using the AR system. In some examples, audio wake-up command classifier 470 identifies a conservative approximation of the audio wake-up command issued by the user with high recall but low precision. In some examples, audio wake-up command classifier 470 is a neural network trained using machine learning methods to recognize audio wake-up commands.

[0068] In some examples, the hand tracking input pipeline 458 operates continuously to generate and publish symbol data 414, audio data 466, skeletal model data 432, and coordinate transformation data 430 based on real-world scene frame data 426 generated by one or more cameras of the AR system.

[0069] Figure 5 is an activity diagram of a visual wakeup and suspend process 518 according to some examples. The visual wakeup and suspend process 518 is performed by the AR system when the AR system is operating in a suspend mode or a sleep mode.

[0070] In operation 502, the AR system suspends one or more components of the hand tracking input pipeline 458, that is, puts one or more components into sleep or low power mode, and the camera device component 402 is placed in a limited operating mode. In some examples, the skeletal model inference component 404, the hand classification item inference component 406, and the posture inference component 408 are suspended. In some examples, the limited operating mode includes setting the camera device component 402 to capture video frame data at a limited frame rate, which is a reduced frame rate less than the full operating frame rate. In some examples, the camera device component 402 is set to capture video frame data at a limited resolution, which is a reduced resolution less than the full operating resolution. In some examples, the reduced frame rate of the limited operating mode is 5 frames per second. In some examples, the full operating frame rate is 30 frames per second.

[0071] In some examples, the AR system places the hand tracking input pipeline 458 in a suspended mode by generating a suspend command 520 and transmitting the suspend command 520 to one or more components of the hand tracking input pipeline 458. In some examples, the AR system places the camera component 402 in a limited operating mode by generating a limited operating mode command 528 and transmitting the limited operating mode command 528 to the camera component 402.

[0072] The camera component 402 receives the limited operation mode command 528 and begins to operate in the limited operation mode. In the limited operation mode, the camera component 402 captures real-world scene frame data at a limited frame rate, while the rest of the hand tracking input pipeline is in suspended mode. The real-world scene frame data includes gesture video frame data of a visual wake-up command 526 made by a user 516 as a wake-up command. When in the limited operation mode, the visual wake-up command classifier 456 uses an artificial intelligence method and a visual wake-up command classifier 456 trained using a machine learning method to identify gestures as possible visual wake-up commands in the tracking video frame data. In some examples, the visual wake-up command classifier 456 identifies a conservative approximation of the gesture made by the user with high recall but low precision. The camera component 402 generates possible visual wake-up command data 522 based on the possible visual wake-up commands identified by the visual wake-up command classifier 456.

[0073] In operation 504, the AR system reads possible visual wake-up command data 522 generated by the camera component 402. In operation 506, the AR system detects whether the user 516 is issuing a possible visual wake-up command based on the possible visual wake-up command data 522. In operation 506, based on detecting that the user 516 is not issuing a possible visual wake-up command, the AR system resumes operation 504 and continues to read possible visual wake-up command data 522. In operation 508, based on detecting that the user 516 is issuing a possible visual wake-up command, the AR system wakes up components of the hand tracking input pipeline 458. In some examples, the AR system generates a wake-up command 524 and transmits the wake-up command 524 to the hand tracking input pipeline 458. In operation 508, the AR system commands the camera component 402 to enter a full operational mode. In some examples, the AR system generates a full operational mode command 530 and transmits the full operational mode command 530 to the camera component 402. In the full operational mode, the camera sensor of the camera assembly 402 captures video frame data at a full operational frame rate that is higher than the reduced frame rate of the limited operational mode. In some examples, the camera sensor of the camera assembly 402 captures video frame data at a full operational resolution that is higher than the reduced frame rate of the limited operational mode. In some examples, the AR system generates a full operational mode command 530 and transmits the full operational mode command 530 to the camera assembly 402.

[0074] The hand tracking input pipeline 458 receives the wake-up command 524 and wakes up one or more sleeping components of the hand tracking input pipeline 458. In some examples, the skeletal model inference component 404, the hand classification item inference component 406, and the posture inference component 408 are awakened and start operating in full power mode. The camera component 402 receives the full operation mode command 530 and starts capturing video frame data at a full operation frame rate. The hand tracking input pipeline 458 processes the real world scene frame data 426 generated by the camera component 402 at the full operation frame rate to generate audio data 466 for the AR system to use to determine whether the visual wake-up command 526 issued by the user 516 and identified as a possible visual wake-up command is actually a visual wake-up command. The hand tracking input pipeline 458 generates audio data 466 based on the real world scene frame data 426 generated by the camera component 402. The posture inference component 408 of the hand tracking input pipeline 458 transmits the audio data 466 to the AR system. In operation 510 , the AR system reads the audio data 466 from the hand tracking input pipeline 458 .

[0075] In operation 512, the AR system verifies whether the user 516 is actually issuing a visual wake-up command based on the gesture data 472. That is, the AR system verifies that the possible visual wake-up command detected in operation 506 based on the possible visual wake-up command data 522 generated by the camera device component 402 in the limited operation mode using the visual wake-up command classification item 456 is an actual visual wake-up command based on the gesture data 472 generated by the hand tracking input pipeline 458 in the full operation mode. Based on verifying that the user 516 is actually issuing a visual wake-up command, the AR system enters the full operation mode, and wakes up the components of the AR system in operation 514, and initiates an interactive AR application presented to the user of the AR system, such as the AR interactive application component 420. In some examples, the AR system operates in a suspended mode by reducing the clock frequency of one or more processors (such as the processor 302). When the AR system enters the full operation mode, the AR system increases the clock frequency of the processor 302. In some examples, the AR system operates in the suspended mode by powering off certain components, such as output component 328, biometric component 334, motion component 336, environment component 338, and positioning component 340. When the AR system enters the fully operational mode, the AR system powers on the powered off components.

[0076] In operation 512, upon being unable to verify that the user 516 is issuing a visual wake-up command based on the gesture data 472, the AR system transitions to operation 502. In operation 502, the AR system suspends the hand tracking input pipeline 458 and places the camera component 402 in a limited operating mode, and continues to perform the visual wake-up and suspend process 518 in a low power mode, as described herein.

[0077] In some examples, while the user is still issuing the visual wake-up command 526, the AR system detects a possible visual wake-up command and wakes up the components of the hand tracking input pipeline 458 from the suspended mode and wakes up the camera component 402 from the limited operating mode. This allows the AR system to use the hand tracking input pipeline 458 to verify that the possible visual wake-up command detected by the AR system is actually a visual wake-up command while the user is still issuing the visual wake-up command 526. That is, the fully operational hand tracking input pipeline 458 recognizes the same visual wake-up command 526 issued by the user 516, which was identified as a possible visual wake-up command by the visual wake-up command classification item 456 of the camera component 402 when the hand tracking input pipeline 458 was in the suspended mode. In some examples, the waiting time for the hand tracking input pipeline 458 to transition from the suspended mode to the fully operating mode and the camera component 402 to transition from the limited operating mode to the fully operating mode is 100 milliseconds.

[0078] In some examples, the user issues a visual wake-up command by snapping his fingers using one or both of the user's hands.

[0079] In some examples, visual wake-up command classification item 456 is a component of hand tracking input pipeline 458.

[0080] In some examples, the visual wake-up and suspend process 518 performs operations of the hand tracking input pipeline 458 .

[0081] Figure 6 is an activity diagram of an audio wake-up and suspend process 618 according to some examples. The audio wake-up and suspend process 618 is performed by the AR system when the AR system is operating in a suspend mode or a sleep mode.

[0082] In operation 602, the AR system suspends one or more components of the audio input pipeline 468, i.e., places one or more components in a sleep mode or a low power mode, while the microphone component 460 is placed in a limited operating mode. In some examples, the audio service component 462 is suspended. In some examples, the limited operating mode includes setting the microphone component 460 to capture audio input data 464 at a limited sampling frequency, which is a reduced sampling frequency that is less than the full operating sampling frequency. In some examples, the limited sampling frequency of the limited operating mode is 5000 Hz. In some examples, the full operating sampling frequency is 50000 Hz.

[0083] In some examples, the AR system places the audio input pipeline 468 in a suspended mode by generating a suspend command 620 and transmitting the suspend command 620 to one or more components of the audio input pipeline 468. In some examples, the AR system places the microphone component 460 in a limited operating mode by generating a limited operating mode command 628 and transmitting the limited operating mode command 628 to the microphone component 460.

[0084] Microphone component 460 receives limited operating mode command 628 and begins operating in limited operating mode. In limited operating mode, camera microphone component 460 captures audio data at a limited sampling frequency while the rest of audio input pipeline 468 is in suspended mode. Audio data of audio wake-up command 624 issued by user 616 is used as a wake-up command. In some examples, audio wake-up command 624 is a voice command issued by the user, such as a specific word such as "Snap". In some examples, audio wake-up command 624 is a sound intentionally issued by the user using their hands or other appendages that is intended to be an audio command, such as the user clapping or snapping their fingers.

[0085] When in limited operation mode, microphone component 460 uses artificial intelligence methods and audio wake-up command classifier 470 trained using machine learning methods to identify audio commands as possible audio wake-up commands in audio data. In some examples, audio wake-up command classifier 470 identifies a conservative approximation of the audio command issued by the user with high recall but low precision. Microphone component 460 generates possible audio wake-up command data 622 based on the possible audio wake-up commands identified by audio wake-up command classifier 470.

[0086] In operation 604, the AR system reads the possible audio wake-up command data 622 generated by the microphone component 460. In operation 606, the AR system detects whether the user 616 is issuing a possible audio wake-up command based on the possible audio wake-up command data 622. In operation 606, based on detecting that the user 616 is not issuing a possible audio wake-up command, the AR system resumes operation 604 and continues to read the possible audio wake-up command data 622.

[0087] In operation 608, based on detecting that the user 616 is issuing a possible audio wake-up command, the AR system wakes up components of the audio input pipeline 468. In some examples, the AR system generates a wake-up command 626 and transmits the wake-up command 626 to the audio input pipeline 468. In operation 608, the AR system commands the microphone component 460 to enter a full operational mode. In some examples, the AR system generates a full operational mode command 630 and transmits the full operational mode command 630 to the microphone component 460. In the full operational mode, the audio sensor of the microphone component 460 captures audio data at a full operational sampling frequency that is higher than the limited sampling frequency. In some examples, the AR system generates a full operational mode command 630 and transmits the full operational mode command 630 to the microphone component 460.

[0088] The audio input pipeline 468 receives the wake-up command 626 and wakes up one or more sleeping components of the audio input pipeline 468. In some examples, the audio service component 462 is awakened and begins operating in full power mode. The microphone component 460 receives the full operation mode command 630 and begins capturing audio data at a full operation sampling frequency. The audio input pipeline 468 processes the audio input data 464 generated by the camera component 402 at a full operation sampling frequency to generate audio data 466 for the AR system to use to determine whether the audio wake-up command 624 issued by the user 616 and identified as a possible audio wake-up command is actually an audio wake-up command. The audio input pipeline 468 generates audio data 466 based on the audio input data 464 generated by the microphone component 460. The audio service component 462 of the audio input pipeline 468 transmits the audio data 466 to the AR system.

[0089] In operation 610, the AR system reads the audio data 466 from the audio input pipeline 468. In operation 612, the AR system verifies whether the user 616 is actually issuing an audio wake-up command based on the audio data 466. That is, the AR system verifies that the possible audio wake-up command detected in operation 506 based on the possible audio wake-up command data 622 generated by the microphone component 460 in the limited operation mode using the audio wake-up command classification item 470 is an actual audio wake-up command based on the audio data 466 generated by the audio input pipeline 468 in the full operation mode.

[0090] In operation 614, based on verifying that the user 616 is actually issuing the audio wake-up command, the AR system wakes up components of the AR system and initiates an interactive AR application, such as the AR interactive application component 420, presented to the user of the AR system. In some examples, the AR system operates in a suspended mode by reducing the clock frequency of one or more processors, such as the processor 302. When the AR system enters a fully operational mode, the AR system increases the clock frequency of the one or more processors. In some examples, the AR system operates in a suspended mode by powering off certain components, such as the output component 328, the biometric component 334, the motion component 336, the environment component 338, and the positioning component 340. When the AR system enters a fully operational mode, the AR system powers on the powered-off components.

[0091] Based on the inability to verify that the user 616 is issuing an audio wake-up command, the AR system transitions to operation 602. In operation 602, the AR system suspends the audio input pipeline 468 and places the microphone component 460 in a limited operating mode, and continues to perform the audio wake-up and suspend process 618 in a low power mode as described herein.

[0092] In some examples, while the user is still issuing the audio wake-up command 624, the AR system detects a possible audio wake-up command and wakes up the components of the audio input pipeline 468 from the suspended mode and wakes up the microphone component 460 from the limited operating mode. This allows the AR system to use the audio input pipeline 468 to verify that the possible audio wake-up command detected by the AR system is actually an audio wake-up command while the user is still issuing the audio wake-up command 624. That is, the fully operational audio input pipeline 468 that is awakened recognizes the same audio wake-up command 624 issued by the user 616, which is identified as a possible audio wake-up command by the audio wake-up command classification item 470 of the microphone component 460 when the audio input pipeline 468 is in the suspended mode. In some examples, the waiting time for the audio input pipeline 468 to transition from the suspended mode to the fully operational mode and the microphone component 460 to transition from the limited operating mode to the fully operational mode is approximately 100 milliseconds.

[0093] In some examples, the user issues an audio wake-up command by snapping his fingers using one or both of the user's hands.

[0094] In some examples, the audio wake-up command classification item 470 is a component of the audio input pipeline 468.

[0095] In some examples, the audio wake-up and suspend process 618 performs operations of the audio input pipeline 468 .

[0096] Figure 7 700 is a block diagram illustrating a software architecture 704 that may be installed on any one or more of the devices described herein. The software architecture 704 is supported by hardware, such as a machine 702, including a processor 720, a memory 726, and an I / O component 738. In this example, the software architecture 704 may be conceptualized as a stack of layers in which each layer provides specific functionality. The software architecture 704 includes layers such as an operating system 712, a library 708, a framework 710, and an application 706. In operation, the application 706 invokes an API call 750 through the software stack and receives a message 752 in response to the API call 750.

[0097] The operating system 712 manages hardware resources and provides public services. The operating system 712 includes, for example, a kernel 714, services 716, and drivers 722. The kernel 714 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 714 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 716 can provide other public services to other software layers. Drivers 722 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 722 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0098] The library 708 provides a low-level common infrastructure used by the application 706. The library 708 may include a system library 718 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 708 may include an API library 724, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting two-dimensional (2D) and three-dimensional (3D) graphics content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 708 may also include various other libraries 728 to provide many other APIs to the application 706 .

[0099] The framework 710 provides a high-level common infrastructure used by the applications 706. For example, the framework 710 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 710 may provide a wide range of other APIs that may be used by the applications 706, some of which may be specific to a particular operating system or platform.

[0100] In some examples, applications 706 may include a home application 736, a contacts application 730, a browser application 732, a book reader application 734, a location application 742, a media application 744, a messaging application 746, a game application 748, and a variety of other applications such as third-party applications 740. Application 706 is a program that executes the functions defined in the program. Various programming languages ​​can be used to create one or more of the applications 706 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party application 740 (e.g., a third-party application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the Phone or another mobile operating system. In this example, third-party applications 740 can call API calls 750 provided by the operating system 712 to facilitate the functions described herein.

[0101] Figure 8 830 is a block diagram showing a networked system 800 including details of the glasses 100 according to some examples. The networked system 800 includes the glasses 100, a client device 826, and a server system 832. The client device 826 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 836 and / or a high-speed wireless connection 834. The client device 826 is connected to the server system 832 via a network 830. The network 830 may include any combination of wired and wireless connections. The server system 832 may be one or more computing devices that are part of a service or network computing system. The client device 826 and any elements of the server system 832 and the network 830 may use Figure 7 and Figure 3 The details of the software architecture 704 or machine 300 described respectively are implemented.

[0102] The glasses 100 include a data processor 802, a display 810, one or more cameras 808, and additional input / output elements 816. The input / output elements 816 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 802. Figure 7 and Figure 3 Examples of input / output elements 816 are further discussed. For example, input / output elements 816 may include any I / O component 306, including output component 328, motion component 336, etc. Figure 2 Examples of display 810 are discussed in . In the particular examples described herein, display 810 includes displays for the user's left eye and right eye.

[0103] Data processor 802 includes image processor 806 (eg, video processor), GPU and display driver 838, tracking module 840, interface 812, low power circuitry 804, and high speed circuitry 820. The components of data processor 802 are interconnected by bus 842.

[0104] The interface 812 refers to any source of user commands provided to the data processor 802. In one or more examples, the interface 812 is a physical button that, when pressed, sends a user input signal from the interface 812 to the low-power processor 814. The low-power processor 814 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 814 can process pressing such a button within a first time period as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 812 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the camera 808. In other examples, the interface 812 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 826.

[0105] The image processor 806 includes circuitry for receiving signals from the camera 808 and processing those signals from the camera 808 into a format suitable for storage in the memory 824 or for transmission to the client device 826. In one or more examples, the image processor 806 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from the camera 808, and volatile memory used by the microprocessor in operation.

[0106] The low power circuit system 804 includes a low power processor 814 and a low power wireless circuit system 818. These elements of the low power circuit system 804 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 814 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 814 can accept user input signals from the interface 812. The low power processor 814 can also be configured to receive input signals or command communications from a client device 826 via a low power wireless connection 836. The low power wireless circuit system 818 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 818. In other examples, other low power communication systems may be used.

[0107] High-speed circuit system 820 includes a high-speed processor 822, a memory 824, and a high-speed wireless circuit system 828. High-speed processor 822 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 802. High-speed processor 822 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 834 using high-speed wireless circuit system 828. In some examples, high-speed processor 822 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Figure 7 The high-speed processor 822, which executes the software architecture of the data processor 802, manages data transmission with the high-speed wireless circuit system 828, in addition to any other duties. In some examples, the high-speed wireless circuit system 828 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 828 can implement other high-speed communication standards.

[0108] The memory 824 includes any storage device capable of storing camera data generated by the camera 808 and the image processor 806. Although the memory 824 is shown as being integrated with the high-speed circuitry 820, in other examples, the memory 824 may be a separate, independent element of the data processor 802. In some such examples, electrical wiring may provide a connection from the image processor 806 or the low-power processor 814 to the memory 824 through a chip including the high-speed processor 822. In other examples, the high-speed processor 822 may manage the addressing of the memory 824 so that the low-power processor 814 will direct the high-speed processor 822 whenever a read or write operation involving the memory 824 is desired.

[0109] The tracking module 840 estimates the pose of the glasses 100. For example, the tracking module 840 uses image data and associated inertial data from the camera 808 and the positioning component 340 and GPS data to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene). The tracking module 840 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world scene. The tracking module 840 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 810.

[0110] The GPU and display driver 838 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 810 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 838 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world scene.

[0111] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 826 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 706, such as the messaging application 746.

[0112] Fig. 9 is a block diagram illustrating an example messaging system 900 for exchanging data (e.g., messages and associated content) over a network. The messaging system 900 includes multiple instances of a client device 826 that host several applications including a messaging client 902 and other applications 904. The messaging client 902 is communicatively coupled to other instances of the messaging client 902 (e.g., hosted on respective other client devices 826), a messaging server system 906, and a third-party server 908 via a network 830 (e.g., the Internet). The messaging client 902 may also communicate with the locally hosted application 904 using an application program interface (API).

[0113] The messaging clients 902 are able to communicate and exchange data with other messaging clients 902 and with a messaging server system 906 via the network 830. The data exchanged between the messaging clients 902 and between the messaging clients 902 and the messaging server system 906 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0114] The messaging server system 906 provides server-side functionality to a particular messaging client 902 via the network 830. Although some functions of the messaging system 900 are described herein as being performed by the messaging client 902 or by the messaging server system 906, it may be a design choice whether some functions are located within the messaging client 902 or within the messaging server system 906. For example, it may be technically preferable to initially deploy some technologies and functions within the messaging server system 906, but then migrate the technologies and functions to the messaging client 902 where the client device 826 has sufficient processing power.

[0115] The messaging server system 906 supports various services and operations provided to the messaging clients 902. Such operations include transmitting data to the messaging clients 902, receiving data from the messaging clients 902, and processing data generated by the messaging clients 902. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 900 is activated and controlled by functions available via the user interface (UI) of the messaging client 902.

[0116] Turning now specifically to the messaging server system 906, an application program interface (API) server 910 is coupled to and provides a programming interface to an application server 914. The application server 914 is communicatively coupled to a database server 916, which facilitates access to a database 920 that stores data associated with messages processed by the application server 914. Similarly, a web server 924 is coupled to and provides a web-based interface to the application server 914. To this end, the web server 924 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0117] The application program interface (API) server 910 receives and transmits message data (e.g., commands and message payloads) between the client device 826 and the application server 914. Specifically, the application program interface (API) server 910 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 902 to activate the functions of the application server 914. The application program interface (API) server 910 exposes various functions supported by the application server 914, including: account registration; login functionality; sending messages from a particular messaging client 902 to another messaging client 902 via the application server 914, sending media files (e.g., images or videos) from the messaging client 902 to the messaging server 912, and possible access by another messaging client 902; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 826; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 902).

[0118] The application server 914 hosts several server applications and subsystems, including, for example, a messaging server 912, an image processing server 918, and a social network server 922. The messaging server 912 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 902. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 902. In view of the hardware requirements for other processor- and memory-intensive data processing, such processing can also be performed on the server side by the messaging server 912.

[0119] The application server 914 also includes an image processing server 918 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 912.

[0120] The social network server 922 supports various social networking functions and services and makes these functions and services available to the messaging server 912. To this end, the social network server 922 maintains and accesses an entity graph within the database 920. Examples of functions and services supported by the social network server 922 include identifying other users in the messaging system 900 with whom a particular user has relationships or that the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0121] The messaging client 902 may notify the user of the client device 826 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 902 may provide notifications to participants of a conversation (e.g., a chat session) in the messaging client 902 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0122] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0123] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0124] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0125] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed for use with other components, and are usually part of a program that performs a specific function of a related function. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component can be a special processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be recognized that it can be determined for cost and time considerations whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system to implement the hardware component. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some of the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0126] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0127] "Machine storage media" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media," "device storage media," and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media," "computer storage media," and "device storage media" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media."

[0128] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0129] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0130] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method comprising: placing, by one or more processors of an augmented reality (AR) system, a hand tracking input pipeline of the AR system into a suspended mode; detecting, using a camera component of the hand tracking input pipeline, a possible visual wake-up command issued by a user of the AR system; The one or more processors perform operations based on detecting the possible visual wake-up command, the operations comprising: waking up the hand tracking input pipeline; and placing the camera assembly into a fully operational mode; verifying, by the one or more processors using the hand tracking input pipeline, the possible wake-on-vision command as a wake-on-vision command; and An AR application of the AR system is initiated by the one or more processors based on verifying the possible visual wake-up command.

2. The method according to claim 1, wherein: The possible visual wake-up command and the visual wake-up command are the same gesture performed by the user.

3. The method according to claim 1, wherein: Detecting the possible visual wake-up command further includes: The possible visual wake-up command is identified using a neural network of the camera component.

4. The method according to claim 1, wherein: Placing the hand tracking input pipeline in the suspended mode further comprises: placing, by the one or more processors, one or more components of the hand tracking input pipeline into a suspended mode; and The camera component is placed into a limited operating mode by the one or more processors.

5. The method according to claim 4, wherein: The limited operating mode of the camera component includes a limited frame rate.

6. The method according to claim 1, further comprising: The one or more processors perform operations based on an inability to verify that the possible visual wake-up command is the visual wake-up command, the operations comprising: placing one or more components of the hand tracking input pipeline into the suspended mode; and The camera component is placed in a limited operating mode.

7. The method according to claim 1, wherein: The AR system includes a head-mounted device.

8. A computing device, comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising: Put the AR system's hand tracking input pipeline into suspended mode; detecting, using a camera component of the hand tracking input pipeline, a possible visual wake-up command issued by a user of the AR system; Based on detecting the possible visual wake-up command, an operation is performed, the operation comprising: waking up the hand tracking input pipeline; and placing the camera assembly into a fully operational mode; verifying the possible visual wake-up command as a visual wake-up command using the hand tracking input pipeline; and Based on verifying the possible visual wake-up command, an AR application of the AR system is initiated.

9. The computing device according to claim 8, wherein: The possible visual wake-up command and the visual wake-up command are the same gesture performed by the user.

10. The computing device according to claim 8, wherein: The camera component uses a neural network of the camera component to identify the possible visual wake-up command.

11. The computing device according to claim 8, wherein: The instructions further causing the computing device to perform operations of placing the hand tracking input pipeline in the suspended mode further cause the computing device to perform operations comprising: placing one or more components of the hand tracking input pipeline into a suspended mode; and The camera component is placed in a limited operating mode.

12. The computing device according to claim 11, wherein: The limited operating mode of the camera component includes a limited frame rate.

13. The computing device of claim 8, wherein: The instructions further cause the computing device to perform operations based on failing to verify that the possible visual wake-up command is the visual wake-up command, the operations comprising: placing one or more components of the hand tracking input pipeline into the suspended mode; and The camera component is placed in a limited operating mode.

14. The computing device of claim 8, wherein: The AR system includes a head-mounted device.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: placing, by one or more processors of the AR system, a hand tracking input pipeline of the AR system into a suspended mode; detecting, by a camera component of the hand tracking input pipeline, a possible visual wake-up command issued by a user of the AR system; The one or more processors perform operations based on detecting the possible visual wake-up command, the operations comprising: waking up the hand tracking input pipeline; and placing the camera assembly into a fully operational mode; verifying, by the one or more processors, the possible wake-on-vision command as a wake-on-vision command using the hand tracking input pipeline; and An AR application of the AR system is initiated by the one or more processors based on verifying the possible visual wake-up command.

16. The non-transitory computer readable storage medium of claim 15, wherein: The possible visual wake-up command and the visual wake-up command are the same gesture performed by the user.

17. The non-transitory computer-readable storage medium of claim 15, wherein: Detecting the possible visual wake-up command further includes: The possible visual wake-up command is identified using a neural network of the camera component.

18. The non-transitory computer readable storage medium of claim 15, wherein: Placing the hand tracking input pipeline in the suspended mode further comprises: placing, by the one or more processors, one or more components of the hand tracking input pipeline into a suspended mode; and The camera component is placed into a limited operating mode by the one or more processors.

19. The non-transitory computer readable storage medium of claim 18, wherein: The limited operating mode of the camera component includes a limited frame rate.

20. The non-transitory computer readable storage medium of claim 15, wherein: The instructions further configure the computer to perform operations comprising: The one or more processors perform operations based on an inability to verify that the possible visual wake-up command is the visual wake-up command, the operations comprising: placing one or more components of the hand tracking input pipeline into the suspended mode; and The camera component is placed in a limited operating mode.

21. The non-transitory computer readable storage medium of claim 15, wherein: The AR system includes a head-mounted device.

Images