Hand tracking pipeline adjustment

Abstract

A hand tracking input pipeline adjustment system for an AR system is provided. The AR system deactivates the hand tracking input pipeline and places camera components of the hand tracking input pipeline in a limited mode of operation. The AR system uses the camera component to detect initiation of a gesture by a user of the AR system, and in response to detecting the initiation of the gesture, the AR system activates a hand tracking input pipeline and places the camera component in a full operating mode.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to Greek patent application serial number 20220100508 filed on June 22, 2022 and U.S. patent application serial number 17 / 947,947 filed on September 19, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to user interfaces, and more particularly to user interfaces used in augmented reality and virtual reality. Background Art

[0004] A head-mounted device can be implemented with a transparent or translucent display through which the user of the head-mounted device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or translucent display, and can also see objects (e.g., virtual objects such as renderings of 2D or 3D graphic models, images, videos, text, etc.) generated for display as part of the surrounding environment and / or superimposed on the surrounding environment. This is generally referred to as "augmented reality" or "AR". The head-mounted device can also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is generally referred to as "virtual reality" or "VR". As used herein, unless the context indicates otherwise, the term "AR" refers to either or both of augmented reality and virtual reality as traditionally understood.

[0005] AR systems are designed to be interactive devices that respond immediately to user input. However, keeping the AR system in full operation all the time when the user is not interacting with it wastes power and reduces usage time. Therefore, it is desirable for AR systems to have an "always on" mode of limited operation that saves power and extends usage time. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0007] Figure 1 is a perspective view of an AR system in the form of a head-mounted device according to some examples.

[0008] Figure 2 According to some examples Figure 1 Additional views of the head-mounted device.

[0009] Figure 3is a diagrammatic representation of a machine in the form of a computing device according to some examples within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.

[0010] Figure 4A is a collaboration diagram of a hand tracking input pipeline for an AR system according to some examples.

[0011] Figure 4B is an illustration of the data structure of a hand tracking input pipeline according to some examples.

[0012] Figure 4C is an illustration of another data structure of a hand tracking input pipeline according to some examples.

[0013] Figure 5 is an activity diagram of the processing of a watchdog component of an AR system according to some examples.

[0014] Figure 6 is a block diagram illustrating a software architecture of an AR system within which the present disclosure may be implemented according to some examples.

[0015] Figure 7 is a block diagram illustrating a networked system including details of an AR system according to some examples.

[0016] Figure 8 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network according to some examples. DETAILED DESCRIPTION

[0017] AR systems are limited when it comes to available user input modalities. Indicating user intent and invoking an action or application is more complex for users of AR systems when compared to other mobile devices such as mobile phones. When using a mobile phone, a user can go to the home screen and tap on a specific icon to launch an application. However, due to the lack of physical input devices such as a touch screen or keyboard, such interactions are not easily performed on AR systems. Typically, users can indicate their intent by pressing a limited number of hardware buttons or using a small touchpad. Therefore, it would be desirable to have the following input modality that allows a wider range of inputs that can be used by the user to indicate their intent through user input.

[0018] In some examples, the input modality utilized by the AR system is the recognition of gestures made by the user that do not involve direct manipulation (DMVO) of virtual objects. The gesture is made by the user moving and positioning parts of the user's body, and those parts of the user's body can be detected by the AR system when the user is wearing the AR system. The detectable parts of the user's body may include parts of the user's upper body, arms, hands, and fingers. The components of the gesture may include the movement of the user's arms and hands, the position of the user's arms and hands in space, and the positioning of the user holding his upper body, arms, hands, and fingers. The gesture is useful in providing an AR experience to the user because the gesture provides a way to provide user input to the AR system during the AR experience without the user diverting his attention from the AR experience. As an example, in an AR experience that is an operating manual for a machine, the user can simultaneously view the machine in a real-world scene through the lens of the AR system, view the AR overlay on the real-world scene view of the machine, and provide user input to the AR system.

[0019] AR systems have limited power and thermal budgets. To save power, an AR system can put itself into a suspended mode and enter a low-power state when not in use. It is expected that the user can signal the AR system to exit the suspended mode so that the user can interact with the AR system. Such a signal can be a hand gesture similar to other gestures that form the AR system's hand tracking interaction language. However, recognizing hand gestures will typically require power for computing resources that may not be available in the suspended mode.

[0020] The examples described herein solve these and other problems by providing a hand tracking input pipeline that can be dimmed. In some examples, an AR system includes a hand tracking input pipeline that provides an input modality that can be used for all applications executed by the AR system. During operation of the AR system, the AR system deactivates most of the components of the hand tracking input pipeline to save power. The AR system instructs the camera component to enter a limited operating mode, in which the camera will provide sufficient information to detect the initiation of a gesture. When the AR system detects the initiation of a gesture performed by a user of the AR system, the AR system activates the hand tracking input pipeline and instructs the camera component to enter a full operating mode. Detecting the initiation of a gesture is implemented using a binary gesture classifier that detects the initiation of a gesture without performing further classification of the detected gesture. The AR system sets a timer, and when the timer expires, the AR system returns to a low power mode by deactivating the hand tracking input pipeline and placing the camera in a limited operating mode.

[0021] Other technical features will be readily apparent to those skilled in the art from the following drawings, descriptions and claims.

[0022] Figure 1 is a head mounted AR system according to some examples (e.g., Figure 1 100). The glasses 100 may include a frame 102, which is made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a bridge 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0023] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0024] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 702 discussed below.

[0025] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0026] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0027] In some examples, left camera 114 and right camera 116 provide video frame data for use by glasses 100 to extract 3D information from a real scene.

[0028] The glasses 100 may also include a touchpad 126 mounted to one or both of the left temple piece 122 and the right temple piece 124, or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally arranged vertically, and the touchpad 124 is approximately parallel to the temple of the user in some examples. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. Additional user input can be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide the following means by which the glasses 100 can receive input from the user of the glasses 100.

[0029] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.

[0030] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0031] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image superimposed on or in the right optical element 110 with a view of the real-world scene seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image superimposed on or in the left optical element 108 with a view of the real-world scene seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides an optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's view of the real-world scene, including displaying a user interface to the user of the glasses 100.

[0032] However, it should be understood that other display technologies or configurations may be utilized within the optical engine to display images to a user within the user's field of view. For example, instead of providing a projector 204 and a waveguide, an LCD, LED or other display panel or surface may be provided.

[0033] In use, the user of the glasses 100 will be presented with information, content, and various user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Figure 7 The user may interact with the glasses 100 through voice input or touch input on a client device 726 shown in FIG. 1 , and / or hand movements, positions, and locations recognized by the glasses 100 .

[0034] Figure 3 is a diagrammatic representation of a machine 300 (such as a computing device) within which instructions 310 (e.g., software, programs, applications, applet, app, other executable code) may be executed that cause the machine 300 to perform any one or more of the methodologies discussed herein. The machine 300 may be used as Figure 1The computer 120 of the glasses 100. For example, the instructions 310 can cause the machine 300 to perform any one or more of the methods described herein. The instructions 310 transform a general, unprogrammed machine 300 into a specific machine 300 that is programmed to perform the functions described and shown in the described manner. The machine 300 can operate as a standalone device, or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 300 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 300 may include, but is not limited to: a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a notebook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 310 specifying actions to be taken by the machine 300. Further, while a single machine 300 is illustrated, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute instructions 310 to perform any one or more of the methodologies discussed herein.

[0035] Machine 300 may include processor 302, memory 304, and I / O components 306, which may be configured to communicate with each other via bus 344. In some examples, processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 308 and processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but machine 300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0036] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via the bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. During execution of the instructions 310 by the machine 300, the instructions 310 may also reside, in whole or in part, within the main memory 314, within the static memory 316, within the machine-readable medium 320, within the storage unit 318, within one or more of the processors 302 (e.g., within a processor's cache memory), or within any suitable combination thereof.

[0037] I / O components 306 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is to be understood that I / O components 306 may include Figure 3 306. In various examples, the I / O component 306 may include an output component 328 and an input component 332. The output component 328 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component 332 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a pointing-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides a touch position and / or force or a touch gesture, or other tactile input component), an audio input component (e.g., a microphone), etc.

[0038] In another example, the I / O component 306 may include: a biometric component 334, a motion component 336, an environmental component 338, or a positioning component 340, as well as various other components. For example, the biometric component 334 includes components for recognizing expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 336 may include an inertial measurement unit (IMU), an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental components 338 include, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect concentrations of hazardous gases for safety or measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals associated with the surrounding physical environment. Positioning components 340 include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), etc.

[0039] A variety of technologies may be used to implement communications. I / O components 306 also include communication components 342 that are operable to couple machine 300 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication components 342 may include a network interface component or another suitable device that interfaces with network 322. In other examples, communication components 342 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low power consumption), Device 324 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0040] In addition, the communication component 342 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Hypercode, UCC RSS-2D barcode and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 342, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0041] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 310) when executed by processor 302 cause various operations to implement the disclosed examples.

[0042] Instructions 310 may be transmitted or received over network 322 via a network interface device (e.g., a network interface component included in communications component 342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 310 may be transmitted or received to device 324 via coupling 326 (e.g., a peer-to-peer coupling) using a transmission medium.

[0043] Figure 4A is a collaboration diagram of a hand tracking input pipeline 454 for an AR system (such as glasses 100) according to some examples, and Figure 4B and Figure 4C is a diagram of a data structure. The AR system uses a hand tracking input pipeline 454 to track hand movements and hand positions of a user 462 using the AR system.

[0044] The camera component 402 of the hand tracking input pipeline 454 uses one or more cameras of the AR system such as Figure 1The cameras 114 and 116 generate real-world scene video frame data 424 of a real-world scene from the perspective of the user 462. The real-world scene video frame data 424 includes hand tracking video frame data of detectable parts of the user's body, including parts of the user's upper torso, arms, hands, and fingers. The hand tracking video frame data includes: video frame data of the movement of parts of the user's upper torso, arms, and hands when the user makes gestures or moves their hands and fingers to interact with the real-world scene; video frame data of the position of the user's arms and hands in space when the user makes gestures or moves their hands and fingers to interact with the real-world scene; and video frame data of the user maintaining the positioning of his upper torso, arms, hands, and fingers when the user makes gestures or moves his hands and fingers to interact with the real-world scene.

[0045] The camera component 402 passes the real-world scene video frame data 424 to the skeleton model inference component 404. The skeleton model inference component 404 generates the skeleton model data 428 based on the real-world scene video frame data 424. In some examples, the skeleton model inference component 404 receives the real-world scene video frame data 424 from the camera component 402, and extracts the features of the user's upper body, arms, and hands from the hand tracking video frame data included in the real-world scene video frame data 424. In some examples, the skeleton model inference component 404 generates the skeleton model data 428 based on the real-world scene video frame data 424 using a geometric method and one or more previously generated skeleton classifier models. In some examples, the skeleton model inference component 404 generates the skeleton model data 428 based on classifying the real-world scene video frame data 424 using an artificial intelligence method and a skeleton classifier model previously generated using a machine learning method. In some examples, the skeleton classifier model may include but is not limited to a neural network, a learning vector quantization network, a logistic regression model, a support vector machine, a random decision forest, a naive Bayes model, a linear discriminant analysis model, and a K-nearest neighbor model. In some examples, the machine learning method may include but is not limited to supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection.

[0046] The generated skeletal model data 428 includes landmark data including landmark identification, location in a real-world scene, and classification information of one or more landmarks associated with the user's upper body, arms, and hands.

[0047] The skeletal model inference component 404 transmits the skeletal model data 428 to the hand classification term inference component 406. In some examples, the skeletal model inference component 404 makes the skeletal model data 428 available to components and applications external to the hand tracking input pipeline 454.

[0048] The hand classification term inference component 406 receives the skeletal model data 428 from the skeletal model inference component 404 and generates the hand classification term probability data 426 based on the skeletal model data 428. In some examples, the pose is specified by the hand tracking input pipeline 454 according to a combination of hand classification terms. The hand classification terms are in turn composed of combinations and relationships of landmarks included in the skeletal model data 428. Because the hand tracking input pipeline 454 extracts the hand classification terms from the skeletal model data 428 by the hand tracking input pipeline 454 in a different layer than assembling hand movements into poses, the designer of the AR system can create new poses constructed from existing hand classification terms that constitute already known poses without having to retrain the machine learning component of the hand tracking input pipeline 454. In some examples, the hand classification term inference component 406 compares one or more skeletal models included in the skeletal model data 428 with previously generated hand classification term models, and generates one or more hand classification term probabilities based on the comparison. In some examples, the hand classification item inference component 406 classifies the skeleton model based on the hand classification item model generated by the artificial intelligence method and the machine learning method previously. Determine one or more hand classification item probabilities. In some examples, the hand classification item model may include but is not limited to a neural network, a learning vector quantization network, a logistic regression model, a support vector machine, a random decision forest, a naive Bayes model, a linear discriminant analysis model, and a K-nearest neighbor model. In some examples, the machine learning method may include but is not limited to supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection. In some examples, the hand classification item inference component 406 generates skeletal hand classification item probability data 426 based on the skeleton model data 428 using a geometric method and one or more previously generated hand classification item models.

[0049] The one or more hand classification term probabilities indicate probabilities that specified hand classification term components of a gesture can be identified from the skeletal model data 428. The hand classification term inference component 406 transmits the hand classification term probability data 426 to the gesture inference component 408 and the gesture text input recognition component 410.

[0050] The gesture inference component 408 receives the hand classification term probability data 426 and determines the gesture data 422 based on the hand classification term probability data 426. In some examples, the gesture inference component 408 compares the hand classification term identified in the hand classification term probability data 426 with the gesture identification data identifying the specific gesture. The gesture identification consists of one or more hand classification terms corresponding to the specific gesture. The gesture identification is defined using a grammar whose symbols correspond to the hand classification terms. For example, the posture identifier of a posture is "LEFT_PALMAR_FINGERSEXTENDED_RIGHT PALMAR_FINGERS_EXTENDED", where: "LEFT" is a symbol corresponding to the hand classification item indicating that the user's left hand has been recognized; "PALMAR" is a symbol corresponding to the hand classification item indicating that the palm of the user's hand has been detected and modifies "LEFT" to indicate that the user's left palm has been recognized; "FINGERS" is a symbol corresponding to the hand classification item indicating that the user's fingers have been recognized; and "EXTENDED" is a symbol corresponding to the hand classification item indicating that the user's fingers are extended and modifies "FINGERS". In some examples, the posture identifier is a single tag such as a number that identifies the posture based on the component hand classification items of the posture. Posture recognition identifies the posture in the context of a physical description of the posture. The posture inference component 408 transmits 422 to the system framework component 414.

[0051] The gesture inference component 408 transmits the gesture data 422 to the system framework component 414. The system framework component 414 receives the gesture data 422 and generates the directional input event data 450 or the directional input event data 450 based on the gesture data 422. The input event of the directional input event data 450 can be one of a plurality of categories. Undirected input events belonging to the undirected category are routed to operating system level components such as the system user interface component 416. Directional input events belonging to the directional category are routed to specific components such as the AR application component 418.

[0052] The gesture text input recognition component 410 receives the hand classification term probability data 426 and generates the symbol data 412 based on the hand classification term probability data 426. In some examples, the gesture inference component 408 compares the hand classification term identified in the hand classification term probability data 426 with the symbol data identifying specific characters, words, and commands. For example, the symbol data of the gesture is the character "V" of the gesture as a sign language symbol in American Sign Language (ASL). A single hand classification item for a gesture may be "LEFT" for the left hand, "PALMAR" for the palm of the left hand, "INDEXFINGER" for the index finger, "EXTENDED" for the modified "INDEXFINGER", "MIDDLEFINGER" for the middle finger, "EXTENTED" for the modified "MIDDLEFINGER", "RINGFINGER" for the ring finger, "CURLED" for the modified "RINGFINGER", "LITTLEFINGER" for the little finger, "CURLED" for the modified "LITTLEFINGER", "THUMB" for the thumb, and "CURLED" for the modified "THUMB".

[0053] In some examples, the entire word may also be identified by the gesture text input recognition component 410 based on the hand classification terms indicated by the hand classification term probability data 426. In some examples, commands such as commands corresponding to a specified set of keystrokes in an input system having a keyboard may be identified by the gesture text input recognition component 410 based on the hand classification terms indicated by the hand classification term probability data 426.

[0054] The gesture inference component 408 and the gesture text input recognition component 410 transmit gesture data 422 and symbol data 412, respectively, to the system framework component 414. The system framework component 414 receives the gesture data 422 and the symbol data 412 (collectively and individually referred to as "input event data") and generates undirected input event data 448 or directional input event data 450 based in part on the input event data. Undirected input events belonging to the undirected category of input events are routed to operating system level components such as the system user interface component 416. Directional input events belonging to the directional category of input events are routed to target components such as the AR application component 418.

[0055] In some examples of processing input data received from gesture inference component 408 and gesture text input recognition component 410 that can be classified as undirected input event data 448, system framework component 414 classifies the input data as undirected input event data 448 based on the input data and component registration data described below. System framework component 414 routes the input data as undirected input event data 448 to system user interface component 416 based on classifying the input data as undirected input event data 448.

[0056] The system user interface component 416 receives the undirected input event data 448 and determines a target component based on an indication or selection of a virtual object associated with the target component by the user when making a gesture corresponding to the undirected input event data 448. In some examples, the system user interface component 416 determines the position of the user's hand in the real-world scene when the gesture is made. The system user interface component 416 determines a set of virtual objects currently provided to the user by the AR system in the AR experience. The system user interface component 416 determines a virtual object whose apparent position in the real-world scene is related to the position of the user's hand in the real-world scene when the gesture is made. The system user interface component 416 determines the target AR application component based on looking up the AR application component associated with the virtual object in an internal data structure of the AR system, and determines the AR application component as the target AR application component.

[0057] The system user interface component 416 registers with the system framework component 414 the target AR application component to which the directional input event data 450 is to be routed. The system framework component 416 registers the component registration data such as Figure 4B The component registration data 438 is stored in a data store to be accessed during operation of the system framework component 414. The component registration data 438 includes a component ID field 430 identifying a target AR application component, a registered language field 436 identifying a language model to be associated with the target AR application component, and one or more registered gesture fields 432 and / or registered symbol fields 434 indicating gestures and symbols to be routed to the registered AR application component. As shown, the component ID field 430 includes the AR application component identifier "TEXTENTRY"; the registered language field 436 identifies the language associated with the registered AR application component, namely "ENGLISH"; the registered posture field 432 includes the posture identifier routed to the registered target AR application component, namely "LEFT_PALMAR_FINGERS EXTENDED_RIGHT_PALMAR_FINGERS_EXTENDED", and the registered symbol field 434 identifies a set of symbols routed to the registered AR application component (i.e., "[*]" representing all symbols).

[0058] As another example of component registration data, Figure 4C The component registration data 440 includes: a component ID field 442, which includes the AR application component identifier "EMAIL"; a registered language field 436, which identifies the language associated with the registered AR application component, namely "ASL"; and a registered symbol field 444, which identifies a set of symbols (i.e., the word "EMAIL") that are routed to the registered AR application component.

[0059] Referring again to the system framework component 414 processing input data that can be classified as undirected input event data 448 received from the gesture inference component 408 and the gesture text input recognition component 410, the system framework component 414 classifies the input data received from the gesture inference component 408 and the gesture text input recognition component 410 as undirected input event data 448 or directional input event data 450 based on the input data and the component registration data. In some examples, when processing the symbol data 412, the system framework component 414 searches the registered symbol field of the component registration data, such as the registered symbol field 434 of the component registration data 438, for a registered symbol that matches the symbol data. When the system framework component 414 determines a match, the system framework component 414 determines that the symbol data is directional input event data 450. The system framework component 414 also determines the target AR application component based on the target AR application component identified in the component ID field (such as the component ID field 430) of the component registration data that includes the matching registered symbol. In a similar manner, when processing gesture data 422, system framework component 414 searches a registered gesture field of component registration data, such as registered gesture field 432 of component registration data 438, for a registered gesture that matches the gesture input data. When system framework component 414 determines a match, system framework component 414 determines that the gesture input data is directional input event data 450, and also determines a target AR application component to which directional input event data 450 is to be routed. In the event that system framework component 414 determines that symbol data and / or gesture input data for the input data is not found in the component registration data, 414 determines that the input data is to be classified as undirectional input event data 448, and is to be routed to system user interface component 416.

[0060] In another example of processing directional input event data 450, an AR application component (e.g., AR application component 418) registers itself with the system framework component 414. To do so, the AR application component sends component registration data (e.g., Figure 4BThe component registration data 438 of the AR application component is transmitted to the system framework component 414. The system framework component 414 receives the component registration data and stores the component registration data in a data store for use in routing the directional input event data 450 to the AR application component.

[0061] In another example of processing the directional input event data 450, the AR system determines based on the hint that the directional input event data 450 is to be routed to the AR application component. For example, if the AR system executes the current AR application component in the single application mode state, the current AR application component is hinted as the AR application component to which the directional input event data 450 is to be routed.

[0062] In some examples, the system framework component 414 transmits the language model feedback data 420 to the hand classification term inference component 406 and the gesture inference component 408 to improve the accuracy of the inferences made by the hand classification term inference component 406 and the gesture inference component 408. In some examples, the system framework component 414 generates the language model feedback data 420 based on the user context data (e.g., component registration data that registers the AR application component and data about the hand classification terms that constitute the registered gestures and the hand classification terms that constitute the gestures associated with the registered symbols). The component registration data includes information about the gestures and symbols in the gesture data 422 and the symbol data 412 that are routed to the AR application component as part of the directional input event data 450, as well as the language of the symbols. In addition, the system framework component 414 includes information about the composition of a particular gesture including the hand classification terms associated with the gestures and symbols.

[0063] In another example of processing the language model feedback data 420, the system framework component 414 transmits the prompt as part of the language model feedback data 420 to the hand classification term inference component 406, the gesture inference component 408, and the gesture text input recognition component 410. The system framework component 414 generates the prompt based on the language model associated with the AR application component, for example, by the language specified in the registration language field 446 in the component registration data 440. The gesture text input recognition component 410 determines the possible next symbol N based on the previous characters N-1, N-2, etc. and the language model. In some examples, the system framework component 414 generates the prompt based on the language model, which is a hidden Markov model that predicts what the next symbol N is based on one or more of the previous characters N-1, N-2, etc. In another example, the gesture text input recognition component 410 generates the next symbol N using an artificial intelligence method based on a language model generated using a machine learning method. In some examples, the language model may include, but is not limited to, a neural network, a learning vector quantization network, a logistic regression model, a support vector machine, a random decision forest, a naive Bayes model, a linear discriminant analysis model, and a K nearest neighbor model. In some examples, machine learning methods may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection. System framework component 414 generates a prompt based on the next symbol N. In some examples, system framework component 414 determines the next gesture associated with the next symbol N by mapping the next symbol N to the next gesture based on a lookup table that associates symbols with gestures. System framework component 414 decomposes the next gesture into a set of one or more next hand classification items. System framework component 414 transmits the next gesture as part of language model feedback data 420 to gesture inference component 408, and transmits a set of next hand classification items as part of language model feedback data 420 to hand classification item inference component 406.

[0064] AR application components executed by the AR system, such as the watchdog component 456, the system user interface component 416, and the AR application component 418, are consumers of data (e.g., limited operational real-world scene video frame data 460, skeletal model data 428, gesture data 422, and symbolic data 412) generated by the hand tracking input pipeline 454. The AR system executes the system user interface component 416 to provide a system-level user interface, such as a command console, etc., to a user of the AR system using gestures as an input modality. The AR system executes the AR application component 418 to provide a user interface, such as an AR experience, to a user of the AR system using gestures as an input modality.

[0065] The AR system includes a watchdog component 456 that the AR system uses to activate and deactivate the hand tracking input pipeline 454 based on recognition of the initiation of a gesture, so that the AR system can provide an "always-on" gesture input function and still save power. The watchdog component 456 receives limited operational real-world scene video frame data 460 from the camera component 402 and generates instruction data 458 that is transmitted to the hand tracking input pipeline 454 based on the limited operational real-world scene video frame data 460.

[0066] The watchdog component 456 uses a binary gesture classifier 452 to identify the initiation of a gesture being made by the user based on the limited operation real world scene video frame data 460. The binary gesture classifier 452 identifies a conservative approximation of the initiation of a gesture with a high recall rate. The output of the binary gesture classifier 452 is a determination of whether the user 462 has initiated a gesture intended to be an input into the AR system, without determining what the gesture is or what the user's intention is when making the gesture input. This allows the binary gesture classifier 452 to operate on limited operation frame rate and limited operation resolution video frame data, and still determine the initiation of the gesture with speed and accuracy. In some examples, the binary gesture classifier 452 uses an artificial intelligence method and a binary gesture classification model previously generated using a machine learning method to classify the limited operation real world scene video frame data 460 into video frame data including the initiation of a gesture or video frame data not including the initiation of a gesture. In some examples, the binary gesture classification model may include, but is not limited to, a neural network, a learning vector quantization network, a logistic regression model, a support vector machine, a random decision forest, a naive Bayes model, a linear discriminant analysis model, and a K-nearest neighbor model. In some examples, machine learning methods may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection. In some examples, binary gesture classifier 452 classifies limited operation real-world scene video frame data 460 using a geometric method and one or more previously generated binary gesture classification models.

[0067] In some examples, binary gesture classifier 452 identifies a conservative approximation of the initiation of a gesture with high recall but low precision.

[0068] In some examples, the camera component 402 and the skeletal model inference component 404 communicate using an automatically synchronized shared memory buffer. In addition, the camera component 402 and the skeletal model inference component 404 publish the limited operation real world scene video frame data 460 and the skeletal model data 428 on the memory buffer, which can be accessed by components and applications (e.g., watchdog component 456) outside the hand tracking input pipeline 454.

[0069] In some examples, binary gesture classifier 452 recognizes the presence of a user's hand and acts as a hand detector. Whenever a hand appears in the field of view of the camera of camera component 402, watchdog component 456 activates the full hand tracking input pipeline 454.

[0070] In some examples, binary gesture classifier 452 recognizes a specific gesture, and watchdog component 456 activates the full hand tracking input pipeline 454 whenever the specific gesture is detected.

[0071] In many examples, the hand classification term inference component 406, the gesture inference component 408, and the gesture text input recognition component 410 transmit the hand classification term probability data 426, the gesture data 422, and the symbol data 412, respectively, via an inter-process communication method.

[0072] In some examples, the hand tracking input pipeline 454 operates to continuously generate and publish symbolic data 412, gesture data 422, and skeletal model data 428 based on real-world scene video frame data 424 generated by one or more cameras of the AR system.

[0073] Figure 5 is an activity diagram of watchdog processing performed by the watchdog component 456 according to some examples. The AR system uses the watchdog component 456 to save power while still providing always-on input functionality to the user.

[0074] In operation 502, the watchdog component 456 deactivates one or more components of the hand tracking input pipeline 454 by instructing the one or more components of the hand tracking input pipeline 454 to enter a deactivation mode. In some examples, one or more of the skeletal model inference component 404, the hand classification term inference component 406, the gesture inference component 408, and the gesture text input recognition component 410 of the hand tracking input pipeline 454 are deactivated. In some examples, the AR system deactivates a component of the hand tracking input pipeline by instructing the component to enter a standby mode, in which the component stops processing input data for generating output data and waits for instructions from the AR system to resume processing data. In the deactivation mode, the component consumes minimal resources of the AR system. In some examples, the AR system instructs one or more components of the hand tracking input pipeline 454 to enter a deactivation mode by generating deactivation instruction data 520 and transmitting the deactivation instruction data 520 as part of the instruction data 458 to the one or more components of the hand tracking input pipeline 454.

[0075] In operation 504, the watchdog component 456 places the camera component 402 of the hand tracking input pipeline 454 into a limited operating mode. In some examples, the limited operating mode includes instructing the camera component 402 to capture video frame data at a limited operating frame rate, which is a frame rate that is lower than the full operating frame rate. In some examples, the camera component 402 is instructed to capture video frame data at a limited operating resolution, which is a reduced resolution that is lower than the full operating resolution. In some examples, the reduced frame rate of the limited operating mode is 5 frames per second. In some examples, the full operating frame rate is 30 frames per second. In some examples, the AR system instructs the camera component 402 to enter the limited operating mode by generating limited operating mode instruction data 524 and transmitting the limited operating mode instruction data 524 to the camera component 402 as part of the instruction data 458.

[0076] The camera component 402 receives the limited operation mode instruction data 524 and begins to operate in the limited operation mode. In the limited operation mode, the camera component 402 generates limited operation real-world scene video frame data 460 at a limited operation frame rate. In some examples, the camera component 402 generates limited operation real-world scene video frame data 460 at a limited operation resolution. The real-world scene video frame data includes gesture video frame data of a gesture 464 being made by a user 462. The camera component 402 generates the limited operation real-world scene video frame data 460 and transmits the limited operation real-world scene video frame data 460 to the binary gesture classifier 452 of the watchdog component 456.

[0077] In operation 506, the watchdog component 456 receives the limited operation real-world scene video frame data 460, and uses the binary gesture classifier 452 to classify the limited operation real-world scene video frame data 460 from the camera device component 402 as video frame data that includes or does not include the initiation of a gesture performed by the user 462 to detect the initiation of a gesture.

[0078] In operation 508, based on the failure to detect the initiation of a gesture by classifying the limited operation real-world scene video frame data 460 as video frame data that does not include the initiation of a gesture performed by the user 462 (as represented by the [No] branch of operation 508), the watchdog component 456 returns to operation 506 and continues to classify the limited operation real-world scene video frame data 460 as video frame data that includes or does not include the initiation of a gesture.

[0079] In operation 510, based on detecting the initiation of a gesture by classifying the limited operation real world scene video frame data 460 as video frame data including the initiation of a gesture performed by the user 462 (as represented by the [yes] branch of operation 508), the watchdog component 456 activates one or more deactivated components of the hand tracking input pipeline 454. In some examples, one or more of the skeletal model inference component 404, the hand classification term inference component 406, the gesture inference component 408, and the gesture text input recognition component 410 of the hand tracking input pipeline 454 are activated. In some examples, the deactivated components of the hand tracking input pipeline 454 operate in a standby mode in which the components do not process input data for generating output data. Instead, the components wait for an activation instruction. When the component receives the activation instruction, the component resumes receiving and processing input data for generating output data. In some examples, the watchdog component 456 generates activation instruction data 522 and transmits the activation instruction data 522 to the hand tracking input pipeline 454 as part of the instruction data 458. One or more deactivation components to be activated in the hand tracking input pipeline 454 receive the activation instruction data 522 and are activated to operate in the full operational mode.

[0080] In operation 512, the watchdog component 456 instructs the camera component 402 to enter the full operational mode. In some examples, the watchdog component 456 generates full operational mode instruction data 526 and transmits the full operational mode instruction data 526 to the camera component 402. The camera component 402 receives the full operational mode instruction data 526 and begins operating in the full operational mode. In some examples, when in the full operational mode, the camera component 402 generates real-world scene video frame data 424 at a full operational frame rate that is greater than the limited operational frame rate.

[0081] In some examples, when in the full operational mode, the camera component 402 generates real-world scene video frame data 424 at a full operational resolution that is greater than a limited operational resolution.

[0082] In some examples, the camera component 402 includes multiple cameras, and the camera component 402 selectively turns the multiple cameras on and off. In a limited operating mode, the camera component 402 selectively turns off one or more of the multiple cameras and operates with a subset of the multiple cameras. In a full operating mode, the camera component 402 operates with a larger subset or all of the multiple cameras.

[0083] In the case where the hand tracking input pipeline 454 including the camera component 402 is fully operational, the hand tracking input pipeline 454 processes the initiation of a gesture detected by the watchdog component 456 while the camera component 402 is in the limited operational mode. In some examples, the watchdog component 456 detects the initiation of a gesture and, while the user is still performing the gesture that is detected by the watchdog component 456, activates the components of the hand tracking input pipeline 454 from the deactivated mode and activates the camera component 402 from the limited operational mode. This allows the AR system to use the hand tracking input pipeline 454 to recognize the gesture being performed by the user 462 while the user is still performing the gesture. That is, the fully operational hand tracking input pipeline 454 recognizes the same gesture being performed by the user 462 that was identified as a possible gesture by the watchdog component 456 using the binary gesture classifier 452 when the hand tracking input pipeline 454 was in the deactivated mode. In some examples, the delay for the hand tracking input pipeline 454 to switch from the disabled mode to the fully operational mode and the camera component 402 to switch from the limited operational mode to the fully operational mode is 100 milliseconds.

[0084] In operation 514, watchdog component 456 sets a timer that signals the AR system to disable hand tracking input pipeline 454 and place camera component 402 in limited operating mode when the timer expires. In operation 516, watchdog component 456 monitors the timer. In operation 518, based on determining in operation 516 that the timer has not expired, watchdog component 456 returns to operation 516 and continues to monitor the timer. Based on determining in operation 518 that the timer has expired, watchdog component 456 returns to operation 502 and disables components of hand tracking input pipeline 454, and instructs camera component 402 to enter limited operating mode.

[0085] In some examples, binary gesture classifier 452 is a component of hand tracking input pipeline 454 .

[0086] In some examples, binary gesture classifier 452 is a component of the AR system.

[0087] In some examples, the AR system performs the functionality of watchdog component 456 .

[0088] In some examples, watchdog component 456 is a component of hand tracking input pipeline 454 .

[0089] In some examples, hand tracking input pipeline 454 performs the operations of watchdog component 456 .

[0090] In some examples, watchdog component 456 is activated by a user of the AR system using a physical input device on the AR system such as, but not limited to, touchpad 126 .

[0091] Figure 6 6 is a block diagram 600 illustrating a software architecture 604 that may be installed on any one or more of the devices described herein. The software architecture 604 is supported by hardware, such as a machine 602, including a processor 620, a memory 626, and an I / O component 638. In this example, the software architecture 604 may be conceptualized as a stack of layers in which each layer provides specific functionality. The software architecture 604 includes layers such as an operating system 612, a library 608, a framework 610, and an application 606. In operation, the application 606 invokes an API call 650 through the software stack and receives a message 652 in response to the API call 650.

[0092] The operating system 612 manages hardware resources and provides public services. The operating system 612 includes, for example, a kernel 614, services 616, and drivers 622. The kernel 614 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 614 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 616 can provide other public services to other software layers. Drivers 622 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 622 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0093] The library 608 provides a low-level common infrastructure used by the application 606. The library 608 may include a system library 618 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 608 may include an API library 624, such as a media library (e.g., a library for supporting presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting two-dimensional (2D) and three-dimensional (3D) graphics content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 608 may also include various other libraries 628 to provide many other APIs to the application 606 .

[0094] The framework 610 provides a high-level common infrastructure used by the applications 606. For example, the framework 610 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 610 can provide a wide range of other APIs that can be used by the applications 606, some of which can be specific to a particular operating system or platform.

[0095] In some examples, applications 606 may include home applications 636, contact applications 630, browser applications 632, book reader applications 634, location applications 642, media applications 644, messaging applications 646, game applications 648, and a variety of other applications such as third-party applications 640. Applications 606 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 606 constructed in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C language or assembly language). In a specific example, third-party applications 640 (e.g., applications created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the Phone or another mobile operating system. In this example, third-party applications 640 can call API calls 650 provided by the operating system 612 to facilitate the functions described herein.

[0096] Figure 7 is a block diagram showing a networked system 700 including details of the glasses 100 according to some examples. The networked system 700 includes the glasses 100, a client device 726, and a server system 732. The client device 726 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 736 and / or a high-speed wireless connection 734. The client device 726 is connected to the server system 732 via a network 730. The network 730 may include any combination of wired and wireless connections. The server system 732 may be one or more computing devices that are part of a service or network computing system. The client device 726 and any elements of the server system 732 and the network 730 may use Figure 6 and Figure 3 The details of the software architecture 604 or machine 300 described respectively are implemented.

[0097] The glasses 100 include a data processor 702, a display 710, one or more cameras 708, and additional input / output elements 716. The input / output elements 716 may include a microphone, an audio speaker, a biometric sensor, additional sensors, or additional display elements integrated with the data processor 702. Figure 6 and Figure 3 Examples of input / output elements 716 are further discussed. For example, input / output elements 716 may include any I / O component 306, including output component 328, motion component 336, etc. Figure 2 Examples of display 710 are discussed in . In the particular examples described herein, display 710 includes displays for the user's left eye and right eye.

[0098] Data processor 702 includes image processor 706 (eg, video processor), GPU and display driver 738, tracking module 740, interface 712, low power circuitry 704, and high speed circuitry 720. Components of data processor 702 are interconnected by bus 742.

[0099] The interface 712 refers to any source of user commands provided to the data processor 702. In one or more examples, the interface 712 is a physical button that, when pressed, sends a user input signal from the interface 712 to the low-power processor 714. The low-power processor 714 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 714 can process pressing such a button within a first time period as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 712 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the camera 708. In other examples, the interface 712 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 726.

[0100] Image processor 706 includes circuitry for receiving signals from camera 708 and processing those signals from camera 708 into a format suitable for storage in memory 724 or for transmission to client device 726. In one or more examples, image processor 706 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 708, and volatile memory used by the microprocessor in operation.

[0101] The low power circuit system 704 includes a low power processor 714 and a low power wireless circuit system 718. These elements of the low power circuit system 704 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 714 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 714 can accept user input signals from the interface 712. The low power processor 714 can also be configured to receive input signals or command communications from the client device 726 via the low power wireless connection 736. The low power wireless circuit system 718 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 718. In other examples, other low power communication systems may be used.

[0102] High-speed circuit system 720 includes a high-speed processor 722, a memory 724, and a high-speed wireless circuit system 728. High-speed processor 722 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 702. High-speed processor 722 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 734 using high-speed wireless circuit system 728. In some examples, high-speed processor 722 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Figure 6 The high-speed processor 722, which executes the software architecture of the data processor 702, manages data transmission with the high-speed wireless circuit system 728, in addition to any other duties. In some examples, the high-speed wireless circuit system 728 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 728 can implement other high-speed communication standards.

[0103] The memory 724 includes any storage device capable of storing camera data generated by the camera 708 and the image processor 706. Although the memory 724 is shown as being integrated with the high-speed circuit system 720, in other examples, the memory 724 may be a separate, independent element of the data processor 702. In some such examples, electrical wiring may provide a connection from the image processor 706 or the low-power processor 714 to the memory 724 through a chip including the high-speed processor 722. In other examples, the high-speed processor 722 may manage the addressing of the memory 724 so that the low-power processor 714 will direct the high-speed processor 722 whenever a read or write operation involving the memory 724 is desired.

[0104] The tracking module 740 estimates the pose of the glasses 100. For example, the tracking module 740 uses image data and associated inertial data from the camera 708 and the positioning component 340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene). The tracking module 740 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world scene. The tracking module 740 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 710.

[0105] The GPU and display driver 738 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 710 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 738 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world scene.

[0106] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 726 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 606, such as the messaging application 646.

[0107] Figure 8 is a block diagram illustrating an example messaging system 800 for exchanging data (e.g., messages and associated content) over a network. The messaging system 800 includes multiple instances of a client device 726 that host several applications including a messaging client 802 and other applications 804. The messaging client 802 is communicatively coupled to other instances of the messaging client 802 (e.g., hosted on respective other client devices 726), a messaging server system 806, and a third-party server 808 via a network 730 (e.g., the Internet). The messaging client 802 may also communicate with the locally hosted application 804 using an application program interface (API).

[0108] The messaging clients 802 are able to communicate and exchange data with other messaging clients 802 and with a messaging server system 806 via the network 730. The data exchanged between the messaging clients 802 and between the messaging clients 802 and the messaging server system 806 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0109] The messaging server system 806 provides server-side functionality to a particular messaging client 802 via the network 730. Although some functions of the messaging system 800 are described herein as being performed by the messaging client 802 or by the messaging server system 806, the location of some functions within the messaging client 802 or within the messaging server system 806 may be a design choice. For example, it may be technically preferable to initially deploy some technologies and functions within the messaging server system 806, but then migrate the technologies and functions to the messaging client 802 where the client device 726 has sufficient processing power.

[0110] The messaging server system 806 supports various services and operations provided to the messaging clients 802. Such operations include transmitting data to the messaging clients 802, receiving data from the messaging clients 802, and processing data generated by the messaging clients 802. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 800 is activated and controlled by functions available via the user interface (UI) of the messaging client 802.

[0111] Turning now specifically to the messaging server system 806, an application program interface (API) server 810 is coupled to and provides a programming interface to an application server 814. The application server 814 is communicatively coupled to a database server 816, which facilitates access to a database 820 that stores data associated with messages processed by the application server 814. Similarly, a web server 824 is coupled to and provides a web-based interface to the application server 814. To this end, the web server 824 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0112] The application program interface (API) server 810 receives and transmits message data (e.g., commands and message payloads) between the client device 726 and the application server 814. Specifically, the application program interface (API) server 810 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 802 to activate the functions of the application server 814. The application program interface (API) server 810 exposes various functions supported by the application server 814, including: account registration; login functionality; sending messages from a particular messaging client 802 to another messaging client 802 via the application server 814, sending media files (e.g., images or videos) from the messaging client 802 to the messaging server 812, and possible access by another messaging client 802; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 726; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 802).

[0113] The application server 814 hosts several server applications and subsystems, including, for example, a messaging server 812, an image processing server 818, and a social network server 822. The messaging server 812 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 802. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 802. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 812.

[0114] The application server 814 also includes an image processing server 818 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 812.

[0115] The social network server 822 supports various social networking functions and services and makes these functions and services available to the messaging server 812. To this end, the social network server 822 maintains and accesses an entity graph within the database 820. Examples of functions and services supported by the social network server 822 include identifying other users in the messaging system 800 with whom a particular user has relationships or that the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0116] The messaging client 802 may notify the user of the client device 726 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 802 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 802 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0117] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

[0118] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0119] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0120] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed for use with other components, and are usually part of a program that performs a specific function of a related function. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some of the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0121] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0122] "Machine storage media" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media," "device storage media," and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media," "computer storage media," and "device storage media" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media."

[0123] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes called "cores") that can execute instructions simultaneously.

[0124] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0125] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method comprising: deactivating, by one or more processors of an augmented reality (AR) system, a hand tracking input pipeline of the AR system; instructing, by the one or more processors, a camera component of the AR system to enter a limited operating mode; detecting, by the one or more processors using the camera component, initiation of a gesture by a user of the AR system; as well as The one or more processors perform the following operations based on detecting the initiation of the gesture, the operations comprising: activating the hand tracking input pipeline; as well as The camera component is instructed to enter a fully operational mode.

2. The method according to claim 1, wherein: Detecting the initiation of the gesture further comprises: The initiation of the gesture is identified using a binary gesture classifier.

3. The method according to claim 1, wherein: The limited operating mode of the camera component includes a limited operating frame rate.

4. The method according to claim 3, wherein: The limited operating frame rate is a frame rate that is less than the full operating frame rate.

5. The method according to claim 1, wherein: The camera device component includes a plurality of cameras, and Wherein, instructing the camera device component of the AR system to enter the limited operation mode further includes: The camera assembly is instructed to selectively shut down one or more cameras of the camera assembly.

6. The method according to claim 1, further comprising: Based on detecting the initiation of the gesture, performing the following operations includes: Setting timers; and Based on determining that the timer has expired, perform the following operations, the operations comprising: deactivating one or more components of the hand tracking input pipeline; and The camera component is instructed to enter the limited operating mode.

7. The method according to claim 1, wherein: The AR system includes a head-mounted device.

8. A computing device, comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising: Deactivating a hand tracking input pipeline of the AR system; Instructing a camera component of the AR system to enter a limited operation mode; detecting, using the camera component, initiation of a gesture by a user of the AR system; and Based on detecting the initiation of the gesture, performing the following operations includes: activating the hand tracking input pipeline; and The camera component is instructed to enter a fully operational mode.

9. The computing device according to claim 8, wherein: The instructions, which when executed by the one or more processors, cause the computing device to perform operations for detecting initiation of the gesture further include: The initiation of the gesture is identified using a binary gesture classifier.

10. The computing device according to claim 8, wherein: The limited operating mode of the camera component includes a limited operating frame rate.

11. The computing device according to claim 10, wherein: The limited operating frame rate is a frame rate that is less than the full operating frame rate.

12. The computing device according to claim 8, in, The camera device component includes a plurality of cameras, and Wherein, the instructions that, when executed by the one or more processors, cause the computing system to perform operations including instructing a camera component of the AR system to enter a limited operating mode further cause the computing system to perform operations including: The camera assembly is instructed to selectively shut down one or more cameras of the camera assembly.

13. The computing device of claim 8, wherein: The instructions, when executed by the one or more processors, further cause the computing device to perform operations including: The one or more processors perform the following operations based on detecting the initiation of the gesture, the operations comprising: setting, by the one or more processors, a timer; Based on determining that the timer has expired, perform the following operations, the operations comprising: deactivating one or more components of the hand tracking input pipeline; and The camera assembly is placed in a limited operating mode.

14. The computing device of claim 8, wherein: The AR system includes a head-mounted device.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a computer, cause the computer to perform operations comprising: Deactivating a hand tracking input pipeline of the AR system in a deactivation mode; placing a camera component of the AR system in a limited operating mode; detecting, using the camera component, initiation of a gesture by a user of the AR system; as well as Based on detecting the initiation of the gesture, performing the following operations includes: activating the hand tracking input pipeline; as well as Place the camera assembly into full operational mode.

16. The non-transitory computer readable storage medium of claim 15, wherein: The instructions, which when executed by one or more processors of a computer, cause the computer to perform operations of detecting initiation of the gesture further include: The initiation of the gesture is identified using a binary gesture classifier.

17. The non-transitory computer-readable storage medium of claim 15, wherein: The limited operating mode of the camera component includes a limited operating frame rate.

18. The non-transitory computer readable storage medium of claim 17, wherein: The limited operating frame rate is a frame rate that is less than the full operating frame rate.

19. The non-transitory computer readable storage medium of claim 15, in, The camera device component includes a plurality of cameras, and Wherein, the instructions that, when executed by the one or more processors, cause the computer to perform operations including instructing a camera component of the AR system to enter a limited operating mode further cause the computer to perform operations including: The camera assembly is instructed to selectively shut down one or more cameras of the camera assembly.

20. The non-transitory computer readable storage medium of claim 15, wherein: When executed by one or more processors of a computer, the instructions cause the computer to perform the following operations, the operations comprising: Based on detecting the initiation of the gesture, performing the following operations includes: activating the hand tracking input pipeline; and Place the camera assembly into full operational mode.

21. The non-transitory computer readable storage medium of claim 15, wherein: The AR system includes a head-mounted device.

Images