Gesture-based application invocation
Through posture-based input mode and contactless imaging sensors, the problem of user interaction complexity of AR devices is solved, a more efficient and natural user experience is achieved, and the interactiveness of the virtual environment is enhanced.
Patent Information
- Application Number
- CN202380032534.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-04
- Filing Date
- 2023-03-30
- Publication Date
- 2024-11-15
AI Technical Summary
Existing head-mounted AR devices have complexities in user intentions and application calls, lacking effective input modes, especially in the absence of physical input devices, it is difficult to achieve efficient user interaction with virtual objects.
Using a posture-based input mode, by detecting the posture and hand movement of the user's body, a contactless imaging sensor provides a silent or contactless user interface, allowing the user to call applications and perform actions through posture and hand tracking methods, Combines the action manager to parse and map poses to the corresponding applications and actions.
It improves the operational convenience and interactivity of AR devices, reduces dependence on visual, auditory and tactile prompts, saves computing resources, provides a more natural and non-interference user experience, and enhances the user's interaction ability with the virtual environment.
Smart Images

Figure CN118974687A_ABST
Abstract
Description
[0001] Priority claim
[0002] This application claims the benefit of priority to U.S. patent application serial number 17 / 657,911, filed on April 4, 2022, which is incorporated herein by reference in its entirety. Technical Field
[0003] The present disclosure relates generally to software application management, and more particularly to management of applications used in augmented reality and virtual reality. Background Art
[0004] A head-mounted device may be implemented with a transparent or translucent display through which the user of the head-mounted device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or translucent display, and also to see objects (e.g., virtual objects such as renderings of 2D or 3D graphics models, images, videos, text, etc.) generated for display as part of the surrounding environment and / or superimposed on the surrounding environment. This is commonly referred to as "augmented reality" or "AR". A head-mounted device may also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is commonly referred to as "virtual reality" or "AR". As used herein, unless the context indicates otherwise, the term "AR" refers to either or both of augmented reality and virtual reality as traditionally understood.
[0005] Users of head mounted devices can access and use computer software applications to perform various tasks or participate in entertainment activities. In order to use the computer software application, the user interacts with a user interface provided by the head mounted device. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.
[0007] Figure 1 is a perspective view of a head mounted device according to some examples.
[0008] Figure 2 Based on some examples Figure 1 Additional views of the head-mounted device.
[0009] Figure 3 is a diagrammatic representation of a machine in the form of a computing device according to some examples within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.
[0010] Figure 4is a process flow diagram of a gesture-based application invocation process for an AR device according to some examples.
[0011] Figure 5 is a sequence diagram of an example gesture-based application invocation process for an AR device according to some examples.
[0012] Fig. 6A is a diagram of the data structure of posture metadata according to some examples.
[0013] Figure 6B is a data structure diagram of application and action metadata according to some examples.
[0014] Figure 6C is a process flow diagram of an application management process for an AR device according to some examples.
[0015] Fig. 7A is a sequence diagram of a prompt user interface process of an AR device according to some examples.
[0016] Figure 7B is a prompt user interface for an AR device according to some examples.
[0017] Fig. 8A Depicted is a gesture indicating that a user wants to see a battery level displayed, according to some examples.
[0018] Figure 8B Depicted are gestures indicating that a user wants to invoke a camera application, according to some examples.
[0019] Figure 8C Depicted are gestures indicating that a user wants to invoke a navigation application, according to some examples.
[0020] Fig.8D Depicted is a gesture indicating that a user wants to invoke a time or watch application, according to some examples.
[0021] Fig. 8E Depicted are gestures indicating that a user wants to invoke a health monitoring application, according to some examples.
[0022] Figure 8F Depicted is a gesture indicating that a user wants to close an application, according to some examples.
[0023] Fig. 9 It is shown that according to some examples, Figure 1 Block diagram of networking system 900 showing details of a head mounted device.
[0024] Fig.10 is a diagrammatic representation of a networked environment in which the present disclosure may be deployed, according to some examples.
[0025] Fig.11is a block diagram illustrating a software architecture within which the present disclosure may be implemented according to some examples. DETAILED DESCRIPTION
[0026] AR devices may be limited when it comes to available computing resources and user input modalities. Unlike other mobile devices such as mobile phones, indicating user intent and invoking actions or applications is more complicated for users of AR devices. When using a mobile phone, a user can go to the home screen and tap on a specific icon to launch an application. However, due to the lack of physical input devices such as a touch screen or keyboard, such interactions are not easy to perform on AR devices. Typically, users can indicate their intent by pressing a limited number of hardware buttons or using a small touchpad. Therefore, it would be desirable to have the following input modality that allows a wider range of inputs that can be used by users to indicate their intent through user input.
[0027] An input modality that can be used with an AR device is hand tracking, where a user is provided with a user interface that is displayed to the user in an AR overlay with a 2D or 3D rendering. The rendering is a graphical model in 2D or 3D, where virtual objects located in the model correspond to interactive elements of the user interface. In this way, the user perceives the virtual objects as objects within the overlay in the user's field of view of the real world scene when wearing the AR device, or perceives the virtual objects as objects within the virtual world as seen by the user when wearing the AR device. In order to allow the user to interact with the virtual objects, the AR device detects the user's hands and tracks the movement, position, and / or location of the hands to determine the user's interaction with the virtual objects.
[0028] The generation and display of images that form the hand tracking user interface utilizes computational and power resources that may be limited on the AR device. Furthermore, the generation of the user interface means that the application may already be running on the AR device that is capable of generating the user interface. Therefore, it would be desirable to have an input modality that does not utilize the visual component of the user interface, which relies on the generation of images and their subsequent presentation to the user to prompt user interaction.
[0029] Utilizing gesture-based interactions as described herein provides improvements in the operation of an AR device. Gestures made by a user have the ability to provide a greater range of user input than with a keyboard or touchpad. For example, considering only a single hand with 5 fingers, there are 2 5 different possible inputs.
[0030] Gestures provide an input modality suitable for use with AR devices. Gestures are made by the user who moves and positions parts of the user's body, and those parts of the user's body can be detected by the AR device when the user is wearing the AR device. Detectable parts of the user's body may include parts of the user's upper body, arms, hands, and fingers. The components of the gesture may include the movement of the user's arms and hands, the position of the user's arms and hands in space, and the positioning of the user holding his upper body, arms, hands, and fingers. Gestures are useful in providing AR experiences to users because gestures provide a way to provide user input to the AR device during the AR experience without allowing the user to divert their attention from the AR experience. As an example, in an AR experience that is an operating manual for a machine, a user can simultaneously view a machine in a real-world scene through the lens of the AR device, view the AR overlay on the real-world scene view of the machine, and provide user input to the AR device.
[0031] As further described in detail herein, gestures are used as user input modalities in various types of user interfaces. In some examples, gestures are used in silent user interfaces that do not provide prompts to the user to elicit user input or guide the user through a user interface dialog box, that is, the AR device can provide a silent user interface and detect silent gestures made by the user. Therefore, the use of silent gestures reduces the use of visual, auditory, or tactile cues presented to the user to allow a user interface to be implemented that does not use the imaging forming elements of the AR device or consume computing resources to generate and implement a user interface with visual, auditory, and / or tactile cues. For example, silent gestures are detected and utilized by the AR device without using the visual, auditory, and / or tactile components of the user interface provided to the user to prompt the user for user input or guide the user through a user interface dialog box.
[0032] In one example, gestures are used as a user input modality in a prompting user interface that provides visual, auditory, and / or tactile components that prompt or guide the user to input a prompting gesture.
[0033] In another example, gestures are used in a user interface dialog box having a user interface displayed to a user with an AR overlay having a rendering. Virtual objects are provided in the rendering as interactive elements with which the user can interact using the detected gestures.
[0034] In some examples, gestures and hand tracking methods are combined to implement complex user interface dialog boxes. As an example, a user interface dialog box can be started using a silent gesture, and within the user interface dialog box, a hand tracking method can be used to prompt the user to make a prompt gesture.
[0035] In another example, gestures are detected using a non-contact method, and these gestures are referred to herein as "non-contact gestures". Non-contact gestures are gestures detected without the use of a physical input device such as a keyboard, touch screen, keypad, switch, etc., which are manipulated by the user by contacting the physical input device. Non-contact gestures are sensed or detected using a non-contact imaging sensor that does not make physical contact with the part of the user's body being detected. An example non-contact imaging sensor with temporal and spatial resolution is a camera operating in the visible or infrared spectrum that is sufficient to capture details of a part of the user's body for producing spatial resolution. Therefore, one or more visible or infrared cameras that detect user gestures can be used to detect the user's non-contact gestures without the use of a physical input device. Other types of non-contact imaging sensors with sufficient temporal and spatial resolution, such as sensors that utilize ultrasonic or electromagnetic waves outside the visible and infrared spectrums, may also be used.
[0036] In some examples, hand tracking is implemented using a contactless method in which one or more visible or infrared cameras are used to detect the movement, position, and location of a user's hand without using a physical input device such as a sensor glove, a keyboard, a keypad, a touch screen, a touchpad, one or more switches, and the like.
[0037] In an example use of the gesture and hand tracking method, a system for invoking an application and / or instructing an application to perform a specified action combines a silent gesture-based user interface as a startup process and a prompt hand tracking user interface to address the need for prompt input in portions of a user interface dialog box.
[0038] During the application registration process, gestures can be used for applications and actions on the AR device, and applications can register to be notified of relevant gestures and provide interactive applications associated with those gestures to the user. As a result of the application registration process, a set of registered applications and actions are created and stored, the set of registered applications and actions having metadata associating applications and application actions with gestures, and a set of qualified tags having semantic metadata identifying gestures can be used to help map specific gestures to applications and actions.
[0039] During the scanning process of the AR device, a silent user interface using gestures as an input modality is provided to the user. To invoke an application or an action of an application, the user performs a specified gesture detected by the AR device. The AR device provides the detected gesture to the action manager, which associates the application or application action with the detected gesture.
[0040] If there is one application or action associated with the detected gesture, the action manager invokes the application or action. If more than one application or action is registered in association with a particular gesture to cause ambiguity in the user's intent, the action manager disambiguates the user's intent using a prompting user interface that uses prompts and hand tracking to resolve the ambiguity. The prompting user interface provides identification information of applications and / or application actions to the user as a visual prompt so that the user can select a desired application or action.
[0041] When the action manager calls the application, the action manager can pass the captured video frame data and other context and operation information (such as action type and any detected tags) as input parameters. The called application can receive the input parameter data using an API hook that can be triggered by the manager when the application is launched. The application can then start executing or perform the intended action.
[0042] Other technical features will be readily apparent to those skilled in the art from the following drawings, descriptions and claims.
[0043] Figure 1 is a head mounted AR device according to some examples (e.g. Figure 1 100). The glasses 100 may include a frame 102, which is made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a bridge 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.
[0044] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or unitary construction.
[0045] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 902 discussed below.
[0046] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.
[0047] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.
[0048] In some examples, left camera 114 and right camera 116 provide video frame data for use by glasses 100 to extract 3D information from a real scene.
[0049] The glasses 100 may also include a touchpad 126 mounted to one or both of the left temple piece 122 and the right temple piece 124, or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally arranged vertically, and the touchpad 124 is approximately parallel to the temple of the user in some examples. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. Additional user input can be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide the following means by which the glasses 100 can receive input from the user of the glasses 100.
[0050] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.
[0051] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .
[0052] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image superimposed on or in the right optical element 110 with a view of the real-world scene seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image superimposed on or in the left optical element 108 with a view of the real-world scene seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides an optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's view of the real-world scene, including displaying a user interface to the user of the glasses 100.
[0053] However, it should be understood that other display technologies or configurations may be utilized within the optical engine to display images to a user within the user's field of view. For example, instead of providing a projector 204 and a waveguide, an LCD, LED or other display panel or surface may be provided.
[0054] In use, the user of the glasses 100 will be presented with information, content, and various user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Fig. 9 The user may interact with the glasses 100 via voice input or touch input on a client device 926 shown in FIG. 1 , and / or hand movements, positions, and locations detected by the glasses 100 .
[0055] Figure 3 is a diagrammatic representation of a machine 300 within which instructions 310 (e.g., software, programs, applications, applet, app, other executable code) may be executed for causing the machine 300 to perform any one or more of the methodologies discussed herein. The machine 300 may be used as Figure 1 The computer 120 of the glasses 100. For example, the instructions 310 can cause the machine 300 to perform any one or more of the methods described herein. The instructions 310 transform a general, unprogrammed machine 300 into a specific machine 300 that is programmed to perform the functions described and shown in the described manner. The machine 300 can operate as a standalone device, or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 300 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 300 may include, but is not limited to: a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a notebook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 310 specifying actions to be taken by the machine 300. Further, while a single machine 300 is illustrated, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute instructions 310 to perform any one or more of the methodologies discussed herein.
[0056] Machine 300 may include processor 302, memory 304, and I / O components 306, which may be configured to communicate with each other via bus 344. In an example, processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 308 and processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but machine 300 may include a single processor with a single core, a single processing machine with multiple cores (eg, a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.
[0057] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via the bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. During execution of the instructions 310 by the networked system 900, the instructions 310 may also reside, in whole or in part, within the main memory 314, within the static memory 316, within the machine-readable medium 320, within the storage unit 318, within one or more of the processors 302 (e.g., within a cache memory of a processor), or within any suitable combination thereof.
[0058] I / O components 306 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is to be understood that I / O components 306 may include Figure 3306. In various examples, the I / O component 306 may include an output component 328 and an input component 332. The output component 328 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component 332 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a pointing-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides a touch position and / or force or a touch gesture, or other tactile input component), an audio input component (e.g., a microphone), etc.
[0059] In another example, the I / O component 306 may include: a biometric component 334, a motion component 336, an environmental component 338, or a positioning component 340, as well as various other components. For example, the biometric component 334 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 336 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental components 338 include, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect concentrations of hazardous gases for safety or measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals associated with the surrounding physical environment. Positioning components 340 include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), etc.
[0060] A variety of technologies may be used to achieve communication. I / O components 306 also include communication components 342 that are operable to couple networked system 900 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication components 342 may include a network interface component or another suitable device that interfaces with network 322. In other examples, communication components 342 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Components (e.g. Low power consumption), Device 324 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).
[0061] In addition, the communication component 342 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Hypercode, UCC RSS-2D barcode and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 342, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.
[0062] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 310) when executed by processor 302 cause various operations to implement the disclosed examples.
[0063] Instructions 310 may be transmitted or received over network 322 via a network interface device (e.g., a network interface component included in communications component 342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 310 may be transmitted or received to device 324 via coupling 326 (e.g., a peer-to-peer coupling) using a transmission medium.
[0064] Figure 4 4 is a process flow diagram of a gesture-based application invocation process 400 of an AR device (eg, glasses 100) according to some examples. The gesture-based application invocation process 400 can be used by an AR device to invoke an application or invoke an action of an application based on user input of one or more gestures.
[0065] In operation 402, the AR device receives a manual or verbal request to initiate a gesture scanning process. For example, the AR device detects audio received via the acoustic sensor component 338 of the AR device and determines that the received audio corresponds to a request to initiate a gesture scanning process (e.g., "start glasses"). In another example, the AR device detects movement of the user, such as movement of the user's fingers, hands, arms, or other body parts, and determines that the user's movement corresponds to a request to initiate a gesture scanning process. The AR device can determine that the audio or movement corresponds to a request to initiate a gesture scanning process by mapping the audio or gesture to an action or application in a data store on the AR device or computing device or server system.
[0066] In operation 404, the AR device begins a gesture scanning process, and in operation 406, the AR device sets a timeout timer for a specified amount of time in a timeout process, subprocess, or thread 414. The timeout timer ensures that the gesture-based application call process 400 does not hang if no gesture is detected within the specified amount of time.
[0067] In operation 408, the AR device scans the gesture and invokes an application or an action of the application associated with the detected gesture, such as referring to Figure 5 The scanning and invoking process 500 is described more fully in FIG. If the AR device does not detect a gesture within a specified amount of time, or does not select an application or action, the scanning and invoking process times out in operation 410 , and the AR device stops scanning in operation 412 .
[0068] Figure 55 is a sequence diagram of a scanning and invoking process 500 for scanning gestures and invoking applications or actions based on detected gestures used by an AR device, such as glasses 100, according to some examples. During a subprocess 514, the AR device uses one or more cameras 516 and a gesture recognition engine 520 to scan and detect gestures made by a user. In operation 502 of the subprocess 514, the AR device generates real-world scene video frame data 518 of a real-world scene from the perspective of a user of the AR device using one or more cameras 516 of the AR device. Included in the real-world scene video frame data 518 is gesture video frame data of detectable parts of the user's body, including the user's upper body and parts of the user's arms, hands, and fingers. The components of a gesture may include the movement of parts of the user's upper body, arms, and hands when the user makes a gesture; the position of the user's arms and hands in space when the user makes a gesture; and the positioning of the user holding his upper body, arms, hands, and fingers when the user makes a gesture. One or more cameras 516 transmit real-world scene video frame data 518 to gesture recognition engine 520 .
[0069] The gesture recognition engine 520 receives the real-world scene video frame data 518, and uses the real-world scene video frame data 518 in operation 504 to detect gestures based on the gesture video frame data included in the real-world scene video frame data 518. In an example, the gesture recognition engine 520 extracts feature data of the user's upper body, arms, hands, and fingers from the gesture video frame data using a feature extraction method. The gesture recognition engine 520 uses an object recognition method to identify landmarks on the user's upper body, arms, hands, and fingers based on the feature data, and generates a current skeletal model of the user's upper body, arms, hands, and fingers based on the landmarks. The gesture recognition engine 520 compares the current skeletal model with a gesture skeletal model previously generated using a machine learning method. The gesture recognition engine 520 determines the detected gesture based on the comparison of the current skeletal model with the gesture skeletal model.
[0070] In operation 506, gesture recognition engine 520 generates detected gesture data 524 based on the detected gesture. Detected gesture data 524 includes gesture identification data identifying the detected gesture.
[0071] The gesture recognition engine 520 transmits the detected gesture data 524 to the application manager 526. In operation 508, the application manager 526 selects an application or action 530 from a set of registered applications and actions, as shown in FIG. Figure 6C The selected application or selected action is selected based on an association with the detected gesture identified by the gesture identification data included in the detected gesture data 524.
[0072] In operation 510, the application manager 526 determines the call data 528 to be used by the called selected application or action 530 when launched. The call data 528 includes input parameter data of the called selected application or the selected action, such as but not limited to the real scene video frame data 518 captured during the gesture scanning subprocess 514, context information related to the operating environment of the AR device, the physical location determined from the GPS sensor, etc., the head positioning of the user wearing the AR device detected by the accelerometer or other sensor of the AR device, and the operation information such as the action type or function and any detected tags.
[0073] In operation 512, the application manager 526 invokes a selected application or a selected action selected from a set of registered applications and actions 530 known to the application manager 526. The application manager 526 invokes the selected application or action 530 and communicates invocation data 528 to the invoked selected application or action 530.
[0074] In some examples, when an application is selected, the AR device loads the selected application from the data store into a working memory and begins executing the selected application while passing the input parameter data to the called application. In another example, when the selected application 530 is called, the called application receives the input parameter data using an API hook triggered by the application manager 526.
[0075] In some examples, when an action is selected, the action may be an action of a software application, service, or function already running in another process of the AR device, and invoking the selected action includes passing a message to the already running application to perform the selected action.
[0076] In the example, refer to Fig. 8A In the embodiment of the present invention, the AR device detects a gesture 802 in which the user holds his hand with his index finger extended and the remaining fingers curled on his respective palms and the user touches the distal phalanx of each index finger together. The AR device determines that the user's intention in making the gesture is to instruct the AR device to call a battery level application. Based on the detected gesture, the AR device selects an action of the battery level application so as to display the charge level of the battery of the AR device.
[0077] In another example, refer to Figure 8BIn the embodiment of the present invention, the user holds his hand in a position indicating a rectangular frame, which indicates that the user is viewing a real-world scene on the screen of a smartphone used in camera mode. The AR device determines that the user's intention in making the gesture is to instruct the AR device to call a camera application and take a picture using one or more cameras of the AR device. Based on the detected gesture, the AR device selects an action for the camera application and records an image using one or more cameras.
[0078] In another example, refer to Figure 8C In the embodiment of the present invention, the user holds his hand as if he is holding the steering wheel of a car. The AR device determines that the user's intention in making the gesture is to instruct the AR device to call a navigation application. Based on the detected gesture, the AR device selects a navigation application and calls the navigation application.
[0079] In another example, refer to Fig.8D In the embodiment of the present invention, the user makes a fist with his hand and then bends the corresponding arm at the elbow to make it look like the user is looking at a watch. The AR device determines that the user's intention in making the gesture is to instruct the AR device to call a clock application. Based on the detected gesture, the AR device selects an action of the clock application that provides the user with the time of day and calls the action.
[0080] In another example, refer to Fig. 8E The detected gesture 810 is a gesture in which the user partially curls their fingers over the respective palms of the fingers, touches the top surfaces of the distal phalanges of the fingers together, and touches each thumb together at the distal phalanges of the thumbs so that the user's hand outlines the shape of a heart. The AR device determines that the user's intention in making the gesture is to instruct the AR device to invoke a fitness application. Based on the detected gesture 812, the AR device selects a fitness application and invokes the fitness application.
[0081] In another example, refer to Figure 8F The detected gesture 812 may be a gesture in which the user extends the fingers of each hand, rotates the hand so that the user looks at the palm, and touches the outer edge of each pinky finger and the palm together, as if the user is covering his eyes. The AR device determines that the user's intention in making the gesture is to instruct the AR device to turn off. Based on the detected gesture, the AR device selects an action of the AR device to close the application, and turns off the AR device.
[0082] In some examples, application manager 526 performs the functions of a gesture recognition engine by utilizing various APIs and system libraries to receive and process real-world scene video frame data 518 from one or more cameras 516 to detect gestures and generate detected gesture data 524 .
[0083] In some examples, the AR device provides a silent user interface to request user input in the form of a gesture. For example, the user is not prompted for user input, or the user is guided through the user interface without using the visual, auditory, and / or tactile components of the user interface provided to the user. To this end, the visual cues of the user interface are not displayed to the user, so that the user is not prompted to make user input by interacting with the visual components. As another example, the auditory cues of the user interface are not provided to the user, so that the user is not prompted to make user input by making a specific gesture. As another example, the tactile cues of the user interface are not provided to the user, so that the user is not prompted to make user input by making a specific gesture. In some such examples, when the AR device scans the gesture, the AR device provides a silent user interface during subprocess 514.
[0084] In some embodiments, the AR device provides a user interface that does not use a physical input device (e.g., a sensor glove, a keyboard, a keypad, a touch screen, a touchpad, one or more switches, etc.) to detect user input in the form of non-contact gestures. Such an interface is referred to herein as a "non-contact user interface". The non-contact user interface utilizes a non-contact imaging sensor that is not in physical contact with a part of the user's body. An example non-contact imaging sensor with temporal and spatial resolution is a camera operating in the visible spectrum or infrared spectrum, which is sufficient to capture details of a part of the user's body for producing spatial resolution. Other types of non-contact imaging sensors with sufficient temporal and spatial resolution can also be used, such as sensors that utilize ultrasonic or electromagnetic waves outside the visible and infrared spectrums.
[0085] In some examples, the AR device scans for gestures while providing a user interface that is both contactless and silent (referred to herein as a "contactless silent user interface") to the user to request a contactless silent gesture from the user. As an example, during the scanning process, the AR device does not provide any prompts to the user, whether visual, auditory, or tactile, and the AR device scans for gestures using a contactless imaging sensor. The gestures detected while providing the contactless silent user interface are referred to herein as "contactless silent gestures".
[0086] In some examples, the AR device provides a non-contact and prompted user interface (referred to herein as a "non-contact prompted user interface"). For example, the AR device provides visual, auditory, and / or tactile prompts to the user, and uses a non-contact imaging sensor to detect gestures made by the user. The gestures detected when providing the non-contact prompted user interface are referred to herein as "non-contact prompted gestures".
[0087] Fig. 6A 6 is a data structure diagram of gesture metadata in the form of a set of qualified tags 624 used by the application or action selection process 600 to determine qualified tags for gestures detected by the AR device according to some examples. Qualified tags, such as qualified tags 630a and 630b, include metadata that associates gesture identification data 626 with semantic data 628, where the gesture identification data 626 identifies the detected gesture based on physical attributes of the detected gesture, and semantic data 628 describes the detected gesture at a semantic level within the context of the application or action selection process 600. For example, the gesture identification data 626 associates the detected gesture with semantic data 628. Fig.8D The posture 808 is identified as "RIGHT_ARM_BENT_ELBOW_CLOSED_HAND", and the qualified tag 630a associates the posture identification data 626 with the semantic data 628 "WRISTWATCH", which indicates that the posture 808 is given the meaning of "watch" when detected by the AR device.
[0088] In some examples, semantic data will be associated with a single set of gesture identification data for a gesture, such as associating semantic data 632 "POWER_DOWN" with a gesture (e.g., Figure 8F The posture identification data 634 “PALMAR_LEFT_HAND_EXTENDED_FINGERS_PALMAR_RIGHT_HAND_EXTENDED_FINGERS” of the posture 812 is associated with the posture identification data 634 “PALMAR_LEFT_HAND_EXTENDED_FINGERS_PALMAR_RIGHT_HAND_EXTENDED_FINGERS”.
[0089] In another example, the use of qualified tags provides a method for associating multiple posture identification data with a single set of semantic data, thereby allowing posture synonyms. For example, the posture identification data 636 "LEFT_ARM_BENT_ELBOW_OPEN_HAND" is associated with the posture identification data 626 "RIGHT_ARM_BENT_ELBOW_HOLD_HAND" through qualified tag 630a and semantic data 628 "WATCH". By associating the two posture identification data with the same semantic data, the postures identified as "RIGHT_ARM_BENT_ELBOW_HOLD_HAND" and "LEFT_ARM_BENT_ELBOW_OPEN_HAND" are defined as posture synonyms.
[0090] Figure 6B6 is a data structure diagram of application and action metadata in the form of a set of registered applications and actions 638 according to some examples, which are used by the application or action selection process 600 to map qualified tags to applications or actions. Registered tags, such as registered tags 658a, 658b, and 658c, include metadata that associates application identification data (e.g., application identification data 640) of an application and action identification data (e.g., action identification data 642) of an action with semantic data (e.g., semantic data 644) of a gesture. For example, application identification data 640, i.e., "TIME", identifying an application that provides a timing service within an AR device, and action identification data 642, i.e., "DISPLAY_CURRENT_TIME", identifying an action of the application, are associated with semantic data 644, "watch". When the AR device detects a gesture associated with semantic data 644, "watch", the AR device calls the "time" application, and the "time" application performs the "display_current_time" action to display the current time to the user of the AR device.
[0091] In some examples, two or more applications or actions are associated with a set of semantic data. As shown, semantic data 644 and semantic data 646 have the same value, namely "watch". Semantic data 644 is associated with application identification data 640 "time" and action identification data 642 "display_current_time", while semantic data 646 is associated with application identification data 648 "calendar (CALENDAR)" and action 650 with a null value. When the AR device detects a gesture associated with semantic data 644 "watch", the AR device, as described below: calls the "time" application and the "time" application performs the "display_current_time" action; or calls the "calendar" application and provides the "calendar" application to the user of the AR device.
[0092] In some examples, an application is registered along with the application's actions. This serves as an instruction to the AR device when a gesture is detected to call the registered application and instruct the application to perform the application's identified action. For example, the registration tag 658a includes application identification data 640 identifying the application "time" and action identification data 642 identifying the action "show_current_time". The registration tag 658a also includes semantic data 644 associating a gesture with the meaning of "watch" with the application "time" and the action "show_current_time". When the AR device detects a gesture with the meaning of "watch", the AR device calls the application "time" and instructs the called application "time" to perform the operation "show_current_time".
[0093] In some examples, the application is not registered with the identification action. This serves as an instruction to the AR device when a gesture is detected to call the registered application without instructing the application to perform a specific action. For example, the registration tag 658b includes application identification data 648 identifying the application "Calendar" and action identification data 650 that is "NULL" or empty. The registration tag 658b also includes semantic data 646 that associates a gesture with the meaning of "watch" with the application "calendar". When the AR device detects a gesture with the meaning of "watch", the AR device calls the application "calendar" without instructing the application "calendar" to perform any specific operation.
[0094] In some examples, the action is not registered with the application. This serves as an instruction to the AR device when a gesture is detected to directly call or execute the action, such as by making a system-level call to a system function. For example, the registration tag 658c includes action identification data 654 that identifies the action "SYSTEM_POWER_DOWN", while the application identification data 652 is "NULL" or absent. The registration tag 658c also includes semantic data 656 that associates a gesture with the meaning of "POWER_DOWN" with the action "SYSTEM_POWER_DOWN". When the AR device detects a gesture with the meaning of "POWER_DOWN", the AR device performs a shutdown function without calling another application.
[0095] Figure 6C 6 is a process flow diagram of an application or action selection process 600 used by an AR device according to some examples. The application or action selection process 600 selects detected gesture data (e.g., Figure 5 The detected gesture data 524) is mapped to one or more applications or actions. If the detected gesture data is mapped to more than one application or action, the application or action selection process 600 provides a user interface to the user so that the user selects an application or action.
[0096] In operation 602, the AR device receives detected gesture data of a detected gesture. In operation 604, the AR device maps the detected gesture data 524 to a qualified tag in a set of qualified tags 624. In an example, the AR device matches gesture identification data included in the detected gesture data with gesture identification data included in the qualified tags.
[0097] In operation 606, the AR device selects a set of identified registered applications and actions based on registered applications or actions in the set of registered applications or actions 638 that have registration tags that match the qualified tags. In an example, the registration tags include application identification data and action identification data associated with the semantic data of the gesture. The qualified tags include the semantic data of the detected gesture. When the semantic data of the qualified tags matches the semantic data of the registration tags, the AR device determines that the qualified tags match the registration tags.
[0098] In operation 608, the AR device determines whether there is a match between the qualified tag and one or more registered tags, and therefore determines one or more applications or actions during operation 606. If there is no match, then in operation 616 the AR device issues a timeout signal.
[0099] If a match is made in operation 608, the AR device determines in operation 610 whether there is a default application or action for the qualified tag. The default application or action is a preset application or action that is designated as the default application or action for the detected posture during the configuration process. In some examples, the configuration process can be part of the application installation process when the application is installed on the AR device. In other examples, the default application or action can be specified in a configuration file located on the AR device by the user using a configuration function of the application installed on the AR device, or can be specified in a global configuration file that can specify default values for multiple postures. If the AR device determines that there is a default application or action specified for the detected posture, then in operation 620, the AR device sets the default application or action to the selected application or the selected action.
[0100] If there is no default application or action as determined in operation 610, the AR device determines whether there are multiple identified registered applications or application actions in the set of identified registered applications and actions. If there is one identified registered application or action in the set of identified registered applications and actions, in operation 620, the AR device sets one identified registered application or action in the set of identified registered applications and actions as the selected application or the selected action.
[0101] If, in operation 612, the AR device determines that there are multiple identified registered applications or application actions with tags matching the qualified tags in the set of identified registered applications and actions, then the AR device provides a prompt user interface to the user of the AR device in operation 614. The prompt user interface allows the user to select the identified registered application or action as the selected action from the list of multiple identified registered applications or application actions, such as Fig. 7A and Figure 7B More fully described in .
[0102] In operation 618, the AR device determines whether the user selects the identified registered application or action from the plurality of identified registered applications or application actions. If the AR device determines that the user has selected the identified registered application or action, the AR device sets the identified registered application or action as the selected application or the selected action in operation 620. If the AR device determines that the user has not selected the identified registered application or action, the AR device issues a timeout signal in operation 616.
[0103] In some examples, gesture data generated for a detected gesture includes a gesture identification number.
[0104] In some examples, the gesture data includes a textual gesture identifier, such as "hands_forming_heart" or "two_index_fingers_touching."
[0105] In some examples, an application may register a gesture or multiple predefined gestures. An application may implicitly register an action by defining an action exposed via a settings file in a binding to a file.
[0106] In some examples, a dynamic registration model is used to enable multiple applications or actions to be executed in parallel. In such examples, the action manager scans available applications and tracks application actions associated with the applications. Thus, the action manager can act as a global registry of application options, abstracting applications from the global scanning model, and controlling the registration of applications.
[0107] In some examples, the application manager can decide which applications are allowed to register actions (and for which gestures and / or tags), and can also act as a secure forwarding "proxy" to prevent sensitive data from being passed to third-party applications.
[0108] In some such examples, the action manager may provide a dedicated user interface to the user or extend an existing settings application to add or disable applications being invoked during a scan.
[0109] Now refer to Fig. 7A and Figure 7B , Fig. 7A Depicts an application prompt user interface process 700 and an application action selection process (eg, Figure 6C A sequence diagram of an application or action selection process 600), and Figure 7B A prompt user interface 726 is depicted according to some examples. When more than one of the registered applications and actions matches a qualifying tag and there is no defined default application or action, the AR device 728 (e.g., Figure 1The glasses 100 of the present invention utilize the prompt user interface process 700 and the prompt user interface 726 to determine the selection of an application or action by the user. The AR device 728 utilizes a hand tracking method to detect the user's direct manipulation of a virtual object included in the prompt user interface 726. In some examples, the AR device 728 utilizes the detected gesture as a user input into the prompt user interface.
[0110] In operation 708, the application manager 704 of the AR device 728 generates a prompt user interface 726 including multiple virtual objects 730, wherein the multiple virtual objects 730 constitute an interactive element with text information, and the text information identifies multiple registered applications and actions from a set of registered applications and actions, wherein the multiple registered applications and actions have corresponding registration tags that match the qualified tags of the detected posture.
[0111] The application manager 704 generates a rendering of the cue user interface 726 and transmits the rendering data 710 to the optical engine 706 of the AR device 728. In operation 712, the optical engine 706 provides the cue user interface 726 in a display to a user of the AR device 728 based on the rendering data.
[0112] In operation 714, one or more cameras 702 of the AR device 728 generate real-world scene video frame data 716 of a real-world scene 732 viewed by a user of the AR device 728. The real-world scene video frame data 716 includes hand movement video frame data indicating how the user moves one or more of their hands, hand positioning video frame data indicating what position the user holds their hands in while making the movement, and hand position video frame data indicating the position of one or more of the user's hands 734 in the real-world scene (collectively referred to as hand movement, positioning, and position video frame data) from the user's perspective while wearing the AR device 728 and viewing a rendered projection of the cue user interface 726 provided by the optical engine 706. The one or more cameras 702 transmit the real-world scene video frame data 716 to the application manager 704.
[0113] In order to provide a way for the user to interact with one or more virtual objects 730 of the prompt user interface 726, one or more user-movable colliders are provided based on movable objects in the real world scene (e.g., one or more of the user's hands). In operation 718, the application manager 704 extracts features of the user's hands from the hand movement, location, and position video frame data included in the real world scene video frame data 716 using a feature extraction method. The application manager 704 detects landmarks associated with the user's hand 734 when the user moves his hand in the real world scene using an object detection method and the features of the user's hand extracted from the hand movement, location, and position video frame data. In operation 720, the application manager 704 generates one or more landmark colliders 736 corresponding to one or more detected landmarks on the user's hand 734 in the prompt user interface 726. The user can move one or more parts of his hand to move the one or more landmark colliders 736 within the prompt user interface 726.
[0114] In operation 722, the application manager 704 detects a collision between one or more landmark colliders 736 and one or more virtual objects 730. In operation 724, the application manager 704 uses the collision to determine a user interaction of the user with the virtual object 730, and determines the user's selection of one of the plurality of identified registered applications and actions or an action as a selected application or a selected action.
[0115] In some examples, the AR device uses hand tracking methods and a user interface provided to the user, such as prompt user interface 726. The user interface includes interactive elements in the form of virtual objects (e.g., virtual object 730) with which the user can interact. Figure 5 Compared to the unprompted gesture-based user interface described in Figure 6 that scans and detects unprompted contactless gestures as user input, the hand tracking-based user interface uses interactive components in the form of virtual objects to prompt and guide the user to prompt user input. Although there are prompts, the prompting user input is still based on the contactless method as described herein.
[0116] In some examples, gesture detection methods are used instead of hand tracking methods to detect user input. For example, a user is provided with a list of options for identified registered applications or registered applications and actions in a vertical format, and the user uses a "thumbs up" gesture to move the selector virtual object up the list, a "thumbs down" gesture to move the virtual selection down the list, and an "OK" gesture to indicate that the selector is on the application or action that the user wants to select.
[0117] Fig. 9is a block diagram showing a networked system 900 including details of the glasses 100 according to some examples. The networked system 900 includes the glasses 100, a client device 926, and a server system 932. The client device 926 may be a smartphone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 936 and / or a high-speed wireless connection 934. The client device 926 is connected to the server system 932 via a network 930. The network 930 may include any combination of wired and wireless connections. The server system 932 may be one or more computing devices that are part of a service or network computing system. The client device 926 and any elements of the server system 932 and the network 930 may use Fig.11 and Figure 3 The details of the software architecture 1104 or machine 300 described respectively are implemented.
[0118] The glasses 100 include a data processor 902, a display 910, one or more cameras 908, and additional input / output elements 916. The input / output elements 916 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 902. Fig.11 and Figure 3 Examples of input / output elements 916 are further discussed. For example, input / output elements 916 may include any I / O component 306, including output component 328, motion component 336, etc. Figure 2 Examples of display 910 are described in the discussion above. In the specific examples described herein, display 910 includes displays for the left and right eyes of a user.
[0119] Data processor 902 includes image processor 906 (eg, video processor), GPU and display driver 938, tracking module 940, interface 912, low power circuitry 904, and high speed circuitry 920. Components of data processor 902 are interconnected by bus 942.
[0120] The interface 912 refers to any source of user commands provided to the data processor 902. In one or more examples, the interface 912 is a physical button that, when pressed, sends a user input signal from the interface 912 to the low-power processor 914. The low-power processor 914 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 914 can process pressing such a button within a first time period as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 912 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the camera 908. In other examples, the interface 912 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 926.
[0121] Image processor 906 includes circuitry for receiving signals from camera 908 and processing those signals from camera 908 into a format suitable for storage in memory 924 or for transmission to client device 926. In one or more examples, image processor 906 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 908, and volatile memory used by the microprocessor in operation.
[0122] The low power circuit system 904 includes a low power processor 914 and a low power wireless circuit system 918. These elements of the low power circuit system 904 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 914 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 914 can accept user input signals from the interface 912. The low power processor 914 can also be configured to receive input signals or command communications from a client device 926 via a low power wireless connection 936. The low power wireless circuit system 918 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 918. In other examples, other low power communication systems may be used.
[0123] High-speed circuit system 920 includes a high-speed processor 922, a memory 924, and a high-speed wireless circuit system 928. High-speed processor 922 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 902. High-speed processor 922 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 934 using high-speed wireless circuit system 928. In some examples, high-speed processor 922 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Fig.11 The high-speed processor 922 uses the software architecture of the data processor 902 to manage data transmission with the high-speed wireless circuit system 928, in addition to any other duties. In some examples, the high-speed wireless circuit system 928 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to as Wi-Fi herein. In other examples, the high-speed wireless circuit system 928 can implement other high-speed communication standards.
[0124] The memory 924 includes any storage device capable of storing camera data generated by the camera 908 and the image processor 906. Although the memory 924 is shown as being integrated with the high-speed circuitry 920, in other examples, the memory 924 may be a separate, independent element of the data processor 902. In some such examples, electrical wiring may provide a connection from the image processor 906 or the low-power processor 914 to the memory 924 through a chip including the high-speed processor 922. In other examples, the high-speed processor 922 may manage the addressing of the memory 924 so that the low-power processor 914 will direct the high-speed processor 922 whenever a read or write operation involving the memory 924 is desired.
[0125] The tracking module 940 estimates the pose of the glasses 100. For example, the tracking module 940 uses image data and associated inertial data from the camera 908 and the positioning component 340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene environment). The tracking module 940 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world scene environment. The tracking module 940 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 910.
[0126] The GPU and display driver 938 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 910 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 938 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real scene environment.
[0127] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 926 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 1106, such as the messaging application 1146.
[0128] Fig.10 is a block diagram illustrating an example messaging system 1000 for exchanging data (e.g., messages and associated content) over a network. The messaging system 1000 includes multiple instances of a client device 926 that host several applications including a messaging client 1002 and other applications 1004. The messaging client 1002 is communicatively coupled to other instances of the messaging client 1002 (e.g., hosted on respective other client devices 926), a messaging server system 1006, and a third-party server 1008 via a network 930 (e.g., the Internet). The messaging client 1002 may also communicate with the locally hosted application 1004 using an application program interface (API).
[0129] The messaging clients 1002 are able to communicate and exchange data with other messaging clients 1002 and with a messaging server system 1006 via the network 930. The data exchanged between the messaging clients 1002 and between the messaging clients 1002 and the messaging server system 1006 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).
[0130] The messaging server system 1006 provides server-side functionality to a particular messaging client 1002 via the network 930. Although some functions of the messaging system 1000 are described herein as being performed by the messaging client 1002 or by the messaging server system 1006, the location of some functions within the messaging client 1002 or within the messaging server system 1006 may be a design choice. For example, it may be technically preferred to initially deploy some technologies and functions within the messaging server system 1006, but then migrate the technologies and functions to the messaging client 1002 where the client device 926 has sufficient processing power.
[0131] The messaging server system 1006 supports various services and operations provided to the messaging clients 1002. Such operations include transmitting data to the messaging clients 1002, receiving data from the messaging clients 1002, and processing data generated by the messaging clients 1002. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 1000 is activated and controlled by functions available via the user interface (UI) of the messaging client 1002.
[0132] Turning now specifically to the messaging server system 1006, an application program interface (API) server 1010 is coupled to and provides a programming interface to an application server 1014. The application server 1014 is communicatively coupled to a database server 1016, which facilitates access to a database 1020 that stores data associated with messages processed by the application server 1014. Similarly, a web server 1024 is coupled to and provides a web-based interface to the application server 1014. To this end, the web server 1024 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.
[0133] The application program interface (API) server 1010 receives and transmits message data (e.g., commands and message payloads) between the client device 926 and the application server 1014. Specifically, the application program interface (API) server 1010 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 1002 to activate the functions of the application server 1014. The application program interface (API) server 1010 exposes various functions supported by the application server 1014, including: account registration; login functionality; sending messages from a particular messaging client 1002 to another messaging client 1002 via the application server 1014, sending media files (e.g., images or videos) from the messaging client 1002 to the messaging server 1012, and possible access by another messaging client 1002; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 926; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 1002).
[0134] The application server 1014 hosts several server applications and subsystems, including, for example, a messaging server 1012, an image processing server 1018, and a social network server 1022. The messaging server 1012 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 1002. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 1002. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 1012.
[0135] The application server 1014 also includes an image processing server 1018 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 1012.
[0136] The social network server 1022 supports various social networking functions and services and makes these functions and services available to the messaging server 1012. To this end, the social network server 1022 maintains and accesses an entity graph within the database 1020. Examples of functions and services supported by the social network server 1022 include identifying other users in the messaging system 1000 with whom a particular user has relationships or that the particular user is "following", and also includes identifying interests and other entities of a particular user.
[0137] The messaging client 1002 may notify the user of the client device 926 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 1002 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 1002 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.
[0138] Fig.111100 is a block diagram illustrating a software architecture 1104 that may be installed on any one or more of the devices described herein. The software architecture 1104 is supported by hardware, such as a machine 1102, including a processor 1120, a memory 1126, and an I / O component 1138. In this example, the software architecture 1104 may be conceptualized as a stack of layers, in which each layer provides specific functionality. The software architecture 1104 includes layers such as an operating system 1112, a library 1108, a framework 1110, and an application 1106. In operation, the application 1106 invokes an API call 1150 through the software stack and receives a message 1152 in response to the API call 1150.
[0139] The operating system 1112 manages hardware resources and provides public services. The operating system 1112 includes, for example, a kernel 1114, services 1116, and drivers 1122. The kernel 1114 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1114 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1116 can provide other public services to other software layers. Drivers 1122 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1122 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.
[0140] The library 1108 provides a low-level common infrastructure used by the application 1106. The library 1108 may include a system library 1118 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1108 may include an API library 1124, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting two-dimensional (2D) and three-dimensional (3D) image content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1108 may also include various other libraries 1128 to provide many other APIs to the application 1106 .
[0141] The framework 1110 provides a high-level common infrastructure used by the applications 1106. For example, the framework 1110 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1110 may provide a wide range of other APIs that may be used by the applications 1106, some of which may be specific to a particular operating system or platform.
[0142] In an example, applications 1106 may include a home application 1136, a contacts application 1130, a browser application 1132, a book reader application 1134, a location application 1142, a media application 1144, a messaging application 1146, a game application 1148, and a variety of other applications such as third-party applications 1140. Applications 1106 are programs that execute functions defined in the program. Various programming languages may be used to create one or more of the applications 1106 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1140 (e.g., those created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, the third-party application 1140 can call the API call 1150 provided by the operating system 1112 to facilitate the functions described in this article.
[0143] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.
[0144] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.
[0145] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.
[0146] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed for use with other components, and are usually part of a program that performs a specific function of a related function. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some of the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.
[0147] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.
[0148] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term "signal media".
[0149] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.
[0150] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.
[0151] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.
Claims
1. A computer-implemented method, include: detecting, by one or more processors of an augmented reality (AR) device, a gesture performed by a user of the AR device using one or more cameras of the AR device; generating, by one or more processors of the AR device, detected gesture data identifying the gesture; selecting, by one or more processors of the AR device, a selected application or action from a set of registered applications and actions based on the detected gesture data; as well as The selected application or action is called by one or more processors of the AR device so that the selected application or action is executed by the one or more processors of the AR device.
2. The computer-implemented method of claim 1, in, Selecting the selected application or action also includes: mapping the detected gesture data to a qualified label in the set of qualified labels based on the detected gesture data and the set of qualified labels; matching the qualified tags with registered tags associated with a registered application or an action in the set of registered applications and actions; and A registered application or action with a matching registration tag among the set of registered applications and actions is selected as the selected application or action.
3. The computer-implemented method of claim 1, in, Selecting the selected application or action also includes: mapping the detected gesture data to a qualified tag among one or more qualified tags, wherein a plurality of sets of registered applications and actions have registered tags that match the qualified tags; providing a prompting user interface having a display of the plurality of groups of registered applications and actions selectable by the user; and A user selection of a selected application or action from among the plurality of sets of registered applications and actions is determined.
4. The computer-implemented method of claim 1, in, Selecting the selected application or action also includes: mapping the gesture to a qualified label in the set of qualified labels based on the gesture data and the set of qualified labels; matching the qualified tag with a registered tag associated with a registered application or an action in the set of registered applications and actions; selecting an application or action from the set of registered applications and actions that has a registered tag that matches the qualifying tag; and A determination is made as to whether the selected application is a preset default application or action associated with the gesture.
5. The computer-implemented method of claim 1, in, The gesture is a silent, contactless gesture.
6. The computer-implemented method of claim 1, in, The gesture is a contactless gesture.
7. The computer-implemented method of claim 1, in, The AR device is glasses.
8. A computing device, include: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, configure the computing device to perform operations comprising: detecting, by the one or more processors using one or more cameras of the AR device, a gesture performed by a user of the AR device; generating, by the one or more processors, detected gesture data identifying the gesture; selecting, by the one or more processors, a selected application or action from a set of registered applications and actions based on the detected gesture data; and The selected application or action is called by the one or more processors so that the selected application or action is executed by the one or more processors.
9. The computing device according to claim 8, in, Selecting the selected application or action also includes: mapping the detected gesture data to a qualified label in the set of qualified labels based on the detected gesture data and the set of qualified labels; matching the qualified tags with registered tags associated with a registered application or an action in the set of registered applications and actions; and A registered application or action with a matching registration tag among the set of registered applications and actions is selected as the selected application or action.
10. The computing device according to claim 8, in, Selecting the selected application or action also includes: mapping the detected gesture data to a qualified tag among one or more qualified tags, wherein a plurality of sets of registered applications and actions have registered tags that match the qualified tags; providing a prompting user interface having a display of the plurality of groups of registered applications and actions selectable by the user; and A user selection of a selected application or action from among the plurality of sets of registered applications and actions is determined.
11. The computing device according to claim 8, in, Selecting the selected application or action also includes: mapping the gesture to a qualified label in the set of qualified labels based on the gesture data and the set of qualified labels; matching the qualified tag with a registered tag associated with a registered application or an action in the set of registered applications and actions; selecting an application or action from the set of registered applications and actions that has a registered tag that matches the qualifying tag; and A determination is made as to whether the selected application is a preset default application or action associated with the gesture.
12. The computing device according to claim 8, in, The gesture is a silent, contactless gesture.
13. The computing device according to claim 8, in, The gesture is a contactless gesture.
14. The computing device according to claim 8, in, The AR device is glasses.
15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations, the operations include: detecting, by one or more processors of an AR device, a gesture performed by a user of the AR device using one or more cameras of the AR device; generating, by one or more processors of the AR device, detected gesture data identifying the gesture; selecting, by one or more processors of the AR device, a selected application or action from a set of registered applications and actions based on the detected gesture data; as well as The selected application or action is called by one or more processors of the AR device so that the selected application or action is executed by the one or more processors of the AR device.
16. The non-transitory computer readable storage medium of claim 15, in, Selecting the selected application or action also includes: mapping the detected gesture data to a qualified label in the set of qualified labels based on the detected gesture data and the set of qualified labels; matching the qualified tags with registered tags associated with a registered application or an action in the set of registered applications and actions; and A registered application or action with a matching registration tag among the set of registered applications and actions is selected as the selected application or action.
17. The non-transitory computer readable storage medium of claim 15, in, Selecting the selected application or action also includes: mapping the detected gesture data to a qualified tag among one or more qualified tags, wherein a plurality of sets of registered applications and actions have registered tags that match the qualified tags; determining a plurality of sets of registered applications and actions having registration tags matching the qualifying tags; providing a prompting user interface having a display of the plurality of groups of registered applications and actions selectable by the user; and A user selection of a selected application or action from among the plurality of sets of registered applications and actions is determined.
18. The non-transitory computer readable storage medium of claim 15, in, Selecting the selected application or action also includes: mapping the gesture to a qualified label in the set of qualified labels based on the gesture data and the set of qualified labels; matching the qualified tag with a registered tag associated with a registered application or an action in the set of registered applications and actions; selecting an application or action from the set of registered applications and actions that has a registered tag that matches the qualifying tag; and A determination is made as to whether the selected application is a preset default application or action associated with the gesture.
19. The non-transitory computer readable storage medium of claim 15, in, The gesture is a silent, contactless gesture.
20. The non-transitory computer readable storage medium of claim 15, in, The gesture is a contactless gesture.
21. The non-transitory computer readable storage medium of claim 15, in, The AR device is glasses.
