Multi-modal ui with semantic events
By tracking the input pipeline, posture input and voice input modes by hand, combined with the hand item system and bone model inference, the problem of user input mode limitation in the AR system is solved, and more efficient and intuitive user interface interaction is achieved Experience.
Patent Information
- Application Number
- CN202380036076.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-28
- Filing Date
- 2023-04-26
- Publication Date
- 2024-12-03
AI Technical Summary
Existing augmented reality (AR) systems have limitations in user input modes, especially in the absence of physical input devices, which makes it difficult for users to effectively indicate intentions and call actions, resulting in reduced experience and increased resource consumption.
Using hand tracking input pipeline and posture input mode, it provides multiple input modes to improve the user interface by detecting the user's hand movement and body posture, combining voice input, including hand part item system and bone model inference, to achieve Accurate identification and processing of user intentions.
It improves the user interface interactivity and experience quality of the AR system, reduces resource consumption, provides a more intuitive and efficient input method, and enhances the user's interaction ability with virtual objects.
Smart Images

Figure CN119072675A_ABST
Abstract
Description
[0001] Priority claim
[0002] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 334,841, filed on April 26, 2022, and U.S. Provisional Application Serial No. 63 / 363,732, filed on April 28, 2022, each of which is incorporated herein by reference in its entirety. Technical Field
[0003] The present disclosure relates generally to user interfaces, and more particularly to user interfaces used in augmented reality and virtual reality. Background Art
[0004] A head-mounted device can be implemented with a transparent or translucent display through which the user of the head-mounted device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or translucent display, and can also see objects (e.g., virtual objects such as renderings of 2D or 3D graphic models, images, videos, text, etc.) generated for display as part of the surrounding environment and / or superimposed on the surrounding environment. This is generally referred to as "augmented reality" or "AR". The head-mounted device can also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is generally referred to as "virtual reality" or "VR". As used herein, unless the context indicates otherwise, the term "AR" refers to either or both of augmented reality and virtual reality as traditionally understood.
[0005] A user of a head mounted device can access and use computer software applications to perform various tasks or participate in recreational activities. Performing tasks or participating in recreational activities may include entering text. To enter text, the user interacts with a text input user interface provided by the head mounted device. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.
[0007] Figure 1 is a perspective view of a head mounted device according to some examples.
[0008] Figure 2 According to some examples Figure 1 Additional views of the head-mounted device.
[0009] Figure 3is a diagrammatic representation of a machine in the form of a computing device according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.
[0010] Figure 4A is a collaboration diagram of a hand tracking input pipeline for an AR system (such as glasses) according to some examples.
[0011] Figure 4B is a diagram of a data structure according to some examples.
[0012] Figure 4C is an illustration of another data structure according to some examples.
[0013] Figure 5A is a sequence diagram of a sign language text input process for an AR system according to some examples.
[0014] Figure 5B is an illustration of a start / stop text input gesture according to some examples.
[0015] Figure 5C is an illustration of sign language symbols based on some examples.
[0016] Figure 5D is an illustration of an AR overlay according to some examples.
[0017] Fig. 6A is a deployment diagram of a voice user interface pipeline based on some examples.
[0018] Figure 6B is a diagram of a data structure according to some examples.
[0019] Figure 7 is a sequence diagram of a voice user interface pipeline process according to some examples.
[0020] Figure 8 is a sequence diagram of a voice input processing method according to some examples.
[0021] Fig. 9 is a block diagram illustrating a software architecture within which the present disclosure may be implemented according to some examples.
[0022] Fig.10 is a block diagram showing a networked system including details of a head-mounted AR system according to some examples.
[0023] Fig.11 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network according to some examples. DETAILED DESCRIPTION
[0024] AR systems are limited when it comes to user input modalities. Indicating user intent and invoking actions or applications is more complex for users of AR systems when compared to other mobile devices such as mobile phones. When using a mobile phone, a user can go to the home screen and tap on a specific icon to launch an application. However, due to the lack of physical input devices such as a touch screen or keyboard, such interactions are not easily performed on AR systems. Typically, users can indicate their intent by pressing a limited number of hardware buttons or using a small touchpad. However, such operations often reduce the overall enjoyment and experience, and may involve navigating away from the current interface, which consumes a lot of resources and time.
[0025] In some examples, an input modality that can be used with an AR system is hand tracking combined with direct manipulation of virtual objects (DMVO), where a user is provided with a user interface displayed to the user in an AR overlay with a 2D or 3D rendering. The rendering is a graphical model in 2D or 3D, where virtual objects located in the model correspond to interactive elements of the user interface. In this way, the user perceives the virtual objects as objects within the overlay in the user's field of view of the real world scene when wearing the AR system, or perceives the virtual objects as objects within the virtual world as seen by the user when wearing the AR system. In order to allow the user to manipulate the virtual objects, the AR system detects the user's hands and tracks the movement, position, and / or location of the hands to determine the user's interaction with the virtual objects.
[0026] In another example, gestures made by the user that do not involve the DMVO method provide another input modality suitable for use with the AR system. The gesture input is made by the user who moves and positions parts of the user's body, and those parts of the user's body can be detected by the AR system when the user is wearing the AR system. The detectable parts of the user's body may include parts of the user's upper body, arms, hands, and fingers. The components of the gesture input may include the movement of the user's arms and hands, the position of the user's arms and hands in space, and the user maintaining the positioning of his upper body, arms, hands, and fingers. Gesture input is useful in providing an AR experience to the user because gesture input provides a way to provide user input to the AR system during the AR experience without allowing the user to divert his attention from the AR experience. As an example, in an AR experience that is an operating manual for a machine, the user can simultaneously view a machine in a real-world scene through the lens of the AR system, view the AR overlay on the real-world scene view of the machine, and provide user input to the AR system.
[0027] In further examples, audio input or voice input provides another input modality. Voice commands spoken by the user during the AR experience are considered user input into the voice user interface.
[0028] In some examples, an improved user interface is provided to users of an AR system by combining voice, DMVO, and gesture input modalities.
[0029] In some examples, an application programming interface (API) is provided whereby input events are readily consumed by AR applications as semantic events, meaning that the AR application can simply register the events it wants to listen to and transparently consume the events accordingly.
[0030] In some examples, audio processing can be performed in multiple components of the AR system (e.g., framework layer, application layer) and in the cloud via a cloud-based speech recognition and natural language processing (NLP) backend.
[0031] In some examples, the AR system provides a framework for modality agnosticism through a universal input system, where hand-based input is one of several choices for input modality.
[0032] In some examples, the AR system provides an organizational scheme for input recognition, processing, and dispatching at the operating system level as a whole, and leaves the specific implementation of device input to device and interaction designers.
[0033] In some examples, AR systems provide general and standardized semantic events, with application options using “escape hatches” to consume raw, modality-specific input data when appropriate.
[0034] In some examples, the AR system provides a hand tracking input pipeline function as a client of a generic camera service component to continuously process a buffer of frame data and infer a skeletal model that is continuously updated with the frame data. The skeletal model (published as a subnet IP address (Snips) of an automatically synchronized shared memory buffer) is then consumed by components that interpret the input at increasingly higher levels of abstraction. The hand tracking input pipeline terminates with a module that recognizes generic gestures and text editing operations in a hand classification item stream to maintain sufficient history internally to integrate short histories into recognized gestures of the corresponding type. At the moment when the recognizer module recognizes some types of gestures, the recognizer module notifies the appropriate framework level service (e.g., system framework component) via a system on chip (SoC) Snips call. This portion of the hand tracking input pipeline is independent of the AR application component model of the operating system.
[0035] In some examples, the AR system provides a voice input pipeline that continuously processes audio data to detect voice commands. The voice input pipeline generates semantic event data including voice commands and transmits the voice commands to the AR application component.
[0036] In some examples, the AR system provides a system framework component that receives high-level (i.e., symbols and gestures) but untargeted (not related to a specific AR application) input events along with metadata and figures out the process that expects the event. After determining the target of the gesture input event, the system framework component forwards the event as a Snips method call to the process that is intended to receive the event. Security is provided by the indirection through the system framework component. If the OS runs multiple untrusted applications, the system framework component will route the input according to its intended recipient.
[0037] In some examples, the AR system provides a skeletal model inference component that publishes skeletal samples as Snips shared memory buffers to allow consumers to read each processed skeletal update with the lowest possible latency. AR applications choose to consume raw hand skeletal samples rather than "cooked" gesture input events read from the same hand skeletal shared memory buffer, which again minimizes latency and facilitates operations that want to closely correlate graphics and hand movement observation.
[0038] In some examples, the AR system provides a system framework component that knows which AR applications are visible or otherwise available for input in some way, and which knows which specific gestures consisting of which specific hand classification items these AR applications expect to receive. The system framework component uses high-level user context information from the operating system framework to guide the low-level inference system.
[0039] In some examples, the AR system provides multiple feedback channels so that the AR system as a whole has more information about the user's likely behavior than any single component alone, to estimate whether any particular interpretation of user input is correct by using multiple components of the AR system to allow reading of user signals that are closer to the "noise floor."
[0040] In some examples, the AR system provides facilities to support direct manipulation of virtual objects (DMVO), in which a user interacts with a virtual object using his or her hands as if the virtual object were a physical object. To this end, the AR system provides a continuously updated hand skeletal model that includes hand configuration information (e.g., joint angles) and a continuously updated transformation from the skeletal model coordinate system to the device user coordinate system. The continuously updated hand skeletal model is the output of one of the early stages of the hand input data processing pipeline, which is delivered to the AR application without a higher-level semantic interpretation.
[0041] In some examples, the AR system provides directional gestures that involve the user issuing commands at a target that indicate a gesture consisting of hand classification terms. The target can be a specific virtual object or the system as a whole, and the target describes the gesture it wants to recognize.
[0042] In some examples, the AR system provides for using hand classification terms as gesture building blocks to allow application developers to specify gestures based on combinations of hand classification terms. The AR system processes the hand classification terms extracted from the hand tracking data stream in a layer different from assembling hand movements into gestures, and the AR application can utilize the newly created symbols built from existing gestures without retraining the machine learning component of the hand tracking input system.
[0043] In some examples, the AR system provides specific gestures that become standardized across AR applications and scenarios. In some examples, the OS as a whole may support a universal "back" gesture that activates a system-level user interface for switching between applications in some manner. Thus, the hand classification term system provided by the AR system allows the interaction language to be built from reusable modular components that free developers from the need to retrain machine learning models to provide new commands or actions.
[0044] In some examples, an AR system provides custom gestures using hand classification items to allow developers of AR applications to create gestures for specific tasks without modifying or retraining the hand tracking input pipeline itself. In some examples, a database exploration application describes a gesture language specifically for expressing some classification of data transformation operations to allow users to "move" through data by simply forming certain hand shapes, rather than selecting menu items or clicking buttons as users would in the mobile or desktop world. In some examples, the relationship between hand classification items and gestures is the same as the relationship between letters and words, and the hand classification item system allows for the recognition of novel gestures composed of already known hand classification items. In other examples, a large number of novel gestures are composed from a fixed set of classification items.
[0045] In some examples, the AR system provides recognition of directional gesture input in both "on-screen" and "off-screen" scenarios. Gestures can be made off-screen, with the hand in a comfortable and unobtrusive positioning.
[0046] In some examples, the AR system allows a user to input non-trivial amounts of text in a sustainable, reliable manner via a platform-level API provided in the operating system.
[0047] In some examples, the AR system generates input event data using input data captured by an input device of the AR system. The AR system uses the input event data and component registration data of one or more AR application components of the AR system to determine a target AR application component of the AR system, and transmits the input data to the target AR application component.
[0048] In some examples, the input event data includes gesture input data of a gesture being made by a user of the AR system.
[0049] In some examples, the input event data includes symbol input data of a sign language symbol being made by a user of the AR system.
[0050] In some examples, input event data is generated by a hand tracking input pipeline of the AR system.
[0051] In some examples, the input event data includes voice commands being issued by a user of the AR system.
[0052] In some examples, input event data is generated by a speech input pipeline of the AR system.
[0053] In some examples, the speech input pipeline includes a speech machine learning (ML) service.
[0054] Figure 1 According to some examples, a head mounted device (e.g., Figure 1 100). The glasses 100 may include a frame 102 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first or left optical element holder 104 (e.g., a display or lens holder) and a second or right optical element holder 106 connected by a bridge 112. A first or left optical element 108 and a second or right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.
[0055] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or unitary construction.
[0056] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 1002 discussed below.
[0057] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.
[0058] The glasses 100 include one or more cameras, such as a first or left camera 114 and a second or right camera 116. The one or more cameras can operate as visual input devices to capture images of real-world scenes and images of the user's hands as the user makes gestures. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, in addition to the left camera 114 and the right camera 116, the glasses 100 also include any number of input sensors or other input / output devices. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, and the like.
[0059] In some examples, left camera 114 and right camera 116 provide video frame data for use by glasses 100 to extract 3D information from a real scene.
[0060] The glasses 100 may also include a touchpad 126 mounted to or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally arranged vertically, with the touchpad 126 being approximately parallel to the temple of the user in some examples. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. Additional user input may be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide a device by which the glasses 100 can receive input from the user of the glasses 100.
[0061] In some examples, the glasses 100 also include one or more audio speakers, such as audio speakers 130 and 132. The one or more audio speakers may include, but are not limited to, earphones, earbuds, headphones, in-ear monitors, in-ear earbuds, wireless earbuds, over-ear headphones, over-ear headphones, etc. The audio speakers are operable to provide audio signals to the user, such as speech or voice prompts or messages.
[0062] In some examples, glasses 100 also include one or more audio input devices, such as one or more microphones as illustrated by microphone 134 and microphone 136. The one or more audio input devices are operable to capture speech or voice commands of a user.
[0063] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured in left and right optical element holders 104, 106, respectively.
[0064] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .
[0065] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 superimposed with a view of the real-world scene seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 superimposed with a view of the real-world scene seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides an optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's view of the real-world scene, including displaying a user interface to the user of the glasses 100.
[0066] However, it should be understood that other display technologies or configurations may be utilized within the optical engine to display images to a user in the user's field of view. For example, instead of providing a projector 204 and a waveguide, an LCD, LED or other display panel or surface may be provided.
[0067] In use, the user of the glasses 100 will be presented with information, content, and various user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Fig.10 Interact with the glasses 100 through voice input or touch input on a client device 1026 shown in FIG. 1 and / or hand movements, position, and orientation detected by the glasses 100 .
[0068] Figure 3 is a diagrammatic representation of a machine 300 (e.g., a computing device) in which instructions 310 (e.g., software, programs, applications, applet, app, or other executable code) may be executed that cause the machine 300 to perform any one or more of the methodologies discussed herein. The machine 300 may be used as Figure 1The computer 120 of the glasses 100. For example, the instructions 310 can cause the machine 300 to perform any one or more of the methodologies described herein. The instructions 310 transform a general purpose, unprogrammed machine 300 into a specific machine 300 that is programmed to perform the functions described and illustrated in the manner described. The machine 300 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 300 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, the instructions 310 specifying actions to be taken by the machine 300. In addition, while a single machine 300 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute the instructions 310 to perform any one or more of the methodologies discussed herein.
[0069] Machine 300 may include processor 302, memory 304, and I / O component 306, which may be configured to communicate with each other via bus 344. In some examples, processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 308 and processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but machine 300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.
[0070] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via the bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. During execution of the instructions 310 by the machine 300, the instructions 310 may also reside, in whole or in part, within the main memory 314, within the static memory 316, within the machine-readable medium 320, within the storage unit 318, within one or more of the processors 302 (e.g., within a processor's cache memory), or within any suitable combination thereof.
[0071] I / O components 306 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is to be understood that I / O components 306 may include Figure 3 306. In various examples, the I / O components 306 may include output components 328 and input components 332. The output components 328 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 332 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.
[0072] In another example, the I / O component 306 may include: a biometric component 334, a motion component 336, an environmental component 338, or a positioning component 340, as well as various other components. For example, the biometric component 334 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 336 may include an inertial measurement unit (IMU), an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 338 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals associated with the surrounding physical environment. Positioning component 340 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can provide altitude), an orientation sensor component (e.g., a magnetometer), etc.
[0073] A variety of technologies may be used to implement communications. I / O components 306 also include communication components 342 that are operable to couple machine 300 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication components 342 may include a network interface component or another suitable device that interfaces with network 322. In other examples, communication components 342 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Components (e.g. Low power consumption), Device 324 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).
[0074] In addition, the communication component 342 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Hypercode, UCC RSS-2D barcode and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 342, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.
[0075] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 310) when executed by processor 302 cause various operations to implement the disclosed examples.
[0076] Instructions 310 may be transmitted or received via a network interface device (e.g., a network interface component included in communication component 342) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 322. Similarly, instructions 310 may be transmitted or received to device 324 via coupling 326 (e.g., a peer-to-peer coupling) using a transmission medium.
[0077] Figure 4A is a collaboration diagram of a hand tracking input pipeline 440 of an AR system (e.g., glasses 100) according to some examples, and Figure 4B and Figure 4C is a diagram of a data structure. The camera service component 402 uses one or more cameras (eg, Figure 1The camera devices 114 and 116 of the AR system generate real-world scene environment frame data 428 of the real-world scene environment from the perspective of the user of the AR system. One or more cameras are used as visual input devices in the hand tracking input pipeline 440. The camera service component 402 transmits the real-world scene environment frame data 428 to the skeleton model inference component 404. The real-world scene environment frame data 428 includes tracking video frame data of detectable parts of the user's body (including parts of the user's upper body, arms, hands, and fingers). The tracking video frame data includes: video frame data of the movement of parts of the user's upper body, arms, hands, and fingers when the user makes gestures or moves his hands and fingers to interact with the real-world scene environment; video frame data of the position of the user's arms and hands in space when the user makes gestures or moves his hands and fingers to interact with the real-world scene environment; and video frame data of the user maintaining the positioning of his upper body, arms, hands, and fingers when the user makes gestures or moves his hands and fingers to interact with the real-world scene environment.
[0078] The skeleton model inference component 404 receives the real world scene environment frame data 428 and generates the skeleton model data 434 based on the real world scene environment frame data 428. The skeleton model data 434 includes data representing the geometric skeleton model of one or more parts of the user's upper body, arms, hands and fingers. In some examples, the skeleton model inference component 404 extracts the features of the user's upper body, arms and hands from the tracking video frame data included in the real world scene environment frame data 428. The extracted features may include landmark data, which includes landmark identification, location in the real world scene environment, and classification information of one or more landmarks associated with the user's upper body, arms and hands. The skeleton model inference component 404 generates the skeleton model data 434 based on the extracted features using a geometric method and one or more previously generated skeleton models. In another example, the skeleton model inference component 404 generates the skeleton model data 434 based on the classification of the real world scene environment frame data 428 using an artificial intelligence method and one or more skeleton models previously generated using a machine learning method.
[0079] The skeletal model inference component 404 transmits the skeletal model data 434 to the hand classification term inference component 406. In addition, the skeletal model inference component 404 makes the skeletal model data 434 available to applications executing on the AR system (eg, the AR DMVO application component 422).
[0080] The camera service component 402 transmits the real-world scene environment frame data 428 to the rough hand positioning inference component 412. The rough hand positioning inference component 412 generates coordinate transformation data 432 based on the real-world scene environment frame data 428. The coordinate transformation data 432 includes a continuously updated transformation of the coordinate system of the skeletal model from the skeletal model data 434 and the coordinate system of the user coordinate system of the AR system. In the example, the object rough hand positioning inference component 412 receives the real-world scene video real-world scene environment frame data 428 of the real-world scene environment, and extracts features of objects (including the user's upper body, arms and hands) in the real-world scene environment from the real-world scene environment video frame data. The object rough hand positioning inference component 412 generates coordinate transformation data 432 based on the extracted features. The rough hand positioning inference component 412 transmits the coordinate transformation data 432 to the AR DMVO application component 422.
[0081] The hand classification term inference component 406 receives the skeleton model data 434 from the skeleton model inference component 404, and generates the hand classification term probability data 430 based on the skeleton model data 438. In some examples, the posture is specified by the hand tracking input pipeline 440 according to the combination of hand classification terms. The hand classification terms are in turn composed of the combination and relationship of the features of the skeleton model included in the skeleton model data 434. In some examples, the hand classification term inference component 406 compares one or more skeleton models included in the skeleton model data 434 with the previously generated hand classification term hand classification term models, and generates one or more hand classification term probabilities based on the comparison. The one or more hand classification term probabilities indicate the probability that the specified hand classification term can be identified according to the skeleton model data 434. The hand classification term inference component 406 generates the hand classification term probability data 430 based on the one or more hand classification term probabilities. In another example, the hand classification term inference component 406 determines one or more hand classification term probabilities based on classifying the skeleton model using an artificial intelligence method and a hand classification term model previously generated using a machine learning method.
[0082] In some examples, the hand tracking input pipeline 440 determines hand classification terms from the skeletal model data 434 in a layer different from determining gestures from hand movements made by a user of the AR system. Thus, a designer of an AR application component executed by the AR system can create new gestures that are constructed from existing hand classification terms composed of already known gestures without having to retrain the machine learning component of the hand tracking input pipeline 440, or generate an implementation-specific hand classification term model. In some examples, the hand classification term model used to generate the hand classification term probability data 430 is not implementation-specific. That is, the hand classification term model is not trained to recognize only specific hand classification terms for the purposes of a particular implementation. Instead, the hand classification term model is trained to recognize as many hand classification terms as possible, without regard to what those hand classification terms mean in the context of a particular implementation (e.g., a task-specific AR application component).
[0083] The hand classification term inference component 406 transmits the hand classification term probability data 430 to the posture inference component 408 and the posture text input recognition component 410. The posture inference component 408 receives the hand classification term probability data 430 and generates semantic event data in the form of posture input data 426 based on the hand classification term probability data 430. In the example, the posture inference component 408 compares the hand classification term identified in the hand classification term probability data 430 with the posture identification data identifying the specific posture. The posture identification consists of one or more hand classification terms corresponding to the specific posture. The posture identification is defined using a grammar whose symbols correspond to the hand classification terms. For example, the posture identifier of a posture is "LEFT_PALMAR_FINGERS EXTENDED_RIGHT PALMAR_FINGERS_EXTENDED", where: "LEFT" is a symbol corresponding to the hand classification item indicating that the user's left hand has been detected; "PALMAR" is a symbol corresponding to the hand classification item indicating that the palm of the user's hand has been detected and modifies "LEFT" to indicate that the user's left palm has been detected; "FINGERS" is a symbol corresponding to the hand classification item indicating that the user's fingers have been detected; and "EXTENDED" is a symbol corresponding to the hand classification item indicating that the user's fingers are extended and modifies "FINGERS". In another example, the posture identifier is a single tag such as a number that identifies the posture based on the component hand classification item of the posture. The posture identifier identifies the posture in the context of a physical description of the posture. The posture inference component 408 transmits 426 to the system framework component 416.
[0084] The gesture inference component 408 transmits the gesture input data 426 to the system framework component 416. The system framework component 416 receives the gesture input data 426 and generates directional input event data 438 based on the gesture input data 426, etc. The input event of the directional input event data 438 can be one of a plurality of categories. Undirectional input events belonging to the undirectional category are routed to operating system level components such as the system user interface component 418. Directional input events belonging to the directional category are routed to specific components such as the AR gesture application component 420. In some examples, the system framework component 416 processes all incoming input events, such as, but not limited to, voice commands received from a user, keyboard or button presses received from a user, etc.
[0085] The gesture text input recognition component 410 receives the hand classification term probability data 430 and generates semantic event data in the form of symbol input data 414 based on the hand classification term probability data 430. In an example, the gesture inference component 408 compares the hand classification term identified in the hand classification term probability data 430 with symbol data identifying specific characters, words, and commands. For example, the symbol data of the gesture is the character "V" of the gesture as a sign language symbol in American Sign Language (ASL). A single hand classification item for a gesture may be "LEFT" for the left hand, "PALMAR" for the palm of the left hand, "INDEXFINGER" for the index finger, "EXTENDED" for the modified "INDEXFINGER", "MIDDLEFINGER" for the middle finger, "EXTENTED" for the modified middle finger, "RINGFINGER" for the ring finger, "CURLED" for the modified "RINGFINGER", "LITTLEFINGER" for the little finger, "CURLED" for the modified "LITTLEFINGER", "THUMB" for the thumb, and "CURLED" for the modified "THUMB".
[0086] In some examples, the gesture text input recognition component 410 may also identify the entire word based on the hand classification term indicated by the hand classification term probability data 430. In another example, the gesture text input recognition component 410 may identify a command, such as a command corresponding to a specified set of keystrokes in an input system having a keyboard, based on the hand classification term indicated by the hand classification term probability data 430.
[0087] The gesture inference component 408 and the gesture text input recognition component 410 transmit gesture input data 426 and symbol input data 414, respectively, to the system framework component 416. The system framework component 416 receives the gesture input data 426 and the symbol input data 414 (collectively and individually referred to as "input event data") and generates undirected input event data 436 or directional input event data 438 based in part on the input event data. Undirected input events belonging to the undirected category of input events are routed to operating system level components such as the system user interface component 418. Directional input events belonging to the directional category of input events are routed to target components such as the AR gesture application component 420.
[0088] In an example of processing input data that can be classified as undirected input event data 436 received from gesture inference component 408 and gesture text input recognition component 410, system framework component 416 classifies the input data as undirected input event data 436 based on the input data and component registration data described below. System framework component 416 routes the input data as undirected input event data 436 to system user interface component 418 based on classifying the input data as undirected input event data 436.
[0089] The system user interface component 418 receives the undirected input event data 436 and determines the target component based on the user's indication or selection of the virtual object associated with the target component when making a gesture corresponding to the undirected input event data 436. In an example, the system user interface component 418 determines the position of the user's hand in the real-world scene environment when making the gesture. The system user interface component 418 determines a set of virtual objects currently provided to the user by the AR system in the AR experience. The system user interface component 418 determines a virtual object whose apparent position in the real-world scene environment is related to the position of the user's hand in the real-world scene environment when making the gesture. The system user interface component 418 determines the target AR application component based on finding the AR application component associated with the virtual object in the internal data structure of the AR system, and determines the AR application component as the target AR application component.
[0090] The system user interface component 418 registers with the system framework component 416 the target AR application component to which the directional input event data 438 is to be routed. The system framework component 416 registers the component registration data, such as Figure 4BThe component registration data 450 is stored in a data store to be accessed during operation of the system framework component 416. The component registration data 450 includes a component ID field 442 identifying a target AR application component, a registered language field 448 identifying a language model to be associated with the target AR application component, and one or more registered gesture fields 444 and / or registered symbol fields 446 indicating gestures and symbols to be routed to the registered AR application component. As shown, the component ID field 442 includes the AR application component identifier "TEXTENTRY"; the registered language field 448 identifies the language associated with the registered AR application component, namely "ENGLISH"; the registered posture field 444 includes the posture identifier that should be routed to the registered target AR application component, namely "LEFT_PALMAR_FINGERS EXTENDED_RIGHT_PALMAR_FINGERS_EXTENDED", and the registered symbol field 446 identifies a set of symbols that should be routed to the registered AR application component (i.e., "[*]" representing all symbols).
[0091] As another example of component registration data, Figure 4C The component registration data 452 includes: a component ID field 454, which includes the AR application component identifier "EMAIL"; a registered language field 458, which identifies the language associated with the registered AR application component, namely "ASL", and a registered symbol field 456, which identifies a set of symbols (i.e., the word "EMAIL") that should be routed to the registered AR application component.
[0092] In some examples, an API is provided by the AR system so that the AR application component can register itself with the system framework component 416, such as but not limited to registering itself when the AR application component is installed.
[0093] Referring again to the system framework component 416 processing input data that can be classified as non-directional input event data 436 received from the gesture inference component 408 and the gesture text input recognition component 410, the system framework component 416 classifies the input data received from the gesture inference component 408 and the gesture text input recognition component 410 as non-directional input event data 436 or directional input event data 438 based on the input data and the component registration data. In an example, when processing the symbol input data 414, the system framework component 416 searches the registered symbol field of the component registration data, such as the registered symbol field 446 of the component registration data 450, for a registered symbol that matches the symbol input data. When the system framework component 416 determines a match, the system framework component 416 determines that the symbol input data is the directional input event data 438. The system framework component 416 also determines the target AR application component based on the target AR application component identified in the component ID field (e.g., component ID field 442) of the component registration data that includes the matching registration symbol. In a similar manner, when processing gesture input data 426, system framework component 416 searches the registered gesture field of component registration data, such as registered gesture field 444 of component registration data 450, for a registered gesture that matches the gesture input data. When system framework component 416 determines a match, system framework component 416 determines that the gesture input data is directional input event data 438, and also determines a target AR application component to which directional input event data 438 is to be routed. In the event that system framework component 416 determines that symbolic input data and / or gesture input data of the input data is not found in the component registration data, 416 determines that the input data is to be classified as undirected input event data 436 and is to be routed to system user interface component 418.
[0094] In another example of processing directional input event data 438, an AR application component (e.g., AR gesture application component 420) registers itself with system framework component 416. To do so, the AR application component registers component registration data (e.g., Figure 4B The component registration data 450 is transmitted to the system framework component 416. The system framework component 416 receives the component registration data and stores the component registration data in a data store for use in routing the directional input event data 438 to the AR application component.
[0095] In another example of processing directional input event data 438, the AR system determines based on hinting that the directional input event data 438 is to be routed to the AR application component. For example, if the AR system executes the current AR application component in a single application mode state, the current AR application component is hinted as the AR application component to which the directional input event data 438 should be routed.
[0096] In some examples, an API is provided by the AR system so that the AR application component can access gesture input data 426 and symbol input data 414 from the system framework component 416 .
[0097] In some examples, the system framework component 416 transmits the language model feedback data 424 to the hand classification term inference component 406 and the gesture inference component 408 to improve the accuracy of the inferences made by the hand classification term inference component 406 and the gesture inference component 408. In an example, the system framework component 416 generates the language model feedback data 424 based on the user context data (e.g., component registration data that registers the AR application component and data about the hand classification terms that constitute the registered gestures and the hand classification terms that constitute the gestures associated with the registered symbols). The component registration data includes information about the gestures and symbols in the gesture input data 426 and the symbol input data 414 that are routed to the AR application component as part of the directional input event data 438, and the language of the symbols. In addition, the system framework component 416 includes information about the composition of a particular gesture including the hand classification terms associated with the gestures and symbols.
[0098] In another example of processing the language model feedback data 424, the system framework component 416 transmits the prompt as part of the language model feedback data 424 to the hand classification term inference component 406, the gesture inference component 408, and the gesture text input recognition component 410. The system framework component 416 generates the prompt based on the language model associated with the AR application component, for example, by the language specified in the registered language field 458 in the component registration data 452. The system framework component 416 determines a possible next symbol N based on the previous characters N-1, N-2, etc. and the language model. In some examples, the system framework component 416 generates the prompt based on the language model, which is a hidden Markov model that predicts what the next symbol N should be based on one or more of the previous characters N-1, N-2, etc. In another example, the system framework component 416 generates the next symbol N using an AI method based on a language model generated using a machine learning method. In an example, the system framework component 416 determines the next gesture associated with the next symbol N by mapping the next symbol N to the next gesture based on a lookup table that associates symbols with gestures. The system framework component 416 decomposes the next gesture into a set of one or more next hand classification terms. The system framework component 416 transmits the next gesture as part of the language model feedback data 424 to the gesture inference component 408, and transmits the set of next hand classification terms as part of the language model feedback data 424 to the hand classification term inference component 406.
[0099] AR application components executed by the AR system, such as the AR DMVO application component 422, the system user interface component 418, and the AR gesture application component 420, are consumers of data (e.g., coordinate transformation data 432, skeletal model data 434, gesture input data 426, and symbolic input data 414) generated by the hand tracking input pipeline 440. The AR system executes the AR DMVO application component 422 to provide a user interface to a user of the AR system using direct manipulation of visual objects within a 2D or 3D user interface. The AR system executes the system user interface component 418 to provide a system-level user interface such as a command console, etc. to a user of the AR system using gestures as an input modality. The AR system executes the AR gesture application component 420 to provide a user interface such as an AR experience to a user of the AR system using gestures as an input modality.
[0100] In some examples, the camera service component 402, the skeletal model inference component 404, and the coarse hand position inference component 412 communicate using an automatically synchronized shared memory buffer. In addition, the skeletal model inference component 404 and the coarse hand position inference component 412 publish the skeletal model data 434 and the coordinate transformation data 432, respectively, on a memory buffer that can be accessed by components and applications (e.g., AR DMVO application component 422) outside the hand tracking input pipeline 440.
[0101] In many examples, the hand classification term inference component 406, the gesture inference component 408, and the gesture text input recognition component 410 transmit the hand classification term probability data 430, the gesture input data 426, and the symbol input data 414, respectively, via an inter-process communication method.
[0102] In some examples, the hand tracking input pipeline 440 operates to continuously generate and publish symbolic input data 414, gesture input data 426, skeletal model data 434, and coordinate transformation data 432 based on real-world scene environment frame data 428 generated by one or more cameras of the AR system.
[0103] Figure 5A is a sequence diagram of a sign language text input process 500 of an AR system (eg, glasses 100) according to some examples, Figure 5B is an illustration of a start / end text input gesture 526, Figure 5C is a diagram of the sign language symbol 528, and Figure 5D is an illustration of an AR overlay 534. During the sign language text input process 500, the AR system utilizes the AR gesture application component 420 to implement the sign language text input process 500 using a gesture recognition method.
[0104] The AR system supplies the symbol input event data 510 and the gesture input event data 522 as part of the directional input event data 552 to the AR gesture application component 420 using the hand tracking input pipeline 440. The gesture text input recognition component 410 generates the symbol input event data 510 based on the sign language model and the hand classification term probability data 430. In some examples, the gesture text input recognition component 410 uses an AI method and a sign language model generated using a machine learning method to classify the hand classification term probability data 430, and assigns the hand classification term probability data 430 to sign language symbols corresponding to letters or characters and numbers in the alphabet, such as Figure 5C The gesture text input recognition component 410 generates the symbol input event data 510 based on the sign language symbol by mapping the sign language symbol to a corresponding character in the alphabet or numbers.
[0105] In another example, the gesture text input recognition component 410 compares the hand classification term probability data 430 with a previously determined set of sign language models, wherein the member sign language models of the set of sign language models correspond to individual sign language symbols. The gesture text input recognition component 410 selects a sign language symbol corresponding to the hand classification term probability data 430 based on the comparison, and generates the symbol input event data 510 based on the sign language symbol by mapping the sign language symbol to a corresponding character in the alphabet or numbers.
[0106] In some examples, additional gestures and symbols (not shown) are made by user 538 to indicate text editing commands. User 538 uses these commands to instruct the AR system to perform editing functions on the input text.
[0107] During use of the AR system by a user 538, when the user 538 wants to input text into a virtual object (e.g., text scene component 502) of an AR gesture application component (e.g., AR gesture application component 420), the user 538 creates an input event by making a start text input gesture 550 (e.g., start / end text input gesture 526). In operation 504, the hand tracking input pipeline 440 generates gesture input event data 522 based on the real-world scene environment video frame data of the start text input gesture 550 being made by the user 538 as described herein. The hand tracking input pipeline 440 transmits the gesture input event data 522 to the system framework component 416.
[0108] In loop operation 540, the system framework component 416 receives and processes each input event included in the gesture input event data 522 and the symbol input event data 510 received from the hand tracking input pipeline 440, as indicated by the guard condition 542. In operation 544, the system framework component 416 generates directional input event data 552 based on the gesture input event data 522 and the component registration data as described herein. In operation 546, the system framework component 416 routes the directional input event data 552 to the AR gesture application component 420.
[0109] In operation 506 , the AR gesture application component 420 receives directional input event data 552 of the gesture input event data 522 including a start text input gesture 550 being made by the user 538 , and detects the start text input gesture 550 being made by the user 538 based on the directional input event data 552 , and initiates text input into the text scene component 502 .
[0110] The user 538 makes one or more sign language symbols 528, and in operation 504, the hand tracking input pipeline 440 generates sign input event data 510 for the sign language symbol 528 based on the real world scene environment video frame data generated by one or more cameras as described herein. The hand tracking input pipeline 440 transmits the sign input event data 510 to the system framework component 416.
[0111] In operation 544 of loop operation 540 , system framework component 416 generates directional input event data 552 based on symbol input event data 510 and component registration data as described herein. In operation 546 , system framework component 416 routes directional input event data 552 including symbol input event data 510 to AR gesture application component 420 .
[0112] In loop operation 508 , AR gesture application component 420 performs loop operation 508 until AR gesture application component 420 detects an ending text input gesture 548 being made by the user, such as start / end text input gesture 526 , as indicated by guard condition 512 .
[0113] In alternative operation 514, AR gesture application component 420 receives directional input event data 552 and detects one or more input symbols of symbol input event data 510 in directional input event data 552, as indicated by guard condition 516. One or more input symbols correspond to one or more sign language symbols 528 being made by user 538, such as "V" sign language symbol 530. One or more input symbols include characters corresponding to letters, numbers, and editing commands in the alphabet. Based on detecting the input symbols, in operation 518, AR gesture application component 420 generates input text data 536 based on the one or more input symbols, and transmits the input text data 536 to text scene component 502. In operation 520, text scene component 502 provides input text 532 to the user in text scene component 502 based on input text data 536.
[0114] In some examples, the one or more input symbols correspond to one or more editing commands for the text input by the user 538. Based on determining that the one or more input symbols correspond to the one or more editing commands, the AR gesture application component 420 performs one or more editing functions on the input text 532, such as a cut operation, a copy operation, a paste operation, etc.
[0115] When the user 538 completes entering text into the text scene component 502, the user 538 makes an end text input gesture 548, such as the start / end text input gesture 526. In operation 504, the hand tracking input pipeline 440 generates gesture input event data 522 based on the real-world scene environment video frame data of the end text input gesture 548 being made by the user 538 as described herein. The hand tracking input pipeline 440 transmits the gesture input event data 522 to the system framework component 416.
[0116] In operation 544 , system framework component 416 generates directional input event data 552 including gesture input event data 522 based on gesture input event data 522 and component registration data as described herein. In operation 546 , system framework component 416 routes directional input event data 552 to AR gesture application component 420 .
[0117] In loop operation 508, AR gesture application component 420 receives directional input event data 552. In alternative operation 514, the AR system detects an end text input gesture 548 being made by user 538 based on gesture input event data 522 of directional input event data 552, as indicated by guard condition 524. Based on detecting the end text input gesture 548, AR gesture application component 420 terminates, thereby terminating sign language text input process 500.
[0118] In some examples, AR gesture application component 420 of the AR system performs the functions of hand tracking input pipeline 440 and system framework component 416 by utilizing various APIs and system libraries of the AR system.
[0119] Fig. 6A 6 is a deployment diagram of a voice input pipeline according to some examples. An AR system such as glasses 100 uses a voice input pipeline 602 to receive and process voice commands from a user. The voice input pipeline 602 processes audio signals captured by a microphone hardware and pre-processing audio digital signal processor (aDSP) component 650, which is a component of a set of hardware abstraction layers (HALs), cores, and hardware components 654 of an AR system. The microphone hardware and pre-processing aDSP component 650 uses one or more audio input devices such as ( Figure 1 The microphone 134 and microphone 136 of the glasses 100 are used to capture voice input data from the user of the AR system. The voice input pipeline 602 uses the services of one or more voice machine learning (ML) services 626 to process the audio signal data 620 captured by the microphone hardware and the pre-processing aDSP component 650.
[0120] The HAL, core, and hardware components 654 include a wake word (WW) WW engine audio digital signal processor (aDSP) 652, which is operable to generate an interrupt 618 using audio signal data 620. The WW engine aDSP 652 provides a wake word detection capability to the AR system. During operation, the WW engine aDSP 652 continuously receives audio signal data 620, and detects specific sound patterns or keywords in the audio signal data 620, and generates an interrupt 618 in response. In some examples, the WW engine aDSP 652 is implemented in an audio digital signal processing component. In some examples, the WW engine aDSP 652 uses an AI method to detect a wake word in the audio signal data 620. In some examples, the WW engine aDSP 652 processes tactile inputs, such as, but not limited to, button presses, touch screen touches, key presses, and the like.
[0121] The WW engine aDSP 652 communicates the interrupt 618 to the sound trigger framework / HAL component 642. The sound trigger framework / HAL component 642 acts as an interface between the components of the HAL, core, and hardware components 654 and the voice service component 634. The sound trigger framework / HAL component 642 receives the interrupt 618 and generates a wake-up command 614, which the sound trigger framework / HAL component 642 communicates to the voice service component 634.
[0122] In response to receiving the wake-up command 614, the voice services component 634 begins monitoring microphone data 610 generated by the audio input service 644. The audio input service 644 receives the audio signal data 620 from the microphone hardware and pre-processing aDSP component 650 and routes the audio signal data 620 to the voice services component 634 as microphone data 610. In some examples, the microphone hardware and pre-processing aDSP component 650 includes processing capabilities to implement algorithms that improve the quality of the audio signal data 620 for speech recognition, voice activity detection, and similar processing.
[0123] The speech service component 634 is operatively coupled to one or more speech analysis components of the speech ML service 626. In some examples, the speech ML service 626 is a service provided by the AR system. In some examples, the speech ML service 626 is a service provided by a device or system external to the AR system. In some examples, the speech ML service 626 is a cloud-based service that can be accessed by the AR system via a network. In some examples, the speech ML service 626 includes an automatic speech recognition (ASR) inference engine 628, a natural language processing (NLP) inference engine 630, a text-to-speech (TTS) engine 632, and the like.
[0124] In some examples, the ASR inference engine 628 uses a combination of statistical models and machine learning algorithms to recognize speech patterns and convert the speech patterns into transcription data 612 using the microphone data 610 .
[0125] In some examples, NLP inference engine 630 processes and analyzes transcription data 612 to generate intent data 608 of the user of the AR system. In some examples, NLP inference engine 630 uses a combination of language rules, statistical models, and machine learning algorithms to perform its operations. In some examples, intent data 608 includes named entity identification, sentiment analysis, topic models, language translation, etc.
[0126] In some examples, the TTS engine 632 receives text data as input and analyzes the text to identify the appropriate pronunciation, intonation, and voice for each word or phrase in the text. In some examples, the TTS engine 632 uses a pre-recorded speech database (which contains individual segments of recorded speech for each phoneme, word, or phrase in the target language) to generate text-to-speech data 604 that encodes the text data 606 as audio data. The segments are combined in real time to create a natural-sounding voice that speaks the written text.
[0127] In some examples, the TTS engine 632 uses machine learning algorithms to dynamically generate speech by analyzing the acoustic properties of spoken language and using neural networks to generate speech sounds that mimic the sound patterns of natural language.
[0128] During operation, the speech service component 634 transmits microphone data 610 to the ASR inference engine 628. The ASR inference engine 628 receives the microphone data 610 and generates transcription data 612 using the microphone data 610. The ASR inference engine 628 transmits the transcription data 612 to the speech service component 634. The speech service component 634 receives the transcription data 612 and transmits the transcription data 612 to the NLP inference engine 630. The speech service component 634 receives the intent data 608 and generates semantic event data in the form of a combination of the intent data 608 and the transcription data 612, and transmits the intent data 608 and the transcription data 612 as intent and transcription data 616 to the system framework component 416.
[0129] In some examples, the NLP inference engine 630 receives the transcription data 612 directly from the ASR inference engine 628 and generates the intent data 608 without converting the transcription data 612 back and forth by the speech service component 634. Thus, the components of the speech ML service 626 generate the intent data 608 directly using the microphone data 610.
[0130] In some examples, an API is provided by the AR system so that the AR application component can access the intent data 608 and the transcription data 612 from the system framework component 416 .
[0131] System framework component 416 receives intent and transcription data 616 and uses intent and transcription data 616 to generate directed input event data 438 or undirected input event data 436. Undirected input event data for input events belonging to the undirected category is routed to an operating system level component such as system user interface component 418. Directed input event data for input events belonging to the directional category is routed to a target component such as AR gesture application component 420. System framework component 416 uses intent and transcription data 616 and component registration data to classify intent and transcription data 616, as described below.
[0132] The system framework component 416 registers the component registration data such as ( Figure 6BThe component registration data 668 is stored in a data store to be accessed during operation of the system framework component 416. The component registration data 668 includes: a registered language field 666 that identifies a language model to be associated with the target AR application component; and a component ID field 660 that identifies the target AR application component to which the input event data is to be routed. The component registration data 668 also includes: one or more registered voice command fields 670 that identify recognized voice commands to be associated with the target AR application component; one or more registered gesture fields 662 and / or registered symbol fields 664 that indicate voice commands, gestures, and symbols to be routed to the registered AR application component.
[0133] As shown, the component ID field 660 includes the AR application component identifier "TEXT ENTRY"; the registered language field 666 identifies the language associated with the registered AR application component, namely "ENGLISH"; the registered posture field 662 includes the posture identifier that should be routed to the registered target AR application component, namely "LEFT_PALMAR_FINGERS EXTENDED_RIGHT_PALMAR_FINGERS_EXTENDED", the registered symbol field 664 identifies a set of symbols that should be routed to the registered AR application component (i.e., "[*]" representing all symbols), and the registered voice command that should be routed to the registered AR application, namely "ENTER TEXT".
[0134] In some examples, a system user interface component can be registered as a target AR application component.
[0135] In some examples, an API is provided by the AR system so that the AR application component can register itself with the system framework component 416, such as but not limited to, registering itself when the AR application component is installed.
[0136] Referring again to the system framework component 416 processing the intent and transcription data 616 received from the voice service component 634 when processing the intent and transcription data 616, the system framework component 416 searches the registered voice command field of the component registration data for a registered voice command that matches the intent and transcription data 616. When the system framework component 416 determines a match, the system framework component 416 determines that the intent and transcription data 616 is a directional input event data 438. The system framework component 416 also determines the target AR application component based on the target AR application component identified in the component ID field (e.g., component ID field 660) of the component registration data that includes the matching registered voice command. In the event that the system framework component 416 determines that the intent and transcription data 616 does not match any registered voice command in the component registration data, the system framework component 416 determines that the intent and transcription data 616 will be classified as undirected input event data 436 and will be routed to the system user interface component 418.
[0137] In another example of processing directional input event data 438, the AR system determines based on hinting that the directional input event data 438 is to be routed to the AR application component. For example, if the AR system executes the current AR application component in a single application mode state, the current AR application component is hinted as the AR application component to which the directional input event data 438 should be routed.
[0138] In response to determining that the intent and transcription data 616 include undirected input event data 436, the system framework component 416 transmits the undirected input event data 436 to the system user interface component 418. The system user interface component 418 receives the undirected input event data 436 and determines one or more actions to be performed by the AR system. In some examples, the system user interface component 418 generates task refinement data 622 using the undirected input event data 436 and transmits the undirected input event data 436 to the task execution manager 636. The task execution manager 636 receives the task refinement data 622 and undertakes one or more tasks or processes using resources of the AR system that utilize the task refinement data 622.
[0139] In response to determining that the intent and transcription data 616 include directional input event data 438, the system framework component 416 transmits the directional input event data 438 to the AR voice application component 672. The AR voice application component 672 receives the directional input event data 438 and uses the directional input event data 438 to determine one or more operations to be performed by the AR voice application component 672.
[0140] In some examples, system user interface component 418 generates text data 606 that includes one or more prompts or messages that system user interface component 418 uses to prompt or notify a user of the AR system.
[0141] In some examples, the system user interface component 418 transmits the text data 606 to the TTS engine 632 of the voice ML service 626. The TTS engine 632 receives the text data 606 and generates text-to-speech data 604 using the text data 606. The TTS engine 632 transmits the text-to-speech data 604 to the system user interface component 418. The system user interface component 418 receives the text-to-speech data 604 and transmits the text-to-speech data 604 to the speaker hardware and post-processing component 648 of the HAL, core, and hardware component 654. The speaker hardware and post-processing component 648 receives the text-to-speech data 604 and generates an audible voice prompt or message provided to the user by the AR system. In some examples, the speaker hardware and post-processing component 648 includes processing capabilities to implement algorithms that operate on playback signals to improve playback quality, loudness, etc.
[0142] In some examples, system user interface component 418 receives undirected input event data 436 and uses it to generate display data 624 for a user interface of the AR system. System user interface component 418 transmits display data 624 to optical engine 646 of HAL, core, and hardware component 654. Optical engine 646 receives display data 624 and generates a display of display data 624 in a user interface of the AR system.
[0143] In some examples, the undirected input event data 436 includes instructions for capturing image data, video data, and / or audio data through the AR system. In response to receiving the instructions for capturing image data, video data, and / or audio data, the system user interface component 418 transmits a capture request 656 to the capture service 638. In response, the capture service 638 uses components of the AR system, such as one or more cameras and / or one or more microphones of the glasses 100 to capture image data, video data, and / or audio data. In some examples, the undirected input event data 436 may include instructions to the AR system, and the instructions to the AR system are used to send the captured image data, video data, text data, and / or audio data to the outside of the AR system, such as but not limited to sending a message to a user of an interactive platform, posting to the interactive platform, etc. To this end, the system user interface component 418 sends a send request 658 to the send service 640. The send service 640 uses the resources of the AR system to transmit the captured image data, video data, text data, and / or audio data to a system outside the AR system.
[0144] In some examples, the AR voice application component of the AR system performs the functions of the voice input pipeline 602 and the system framework component 416 by utilizing various APIs and system libraries of the AR system.
[0145] In some examples, the AR application component registers with the system framework component 416 to obtain Fig. 6A ) speech input pipeline 602 and ( Figure 4A The hand tracking input pipeline 440 receives the semantic event data. During operation, the AR application component uses the semantic event data from the voice input pipeline 602 and the hand tracking input pipeline 440 to provide an AR experience to the user.
[0146] Figure 7 is a sequence diagram of the speech input process according to some examples. Figure 7 It is shown that when an AR system (e.g., ( Figure 1 When the glasses 100) are listening for a verbal command issued by a user of the AR system Fig. 6A How the components in a program interact with each other.
[0147] In operation 702, ( Fig. 6A The voice service component 634 of the voice input pipeline 602 transmits the microphone data request 712 to the audio input service 644 of the voice input pipeline 602. In response, the audio input service 644 transmits the audio signal data 620 to the voice service component 634. The audio signal data 620 is transmitted by ( Fig. 6A The microphone hardware and preprocessing aDSP component 650 uses one or more microphones of the AR system to capture and transmit audio signal data 620 to the voice service component 634. The voice service component receives the audio signal data 620 and generates a microphone activity message 714 to the system user interface component 418 via the system framework component 416, as shown in FIG. Fig. 6A In response to receiving microphone activity message 714, system user interface component 418 generates a "microphone activity" display that is displayed to a user of the AR system.
[0148] In loop operation 704, the audio input service 644 transmits the audio data 716 to the speech service component 634. The speech service component 634 receives the audio data 716 and forwards the audio data 716 to the speech ML service 626. The speech ML service 626 receives the audio data 716 and uses the audio data 716 to generate a text transcript 718, as described with reference to Fig. 6A The speech ML service 626 transmits the text transcription 718 to the speech service component 634 .
[0149] In some examples, the speech service component 634 receives the speech service component 634 and transmits the text transcription 718 to the system user interface component 418 via the system framework component 416, as shown in FIG. Fig. 6A The system user interface component 418 receives the text transcription 718 and displays the text transcription 718 to the user of the AR system, as shown in FIG. Fig. 6A described.
[0150] The AR system continues to loop operation 704 until the AR system detects that the user has completed or ended the command, for example but not limited to by detecting a pause in the user's speech, by receiving tactile input from the user, detecting a specific gesture of the user, etc. In response to detecting the completion or end of the command, in operation 706, the voice service component 634 receives the final transcription data 720 and the intent data 722 from the voice ML service 626. Operation 706 transmits the transcription data and the intent data 724 as input event data to the system user interface component 418 via the system framework component 416, as shown in FIG. Figure 4A and Fig. 6A Describe more fully.
[0151] In response to detecting the absence of detectable speech in the audio signal data 738 , or the final transcription data 720 and the intent data 722 do not contain sufficient information to specify an unambiguous command, the AR system performs an alternative loop operation 708 .
[0152] In alternative loop operation 708, the voice service component 634 transmits a no voice detected 726 message to the system user interface component 418 via the system framework component 416, as described with reference to FIG. Fig. 6A In response to receiving no voice detected 726, the system user interface component 418 generates text data 728 of a prompt or message to be provided to the user of the AR system. The system user interface component 418 transmits the text data 728 to the voice ML service 626. The voice ML service 626 Fig. 6A The ) TTS engine 632 generates voice data 730 using the text data 728. The voice ML service 626 transmits the voice data 730 to the system user interface component 418. The system user interface component 418 transmits the voice data 730 to the speaker hardware and post-processing component 648. The speaker hardware and post-processing component 648 receives the voice data 730 and generates an audible prompt or message to the user using the voice data 730. At operation 702, the system user interface component 418 sends a voice request 732 message to the voice service component 634 to request that the voice service component 634 resume operation.
[0153] To end the interaction, in operation 710, the voice service component 634 transmits a stop request 734 to the audio input service 644 and transmits a microphone off message 736 to the system user interface component 418. In response to receiving the microphone off message 736, the system user interface component 418 displays a message to the user of the AR system that the microphone is now off.
[0154] Figure 8 is a sequence diagram of a speech input processing method according to some examples. Figure 1 The glasses 100 use a voice input processing method to process the user's voice input to the AR system.
[0155] In operation 802, the voice service component 634 transmits the intent and transcription data 808 to the system framework component 416. The system framework component 416 attempts to match the intent and transcription data 808 with the AR voice application component or the system user interface component 418, as shown in FIG. Fig. 6A described.
[0156] In response to determining that there is no match or only a partial match, the AR system proceeds to alternative operation 804 .
[0157] In alternative operation 804, the system framework component 416 transmits the command of the unrecognized message 810 to the system user interface component 418. In response to receiving the command of the unrecognized message 810, the system user interface component 418 generates text-to-speech prompt data 812 as a prompt, which will be used to prompt the user to make a new voice input. The system user interface component 418 transmits the text-to-speech prompt data 812 to the voice ML service 626 ( Fig. 6A The speech ML service 626 generates text-to-speech data 814. The system user interface component 418 receives the text-to-speech data 814 and transmits the text-to-speech data 814 to the speaker hardware and post-processing component 648 of the AR system. In response to receiving the text-to-speech data 814, the speaker hardware and post-processing component 648 uses one or more audio speakers of the AR system (e.g., audio speaker 132 and audio speaker 130 ( Figure 1 The system user interface component 418 transmits the additional voice request 816 to the voice service component 634. In response, the voice service component 634 returns to operation 802.
[0158] In response to determining that there is a match between the component registration data and the intent and transcription data 808 , the system framework component 416 proceeds to operation 806 .
[0159] In operation 806, the system user interface component 418 transmits the intent and transcription data 808 as undirected input event data 818 to the system user interface component 418. The AR voice application component 672 receives the undirected input event data 818 and performs one or more operations based on the content of the undirected input event data 818. In the example, the undirected input event data 818 includes a command to capture an image of a real-world scene observed by a user of the AR system and transmit the image as a message to a user of the interactive platform. The undirected input event data 818 includes the address or ID of the user of the interactive platform. The system user interface component 418 generates a task queue that includes tasks to be performed using the AR system ( Fig. 6A The AR system's ( Fig. 6A ) Sending service 640. Fig. 6A The task execution manager 636 captures the image using the capture service 638 and transmits the image to the user of the interactive platform using the sending service 640.
[0160] In some examples, the system framework component 416 determines that the intent and transcription data 808 match data in the component registration data, and the system framework component 416 transmits the intent and transcription data 808 as directed input event data to the AR voice application component associated with the matching data in the component registration data. The AR voice application component receives the intent and transcription data 808 and uses the intent and transcription data 808 to perform one or more operations.
[0161] Fig. 9 900 is a block diagram illustrating a software architecture 904 that may be installed on any one or more of the devices described herein. The software architecture 904 is supported by hardware, such as a machine 902, including a processor 920, a memory 926, and an I / O component 938. In this example, the software architecture 904 may be conceptualized as a stack of layers in which each layer provides specific functionality. The software architecture 904 includes layers such as an operating system 912, a library 908, a framework 910, and an application 906. In operation, the application 906 invokes an API call 950 through the software stack and receives a message 952 in response to the API call 950.
[0162] The operating system 912 manages hardware resources and provides public services. The operating system 912 includes, for example, a kernel 914, a service 916, and a driver 922. The kernel 914 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 914 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. The service 916 can provide other public services for other software layers. The driver 922 is responsible for controlling or interfacing with the underlying hardware. For example, the driver 922 may include a display driver, a camera driver, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.
[0163] The library 908 provides a low-level common infrastructure used by the application 906. The library 908 may include a system library 918 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 908 may include an API library 924, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting two-dimensional (2D) and three-dimensional (3D) graphics content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 908 may also include various other libraries 928 to provide many other APIs to the application 906 .
[0164] The framework 910 provides a high-level common infrastructure used by the applications 906. For example, the framework 910 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 910 can provide a wide range of other APIs that can be used by the applications 906, some of which can be specific to a particular operating system or platform.
[0165] In some examples, applications 906 may include a home application 936, a contacts application 930, a browser application 932, a book reader application 934, a location application 942, a media application 944, a messaging application 946, a game application 948, and a variety of other applications such as third-party applications 940. Application 906 is a program that executes the functions defined in the program. Various programming languages can be used to create one or more of the applications 906 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party application 940 (e.g., a third-party application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, third-party applications 940 can call API calls 950 provided by the operating system 912 to facilitate the functions described herein.
[0166] Fig.10 1 is a block diagram showing a networked system 1000 including details of the glasses 100 according to some examples. The networked system 1000 includes the glasses 100, a client device 1026, and a server system 1032. The client device 1026 may be a smart phone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 1036 and / or a high-speed wireless connection 1034. The client device 1026 is connected to the server system 1032 via a network 1030. The network 1030 may include any combination of wired and wireless connections. The server system 1032 may be one or more computing devices that are part of a service or network computing system. The client device 1026 and any elements of the server system 1032 and the network 1030 may use Fig. 9 and Figure 3 The details of the software architecture 904 or machine 300 described respectively are implemented.
[0167] The glasses 100 include a data processor 1002, a display 1010, one or more cameras 1008, and additional input / output elements 1016. The input / output elements 1016 may include microphones, audio speakers, biometric sensors, additional sensors, or additional display elements integrated with the data processor 1002. Fig. 9 and Figure 3Examples of input / output elements 1016 are further discussed. For example, input / output elements 1016 may include any I / O component 306, including output component 328, motion component 336, etc. Figure 2 Examples of display 1010 are discussed in . In the specific examples described herein, display 1010 includes displays for the left and right eyes of a user.
[0168] Data processor 1002 includes image processor 1006 (eg, video processor), GPU and display driver 1038, tracking module 1040, interface 1012, low power circuitry 1004, and high speed circuitry 1020. Components of data processor 1002 are interconnected by bus 1042.
[0169] The interface 1012 refers to any source of user commands provided to the data processor 1002. In one or more examples, the interface 1012 is a physical button that, when pressed, sends a user input signal from the interface 1012 to the low-power processor 1014. The low-power processor 1014 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. The low-power processor 1014 can process pressing such a button within a first time period as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 1012 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the camera 1008. In other examples, the interface 1012 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 1026.
[0170] Image processor 1006 includes circuitry for receiving signals from camera 1008 and processing those signals from camera 1008 into a format suitable for storage in memory 1024 or for transmission to client device 1026. In one or more examples, image processor 1006 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 1008, and volatile memory used by the microprocessor in operation.
[0171] The low power circuit system 1004 includes a low power processor 1014 and a low power wireless circuit system 1018. These elements of the low power circuit system 1004 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 1014 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 1014 can accept user input signals from the interface 1012. The low power processor 1014 can also be configured to receive input signals or command communications from the client device 1026 via the low power wireless connection 1036. The low power wireless circuit system 1018 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 1018. In other examples, other low power communication systems may be used.
[0172] High-speed circuit system 1020 includes a high-speed processor 1022, a memory 1024, and a high-speed wireless circuit system 1028. High-speed processor 1022 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system used by data processor 1002. High-speed processor 1022 includes processing resources used to manage high-speed data transmission over high-speed wireless connection 1034 using high-speed wireless circuit system 1028. In some examples, high-speed processor 1022 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Fig. 9 The high-speed processor 1022, which executes the software architecture of the data processor 1002, manages data transmission with the high-speed wireless circuit system 1028, in addition to any other duties. In some examples, the high-speed wireless circuit system 1028 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 1028 can implement other high-speed communication standards.
[0173] The memory 1024 includes any storage device capable of storing camera data generated by the camera 1008 and the image processor 1006. Although the memory 1024 is shown as being integrated with the high-speed circuitry 1020, in other examples, the memory 1024 may be a separate, independent element of the data processor 1002. In some such examples, electrical wiring may provide a connection from the image processor 1006 or the low-power processor 1014 to the memory 1024 through a chip including the high-speed processor 1022. In other examples, the high-speed processor 1022 may manage the addressing of the memory 1024 so that the low-power processor 1014 will direct the high-speed processor 1022 whenever a read or write operation involving the memory 1024 is desired.
[0174] The tracking module 1040 estimates the pose of the glasses 100. For example, the tracking module 1040 uses image data and associated inertial data from the camera 1008 and the positioning component 340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene environment). The tracking module 1040 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world scene environment. The tracking module 1040 allows the glasses 100 to visually place virtual objects relative to physical objects within the user's field of view via the display 1010.
[0175] The GPU and display driver 1038 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 1010 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 1038 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the position and orientation of the user relative to physical objects in the user's real-world scene environment.
[0176] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 1026 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 906, such as the messaging application 946.
[0177] Fig.111 is a block diagram illustrating an example messaging system 1100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 1100 includes multiple instances of client devices 1026 that host several applications including a messaging client 1102 and other applications 1104. The messaging client 1102 is communicatively coupled to other instances of the messaging client 1102 (e.g., hosted on respective other client devices 1026), a messaging server system 1106, and a third-party server 1108 via a network 1030 (e.g., the Internet). The messaging client 1102 may also communicate with the locally hosted application 1104 using an application program interface (API).
[0178] The messaging clients 1102 are able to communicate and exchange data with other messaging clients 1102 and with a messaging server system 1106 via the network 1030. The data exchanged between the messaging clients 1102 and between the messaging clients 1102 and the messaging server system 1106 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).
[0179] The messaging server system 1106 provides server-side functionality to a particular messaging client 1102 via the network 1030. Although some functions of the messaging system 1100 are described herein as being performed by the messaging client 1102 or by the messaging server system 1106, the location of some functions within the messaging client 1102 or within the messaging server system 1106 may be a design choice. For example, it may be technically preferred to initially deploy some technologies and functions within the messaging server system 1106, but then migrate the technologies and functions to the messaging client 1102 where the client device 1026 has sufficient processing power.
[0180] The messaging server system 1106 supports various services and operations provided to the messaging clients 1102. Such operations include transmitting data to the messaging clients 1102, receiving data from the messaging clients 1102, and processing data generated by the messaging clients 1102. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 1100 is activated and controlled by functions available via the user interface (UI) of the messaging client 1102.
[0181] Turning now specifically to the messaging server system 1106, an application program interface (API) server 1110 is coupled to and provides a programming interface to an application server 1114. The application server 1114 is communicatively coupled to a database server 1116, which facilitates access to a database 1120 that stores data associated with messages processed by the application server 1114. Similarly, a web server 1124 is coupled to and provides a web-based interface to the application server 1114. To this end, the web server 1124 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.
[0182] The application program interface (API) server 1110 receives and transmits message data (e.g., commands and message payloads) between the client device 1026 and the application server 1114. Specifically, the application program interface (API) server 1110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 1102 to activate the functions of the application server 1114. The application program interface (API) server 1110 exposes various functions supported by the application server 1114, including: account registration; login functionality; sending messages from a particular messaging client 1102 to another messaging client 1102 via the application server 1114, sending media files (e.g., images or videos) from the messaging client 1102 to the messaging server 1112, and possible access by another messaging client 1102; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 1026; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 1102).
[0183] The application server 1114 hosts several server applications and subsystems, including, for example, a messaging server 1112, an image processing server 1118, and a social network server 1122. The messaging server 1112 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 1102. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 1102. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 1112.
[0184] The application server 1114 also includes an image processing server 1118 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 1112.
[0185] The social network server 1122 supports various social networking functions and services and makes these functions and services available to the messaging server 1112. To this end, the social network server 1122 maintains and accesses an entity graph within the database 1120. Examples of functions and services supported by the social network server 1122 include identifying other users in the messaging system 1100 with whom a particular user has relationships or who the particular user is "following," and also includes identifying interests and other entities of a particular user.
[0186] The messaging client 1102 may notify the user of the client device 1026 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 1102 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 1102 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.
[0187] Additional examples include:
[0188] Example 1 is a computer-implemented method, comprising: generating input event data by one or more processors using input data captured by an input device of an AR system; determining a target AR application component of the AR system by the one or more processors using the input event data and component registration data of one or more AR application components of the AR system; and transmitting the input data to the target AR application component by the one or more processors.
[0189] In Example 2, the subject matter of Example 1 includes, wherein the input event data includes gesture input data of a gesture being made by a user of the AR system.
[0190] In Example 3, the subject matter of any one of Examples 1 to 2 includes, wherein the input event data includes symbol input data of a sign language symbol being made by a user of the AR system.
[0191] In Example 4, the subject matter of any one of Examples 1 to 3 includes, wherein the input event data is generated by a hand tracking input pipeline of the AR system.
[0192] In Example 5, the subject matter of any of Examples 1 to 4 includes, wherein the input event data includes a voice command being issued by a user of the AR system.
[0193] In Example 6, the subject matter of any of Examples 1 to 5 includes, wherein the input event data is generated by a speech input pipeline of the AR system.
[0194] In Example 7, the subject matter of any of Examples 1 to 6 includes, wherein the speech input pipeline includes a speech machine learning (ML) service.
[0195] Example 8 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations implementing any of Examples 1-7.
[0196] Example 9 is a device comprising a device implementing any one of Examples 1 to 7.
[0197] Example 10 is a system implementing any one of Examples 1 to 7.
[0198] Example 11 is a method of implementing any one of Examples 1 to 7.
[0199] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.
[0200] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.
[0201] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.
[0202] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed for use with other components, and are usually part of a program that performs a specific function of a related function. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein specific one or more processors are some examples of hardware. For example, some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some of the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, reside in a single machine, and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.
[0203] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.
[0204] "Machine storage media" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. The term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media," "device storage media," and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media," "computer storage media," and "device storage media" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media."
[0205] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.
[0206] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.
Claims
1. A computer-implemented method comprising: generating, by the one or more processors, input event data using input data captured by an input device of the AR system; determining, by the one or more processors, a target AR application component of the AR system using the input event data and component registration data of one or more AR application components of the AR system; as well as The input data is transmitted to the target AR application component by the one or more processors.
2. The computer-implemented method of claim 1 , wherein: The input event data includes gesture input data of a gesture being performed by a user of the AR system.
3. The computer-implemented method of claim 1 , wherein: The input event data includes symbol input data of a sign language symbol being made by a user of the AR system.
4. The computer-implemented method of claim 2, wherein: The input event data is generated by the hand tracking input pipeline of the AR system.
5. The computer-implemented method of claim 1 , wherein: The input event data includes voice commands being issued by a user of the AR system.
6. The computer-implemented method of claim 5, wherein: The input event data is generated by the voice input pipeline of the AR system.
7. The computer-implemented method of claim 6, wherein: The speech input pipeline includes a speech machine learning (ML) service.
8. A machine comprising: one or more processors; as well as a memory storing executable instructions that, when executed by the one or more processors, cause the machine to perform operations comprising: generating input event data using input data captured by an input device of the AR system; determining a target AR application component of the AR system using the input event data and component registration data of one or more AR application components of the AR system; and The input data is transmitted to the target AR application component.
9. The machine according to claim 8, wherein: The input event data includes gesture input data of a gesture being performed by a user of the AR system.
10. The machine according to claim 8, wherein: The input event data includes symbol input data of a sign language symbol being made by a user of the AR system.
11. The machine according to claim 9, wherein: The input event data is generated by the hand tracking input pipeline of the AR system.
12. The machine according to claim 8, wherein: The input event data includes voice commands being issued by a user of the AR system.
13. The machine according to claim 12, wherein: The input event data is generated by the voice input pipeline of the AR system.
14. The machine according to claim 13, wherein: The speech input pipeline includes a speech machine learning (ML) service.
15. A machine storage medium storing executable instructions, wherein the executable instructions, when executed by a machine, cause the machine to perform operations, the operations comprising: generating input event data using input data captured by an input device of the AR system by the one or more processors; determining, by the one or more processors, a target AR application component of the AR system using the input event data and component registration data of one or more AR application components of the AR system; as well as The input data is transmitted to the target AR application component by the one or more processors.
16. The machine storage medium of claim 15, wherein: The operations also include where the input event data includes gesture input data of a gesture being performed by a user of the AR system.
17. The machine storage medium of claim 15, wherein: The operations also include, wherein the input event data includes symbol input data of a sign language symbol being made by a user of the AR system.
18. The machine storage medium of claim 16, wherein: The operations also include, wherein the input event data is generated by a hand tracking input pipeline of the AR system.
19. The machine storage medium of claim 15, wherein: The operations also include where the input event data includes voice commands being issued by a user of the AR system.
20. The machine storage medium of claim 19, wherein: The operations also include, wherein the input event data is generated by a voice input pipeline of the AR system.
21. The machine storage medium of claim 20, wherein: The operations also include, wherein the speech input pipeline includes a speech machine learning (ML) service.
Citation Information
Cited By
Multimodal UI with semantic events
US12656875B2
