Finger spelling text input
Through gesture recognition and finger spelling technology, the problem of user input modes of head-mounted AR system is solved, and rich text input and editing operations are realized in the AR experience, improving the convenience and efficiency of user interaction.
Patent Information
- Application Number
- CN202380036089.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-27
- Filing Date
- 2023-04-27
- Publication Date
- 2024-12-06
AI Technical Summary
The user input mode of existing head-mounted augmented reality (AR) systems is limited and it is difficult to perform complex interactions, especially in the absence of physical input devices, such as a touch screen or a keyboard.
Using gesture recognition and finger spelling as input modes, text input is provided through user's hand and finger motion detection, using hand tracking input pipelines and gesture inference components to identify gestures and finger spelling sign language and convert them into text or editing Order.
It realizes providing richer user input methods in the AR experience, without the need for users to take their attention away from the AR experience, improving the efficiency and accuracy of text input.
Smart Images

Figure CN119096218A_ABST
Abstract
Description
[0001] Priority claim
[0002] This application claims the benefit of priority to U.S. Application No. 17 / 660,965, filed on April 27, 2022, the entire contents of which are incorporated herein by reference. Technical Field
[0003] The present disclosure relates generally to user interfaces, and more particularly to user interfaces used in augmented reality and virtual reality. Background Art
[0004] A head-mounted AR system may be implemented with a transparent or translucent display through which a user of the head-mounted AR system can view the surrounding environment. Such a device enables a user to look through the transparent or translucent display to view the surrounding environment, and also to see objects (e.g., virtual objects, such as renderings of 2D or 3D graphics models, images, videos, text, etc.) that appear to be part of the surrounding environment and / or superimposed on the surrounding environment generated for display. This is generally referred to as "augmented reality" or "AR". A head-mounted AR system may additionally completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is generally referred to as "virtual reality" or "VR". As used herein, unless the context indicates otherwise, the term AR refers to one or both of augmented reality and virtual reality as traditionally understood.
[0005] A user of a head-mounted AR system can access and use computer software applications to perform various tasks or participate in entertainment activities. Performing tasks or participating in entertainment activities can include inputting text. To input text, the user interacts with a text input user interface provided by the head-mounted AR system. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] To easily identify the discussion of any particular element or action, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.
[0007] Figure 1 is a perspective view of a head mounted device according to some examples.
[0008] Figure 2 According to some examples Figure 1 Additional views of the head-mounted device.
[0009] Figure 3 is a diagrammatic representation of a machine in the form of a computing device according to some examples within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.
[0010] Figure 4A is a collaboration diagram of a hand tracking input pipeline 430 of an AR system (eg, glasses 100 ) according to some examples.
[0011] Figure 4B is a diagram of a data structure according to some examples.
[0012] Figure 4C is an illustration of another data structure according to some examples.
[0013] Figure 5A is a sequence diagram of finger-spelling text input processing for an AR system according to some examples.
[0014] Figure 5B is an illustration of a start / stop text input gesture according to some examples.
[0015] Figure 5C is an illustration of fingerspelling sign language based on some examples.
[0016] Figure 5D is an illustration of an AR overlay according to some examples.
[0017] Figure 6 is a block diagram illustrating a software architecture in which the present disclosure may be implemented, according to some examples.
[0018] Figure 7 is a block diagram showing a networked system including details of a head-mounted AR system according to some examples.
[0019] Figure 8 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network according to some examples. DETAILED DESCRIPTION
[0020] When it comes to available user input modalities, AR systems are limited. Compared to other mobile devices, such as mobile phones, it is more complicated for users of AR systems to indicate user intent and invoke actions or applications. When using a mobile phone, a user can go to the home screen and click on a specific icon to launch an application. However, due to the lack of physical input devices such as a touch screen or keyboard, it is not easy to perform such interactions on an AR system. Typically, users can indicate their intent by pressing a limited number of hardware buttons or using a small touchpad. Therefore, it is desirable to have an input modality that allows a larger range of inputs that can be used by users to indicate their intent through user input.
[0021] Gestures are input modalities suitable for use with AR systems. Gestures are made by the user moving and positioning their hands and fingers, and the user's hands can be detected by the AR system if the user is wearing the AR system. The components of a gesture can include the movement of the user's hands and fingers, the positioning of the user's hands in space, and the position of the user's hands and fingers when they are held. Gestures are useful in providing an AR experience for users because they provide a way to provide user input to the AR system during the AR experience without the user shifting their attention away from the AR experience. As an example, in an AR experience that is an operating manual for a machine, the user can simultaneously view the machine in a real-world scene environment through the lens of the AR system, view an AR overlay on the real-world scene environment view of the machine, and provide user input to the AR system.
[0022] Finger spelling is a form of gesture in which the user makes fingerspelling signs with one hand to indicate letters of the alphabet and numbers to be entered into a text field in the AR experience. Other signs can be used to indicate editing commands for entered text.
[0023] In some examples, an AR system utilizes gesture recognition and finger spelling as an input modality to provide text input to an AR experience provided by the AR system. The AR system detects a start text input gesture made by a user and starts text input. The user makes a finger spelling sign language to enter characters and numbers into a text object that is an overlay of the AR experience. When the user finishes entering text, the user makes an end text input gesture to terminate the text input.
[0024] In another example, the system framework component determines whether the gesture or fingerspelling should be directed to the target AR application component, or whether the gesture or fingerspelling is a non-directional gesture or fingerspelling that should be routed to the system user interface component. The system framework component detects the start text input gesture and determines that subsequent gestures and fingerspelling data made by the user should be routed to the fingerspelling text input application component. The fingerspelling text input application component receives subsequent gestures and fingerspelling data made by the user.
[0025] Other technical features may be very obvious to those skilled in the art based on the accompanying drawings, description and claims.
[0026] Figure 1 According to some examples, a head mounted device (e.g., Figure 1100). The glasses 100 may include a frame 102 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first or left optical element holder 104 (e.g., a display or lens holder) and a second or right optical element holder 106 connected by a bridge 112. A first or left optical element 108 and a second or right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.
[0027] Frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, frame 102 can be formed from a single piece of material to have a unitary or unitary construction.
[0028] The glasses 100 may include a computing device such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include one or more processors as well as memory, wireless communication circuitry, and a power source. As discussed below, the computer 120 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 702 discussed below.
[0029] The computer 120 further includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.
[0030] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or multiple (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.
[0031] In some examples, left camera 114 and right camera 116 provide video frame data for glasses 100 to use to extract 3D information from the real-world scene environment scene.
[0032] The glasses 100 may also include a touchpad 126 mounted to one or both of the left temple piece 122 and the right temple piece 124, or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally arranged vertically, and in some examples, the touchpad 126 is approximately parallel to the temple of the user. As used herein, generally vertical alignment means that the touchpad is more vertical than horizontal, although potentially more vertical than the vertical. Additional user input can be provided by one or more buttons 128, and in the example shown, one or more buttons 128 are set on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide a way through which the glasses 100 can receive input from the user of the glasses 100.
[0033] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.
[0034] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .
[0035] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, a lens, or a prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 that overlays the view of the real-world scene environment seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 that overlays the view of the real-world scene environment seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides the optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's view of the real-world scene environment, including displaying a user interface to the user of the glasses 100.
[0036] However, it should be understood that other display technologies or configurations may be utilized within the optical engine to display images to the user in the user's field of view. For example, instead of the projector 204 and waveguide, an LCD, LED or other display panel or surface may be provided.
[0037] In use, information, content, and various user interfaces will be presented to the user of the glasses 100 on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Figure 7 The user may interact with the glasses 100 through voice input or touch input on a client device 726 shown in FIG. 1 and / or hand movements, position, and orientation detected by the glasses 100 .
[0038] Figure 3 is a diagrammatic representation of a machine 300 (e.g., a computing device) within which instructions 310 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 300 to perform any one or more of the methodologies discussed herein. The machine 300 may be used as Figure 1The computer 120 of the glasses 100. For example, the instructions 310 can cause the machine 300 to perform any one or more of the methods described herein. The instructions 310 convert a general, unprogrammed machine 300 into a specific machine 300 that is programmed to perform the functions described and shown in the described manner. The machine 300 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 300 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer (or distributed) network environment. The machine 300 may include, but is not limited to: a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, an AR system, a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 310 of the actions to be taken by the machine 300. Further, while a single machine 300 is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute instructions 310 to perform any one or more of the methodologies discussed herein.
[0039] Machine 300 may include processor 302, memory 304, and I / O component 306, which may be configured to communicate with each other via bus 344. In some examples, processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 308 and processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but machine 300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.
[0040] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via the bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. The instructions 310 may also reside, completely or partially, within the main memory 314, within the static memory 316, within the machine-readable medium 320 within the storage unit 318, within one or more of the processors 302 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 300.
[0041] I / O components 306 may include a variety of components to receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 306 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It will be appreciated that I / O components 306 may include Figure 3 306. In various examples, the I / O component 306 may include an output component 328 and an input component 332. The output component 328 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component 332 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides positioning and / or force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.
[0042] In another example, the I / O component 306 may include a biometric component 334, a motion component 336, an environmental component 338, or a positioning component 340, as well as various other components. For example, the biometric component 334 includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 336 may include an inertial measurement unit (IMU), an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 338 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an auditory sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects dangerous gas concentrations for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals associated with the surrounding physical environment. The positioning component 340 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.
[0043] A variety of technologies may be used to achieve communication. I / O components 306 also include communication components 342 that are operable to couple machine 300 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication components 342 may include a network interface component or another suitable device that interfaces with network 322. In other examples, communication components 342 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Components (e.g. Low power consumption), Device 324 may be another machine or any of a variety of peripherals (eg, a peripheral coupled via USB).
[0044] In addition, the communication component 342 can detect an identifier or include a component that is operable to detect an identifier. For example, the communication component 342 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode such as a universal product code (UPC) barcode, a multi-dimensional barcode such as a Quick Response (QR) code, an Aztec code, a data matrix, a data symbol (Dataglyph), a maximum code (MaxiCode), PDF417, UltraCode, UCC RSS-2D barcode, and other optical codes) or an acoustic detection component (e.g., a microphone for an audio signal of an identification tag). In addition, various information can be obtained via the communication component 342, such as positioning obtained via Internet Protocol (IP) geolocation, Positioning obtained by signal triangulation, positioning obtained by detecting NFC beacon signals that can indicate a specific location, etc.
[0045] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. When executed by processor 302, these instructions (e.g., instructions 310) cause various operations to implement the disclosed examples.
[0046] Instructions 310 may be sent or received via a network interface device (e.g., a network interface component included in communications component 342), utilizing a transmission medium, and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 322. Similarly, instructions 310 may be sent or received via a coupling 326 (e.g., a peer-to-peer coupling) to device 324 using a transmission medium.
[0047] Figure 4A is a collaboration diagram of a hand tracking input pipeline 430 of an AR system such as glasses 100, and Figure 4B and Figure 4C is a diagram of a data structure according to some examples. The camera component 402 utilizes one or more cameras of the AR system, such as Figure 1The cameras 114 and 116 generate real-world scene environment frame data 424 of the real-world scene environment from the perspective of the user of the AR system. The camera component 402 transmits the real-world scene environment frame data 424 to the skeleton model inference component 404. Included in the real-world scene environment frame data 424 is tracking video frame data of the user's hands and fingers. The tracking video frame data includes: video frame data of the movement of the user's hands and fingers when the user makes gestures or moves their hands and fingers to interact with the real-world scene environment; video frame data of the position where the user makes gestures or moves their hands and fingers to interact with the real-world scene environment; and video frame data of the position where the user keeps his upper body, arms, hands, and fingers when the user makes gestures or moves his hands and fingers to interact with the real-world scene environment.
[0048] The skeleton model inference component 404 receives the real-world scene environment frame data 424 and generates the skeleton model data 428 based on the real-world scene environment frame data 424. In some examples, the skeleton model inference component 404 extracts the features of the upper body, arms and hands of the user from the tracking video frame data included in the real-world scene environment frame data 424. The extracted features may include landmark data, and the landmark data includes landmark identification, positioning in the real-world scene environment, and classification information of one or more landmarks associated with the upper body, arms and hands of the user. The skeleton model inference component 404 generates the skeleton model data 428 based on the extracted features and one or more skeleton models previously created using a geometric method. In another example, the skeleton model inference component 404 generates the skeleton model data based on the classification of the real-world scene environment frame data 424 using an artificial intelligence method and one or more skeleton models previously generated using a machine learning method.
[0049] The hand classifier inference component 406 receives the skeletal model data 428 from the skeletal model inference component 404, and generates hand classifier probability data 426 based on the skeletal model data 428. In some examples, the gesture is specified by the hand tracking input pipeline 430 according to a combination of hand classifiers. The hand classifier is composed of a combination and relationship of landmarks included in the skeletal model data 428. In some examples, the hand classifier inference component 406 compares one or more skeletal models included in the skeletal model data 428 with a previously generated hand classifier model, and generates one or more hand classifier probabilities based on the comparison results. The one or more hand classifier probabilities indicate the probability that a specific hand classifier can be identified based on the skeletal model data 428. The hand classifier inference component 406 generates the hand classifier probability data 426 based on the one or more hand classifier probabilities. In another example, the hand classifier inference component 406 determines one or more hand classifier probabilities based on classifying the skeleton model using an artificial intelligence method and a hand classifier model previously generated using a machine learning method.
[0050] In some examples, the hand tracking input pipeline 430 extracts hand classifiers from the skeletal model data 428 by the hand tracking input pipeline 430 in a layer different from assembling hand movements into gestures. Thus, a designer of an AR application component executed by an AR system can create new gestures built on existing hand classifiers that constitute known gestures without having to retrain the machine learning component of the hand tracking input pipeline 430 or generate an implementation-specific hand classifier model. In some examples, the hand classifier model used to generate hand classifier probability data is not implementation-specific. That is, the hand classifier model is not trained to recognize only specific hand classifiers for the purposes of a particular implementation. Instead, the hand classifier model is trained to recognize as many hand classifiers as possible, without regard to what these hand classifiers represent in the context of a particular implementation (e.g., a task-specific AR application component).
[0051] The hand classifier inference component 406 transmits the hand classifier probability data 426 to the gesture inference component 408 and the gesture text input recognition component 410. The gesture inference component 408 receives the hand classifier probability data 426 and determines the gesture input event data 422 based on the hand classifier probability data 426. In some examples, the gesture inference component 408 compares the hand classifier identified in the hand classifier probability data 426 with the gesture identification data identifying the specific gesture. The gesture identification consists of one or more hand classifiers corresponding to the specific gesture. The gesture identification is defined using a grammar whose symbols correspond to the hand classifiers. For example, Figure 5BThe gesture identifier of gesture 522 is "left_palm_finger_stretch_right_palm_finger_stretch", where: "left" is a symbol corresponding to a hand classifier indicating that the user's left hand has been detected; "palm" is a symbol corresponding to a hand classifier indicating that the user's palm has been detected, and "left" is modified to indicate that the user's left palm has been detected; "finger" is a symbol corresponding to a hand classifier indicating that the user's fingers have been detected; and "stretch" is a symbol corresponding to a hand classifier indicating that the user's fingers are stretched and modifies "finger". In another example, a gesture identifier is a single marker, such as a number, that identifies a gesture based on the constituent hand classifiers of the gesture. The gesture identifier identifies the gesture in the context of a physical description of the gesture. The gesture inference component 408 transmits the gesture input event data 422 to the system framework component 414.
[0052] The gesture text input recognition component 410 receives the hand classifier probability data 426 and generates the symbol input event data 412 based on the hand classifier probability data 426. In some examples, the gesture inference component 408 compares the hand classifiers identified in the hand classifier probability data 426 with the symbol data identifying specific characters, words, and commands. For example, the symbol data of the gesture 526 is the character "V" as gesture 526, which is a finger spelling sign in American Sign Language (ASL). The individual hand classifiers for the "V" finger spelling sign 526 can be "left" for the left hand, "palm" for the palm of the left hand, "index finger" for the index finger, "extended" to modify the "index finger", "middle finger" for the middle finger, "extended" to modify the "middle finger", "ring finger" for the ring finger, "bend" to modify the "ring finger", "pinky finger" for the pinky finger, "bend" to modify the "pinky finger", "thumb" for the thumb, and "bend" to modify the "thumb".
[0053] In some examples, the gesture text input recognition component 410 can also identify entire words based on the hand classifiers indicated by the hand classifier probability data 426. In other embodiments, the gesture text input recognition component 410 can identify commands based on the hand classifiers indicated by the hand classifier probability data 426, for example, commands corresponding to a set of specified keystrokes in an input system having a keyboard.
[0054] The gesture inference component 408 and the gesture text input recognition component 410 transmit gesture input event data 422 and symbol input event data 412, respectively, to the system framework component 414. The system framework component 414 receives the gesture input event data 422 and the symbol input event data 412 (collectively referred to as "input event data", and may also be individually referred to as "input event data"), and classifies the gesture input event data 422 and the symbol output event data 412 as non-directional input event data 450 or directional input event data 452 based in part on the input event data. The non-directional input event belongs to the non-directional class of input events, and the system framework component 414 routes the non-directional event input data 450 to an operating system level component, such as a system user interface component 416, based on the classification of the non-directional input event data 450. The directional input event belongs to the directional class of input events, and the system framework component 414 routes the directional input event data 452 to a target component, such as an AR gesture application component 418, based on the classification of the directional input event data 452.
[0055] In an example of classifying and routing input event data received from the gesture inference component 408 and the gesture text input recognition component 410, the system framework component 414 classifies the input event data as non-directional input event data 450 based on the input event data and the component registration data described below. The system framework component 414 receives the input event data and determines the AR application component as the target AR application component based on the user 456's indication or selection of the virtual object 454 associated with the target AR gesture application component 418 when making a gesture corresponding to the input event data. The system framework component 414 determines the position of the user's hand in the real world scene environment when the user makes the gesture based on the skeletal model data 428. The system framework component 414 determines a set of virtual objects that the AR system currently provides to the user in the AR experience. The system framework component 414 determines the following virtual object, the apparent position of which in the real world scene environment is related to the position of the user's hand in the real world scene environment when the user 456 makes the gesture. The system framework component 414 determines a target AR gesture application component, such as the AR gesture application component 418, based on searching the AR gesture application component associated with the virtual object, such as the virtual object 454, in the internal data structure of the AR system, and determines the AR application component, such as the AR gesture application component 418, as the target AR application component.
[0056] In some examples, system framework component 414 registers component data such as Figure 4BThe component registration data 440 is stored in a data repository to be accessed during operation of the system framework component 404 to register a target AR application component to which the directional input event data 452 is to be routed. The component registration data 440 includes a component ID field 432 identifying a target AR application component, a registered language field 438 identifying a language model to be associated with the target AR application component, and one or more registered gesture fields 434 and / or registered symbol fields 436 indicating gestures and symbols to be routed to the registered AR application component. As shown, the component ID field 432 includes an AR application component identification "text input"; the registered language field 438 identifies the language associated with the registered AR application component, namely "English"; the registered gesture field 434 includes a gesture identification that should be routed to the registered target AR application component, namely "left_palm_finger_extend_right_palm_finger_extend", and a registered symbol field 436 identifying a set of symbols, namely "[*]" representing all symbols, that should be routed to the registered AR application component.
[0057] As another example of component registration data, Figure 4C The component registration data 442 includes a component ID field 444, which includes the AR application component identifier "email"; a registered language field 438 identifying the language associated with the registered AR application component, namely "English", and a registered symbol field 446 identifying a set of symbols that should be routed to the registered AR application component, namely the word "email".
[0058] Referring again to the system framework component 414 that processes the input event data received from the gesture inference component 408 and the gesture text input recognition component 410, the system framework component 414 classifies the input event data received from the gesture inference component 408 and the gesture text input recognition component 410 as non-directional input event data 450 or directional input event data 452 based on the input event data and the component registration data. In some examples, when processing the symbol input event data 412 included in the input event data, the system framework component 414 searches for a registered symbol matching the symbol input event data in a registered symbol field of the component registration data, such as the registered symbol field 436 of the component registration data 440. In the event that the system framework component 414 determines a match, the system framework component 404 classifies the symbol input event data as directional input event data 452. The system framework component 414 also determines a target AR application component based on the target AR application component identified in the component ID field of the component registration data including the matched registration symbol, such as the component ID field 432.
[0059] In a similar manner, when processing gesture input event data 422 included in the input event data, the system framework component 414 searches for a registered gesture that matches the gesture input event data in the registered gesture field of the component registration data, such as the registered gesture field 434 of the component registration data 440. In the event that the system framework component 414 determines a match, the system framework component 404 classifies the gesture input event data as directional input event data 452, and also determines the target AR application component to which the directional input event data 452 will be routed. In the event that the system framework component 414 determines that the symbol input event data and / or gesture input event data of the input event data is not found in the component registration data, the system framework component 414 determines that the input event data will be classified as non-directional input event data 450 and will be routed to the system user interface component 416.
[0060] In another example of processing directional input event data 452, an AR application component, such as the AR gesture application component 418, registers itself with the system framework component 414. To this end, the AR application component registers component registration data, such as Figure 4B The component registration data 440 is transmitted to the system framework component 414. The system framework component 414 receives the component registration data and stores the component registration data in a data repository for use in routing the directional input event data 452 to the AR application component.
[0061] In another example, the system framework component 414 classifies the input event data as directional input event data 452, and determines the target AR application component based on the execution context of the AR application component. For example, if the AR system is executing the current AR application component in a single application mode state, it is implied that the current AR application component is the target AR application component to which the directional input event data 452 should be routed. The system framework component 414 determines the target AR application component and classifies the input event data as directional input event data 452 based on the hint, and routes the directional input event data 452 to the target AR application component.
[0062] In some examples, the system framework component 414 transmits the language model feedback data 420 to the hand classifier inference component 406 and the gesture inference component 408 to improve the accuracy of the inferences made by the hand classifier inference component 406 and the gesture inference component 408. In some examples, the system framework component 414 generates the language model feedback data 420 based on user context data, such as component registration data of registered AR application components and data about hand classifiers that make up registered gestures and hand classifiers that make up gestures associated with registered symbols. The component registration data includes information about gestures and symbols in the gesture input event data 422 and the symbol input event data 412 that are routed to the AR application component as part of the directional input event data 452, and the language associated with the AR application component. In addition, the system framework component 414 includes information about the components of a particular gesture that includes a hand classifier associated with the gesture.
[0063] In another example of processing the language model feedback data 420, the system framework component 414 transmits a prompt as part of the language model feedback information 420 to the hand classifier inference component 406, the gesture inference component 408, and the gesture text input recognition component 410. The system framework component 414 generates the prompt based on the language model associated with the AR application component, such as the language specified in the registered language field 448 in the component registration data 442. The gesture text input recognition component 410 determines a possible next symbol N based on the previous characters N-1, N-2, etc. and the language model. In some examples, the system framework component 414 generates the prompt based on the language model as a hidden Markov model, which predicts what the next symbol N should be based on one or more previous characters in the previous characters N-1, N-2, etc. In another example, the gesture text input recognition component 410 generates the next symbol N based on the language model generated using the machine learning method using an AI method. The system framework component 414 generates the prompt based on the next symbol N. In some examples, the system framework component 404 determines a next gesture associated with the next symbol N by mapping the next symbol N to the next gesture based on a lookup table that associates symbols with gestures. The system framework component 414 decomposes the next gesture into a set of one or more next hand classifiers. The system framework component 414 transmits the next gesture to the gesture inference component 408 as part of the language model feedback data 420, and transmits the set of next hand classifiers to the hand classifier inference component 406 as part of the language model feedback data 420.
[0064] AR application components executed by the AR system, such as the system user interface component 416 and the AR gesture application component 418, are consumers of data generated by the hand tracking input pipeline 430, such as the gesture input event data 422 and the symbol input event data 412. The AR system executes the system user interface component 416 to provide a system-level user interface, such as a command console, etc., to a user of the AR system using gestures as an input modality. The AR system executes the AR gesture application component 418 to provide a user interface, such as an AR experience, to a user of the AR system using gestures as an input modality.
[0065] The system framework component 414 routes the input event data as the non-directional input event data 450 to the system user interface component 416 based on classifying the input event data as the non-directional input event data 450 .
[0066] In many examples, hand classifier inference component 406, gesture inference component 408, and gesture text input recognition component 410 each communicate hand classifier probability data 426, gesture input event data 422, and symbol input event data 412 via an inter-process communication (IPC) method.
[0067] In some examples, the hand tracking input pipeline 430 continuously generates and publishes symbol input event data 412 and gesture input event data 422 based on real-world scene environment frame data 424 generated by one or more cameras of the AR system.
[0068] Figure 5A is a sequence diagram of a finger spelling text input process 500 of an AR system such as glasses 100, Figure 5B is a diagram of a start / end text input gesture 522, Figure 5C is a graphic representation of fingerspelling sign language 524, and Figure 5D is an illustration of an AR overlay 530 according to some examples. During the fingerspelling text input process 500, the AR system utilizes the AR gesture application component 418 to implement the fingerspelling text input process 500 using a gesture recognition method.
[0069] The AR system utilizes the hand tracking input pipeline 430 to provide the symbol input event data 412 and the gesture input event data 422 to the AR gesture application component 418 as part of the directional input event data 452 . Figure 4AThe gesture text input recognition component 410 generates the symbol input event data 412 based on the finger spelling model and the hand classification symbol probability data 426. In some examples, the gesture text input recognition component 410 utilizes an AI method and a finger spelling model generated using a machine learning method to classify the hand classification symbol probability data 426 and assign the hand classification probability data 426 to finger spelling signs corresponding to letters or characters in the alphabet and numbers, such as Figure 5C The finger spelling sign language 524. The gesture text input recognition component 410 generates symbol input event data 412 based on the finger spelling sign language by mapping the finger spelling sign language to corresponding characters or numbers in the alphabet.
[0070] In another example, the gesture text input recognition component 410 compares the hand classifier probability data 426 with a previously determined set of finger spelling models, wherein the member finger spelling models of the set of finger spelling models correspond to individual finger spelling signs. The gesture text input recognition component 410 selects the finger spelling sign corresponding to the hand classifier probability information 426 based on the comparison result, and generates symbol input event data 412 based on the finger spelling sign by mapping the finger spelling sign to a corresponding character or number in the alphabet.
[0071] In some examples, user 534 makes additional gestures and sign language (not shown) to indicate text editing commands. User 534 uses these commands to instruct the AR system to perform editing functions on the input text.
[0072] During the use of the AR system by the user 534, when the user 534 wants to input text into a virtual object such as the text scene component 502 of the AR gesture application component such as the AR gesture application component 418, the user 534 creates an input event by making a start text input gesture 546 such as the start / end text input gesture 522. In operation 504, the hand tracking input pipeline 430 generates gesture input event data 422 based on the real world scene environment video frame data of the start text input gesture 546 made by the user 534 as described herein. The hand tracking input pipeline 430 transmits the gesture input event data 422 to the system framework component 414.
[0073] In loop operation 536, as indicated by guard condition 538, system framework component 414 receives and processes each input event included in gesture input event data 422 and symbol input event data 412 received from hand tracking input pipeline 430. In operation 540, system framework component 414 processes the gesture input event data 422 and component registration data, such as the hand tracking input pipeline 430, as described herein. Figure 4BThe component registration data 440 is used to generate the directional input event data 452. In operation 542, the system framework component 414 routes the directional input event data 452 to the AR gesture application component 418.
[0074] In operation 506, the AR gesture application component 418 receives the directional input event data 452 of the gesture input event data 422 including the start text input gesture 546 made by the user 534, and detects that the user 534 is making the start text input gesture 546 based on the directional input event data 452. Based on detecting the start text input gesture 546, the AR gesture application component 418 starts inputting text into the text scene component 502.
[0075] The user 534 makes one or more finger spelling signs 524, and in operation 504, the hand tracking input pipeline 430 generates symbol input event data 412 of the finger spelling sign 524 based on the real world scene environment video frame data generated by one or more cameras as described herein. The hand tracking input pipeline 430 transmits the symbol input event data 412 to the system framework component 414.
[0076] In operation 540 of loop operation 536 , system framework component 414 generates directional input event data 452 based on symbol input event data 412 and component registration data as described herein. In operation 542 , system framework component 414 routes directional input event data 452 including symbol input event data 412 to AR gesture application component 418 .
[0077] In loop operation 508 , as indicated by guard condition 510 , AR gesture application component 418 performs loop operation 508 until AR gesture application component 418 detects that the user is making an end text input gesture 544 , such as start / end text input gesture 522 .
[0078] In alternative operation 512, the AR gesture application component 418 receives the directional input event data 452, and as indicated by the protection condition 514, detects one or more input symbols of the symbol input event data 412 in the directional input event data 452. The one or more input symbols correspond to one or more finger spelling signs 524 being made by the user 534. The one or more input symbols include characters, numbers, and editing commands corresponding to letters in the alphabet. Based on the detection of the input symbol, in operation 516, the AR gesture application component 418 generates input text data 532 based on the one or more input symbols, and transmits the input text data 532 to the text scene component 502. In operation 518, the text scene component 502 provides the user with input text 528 in the text scene component 501 based on the input text data 532.
[0079] In some examples, the one or more input symbols correspond to one or more editing commands for the text input by the user 534. Based on determining that the one or more input symbols correspond to the one or more editing commands, the AR gesture application component 418 performs one or more editing functions on the input text 528, such as a cut operation, a copy operation, a paste operation, etc.
[0080] When the user 534 completes inputting text into the text scene component 502, the user 534 makes an end text input gesture 544, such as the start / end text input gesture 522. In operation 504, the hand tracking input pipeline 430 generates gesture input event data 422 based on the real-world scene environment video frame data of the end text input gesture 544 made by the user 534, as described herein. The hand tracking input pipeline 430 transmits the gesture input event data 422 to the system framework component 414.
[0081] In operation 540 , system framework component 414 generates directional input event data 452 including gesture input event data 422 based on gesture input event data 422 and component registration data as described herein. In operation 542 , system framework component 414 routes directional input event data 452 to AR gesture application component 418 .
[0082] In loop operation 508, AR gesture application component 418 receives directional input event data 452. In alternative operation 512, as indicated by guard condition 520, the AR system detects an end text input gesture 544 made by user 534 based on gesture input event data 422 of directional input event data 452. Based on detecting the end text input gesture 544, AR gesture application component 418 terminates fingerspelling text input process 500.
[0083] In some examples, AR gesture application component 418 of the AR system performs the functions of hand tracking input pipeline 430 and system framework component 414 by utilizing various APIs and system libraries of the AR system.
[0084] Figure 6 600 is a block diagram 604 illustrating a software architecture 604 that may be installed on any one or more of the devices described herein. The software architecture 604 is supported by hardware, such as a machine 602 including a processor 620, a memory 626, and an I / O component 638. In this example, the software architecture 604 may be conceptualized as a stack of layers, wherein each layer provides a specific functionality. The software architecture 604 includes layers such as an operating system 612, a library 608, a framework 610, and an application 606. In operation, the application 606 invokes an API call 650 through the software stack and receives a message 652 in response to the API call 650.
[0085] The operating system 612 manages hardware resources and provides public services. The operating system 612 includes, for example, a kernel 614, services 616, and drivers 622. The kernel 614 acts as an abstraction layer between the hardware and other software layers. For example, the kernel 614 provides functions such as memory management, processor management (e.g., scheduling), component management, network and security settings. Services 616 can provide other public services for other software layers. Drivers 622 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 622 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.
[0086] The library 608 provides a low-level common infrastructure used by the application 606. The library 608 may include a system library 618 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 608 may include an API library 624, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering in two-dimensional (2D) and three-dimensional (3D) graphics content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 608 may also include a variety of other libraries 628 to provide many other APIs to the application 606 .
[0087] The framework 610 provides a high-level common infrastructure used by the applications 606. For example, the framework 610 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 610 may provide a wide range of other APIs used by the applications 606, some of which may be specific to a particular operating system or platform.
[0088] In some examples, applications 606 may include home applications 636, contact applications 630, browser applications 632, book reader applications 634, location applications 642, media applications 644, messaging applications 646, game applications 648, and a broad category of other applications such as third-party applications 640. Applications 606 are programs that execute functions defined in the program. Various programming languages may be used to create one or more of the applications 606 constructed in various ways, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language). In a specific example, third-party applications 640 (e.g., applications created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM , Mobile software running on the mobile operating system of the Phone or another mobile operating system. In this example, third-party applications 640 can activate API calls 650 provided by the operating system 612 to facilitate the functions described herein.
[0089] Figure 7 is a block diagram showing a networked system 700 including details of the glasses 100 according to some examples. The networked system 700 includes the glasses 100, a client device 726, and a server system 732. The client device 726 may be a smart phone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 736 and / or a high-speed wireless connection 734. The client device 726 is connected to the server system 732 via a network 730. The network 730 may include any combination of wired and wireless connections. The server system 732 may be one or more computing devices that are part of a service or network computing system. The client device 726 and any elements of the server system 732 and the network 730 may be configured using the respective wireless devices in the embodiment of the present invention. Figure 6 and Figure 3 The details of the software architecture 604 or machine 300 described in are implemented.
[0090] The glasses 100 include a data processor 702, a display 710, one or more cameras 708, and additional input / output elements 716. The input / output elements 716 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 702. Figure 6 and Figure 3 Examples of input / output elements 716 are further discussed. For example, input / output elements 716 may include any of the I / O components in I / O components 306, including output components 328, motion components 336, etc. Figure 2 Examples of display 710 are discussed in . In the particular examples described herein, display 710 includes displays for the user's left eye and right eye.
[0091] Data processor 702 includes image processor 706 (eg, video processor), GPU and display driver 738, tracking module 740, interface 712, low power circuitry 704, and high speed circuitry 720. Components of data processor 702 are interconnected by bus 742.
[0092] The interface 712 refers to any source of user commands provided to the data processor 702. In one or more examples, the interface 712 is a physical button that sends a user input signal from the interface 712 to the low-power processor 714 when pressed. The low-power processor 714 can process the release of such a button immediately after pressing as a request to capture a single image, and vice versa. The low-power processor 714 can process pressing such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 712 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the camera 708. In other examples, the interface 712 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 726.
[0093] The image processor 706 includes circuitry for receiving signals from the camera 708 and processing those signals from the camera 708 into a format suitable for storage in the memory 724 or for transmission to the client device 726. In one or more examples, the image processor 706 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from the camera 708, and volatile memory used by the microprocessor in operation.
[0094] The low power circuit system 704 includes a low power processor 714 and a low power wireless circuit system 718. These elements of the low power circuit system 704 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 714 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 714 can accept user input signals from the interface 712. The low power processor 714 can also be configured to receive input signals or command communications from the client device 726 via the low power wireless connection 736. The low power wireless circuit system 718 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 718. In other examples, other low power communication systems may be used.
[0095] High-speed circuit system 720 includes a high-speed processor 722, a memory 724, and a high-speed wireless circuit system 728. High-speed processor 722 can be any processor capable of managing high-speed communications and operations for any general-purpose computing system for data processor 702. High-speed processor 722 includes processing resources for managing high-speed data transmission over high-speed wireless connection 734 using high-speed wireless circuit system 728. In some examples, high-speed processor 722 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Figure 6 The high-speed processor 722, which executes the software architecture of the data processor 702, is used to manage data transmission with the high-speed wireless circuit system 728, in addition to any other duties. In some examples, the high-speed wireless circuit system 728 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented by the high-speed wireless circuit system 728.
[0096] The memory 724 includes any storage device capable of storing camera data generated by the camera 708 and the image processor 706. Although the memory 724 is shown as being integrated with the high-speed circuit system 720, in other examples, the memory 724 may be a separate, independent element of the data processor 702. In some such examples, electrical wiring may provide a connection from the image processor 706 or the low-power processor 714 to the memory 724 through a chip including the high-speed processor 722. In other examples, the high-speed processor 722 may manage the addressing of the memory 724 so that the low-power processor 714 will initiate the high-speed processor 722 any time a read or write operation involving the memory 724 is desired.
[0097] The tracking module 740 estimates the pose of the glasses 100. For example, the tracking module 740 uses image data and associated inertial data from the camera 708 and the positioning component 340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene environment). The tracking module 740 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world scene environment. The tracking module 740 allows the glasses 100 to visualize the placement of virtual objects relative to physical objects within the user's field of view via the display 710.
[0098] The GPU and display driver 738 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 710 when the glasses 100 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 738 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the position and orientation of the user relative to physical objects in the user's real-world scene environment.
[0099] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 726 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 606, such as the messaging application 646.
[0100] Figure 8 800 is a block diagram illustrating an example messaging system 800 for exchanging data (e.g., messages and associated content) over a network. The messaging system 800 includes a client device 726 hosting multiple instances of several applications including a messaging client 802 and other applications 804. The messaging client 802 is communicatively coupled to other instances of the messaging client 802 (e.g., hosted on respective other client devices 726), a messaging server system 806, and a third-party server 808 via a network 730 (e.g., the Internet). The messaging client 802 can also communicate with the locally hosted application 804 using an application program interface (API).
[0101] The messaging clients 802 are able to communicate and exchange data with other messaging clients 802 and with a messaging server system 806 via the network 730. The data exchanged between the messaging clients 802 and between the messaging clients 802 and the messaging server system 806 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).
[0102] The messaging server system 806 provides server-side functionality to certain messaging clients 802 via the network 730. Although some functions of the messaging system 800 are described herein as being performed by either the messaging client 802 or the messaging server system 806, the location of certain functions within the messaging client 802 or within the messaging server system 806 may be a design choice. For example, it may be technically preferable to initially deploy some technologies and functions within the messaging server system 806, but then migrate the technologies and functions to the messaging client 802 where the client device 726 has sufficient processing power.
[0103] The messaging server system 806 supports various services and operations provided to the messaging clients 802. Such operations include transmitting data to the messaging clients 802, receiving data from the messaging clients 802, and processing data generated by the messaging clients 802. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 800 is activated and controlled by functions available via the user interface (UI) of the messaging client 802.
[0104] Turning now specifically to the messaging server system 806, an application program interface (API) server 810 is coupled to and provides a programming interface to an application server 814. The application server 814 is communicatively coupled to a database server 816, which facilitates access to a database 820 that stores data associated with messages processed by the application server 814. Similarly, a web server 824 is coupled to and provides a web-based interface to the application server 814. To this end, the web server 824 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.
[0105] The application program interface (API) server 810 receives and transmits message data (e.g., commands and message payloads) between the client device 726 and the application server 814. Specifically, the application program interface (API) server 810 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 802 to activate the functions of the application server 814. The application program interface (API) server 810 exposes various functions supported by the application server 814, including: account registration; login functionality; sending messages from a particular messaging client 802 to another messaging client 802 via the application server 814, sending media files (e.g., images or videos) from the messaging client 802 to the messaging server 812, and possible access by another messaging client 802; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 726; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 802).
[0106] The application server 814 hosts multiple server applications and subsystems, including, for example, a messaging server 812, an image processing server 818, and a social network server 822. The messaging server 812 implements multiple message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 802. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 802. In view of the hardware requirements for other processor and memory intensive processing of data, such processing can also be performed on the server side by the messaging server 812.
[0107] The application server 814 also includes an image processing server 818 that is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 812 .
[0108] The social network server 822 supports various social networking functions and services and makes these functions and services available to the messaging server 812. To this end, the social network server 822 maintains and accesses an entity graph within the database 820. Examples of functions and services supported by the social network server 822 include identifying other users with whom a particular user in the messaging system 800 has a relationship or that the particular user is "following", and also include identifying other entities and interests of a particular user.
[0109] The messaging client 802 may notify the user of the client device 726 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 802 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 802 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.
[0110] Additional examples include:
[0111] Example 1 is a computer-implemented method for text input processing, the method comprising: one or more processors of an augmented reality (AR) system using one or more cameras of the AR system to detect one or more input symbols corresponding to finger spelling sign language made by a user of the AR system; one or more processors generating input text data based on the one or more input symbols; and one or more processors providing text in a text scene component of an AR application component of the AR system based on the input text data.
[0112] In Example 2, the subject matter of Example 1 includes: detecting, by one or more processors, a start text input gesture made by a user of the AR system using one or more cameras of the AR system; initiating, by one or more processors, text input processing based on detecting the start text input gesture; detecting, by one or more processors, an end text input gesture made by the user using one or more cameras of the AR system; and terminating, by one or more processors, text input processing based on detecting the end text input gesture.
[0113] In Example 3, the subject matter of any one of Examples 1-2 includes, wherein the start text input gesture and the end text input gesture are the same gesture.
[0114] In Example 4, the subject matter of any one of Examples 1 to 3 includes, wherein detecting a start text input gesture further includes: determining the AR application component for text input processing as a target AR application component for one or more input symbols based on a user's indication of a virtual object associated with the AR application component when making the start text input gesture.
[0115] In Example 5, the subject matter of any one of Examples 1 to 4 includes, wherein detecting one or more input symbols corresponding to finger spelling sign language made by a user also includes: classifying the symbol input event data as directional input event data based on the symbol input event data of the one or more input symbols and the component registration data of the AR application component; and routing the directional input event data to the AR application component.
[0116] In Example 6, the subject matter of any of Examples 1 to 5 includes, wherein the one or more input symbols include one or more editing commands, and wherein generating the input text data based on the one or more input symbols also includes performing an editing function on the input text data.
[0117] In Example 7, the subject matter of any one of Examples 1 to 6 includes, wherein the AR system includes a head mounted device.
[0118] Example 8 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-7.
[0119] Example 9 is an apparatus comprising means for implementing any of Examples 1 to 7.
[0120] Example 10 is a system for implementing any of Examples 1 to 7.
[0121] Example 11 is a method for implementing any of Examples 1 to 7.
[0122] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate transmission of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.
[0123] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.
[0124] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.
[0125] "Component" refers to a device, physical entity or logic with boundaries defined by function or subroutine calls, branch points, APIs or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be encapsulated functional hardware units designed for use with other components, and are usually part of a program that performs a specific function of a related function. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform some operations and can be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform hardware components of some operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform some operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform some operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be driven for cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform some operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), the hardware component may not be configured or instantiated at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured at different times to be respectively different special purpose processors (e.g., including different hardware components). Software configures one or more specific processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between two or more hardware components or among two or more hardware components. In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., a collection of information). The various operations of the example methods described herein can be performed by temporary configuration (e.g., by software) or permanently configured to perform one or more processors of related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that performs operations to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented in part by a processor, wherein one or more specific processors are examples of hardware. For example, some operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, resident in a single machine and deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.
[0126] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.
[0127] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Thus, the term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media".
[0128] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.
[0129] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.
[0130] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.
Claims
1. A computer-implemented method for text input processing, the method comprising: detecting, by one or more processors of an augmented reality (AR) system, using one or more cameras of the AR system, one or more input symbols corresponding to fingerspelling sign language being performed by a user of the AR system; generating, by the one or more processors, input text data based on the one or more input symbols; as well as Text is provided, by the one or more processors, in a text scene component of an AR application component of the AR system based on the input text data.
2. The computer-implemented method of text input processing according to claim 1 further comprising: detecting, by the one or more processors, using one or more cameras of the AR system, a start text input gesture by a user of the AR system; initiating, by the one or more processors, the text input process based on detecting the start text input gesture; Detecting, by the one or more processors, a gesture of ending text input made by the user using one or more cameras of the AR system; as well as The text input process is terminated, by the one or more processors, based on detecting the end text input gesture.
3. The computer-implemented method of text input processing according to claim 2, wherein: The start text input gesture and the end text input gesture are the same gesture.
4. The computer-implemented method of text input processing according to claim 1, wherein: Detecting the start text input gesture further includes: Based on the user's indication of a virtual object associated with the AR application component when making the start text input gesture, the AR application component of the text input process is determined as a target AR application component of the one or more input symbols.
5. The computer-implemented method of text input processing according to claim 1, wherein: Detecting one or more input symbols corresponding to the fingerspelling sign language being performed by the user further includes: Based on the symbol input event data of the one or more input symbols and the component registration data of the AR application component, classifying the symbol input data as directional input event data; and The directional input event data is routed to the AR application component.
6. The computer-implemented method of text input processing according to claim 1, in, The one or more input symbols include one or more editing commands, and Wherein, generating the input text data based on the one or more input symbols further comprises performing an editing function on the input text data.
7. The computer-implemented method of text input processing according to claim 1, wherein: The AR system includes a head-mounted device.
8. A computing device, comprising: one or more processors; as well as A memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations of text input processing, the operations comprising: Detecting, by the one or more processors, one or more input symbols corresponding to fingerspelling sign language being performed by the user using one or more cameras of the AR system; generating, by the one or more processors, input text data based on the one or more input symbols; and Text is provided, by the one or more processors, in a text scene component of an AR application component of the AR system based on the input text data.
9. The computing device according to claim 8, wherein: The instructions also cause the computing device to perform operations including: detecting, by the one or more processors, utilizing the one or more cameras, a start text input gesture by a user of the AR system; initiating, by the one or more processors, the text input process based on detecting the start text input gesture; Detecting, by the one or more processors, a gesture of ending text input made by the user using one or more cameras of the AR system; as well as The text input process is terminated, by the one or more processors, based on detecting the end text input gesture.
10. The computing device according to claim 9, wherein: The start text input gesture and the end text input gesture are the same gesture.
11. The computing device according to claim 8, wherein: The instructions causing the computing device to perform detection of the start text input gesture further cause the computing device to perform operations including: The one or more processors determine, based on the user's indication of a virtual object associated with the AR application component when making the start text input gesture, the AR application component of the text input processing as a target AR application component of the one or more input symbols.
12. The computing device according to claim 8, wherein: The instructions causing the computing device to perform detection of the one or more input symbols further cause the computing device to perform operations including: classifying, by the one or more processors, the symbol input event data as directional input event data based on the symbol input event data of the one or more input symbols and the component registration data of the AR application component; as well as The directional input event data is routed to the AR application component.
13. The computing device according to claim 8, in, The one or more input symbols include one or more editing commands, and The instruction for causing the computing device to generate the input text data further causes the computing device to perform operations including: An editing function is performed by the one or more processors on the input text data.
14. The computing device of claim 8, wherein: The AR system includes a head-mounted device.
15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations of text input processing, the operations comprising: Detecting, by one or more processors of the AR system, one or more input symbols corresponding to the fingerspelling sign language being performed by the user using one or more cameras of the AR system; generating, by the one or more processors, input text data based on the one or more input symbols; as well as Text is provided, by the one or more processors, in a text scene component of an AR application component of the AR system based on the input text data.
16. The non-transitory computer readable storage medium of claim 15, wherein: The instructions also cause the computer to perform operations including: detecting, by the one or more processors, utilizing the one or more cameras, a start text input gesture by a user of the AR system; initiating, by the one or more processors, the text input process based on detecting the start text input gesture; Detecting, by the one or more processors, a gesture of ending text input made by the user using one or more cameras of the AR system; as well as The text input process is terminated, by the one or more processors, based on detecting the end text input gesture.
17. The non-transitory computer readable storage medium of claim 16, wherein: The start text input gesture and the end text input gesture are the same gesture.
18. The non-transitory computer readable storage medium of claim 15, wherein: The instructions causing the computer to perform detection of the start text input gesture further cause the computer to perform operations including: The one or more processors determine, based on the user's indication of a virtual object associated with the AR application component when making the start text input gesture, the AR application component of the text input processing as a target AR application component of the one or more input symbols.
19. The non-transitory computer-readable storage medium of claim 15, wherein: The instructions causing the computer to perform detection of the one or more input symbols further cause the computer to perform operations comprising: classifying, by the one or more processors, the symbol input event data as directional input event data based on the symbol input event data of the one or more input symbols and component registration data of the AR application component; as well as The directional input event data is routed by the one or more processors to the AR application component.
20. The non-transitory computer readable storage medium of claim 15, in, The one or more input symbols include one or more editing commands, and The instruction for causing the computer to generate the input text data further causes the computer to perform operations including the following operations: An editing function is performed on the input text data.
21. The non-transitory computer readable storage medium of claim 15, wherein: The AR system includes a head-mounted device.
