Finger gesture recognition via acousto-optic sensor fusion

Abstract

A finger gesture recognition system is provided. A finger gesture recognition system includes one or more audio sensors and one or more optical sensors. A finger gesture recognition system uses one or more audio sensors to capture audio signal data of a finger gesture being made by a user, and one or more optical sensors to capture optical signal data of the finger gesture. The finger gesture recognition system recognizes a finger gesture based on the audio signal data and the optical signal data, and transmits finger gesture data of the recognized finger gesture to an augmented reality / combined reality / virtual reality (XR) application.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 375,743, filed on September 15, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to user interfaces, and more particularly to user interfaces used in augmented reality and virtual reality. Background Art

[0004] A head mounted device may be implemented with a transparent or translucent display through which the user of the head mounted device can view the surrounding environment. Such a device enables the user to view the surrounding environment through the transparent or translucent display, and also to see objects (e.g., virtual objects, renderings, images, videos, text, etc., such as 2D or 3D graphics models) generated for display to appear to be part of the surrounding environment and / or superimposed on the surrounding environment. This is generally referred to as "augmented reality" or "AR". The head mounted device may additionally completely block the user's field of view and display a virtual environment that the user can move through or be moved through. This is generally referred to as "virtual reality" or "VR". In a hybrid form, a camera is used to capture a view of the surrounding environment, and then the view is displayed to the user together with the augmentation on a display that blocks the user's eyes. As used herein, unless the context indicates otherwise, the term extended reality (XR) refers to augmented reality, virtual reality, and any mixture of these technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] To easily identify the discussion of any particular element or action, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0006] Figure 1 is a perspective view of a head mounted device according to some examples.

[0007] Figure 2 According to some examples Figure 1 Additional views of the head-mounted device.

[0008] Figure 3 is a diagrammatic representation of a machine according to some examples within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.

[0009] Figure 4 is an illustration of a finger gesture recognition system according to some examples of the present disclosure.

[0010] Figure 5 is an illustration of a set of finger gestures according to some examples of the present disclosure.

[0011] Fig. 6A , Figure 6B and Figure 6C A wrist-worn XR input device incorporating components of a finger gesture recognition system according to some examples of the present disclosure is shown.

[0012] Fig. 7A shows a multimodal finger gesture recognition component, and Figure 7B Methods of operation of a finger gesture recognition system according to some examples are shown.

[0013] Figure 8 is a block diagram illustrating a software architecture in which the present disclosure may be implemented, according to some examples.

[0014] Fig. 9 is a block diagram showing a networked system including details of a head-mounted AR system according to some examples.

[0015] Fig.10 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network according to some examples. DETAILED DESCRIPTION

[0016] Finger gesture recognition is a user input modality for users to interact with AR / VR applications because performing finger gestures is natural and gives users an immersive experience. Recent years have witnessed an explosive growth in XR (i.e., AR, VR, and mixed reality) applications, which have greatly improved productivity and user experience in many scenarios such as remote collaboration, design, entertainment, etc. Although the interactive space is freely extended to the reachable range, XR interaction is still limited and requires great effort from users. Users need to raise their hands to reach the touch surface on a small 2D device or the limited visual space covered by the field of view of a head-mounted tracking camera. Or the user holds a dedicated controller and remembers the corresponding operation of each button.

[0017] Since wrist-worn XR input devices can continuously follow the hand without the need for hand grasping to capture gesture interactions, wrist-worn XR input devices can provide natural and low-input XR interactions. Some wrist-worn XR input devices infer finger gestures by sensing part of the hand (e.g., palm and back of hand) directly from the wrist. The modality used in this solution can directly sense part of the hand, but gesture inference requires most of the hand to be visible, which results in wearing uncomfortable raised off-wrist sensors, or multiple cameras consuming a lot of computing resources. Other wrist-worn XR devices infer finger gestures by capturing physiological signals caused by gestures on the wrist (including wrist deformation, wrist muscle electrical activity, wrist inertia). Compared with other modalities, such user input modalities have complete information of finger gestures. However, these modalities are complex and susceptible to interference (e.g., capacitance and EMG), resulting in limited granularity. In addition, such wrist-worn XR input devices are typically limited to dozens of gestures.

[0018] In some examples, a finger gesture recognition system provides fine-grained gesture recognition. The finger gesture recognition system is built on high-speed, low-power optical motion sensors and improved audio sensors. The optical sensors capture the movement of hand muscles, and the audio sensors are sensitive to the movement of tendons. Since fine-grained gesture sensing can be achieved with a small number of small form factor sensors, the finger gesture recognition system is lightweight, and since finger gesture recognition does not rely on deep learning models, finger gesture recognition is efficient, which brings advantages to gesture recognition in XR applications.

[0019] In some examples, a hands-free fine-grained finger gesture recognition system uses an audio sensor and an optical sensor in the form of a wristband-based XR input device. The finger gesture recognition system includes a sensor fusion network of audio signals and optical signals. The finger gesture recognition system includes a wristband with an audio sensor and a high-speed low-power optical motion sensor to capture in real time audio signals and optical signals from a user during finger gestures performed by the user while interacting with an AR / VR application. The finger gesture recognition system recognizes a set of finger gestures that can be reliably detected from complementary signals generated by the audio sensor and the optical sensor and that can be performed naturally by the user.

[0020] In some examples, the finger gesture recognition system uses a multimodal CNN-Transformer based model for discrete gesture recognition (including flick, pinch, and tap). The finger gesture recognition system detects contact between the thumb and index finger through a contact detection model and continuously tracks the thumb sliding gesture in real time to achieve fine-grained control.

[0021] In some examples, the finger gesture recognition system is suitable for integration into an existing wrist-worn device, such as a smart watch or fitness band.

[0022] Features of finger gesture recognition systems according to some examples of the present disclosure:

[0023] A clear set of finger gestures.

[0024] A multimodal CNN-Transformer model for window-level signal classification.

[0025] Aggregation model for robust pose detection.

[0026] ·Level design to suppress false alarms.

[0027] Other technical features may be very obvious to those skilled in the art based on the accompanying drawings, description and claims.

[0028] Figure 1 is a head mounted AR system according to some examples (e.g., Figure 1 100). The glasses 100 may include a frame 102, which is made of any suitable material such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 102 includes a first optical element holder or a left optical element holder 104 (e.g., a display or lens holder) and a second optical element holder or a right optical element holder 106 connected by a bridge 112. A first optical element or a left optical element 108 and a second optical element or a right optical element 110 may be disposed in the left optical element holder 104 and the right optical element holder 106, respectively. The right optical element 110 and the left optical element 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0029] Frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0030] The glasses 100 may include a computing system such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computing device may be of a suitable size and shape to be partially disposed in one of the temple pieces 122 or temple pieces 124. The computer 120 may include multiple processors, memories, and various communication components that share a common power supply. As discussed below, the various components of the computer 120 may include low-power circuitry, high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 902 discussed below.

[0031] The computer 120 further includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0032] The glasses 100 include a first or left camera 114 and a second or right camera 116. Although two cameras are depicted, other examples contemplate the use of a single or multiple (i.e., more than two) cameras. In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left camera 114 and the right camera 116. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0033] In some examples, left camera 114 and right camera 116 provide video frame data for glasses 100 to use to extract 3D information from the real-world scene environment scene.

[0034] The glasses 100 may also include a touchpad 126 mounted to one or both of the left temple piece 122 and the right temple piece 124, or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is generally arranged vertically, and in some examples, the touchpad 126 is approximately parallel to the temple of the user. As used herein, generally vertical alignment means that although the touchpad is relatively vertical compared to the horizontal, it is potentially more vertical than the horizontal. Additional user input can be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. One or more touchpads 126 and buttons 128 provide a path through which the glasses 100 can receive input from the user of the glasses 100.

[0035] Figure 2 The glasses 100 are shown from the user's perspective. Figure 1 Many of the components shown in FIG. have been omitted. Figure 1 As described, Figure 2 The illustrated eyeglasses 100 include left and right optical elements 108, 110 secured within left and right optical element holders 104, 106, respectively.

[0036] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0037] In some examples, the near-eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, a lens, or a prism). Light 208 emitted by projector 204 encounters the diffractive structure of the waveguide of near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 that overlays the view of the real-world scene environment seen by the user. Similarly, light 214 emitted by projector 212 encounters the diffractive structure of the waveguide of near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 that overlays the view of the real-world scene environment seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides the optical engine of the glasses 100. The glasses 100 use the optical engine to generate an overlay of the user's view of the real-world scene environment, including displaying a user interface to the user of the glasses 100.

[0038] However, it should be understood that other display technologies or configurations may be utilized within the optical engine to display images to the user in the user's field of view. For example, instead of the projector 204 and waveguide, an LCD, LED or other display panel or surface may be provided.

[0039] In use, information, content, and various user interfaces will be presented to the user of the glasses 100 on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g. Fig. 9 The user may interact with the glasses 100 through voice input or touch input on a client device 926 shown in FIG. 1 and / or hand movements, position, and orientation detected by the glasses 100 .

[0040] In some examples, glasses 100 include a standalone AR system that provides an AR experience to a user of glasses 100. In some examples, glasses 100 are a component of an AR system that includes one or more other devices that provide additional computing resources and / or additional user input and output resources. The other devices may include a smartphone, a general-purpose computer, etc.

[0041] Figure 3 is a diagrammatic representation of a machine 300 within which instructions 310 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 300 to perform any one or more of the methodologies discussed herein. The machine 300 may be used as an AR system (e.g., Figure 1The computer 120 of the glasses 100) can be configured to be connected to the computer 120 of the glasses 100. For example, the instructions 310 can cause the machine 300 to perform any one or more of the methods described herein. The instructions 310 convert a general, unprogrammed machine 300 into a specific machine 300 that is programmed to perform the functions described and illustrated in the manner described. The machine 300 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 300 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 300 in combination with other components of the AR system may be used as, but not limited to, a server, client, computer, personal computer (PC), tablet computer, laptop computer, netbook, set-top box (STB), PDA, entertainment media system, cellular phone, smart phone, mobile device, head-mounted device (e.g., smart watch), smart home device (e.g., smart appliance), other smart device, web appliance, network router, network switch, network bridge, or any machine capable of sequentially or otherwise executing instructions 310 for actions to be taken by the machine 300. In addition, while a single machine 300 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute instructions 310 to perform any one or more of the methodologies discussed herein.

[0042] Machine 300 may include processor 302, memory 304, and I / O device interface 306, which may be configured to communicate with each other via bus 344. In an example, processor 302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 308 and processor 312 that execute instructions 310. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 3 Multiple processors 302 are shown, but machine 300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0043] The memory 304 includes a main memory 314, a static memory 316, and a storage unit 318, which are all accessible by the processor 302 via a bus 344. The main memory 304, the static memory 316, and the storage unit 318 store instructions 310 that implement any one or more of the methods or functions described herein. The instructions 310 may also reside, in whole or in part, within the main memory 314, within the static memory 316, within a non-transitory machine-readable medium 320 within the storage unit 318, within one or more of the processors 302 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution of the instructions by the machine 300.

[0044] I / O device interface 306 couples machine 300 to I / O devices 346. One or more of I / O devices 346 may be components of machine 300 or may be separate devices. I / O device interface 306 may include various interfaces to I / O devices 346 used by machine 300 to receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O device interface 306 included in a particular machine will depend on the type of machine. It will be appreciated that I / O device interface 306 may include I / O devices 346. Figure 3 306. In various examples, the I / O device interface 306 may include an output component interface 328 and an input component interface 332. The output component interface 328 may include an interface to a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component interface 332 may include an interface to an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides positioning and / or force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0045] In another example, the I / O device interface 306 may include a biometric component interface 334, a motion component interface 336, an environmental component interface 338, or a positioning component interface 340, as well as various other components. For example, the biometric component interface 334 includes an interface to a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component interface 336 may include an interface to an inertial measurement unit (IMU), an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental component interface 338 may include, for example, an interface to an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an auditory sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects dangerous gas concentrations for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals associated with the surrounding physical environment. The positioning component interface 340 includes an interface to a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0046] A variety of technologies may be used to achieve communication. I / O device interface 306 also includes a communication component interface 342 that is operable to couple machine 300 to network 322 or device 324 via coupling 330 and coupling 326, respectively. For example, communication component interface 342 may include an interface to a network interface component or another suitable device that interfaces with network 322. In other examples, communication component interface 342 may include an interface to a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a wireless communication ... Components (e.g. Low power consumption), Device 324 may be another machine or any of a variety of peripherals (eg, a peripheral coupled via USB).

[0047] In addition, the communication component interface 342 may include an interface to an identifier or a component operable to detect an identifier. For example, the communication component interface 342 may include an interface to a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for an audio signal of an identification tag). In addition, various information may be obtained via the communication component interface 342, such as positioning obtained via Internet Protocol (IP) geolocation, location information obtained via Internet Protocol (IP) geolocation, and other information obtained via Internet Protocol (IP) geolocation. Positioning obtained by signal triangulation, positioning obtained by detecting NFC beacon signals that can indicate a specific location, etc.

[0048] Various memories (e.g., memory 304, main memory 314, static memory 316, and / or memory of processor 302) and / or storage unit 318 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. When executed by processor 302, these instructions (e.g., instructions 310) cause various operations to implement the disclosed examples.

[0049] Instructions 310 may be sent or received via a network interface device (e.g., a network interface component included in communication component interface 342), utilizing a transmission medium, and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 322. Similarly, instructions 310 may be sent or received via coupling 326 (e.g., a peer-to-peer coupling) to device 324 using a transmission medium.

[0050] Figure 4 4 is an illustration of a finger gesture recognition system according to some examples of the present disclosure. The computing system uses a finger gesture recognition system 412 to provide a user input modality to a user 402 when the user interacts with an XR application 410. The finger gesture recognition system 412 includes one or more optical sensors 404 and one or more audio sensors 406. The finger gesture recognition system 412 uses the one or more optical sensors 404 and the one or more audio sensors 406 to respectively capture the user 402's gestures when interacting with the XR application 410 (as shown in FIG. 4 ). Fig. 6A , Figure 6B and Figure 6CMore fully described) the finger gesture movement being made by the user 402 (as described in reference Figure 5 ) of the optically detectable finger gesture component 414 and the auditory detectable finger gesture component 416 as more fully described.

[0051] The one or more optical sensors 404 generate optical signal data 418 and transmit the optical signal data 418 to the finger gesture recognition component 408. The one or more audio sensors 406 generate audio signal data 420 and transmit the audio signal data 420 to the finger gesture recognition component 408. The finger gesture recognition component 408 receives the optical signal data 418 and the audio signal data 420 and detects the finger gesture being made by the user 402 based on the optical signal data 418 and the audio signal data 420 (see FIG. 4 ). Fig. 7A and Figure 7B 4). Finger gesture recognition component 408 transmits the detected finger gesture to XR application 410 as detected finger gesture data 422. XR application 410 receives detected finger gesture data 422 and uses detected finger gesture data 422 as user input into an XR user interface that XR application 410 provides to user 402.

[0052] Figure 5 is an illustration of a set of finger gestures according to some examples of the present disclosure. Finger gestures are natural for most users to perform, and finger gestures can be naturally mapped to control signals (such as clicks and slides) of XR input devices. In addition, these gestures are designed based on sensor signal characteristics. In particular, a flick finger gesture 502, a pinch finger gesture 504, a tap finger gesture 506, a left swipe finger gesture 508, and a right swipe finger gesture 510 are shown. Each gesture is a series of finger movements with defined start and end states of the fingers for data labeling. These definitions allow gestures to be labeled consistently across all sessions labeled by different users.

[0053] Finger gestures are classified into the following categories: fast gestures (i.e., flick finger gesture 502, pinch finger gesture 504, and tap finger gesture 506); and slow / continuous gestures (i.e., left swipe finger gesture 508 and right swipe finger gesture 510). Fast gestures involve faster finger movements and have a large impact on the audio signal, and fast gestures are usually short in duration. Slow gestures like swipes involve continuous movements and usually last longer. In addition, the finger gesture recognition system accurately tracks continuous left and right swipe finger gestures for fine-grained control.

[0054] Fig. 6A , Figure 6B and Figure 6CA wrist-worn XR input device incorporating components of a finger gesture recognition system according to some examples of the present disclosure is shown.

[0055] In some examples, wrist-worn XR input device 602 includes sensors carried by wristband 604. The sensors include one or more optical sensors (e.g., optical sensor 606a and optical sensor 606b (modified PAJ7620U2)), and one or more on-skin or contact audio sensors (e.g., modified MEMS audio sensor 608) in contact with the user's wrist. In some examples, the optical sensor may also be an imaging sensor capable of generating and transmitting images.

[0056] In some examples, a Bluetooth Low Energy (BLE) controller board (Adafruit Feather nRF52840Express) is attached to capture data reported by one or more audio sensors and one or more optical sensors, as well as communicate with the XR device to receive finger gesture input.

[0057] In some examples, one or more audio sensors and one or more optical sensors are soldered on a flexible printed circuit board (FPC) for flexible mounting on wristband 604.

[0058] In some examples, in order to protect the sensor holder and ensure that the sensor is in the designated position when repeatedly worn, a 3D printed wristband skeleton is used. The wristband skeleton consists of separate parts corresponding to the wrist position, namely, the palmar wrist part 610 (VP), the dorsal wrist part 612 (DP) and the radial wrist part 614 (RP). The three pieces are tied together with an elastic band.

[0059] In some examples, wrist anatomy shows that the palmar wrist region is thinner than the other two locations and closer to the tendons that control finger gestures. Therefore, an advantageous placement of one or more audio sensors is on the palmar wrist portion 610 of the wristband to better capture gesture-induced audio signals.

[0060] In some examples, the audio sensor includes a COTS MEMS microphone encapsulated in a metal housing. However, given the small form factor of the MEMS microphone, the contact area between the housing and the user's skin will be limited and prone to capacitance. Therefore, the skin coupling capacitance will interfere with the MEMS microphone (because the MEMS microphone uses internal capacitance to measure sound pressure) and the filter capacitor on the FPC. In addition, the skin coupling noise band may overlap with the frequency band of the finger gesture audio signal and is therefore difficult to remove by signal filtering. In order to reduce noise, the finger gesture recognition system includes a specially designed z-shaped groove on the palm portion of the wristband 604 for microphone mounting. It can be seen that the capacitor of the microphone can be positioned in the z-groove and well isolated from the skin surface.

[0061] In some examples, a thin metal film is applied to the audio sensor 608 (e.g., aluminum foil) to help increase the contact area with the skin, thereby mitigating interference capacitance. In addition, because the metal film covers the opening of the microphone housing, the audio signal from the finger gesture is further amplified by the formed metal drum.

[0062] In some examples, in order to minimize the impact of ambient light, the optical sensor of the finger gesture recognition system operates in the infrared spectrum and includes an IR LED. For example, for each sensing frame, the IR LED first illuminates the sensing area and then dims. The illuminated area is fine-tuned and limited to keep energy consumption low. The optical sensor array captures a frame when the IR LED is turned on and a frame when the IR LED is turned off. Then a subtraction is applied to two adjacent frames to minimize possible IR noise from the environment and generate a single frame.

[0063] In some examples, the optical sensor captures multiple image frames per second, and the finger gesture recognition system performs subtraction between adjacent frames to obtain a dynamic optical sensor data stream that captures moving objects in front of the optical sensor. To detect objects in the field of view, the finger gesture recognition system applies a threshold on the raw image frame and selects pixels above the threshold as valid pixels, and then uses a clustering algorithm to find the largest bright pixel cluster. The finger gesture recognition system uses the center of the cluster to represent the object location and converts the center pixel location to an X / Y coordinate value. The object size is determined by the number of bright pixels, and the brightness is the sum of all pixels.

[0064] In some examples, the optical sensor captures a shroud or back surface of the thumb finger in the field of view of the optical sensor.

[0065] In some of these examples, X / Y coordinates are used to define the position and movement of the thumb.

[0066] In some examples, a finger gesture recognition system uses two optical sensors with slightly different tilt angles to expand the field of view.

[0067] In some examples, the controller board of the finger gesture recognition system captures optical sensor readings and audio sensor readings in a loop to keep all sensors synchronized. The finger gestures recognized by the finger gesture recognition system are sometimes very fast (less than half a second), and the audio sensing modality and the optical sensing modality have different and high sensing rates.

[0068] In some examples, since the sampling rate of the optical sensor is approximately the same and relatively low compared to the audio sensor. Therefore, the finger gesture recognition system controller board accesses the optical sensor via I2C in a synchronous manner. Since the audio sensor sends audio signals to the controller board via pulse density modulation (PDM) at a rate greater than the sampling rate of the optical sensor, the finger gesture recognition system is able to use direct memory access with double buffering to achieve asynchronous transmission of audio signals. Interrupts triggered by buffer full events are minimized to reduce the impact on the loop duration. In some examples, the finger gesture recognition system configures the controller board to send the same amount of data in each cycle to keep the communication overhead consistent.

[0069] In some examples, the finger gesture recognition system enhances the audio signal from the audio sensor through amplitude scaling, jitter noise reduction, time stretching, and / or pitch enhancement.

[0070] In some examples, the finger gesture recognition system applies a median filter to the raw audio signal generated by the audio sensor, converts the raw audio signal into both the frequency domain and the time domain by filtering in the time domain and calculating the spectrogram of the audio signal to capture richer information. For example, the finger gesture recognition system uses a time window of 1024 samples with a step size of 80 samples for calculating the power spectral density map. The finger gesture recognition system records the log of the signal to make it easier to normalize the signal. The finger gesture recognition system normalizes the spectrogram using a min-max scaler. In some examples, the minimum and maximum values ​​are estimated from the entire data set across all users during training. After preprocessing the audio signal, the finger gesture recognition system takes a 2D vector as the audio signal input.

[0071] In some examples, the finger gesture recognition system uses time stretching, jitter reduction, and / or amplitude scaling to enhance the optical signal generated by the optical sensor.

[0072] In some examples, instead of using low-resolution images directly from optical sensors, the finger gesture recognition system uses on-chip computation and receives processed signals such as X / Y coordinates, object size, and brightness from the raw images. The object detection and tracking algorithms are based on image processing techniques to minimize the computational overhead of the main finger gesture recognition system.

[0073] In some examples, the finger gesture recognition system tracks objects at close range and uses a threshold / clustering based object tracking algorithm.

[0074] In some examples, the finger gesture recognition system applies a median filter on the optical sensor data to filter out possible spike artifacts.

[0075] In some examples, to accommodate various wearing positions, the finger gesture recognition system subtracts the original X / Y values ​​from the mean value. The finger gesture recognition system normalizes the values ​​by dividing them by a fixed range, which is the maximum possible range of movement distance estimated from all training data sets. This normalization preserves the amplitude of the gesture and is independent of the wearing position.

[0076] In some examples, the finger gesture recognition system utilizes signals from two optical sensors [X1, Y1, X2, Y2] as inputs to a multi-modal finger gesture recognition component.

[0077] Fig. 7A shows a multimodal finger gesture recognition component, and Figure 7B The method of operation of the finger gesture recognition system according to some examples of the present disclosure is shown. The finger gesture recognition system uses a finger gesture recognition component 730 to recognize finger gestures. The finger gesture recognition component 730 receives audio signal data 726 from one or more audio sensors and optical signal data 728 from one or more optical sensors, and recognizes the finger gesture data 722 gestures, and determines the fine-grained sliding data 724 movement based on the audio signal data 726 and the optical signal data 728.

[0078] In some examples, in order to achieve robust results and provide fine-grained gesture detection (e.g., the distance of a left or right slide), multiple models are used to detect and recognize finger gestures. In some examples, two models are used, and the finger gesture recognition component 730 aggregates the results with logic. For example, the finger gesture recognition component 730 uses a gesture classification model 716 for fast, non-continuous gestures to detect finger gesture data 722 gestures; and another contact binary detection model 718 is used to detect thumb / index finger contact states. If contact is detected, the finger gesture recognition component 730 uses the thumb tracking component 720 to determine the fine-grained sliding data 724 movement for fine-grained control.

[0079] In some examples, during model training, raw gesture samples are generated, and then positive and negative samples are extracted online during training. The start and end positions of each performed gesture are accurately marked on the synchronized acoustic and optical data according to the marking criteria. For each gesture sample, a 3-second signal segment is cropped with a sample in the middle of the entire segment. In some examples, for sliding gestures, samples can be randomly cropped when 80% of the sample gesture signal is ensured to be covered, while for fast motion gesture samples, the complete gesture signal is covered.

[0080] In some examples, a CNN-Transformer based model is designed and trained to classify signal sequences into gestures. The model takes audio and optical signals as multimodal input and classifies the gestures.

[0081] In some examples, for fast gesture recognition, a finger gesture recognition system uses a two-stream CNN model without a self-attention transformer encoder as a baseline.

[0082] In some examples, for slow gesture recognition (swipe left and swipe right) detection, a classifier similar to the fast gesture detection model was trained. Such a model outputs binary results, such as left and right, without continuous fine-grained thumb tracking.

[0083] In some examples, the finger gesture recognition system uses an optical sensor that produces raw object movement coordinates in pixels. The finger gesture recognition system can directly utilize such measurements for fine-grained control. One challenge is that without knowing whether the thumb is touching the index finger, the signal will be noisy because the optical sensor may not detect whether the user's thumb is touching the user's index finger. Therefore, the finger gesture recognition system detects contact of the thumb and finger so that when the thumb is moving in the air, the finger gesture recognition system can filter out these movements. To achieve this goal, another neural network for detecting contact is trained.

[0084] In some examples, audio signals are used to detect contact between a user's thumb and the user's fingers. When the thumb is rubbing against the index finger, the resulting sound can be captured by the finger gesture recognition system's audio sensor, which provides the finger gesture recognition system with an opportunity to identify that the thumb is in contact with the index finger. The movement of the thumb in the air without contacting the index finger can also generate non-trivial sound signals. Therefore, a complex machine learning model is used. We reuse a similar CNN-Transformer model without the optical branch and modify the output to be a binary classifier.

[0085] In some examples, audio signals and optical signals are used to detect contact between a user's thumb and the user's index finger.

[0086] In some examples, the aggregator 734 is used to aggregate the outputs from the posture classification model 716. For example, the aggregation is performed via a finite state machine (FSM). When a ti j-continuous window is predicted to be a window of posture j, a state transition from the current posture i to the new posture j is triggered. Therefore, the posture transition matrix is ​​tij, i, j∈G, where G is the state set of all postures, including the state where no posture is presented. Considering that postures are usually mapped to atomic input operations, the transition between two postures is disabled, that is, only the transition between non-posture states and posture states is allowed. In this way, the aggregation is controlled by a limited set of parameters.

[0087] In some examples, the number of correct poses is used as the objective function where we optimize dozens of aggregation parameters. Correctly recognized poses include poses that fully match or partially match the recognition. Since the solution space is large, random search is used to approximate the optimal solution. The optimization is performed on dedicated data and the results are reported on unseen data.

[0088] In some examples, the finger gesture recognition system captures two audio signal data streams at 41667 Hz and two optical signal data streams at 368 fps, which include X / Y coordinates, object size, and brightness.

[0089] Figure 7B 7 is an illustration of a finger gesture recognition process 732 according to some examples of the present disclosure. The finger gesture recognition system uses the finger gesture recognition process 732 to recognize the finger gestures being made by the user by using the finger gesture recognition component 730.

[0090] In operation 702, a finger gesture recognition system detects readings to identify a finger gesture made by a user by polling one or more audio sensors and one or more optical sensors to obtain signal data. The finger gesture recognition system uses one or more optical sensors and one or more audio sensors to capture optically detectable finger gesture components and auditorily detectable finger gesture components of finger gesture movements made by the user when the user interacts with the XR application, respectively. One or more optical sensors 404 generate optical signal data 728 and transmit the optical signal data 428 to the finger gesture recognition system. One or more audio sensors generate audio signal data 726 and transmit the audio signal data 726 to the finger gesture recognition system.

[0091] In operation 704 , if the finger gesture recognition system detects that the finger gesture recognition system is ready in operation 702 , the finger gesture recognition system attempts to recognize the rapid finger gesture being made by the user based on the audio signal data 726 and the optical signal data 728 using the gesture classification model 716 of the finger gesture recognition component 730 .

[0092] In operation 706 , the finger gesture recognition system determines whether a quick finger gesture has been recognized in operation 704 . If a quick finger gesture has been recognized, the finger gesture recognition system transmits the finger gesture data 722 to the XR application and returns to operation 702 .

[0093] If, in operation 706 , the finger gesture recognition system determines that a rapid finger gesture was not recognized in operation 704 , the finger gesture recognition system transitions to operation 712 .

[0094] In operation 712, the finger gesture recognition system continuously detects sliding finger gestures being made by the user by using the thumb tracking component 720 of the finger gesture recognition component 730. The finger gesture recognition system transmits fine-grained sliding data 724 generated by the thumb tracking component 720 to the XR application.

[0095] During operation 712, a timeout timer (not shown) of the thumb tracking component 720 continuously detects the operation of the thumb tracking component 720 and issues a sliding timeout signal when the user no longer performs the sliding finger gesture. In operation 714, if the timeout timer of the thumb tracking component 720 issues a sliding timeout signal, the finger gesture recognition system transitions to operation 702. If, in operation 714, the timeout timer of the thumb tracking component 720 does not issue a sliding timeout signal, the finger gesture recognition system continues to detect the sliding finger gesture in operation 712.

[0096] Concurrently with operation 704 , in operation 708 , the finger gesture recognition system attempts to detect contact between the user's thumb and fingers based on audio signal data 726 using contact binary detection model 718 of thumb tracking component 720 .

[0097] In operation 710, the finger gesture recognition system determines whether contact is being made between the user's thumb and fingers. If contact is not being made, the finger gesture recognition system transitions to operation 702. If contact is being made, the finger gesture recognition system transitions to operation 712.

[0098] As described above, in operation 712, the finger gesture recognition system continuously detects sliding finger gestures being made by the user by using the thumb tracking component 720 of the finger gesture recognition component 730. The finger gesture recognition system transmits the fine-grained sliding data 724 generated by the thumb tracking component 720 to the XR application.

[0099] During operation 712, a timeout timer (not shown) of the thumb tracking component 720 continuously detects the operation of the thumb tracking component 720 and issues a sliding timeout signal when the user no longer performs the sliding finger gesture. In operation 714, if the timeout timer of the thumb tracking component 720 issues a sliding timeout signal, the finger gesture recognition system transitions to operation 702. If, in operation 714, the timeout timer of the thumb tracking component 720 does not issue a sliding timeout signal, the finger gesture recognition system continues to detect the sliding finger gesture in operation 712.

[0100] Figure 8 800 is a block diagram 804 illustrating a software block diagram 804 that may be installed on any one or more of the devices described herein. The software architecture 804 is supported by hardware, such as a machine 802 including a processor 820, a memory 826, and an I / O component interface 838. In this example, the software architecture 804 may be conceptualized as a stack of layers, wherein each layer provides a specific functionality. The software block diagram 804 includes layers such as an operating system 812, a library 808, a framework 810, and an application 806. In operation, the application 806 invokes an API call 850 through the software stack and receives a message 852 in response to the API call 850.

[0101] The operating system 812 manages hardware resources and provides public services. The operating system 812 includes, for example, a kernel 814, a service 816, and a driver 822. The kernel 814 acts as an abstraction layer between the hardware and other software layers. For example, the kernel 814 provides functions such as memory management, processor management (e.g., scheduling), component management, network and security settings. The service 816 can provide other public services for other software layers. The driver 822 is responsible for controlling the underlying hardware or interfacing with the underlying hardware. For example, the driver 822 may include a display driver, a camera driver, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0102] The library 808 provides a low-level common infrastructure used by the application 806. The library 808 may include a system library 818 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 808 may include an API library 824, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering in two-dimensional (2D) and three-dimensional (3D) graphics content on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 808 may also include a variety of other libraries 828 to provide many other APIs to the application 806 .

[0103] The framework 810 provides a high-level common infrastructure used by the applications 806. For example, the framework 810 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 810 may provide a wide range of other APIs used by the applications 806, some of which may be specific to a particular operating system or platform.

[0104] In an example, applications 806 may include a home application 836, a contacts application 830, a browser application 832, a book reader application 834, a location application 842, a media application 844, a messaging application 846, a game application 848, and a broad category of other applications such as third-party applications 840. Applications 806 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 806 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, third-party applications 840 (e.g., provided by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, third-party applications 840 can activate API calls 850 provided by the operating system 812 to facilitate the functions described herein.

[0105] Fig. 9 is a block diagram showing a networked system 900 including details of the glasses 100 according to some examples. The networked system 900 includes the glasses 100, a client device 926, and a server system 932. The client device 926 may be a smart phone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 936 and / or a high-speed wireless connection 934. The client device 926 is connected to the server system 932 via a network 930. The network 930 may include any combination of wired and wireless connections. The server system 932 may be one or more computing devices that are part of a service or network computing system. The client device 926 and any elements of the server system 932 and the network 930 may be configured using the respective wireless devices in the embodiment of the present invention. Figure 8 and Figure 3 The details of the software architecture 804 or machine 300 described in are implemented.

[0106] The glasses 100 include a data processor 902, a display 910, one or more cameras 908, and additional input / output elements 916. The input / output elements 916 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 902. Figure 8 and Figure 3 Examples of input / output elements 916 are further discussed. For example, input / output elements 916 may include any of the I / O device interfaces in I / O device interfaces 306, including output component interface 328, motion component interface 336, etc. Figure 2 Examples of display 910 are discussed in . In the particular examples described herein, display 910 includes displays for the left and right eyes of a user.

[0107] Data processor 902 includes image processor 906 (eg, video processor), GPU and display driver 938, tracking component 940, interface 912, low power circuitry 904, and high speed circuitry 920. The components of data processor 902 are interconnected by bus 942.

[0108] The interface 912 refers to any source of user commands provided to the data processor 902. In one or more examples, the interface 912 is a physical button that sends a user input signal from the interface 912 to the low-power processor 914 when pressed. The low-power processor 914 can process the release immediately after pressing such a button as a request to capture a single image, and vice versa. The low-power processor 914 can process pressing such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 912 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the camera 908. In other examples, the interface 912 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 926.

[0109] The image processor 906 includes circuitry for receiving signals from the camera 908 and processing those signals from the camera 908 into a format suitable for storage in the memory 924 or for transmission to the client device 926. In one or more examples, the image processor 906 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from the camera 908, and volatile memory used by the microprocessor in operation.

[0110] The low power circuit system 904 includes a low power processor 914 and a low power wireless circuit system 918. These elements of the low power circuit system 904 can be implemented as separate elements or can be implemented on a single IC as part of a single system on a chip. The low power processor 914 includes logic for managing other elements of the glasses 100. As described above, for example, the low power processor 914 can accept user input signals from the interface 912. The low power processor 914 can also be configured to receive input signals or command communications from a client device 926 via a low power wireless connection 936. The low power wireless circuit system 918 includes circuit elements for implementing a low power wireless communication system. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of a low power wireless communication system that may be used to implement the low power wireless circuitry 918. In other examples, other low power communication systems may be used.

[0111] High-speed circuit system 920 includes a high-speed processor 922, a memory 924, and a high-speed wireless circuit system 928. High-speed processor 922 can be any processor capable of managing high-speed communications and operations for any general-purpose computing system for data processor 902. High-speed processor 922 includes processing resources for managing high-speed data transmission over high-speed wireless connection 934 using high-speed wireless circuit system 928. In some examples, high-speed processor 922 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Figure 8 The high-speed processor 922, which executes the software architecture of the data processor 902, is used to manage data transmission with the high-speed wireless circuit system 928, in addition to any other duties. In some examples, the high-speed wireless circuit system 928 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented by the high-speed wireless circuit system 928.

[0112] The memory 924 includes any storage device capable of storing camera data generated by the camera 908 and the image processor 906. Although the memory 924 is shown as being integrated with the high-speed circuit system 920, in other examples, the memory 924 may be a separate, independent element of the data processor 902. In some such examples, electrical wiring may provide a connection from the image processor 906 or the low-power processor 914 to the memory 924 through a chip including the high-speed processor 922. In other examples, the high-speed processor 922 may manage the addressing of the memory 924 so that the low-power processor 914 will initiate the high-speed processor 922 whenever a read or write operation involving the memory 924 is desired.

[0113] The tracking component 940 estimates the pose of the glasses 100. For example, the tracking component 940 uses image data and associated inertial data from the camera 908 and the positioning component interface 340, as well as GPS data, to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene environment). The tracking component 940 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world scene environment. The tracking component 940 allows the glasses 100 to visualize the placement of virtual objects relative to physical objects within the user's field of view via the display 910.

[0114] The GPU and display driver 938 can use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 910 when the glasses 100 are operating in a traditional augmented reality mode. In this mode, the GPU and display driver 938 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world scene environment.

[0115] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 926 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 806, such as the messaging application 846.

[0116] Fig.10 is a block diagram illustrating an example messaging system 1000 for exchanging data (e.g., messages and associated content) over a network. The messaging system 1000 includes a client device 926 hosting multiple instances of several applications including a messaging client 1002 and other applications 1004. The messaging client 1002 is communicatively coupled to other instances of the messaging client 1002 (e.g., hosted on respective other client devices 926), a messaging server system 1006, and a third-party server 1008 via a network 930 (e.g., the Internet). The messaging client 1002 can also communicate with the locally hosted application 1004 using an application program interface (API).

[0117] The messaging clients 1002 are able to communicate and exchange data with other messaging clients 1002 and with a messaging server system 1006 via the network 930. The data exchanged between the messaging clients 1002 and between the messaging clients 1002 and the messaging server system 1006 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0118] The messaging server system 1006 provides server-side functionality to specific messaging clients 1002 via the network 930. Although some functions of the messaging system 1000 are described herein as being performed by the messaging client 1002 or by the messaging server system 1006, the location of specific functions within the messaging client 1002 or within the messaging server system 1006 may be a design choice. For example, it may be technically preferable to initially deploy some technologies and functions within the messaging server system 1006, but then migrate the technologies and functions to the messaging client 1002 where the client device 926 has sufficient processing power.

[0119] The messaging server system 1006 supports various services and operations provided to the messaging clients 1002. Such operations include transmitting data to the messaging clients 1002, receiving data from the messaging clients 1002, and processing data generated by the messaging clients 1002. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 1000 is activated and controlled by functions available via the user interface (UI) of the messaging client 1002.

[0120] Turning now specifically to the messaging server system 1006, an application program interface (API) server 1010 is coupled to and provides a programming interface to an application server 1014. The application server 1014 is communicatively coupled to a database server 1016, which facilitates access to a database 1020 that stores data associated with messages processed by the application server 1014. Similarly, a web server 1024 is coupled to and provides a web-based interface to the application server 1014. To this end, the web server 1024 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0121] The application program interface (API) server 1010 receives and transmits message data (e.g., commands and message payloads) between the client device 926 and the application server 1014. Specifically, the application program interface (API) server 1010 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 1002 to activate the functions of the application server 1014. The application program interface (API) server 1010 exposes various functions supported by the application server 1014, including: account registration; login functionality; sending messages from a particular messaging client 1002 to another messaging client 1002 via the application server 1014, sending media files (e.g., images or videos) from the messaging client 1002 to the messaging server 1012, and possible access by another messaging client 1002; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 926; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 1002).

[0122] The application server 1014 hosts multiple server applications and subsystems, including, for example, a messaging server 1012, an image processing server 1018, and a social network server 1022. The messaging server 1012 implements multiple message processing techniques and functions, particularly those related to aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 1002. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 1002. Given the hardware requirements for other processor and memory intensive processing of data, such processing can also be performed on the server side by the messaging server 1012.

[0123] The application server 1014 also includes an image processing server 1018 that is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 1012.

[0124] The social network server 1022 supports various social networking functions and services and makes these functions and services available to the messaging server 1012. To this end, the social network server 1022 maintains and accesses an entity graph within the database 1020. Examples of functions and services supported by the social network server 1022 include identifying other users in the messaging system 1000 with whom a particular user has a relationship or that the particular user is "following", and also include identifying other entities and interests of a particular user.

[0125] The messaging client 1002 may notify the user of the client device 926 or other users related to such a user (e.g., "friends") of activities occurring in a shared or shareable session. For example, the messaging client 1002 may provide a notification to participants in a conversation (e.g., a chat session) in the messaging client 1002 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0126] Additional examples include:

[0127] Example 1 is a method comprising: capturing, by one or more processors, audio signal data of a finger gesture being made by a user using one or more audio sensors; capturing, by one or more processors, optical signal data of the finger gesture using one or more optical sensors; recognizing, by one or more processors, the finger gesture based on the audio signal data and the optical signal data; and transmitting, by one or more processors, the finger gesture data of the recognized finger gesture to an XR application.

[0128] In Example 2, the subject matter of Example 1 includes, wherein the one or more audio sensors are mounted on a palm-wrist portion of the wrist-worn input device.

[0129] In Example 3, the subject matter of any of Examples 1-2 includes, wherein the one or more optical sensors are mounted on a radial wrist portion of the wrist-worn input device.

[0130] In Example 4, the subject matter of any of Examples 1 to 3 includes, wherein recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: recognizing the finger gesture using a gesture classification model.

[0131] In Example 5, the subject matter of any of Examples 1 to 4 includes, wherein identifying the finger gesture based on the audio signal data and the optical signal data further includes: detecting that the user's thumb is contacting the user's finger based on the audio signal data using a binary detection model.

[0132] In Example 6, the subject matter of any of Examples 1 to 5 includes, wherein identifying the finger gesture based on the audio signal data and the optical signal data further comprises: determining a sliding finger gesture based on the optical signal data using a thumb tracking component.

[0133] In Example 7, the subject matter of any of Examples 1 to 6 includes, wherein recognizing finger gestures based on the audio signal data and the optical signal data further comprises: aggregating outputs of a gesture classification model for recognizing finger gestures using a finite state machine.

[0134] Example 8 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-7.

[0135] Example 9 is an apparatus comprising a method for implementing any of Examples 1 to 7.

[0136] Example 10 is a system for implementing any of Examples 1 to 7.

[0137] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate transmission of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0138] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0139] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0140] "Machine-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "machine-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0141] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Thus, the term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media".

[0142] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0143] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0144] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A system comprising one or more audio sensors; one or more optical sensors; one or more processors operably connected to the one or more audio sensors and the one or more optical sensors; as well as One or more memories operably connected to the one or more processors, the one or more memories storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising: capturing, using the one or more audio sensors, audio signal data of a finger gesture being made by a user; capturing optical signal data of the finger gesture using the one or more optical sensors; recognizing the finger gesture based on the audio signal data and the optical signal data; and Finger pose data of the recognized finger pose is transmitted to an extended reality (XR) application.

2. The system according to claim 1, wherein: The one or more audio sensors are mounted on the palm-wrist portion of the wrist-worn input device.

3. The system according to claim 1, wherein: The one or more optical sensors are mounted on the radial wrist portion of the wrist-worn input device.

4. The system according to claim 1, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: The finger gesture is recognized using a gesture classification model.

5. The system according to claim 1, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: A binary detection model is used to detect that the user's thumb is contacting the user's finger based on the audio signal data.

6. The system according to claim 5, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: A sliding finger gesture is determined based on the optical signal data using a thumb tracking component.

7. The system according to claim 1, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: The output of the gesture classification model for recognizing the finger gesture is aggregated using a finite state machine.

8. A method comprising: capturing, by one or more processors, audio signal data of a finger gesture being made by a user using one or more audio sensors; capturing, by the one or more processors, optical signal data of the finger gesture using one or more optical sensors; recognizing, by the one or more processors, the finger gesture based on the audio signal data and the optical signal data; as well as Finger pose data of the recognized finger pose is transmitted by the one or more processors to the XR application.

9. The method according to claim 8, wherein: The one or more audio sensors are mounted on the palm-wrist portion of the wrist-worn input device.

10. The method according to claim 8, wherein: The one or more optical sensors are mounted on the radial wrist portion of the wrist-worn input device.

11. The method according to claim 8, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: The finger gesture is recognized using a gesture classification model.

12. The method according to claim 8, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: A binary detection model is used to detect that the user's thumb is contacting the user's finger based on the audio signal data.

13. The method according to claim 8, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: A sliding finger gesture is determined based on the optical signal data using a thumb tracking component.

14. The method according to claim 8, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: The output of the gesture classification model for recognizing the finger gesture is aggregated using a finite state machine.

15. A computer storage medium storing instructions that, when executed by a computer, cause the computer to perform operations comprising: capturing audio signal data of finger gestures being made by a user using one or more audio sensors; capturing optical signal data of the finger gesture using one or more optical sensors; recognizing the finger gesture based on the audio signal data and the optical signal data; as well as The finger pose data of the recognized finger pose is transmitted to the XR application.

16. The computer storage medium of claim 15, wherein: The one or more audio sensors are mounted on the palm-wrist portion of the wrist-worn input device.

17. The computer storage medium of claim 15, wherein: The one or more optical sensors are mounted on the radial wrist portion of the wrist-worn input device.

18. The computer storage medium of claim 15, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: The finger gesture is recognized using a gesture classification model.

19. The computer storage medium of claim 15, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: A binary detection model is used to detect that the user's thumb is contacting the user's finger based on the audio signal data.

20. The computer storage medium of claim 15, wherein: Recognizing the finger gesture based on the audio signal data and the optical signal data further comprises: A sliding finger gesture is determined based on the optical signal data using a thumb tracking component.

Images