Multi-finger gestures based on finger manipulation data and limb tracking data

Abstract

A method is performed at an electronic device with one or more processors, non-transitory memory, a display, a limb tracking system, and a communication interface configured to communicate with a finger wearable device. The method includes displaying a computer-generated object on the display. The method includes obtaining finger manipulation data from the finger wearable device via the communication interface. The method includes determining a multi-finger gesture based on a function of limb tracking data from the limb tracking system and the finger manipulation data. The method includes registering an engagement event with respect to the computer-generated object according to the multi-finger gesture.

Description

Technical Field

[0001] This disclosure relates to displaying computer-generated objects, and more specifically, to registering engagement events with respect to computer-generated objects. Background Technology

[0002] Generally speaking, using the current input modality to join computer-generated objects can produce various inaccuracies associated with that joining. For example, a physical object may blur a portion of a computer-generated object, thus reducing tracking accuracy associated with a particular input modality. Similarly, computer-generated objects with greater depth relative to the display (such as background computer-generated objects) may be difficult for the user to join, leading to further tracking inaccuracies. Summary of the Invention

[0003] According to some specific embodiments, a method is performed in an electronic device having one or more processors, non-transitory memory, a display, a limb tracking system, and a communication interface configured to communicate with a finger wearable device. The method includes displaying a computer-generated object on the display. The method includes obtaining finger manipulation data from the finger wearable device via the communication interface. The method includes determining a multi-finger gesture based on limb tracking data from the limb tracking system and the finger manipulation data. The method includes registering engagement events with respect to the computer-generated object based on the multi-finger gesture.

[0004] According to some embodiments, an electronic device includes one or more processors, non-transitory memory, a display, a limb tracking system, and a communication interface configured to communicate with a wearable finger device. One or more programs are stored in the non-transitory memory and configured to be executed by one or more processors. The one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the electronic device, cause the device to perform or cause to perform any of the methods described herein. According to some embodiments, an electronic device includes means for performing or causing to perform any of the methods described herein. According to some embodiments, an information processing apparatus for use in an electronic device includes means for performing or causing to perform any of the methods described herein. Attached Figure Description

[0005] To better understand the various specific implementations described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0006] Figure 1It is a block diagram based on some specific implementation examples of portable multi-functional devices.

[0007] Figure 2 This is a block diagram based on some specific implementations of wearable finger devices.

[0008] Figures 3A to 3M These are examples of electronic devices that register engagement events for computer-generated objects based on corresponding multi-finger gestures, according to some specific implementations.

[0009] Figure 4 This is an example of a flowchart illustrating a method for registering join events of computer-generated objects based on multi-finger gestures, according to some specific implementations. Detailed Implementation

[0010] Electronic devices including integrated input systems can enable users to engage computer-generated objects based on input from the integrated input system. For example, the integrated input system includes a limb-tracking input system and / or an eye-tracking input system. For instance, based on limb-tracking input from the limb-tracking input system, the electronic device determines that a user's corresponding limb meets a proximity threshold relative to a specific computer-generated object. Therefore, the electronic device selects and / or manipulates a specific computer-generated object based on the limb-tracking input. However, using input from an integrated input system to engage computer-generated objects presents several problems. For example, the reliability of limb-tracking input decreases correspondingly when a physical object occludes (e.g., blocks) part of the user's limb. As another example, the limited mobility of the user's eyes and the instability of the user's limbs reduce the efficiency associated with manipulating computer-generated objects. As yet another example, computer-generated objects with relatively deep depth relative to the display (such as computer-generated objects located in the background of a scene) may be difficult for the user to engage, leading to inaccurate limb-tracking and eye-tracking.

[0011] In contrast, the various embodiments disclosed herein include methods, electronic devices, and systems for registering engagement events with respect to a computer-generated object based on limb tracking data and finger manipulation data. For this purpose, the electronic device includes a limb tracking system for acquiring limb tracking data, such as a computer vision-based system performing limb recognition (e.g., semantic segmentation) on image data. Furthermore, for this purpose, the electronic device includes a communication interface for acquiring finger manipulation data from a finger-wearable device. Based on the limb tracking data and finger manipulation data, the electronic device determines the corresponding portions of a multi-finger gesture. For example, in some embodiments, a multi-finger gesture corresponds to a gesture performed by multiple fingers of a user's single hand, such as a pinch gesture (e.g., thumb movement plus index finger movement). As another example, in some embodiments, a multi-finger gesture corresponds to a multi-hand gesture, such as a combination of a tapping gesture (detected based on finger manipulation data) performed by a finger-wearable device worn on the user's left hand and a dragging gesture (detected based on limb tracking data) performed by the user's right hand. The electronic device registers engagement events with respect to a computer-generated object, such as selecting and / or manipulating the computer-generated object, based on the multi-finger gesture.

[0012] Finger-wearable devices can be worn by a user's finger. In some implementations, the electronic device tracks the finger in six degrees of freedom (6DOF) based on finger manipulation data. Therefore, the electronic device continues to receive finger manipulation data from the finger-wearable device even when a physical object partially obscures it. On the other hand, when a physical object blocks the user's limb, other devices utilizing limb tracking cannot track that limb. Furthermore, the electronic device achieves object engagement (e.g., selection, manipulation, etc.) based on finger manipulation data, regardless of the apparent distance between the finger-wearable device and the content manipulation area, resulting in better control and accuracy.

[0013] Reference will now be made in detail to specific embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure various aspects of the embodiments.

[0014] It will also be understood that, although in some cases the terms “first,” “second,” etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be named a second contact, and similarly, a second contact may be named a first contact, without departing from the scope of the various specific embodiments described. Both the first contact and the second contact are contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0015] The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “includes,” “including,” “comprises,” and / or “comprising” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0016] As used herein, depending on the context, the term "if" is optionally interpreted as meaning "when," "at," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if it is determined..." or "if [the stated condition or event] is optionally interpreted as meaning "when it is determined..." or "in response to determination..." or "when [the stated condition or event] is detected," or "in response to the detection of [the stated condition or event]."

[0017] A physical setting is a set of settings in the world that people can sense and / or interact with without using electronic systems. For example, a room is a physical setting that includes physical elements such as physical chairs, physical tables, physical lamps, etc. People can sense and interact with these physical elements of a physical setting through direct touch, taste, sight, smell, and hearing.

[0018] Compared to physical sets, extended reality (XR) sets refer to computer-generated environments, some or all of which are created using computer-generated content. While humans can interact with XR sets using various electronic systems, this interaction utilizes various electronic sensors to monitor human movements and translate those movements into corresponding actions within the XR set. For example, if an XR system detects that a person is looking upwards, it can change its graphics and audio output to present XR content in a manner consistent with upward movement. XR environments can adhere to the laws of physics to simulate physical environments.

[0019] The concept of XR encompasses both Virtual Reality (VR) and Augmented Reality (AR). XR also includes Mixed Reality (MR), which is sometimes used to refer to the spectrum of reality between a physical environment (but not including the physical environment) at one end and VR at the other. XR also includes Augmented Virtual (AV), where a virtual or computer-generated scene integrates sensory input from a physical scene. This input can represent characteristics of the physical scene. For example, virtual objects can display colors captured from the physical environment using image sensors. Alternatively, an AV environment can adopt the current weather conditions of the physical environment.

[0020] Some electronic systems used to implement XR operate in conjunction with opaque displays and one or more imaging sensors for capturing video and / or images of a physical scene. In some implementations, the displayed image is referred to as video pass-through when the system captures an image of the physical scene and uses the captured image to display a representation of the physical scene on an opaque display. Some electronic systems used to implement XR operate in conjunction with transparent or semi-transparent displays (and optionally with one or more imaging sensors). Such displays allow a person to view the physical environment directly through the display and also allow virtual content to be added to the person's field of view by overlaying content onto the physical environment. Some electronic systems used to implement XR operate in conjunction with projection systems that project virtual objects onto a physical scene. For example, a projector may present a hologram onto a physical scene, or project an image onto a physical surface, or project it onto a person's eye (e.g., the retina).

[0021] Electronic systems providing XR environments can have various form factors. Smartphones or tablets can be combined with imaging and display components to provide an XR environment. A head-mounted system may include imaging and display components to provide an XR environment. These systems can provide computing resources for providing an XR environment and can work together to provide an XR environment. For example, a smartphone or tablet can be connected to a head-mounted display to provide an XR environment. Alternatively, a computer can be connected to a home entertainment component or vehicle system to provide an in-vehicle display or head-up display. Electronic systems providing XR environments can utilize display technologies such as LED, OLED, liquid crystal on silicon, laser scanning light sources, digital light projectors, or combinations thereof. Display technologies can employ light-transmitting substrates, including optical waveguides, holographic substrates, optical reflectors, and combiners, or combinations thereof.

[0022] Figure 1 This is a block diagram of an example of a portable multi-functional device 100 (sometimes referred to herein as "electronic device 100" for brevity) according to some specific implementation. Electronic device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, an input / output (I / O) subsystem 106, a speaker 111, a display system 112, an inertial measurement unit (IMU) 130, an image sensor 143 (e.g., a camera), a contact strength sensor 165, an audio sensor 113 (e.g., a microphone), an eye-tracking sensor 164 (e.g., included within a head-mounted device (HMD)), a limb-tracking sensor 150, and other input or control devices 116. In some specific implementations, electronic device 100 corresponds to one of a mobile phone, a tablet computer, a laptop computer, a wearable computing device, a head-mounted device (HMD), a head-mounted housing (e.g., electronic device 100 slides to or is otherwise attached to a head-mounted housing), etc. In some specific implementations, the head-mounted housing is shaped to form a receiver for receiving electronic equipment 100 with a display.

[0023] In some embodiments, the peripheral interface 118, one or more processing units 120, and memory controller 122 are optionally implemented on a single chip, such as chip 103. In other embodiments, they are optionally implemented on separate chips.

[0024] I / O subsystem 106 couples input / output peripherals on electronic device 100, such as display system 112 and other input or control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, image sensor controller 158, intensity sensor controller 159, audio controller 157, eye-tracking controller 160, one or more input controllers 152 for other input or control devices, IMU controller 132, limb tracking controller 180, and privacy subsystem 170, as well as communication interface 190. One or more input controllers 152 receive electrical signals from / send electrical signals to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 152 may optionally be coupled (or not coupled) to any of the following: a keyboard, an infrared port, a universal serial bus (USB) port, a stylus, a finger wearable device, and / or a pointing device such as a mouse. One or more buttons may optionally include volume up / down buttons for volume control of the speaker 111 and / or audio sensor 113. One or more buttons may optionally include a push-button. In some embodiments, other input or control devices 116 may include a positioning system (e.g., GPS) that acquires information about the position and / or orientation of the electronic device 100 relative to a particular object. In some embodiments, other input or control devices 116 may include a depth sensor and / or a time-of-flight sensor that acquires depth information characterizing a particular object.

[0025] Display system 112 provides input and output interfaces between electronic device 100 and user. Display controller 156 receives electrical signals from display system 112 and / or sends electrical signals to display system 112. Display system 112 displays visual output to user. Visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term "visual representation" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0026] In some embodiments, display system 112 corresponds to a touch-sensitive surface. For example, display system 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or tactile contact. Display system 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on display system 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on display system 112. In exemplary embodiments, the contact point between display system 112 and the user corresponds to the user's finger or a finger-wearable device.

[0027] Display system 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other specific embodiments. Display system 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed thereafter, as well as other proximity sensor arrays or other elements for determining one or more points of contact with display system 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies.

[0028] In some embodiments, the user may optionally use any suitable object or accessory, such as a stylus, a wearable finger device, or a finger, to interact with the display system 112. In some embodiments, the user interface is designed to work with finger-based contact and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on a touchscreen. In some embodiments, the electronic device 100 translates coarse finger-based input into precise pointer / cursor positions or commands to perform the user-desired actions.

[0029] Speaker 111 and audio sensor 113 provide an audio interface between the user and electronic device 100. Audio circuitry receives audio data from peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry also receives electrical signals converted from sound waves by audio sensor 113 (e.g., a microphone). Audio circuitry converts the electrical signals into audio data and transmits the audio data to peripheral interface 118 for processing. Audio data is optionally retrieved by peripheral interface 118 from and / or transmitted to memory 102 and / or RF circuitry. In some specific implementations, audio circuitry also includes a headset jack. This headset jack provides an interface between audio circuitry and a removable audio input / output peripheral device, such as an output-only headset or a headset with both outputs (e.g., mono-ear or binaural headphones) and inputs (e.g., a microphone).

[0030] The inertial measurement unit (IMU) 130 includes an accelerometer, a gyroscope, and / or a magnetometer to measure various force, angular rate, and / or magnetic field information relative to the electronic device 100. Therefore, depending on the specific implementation, the IMU 130 detects one or more positional change inputs of the electronic device 100, such as the electronic device 100 being rocked, rotated, or moved in a specific direction.

[0031] Image sensor 143 captures still images and / or video. In some embodiments, optical sensor 143 is located on the back of electronic device 100, opposite to the touchscreen on the front of electronic device 100, allowing the touchscreen to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, another image sensor 143 is located on the front of electronic device 100, enabling the acquisition of images of the user (e.g., for selfies, for video conferencing while the user is viewing other video conferencing participants on the touchscreen, etc.). In some embodiments, the image sensor is integrated within the HMD (Head-Down Display).

[0032] A contact strength sensor 165 detects the strength of a contact on electronic device 100 (e.g., a touch input on a touch-sensitive surface of electronic device 100). The contact strength sensor 165 is coupled to a strength sensor controller 159 in I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the physical environment. In some embodiments, at least one contact strength sensor 165 is arranged juxtaposed with or adjacent to the touch-sensitive surface of electronic device 100. In some embodiments, at least one contact strength sensor 165 is located on the side of electronic device 100.

[0033] Eye-tracking sensor 164 detects the eye gaze of a user of electronic device 100 and generates eye-tracking data indicating the user's eye gaze. In various specific embodiments, the eye-tracking data includes data indicating a fixed point (e.g., a point of attention) of the user on a display panel, such as a display panel within a head-mounted device (HMD), a head-mounted housing, or a head-up display.

[0034] The limb tracking sensor 150 acquires limb tracking data indicating the position of a user's limbs. For example, in some embodiments, the limb tracking sensor 150 corresponds to a hand tracking sensor that acquires hand tracking data indicating the position of a user's hand or fingers within a specific object. In some embodiments, the limb tracking sensor 150 utilizes computer vision techniques to estimate limb pose based on camera images.

[0035] In various embodiments, electronic device 100 includes a privacy subsystem 170 that includes one or more privacy setting filters associated with user information, such as user information included in limb tracking data, eye gaze data, and / or body position data associated with the user. In some embodiments, privacy subsystem 170 selectively prevents and / or restricts electronic device 100 or parts thereof from acquiring and / or transmitting user information. To this end, privacy subsystem 170 receives user preferences and / or choices from the user in response to prompting the user to make user preferences and / or choices. In some embodiments, privacy subsystem 170 prevents electronic device 100 from acquiring and / or transmitting user information unless and until informed consent is obtained from the user. In some embodiments, privacy subsystem 170 anonymizes (e.g., scrambles or obfuscates) certain types of user information. For example, privacy subsystem 170 receives user input specifying which types of user information privacy subsystem 170 anonymizes. As another example, privacy subsystem 170 may include certain types of user information, including sensitive and / or identifying information, through user-specified (e.g., automatic) anonymization.

[0036] Electronic device 100 includes a communication interface 190 configured to communicate with a finger-wearable device (such as...) Figure 2 The finger wearable device 200 shown in the image Figures 3A to 3M The electronic device 100 communicates with the wearable finger device 320. For example, the communication interface 190 corresponds to one of a Bluetooth interface, an IEEE 802.11x interface, a near field communication (NFC) interface, etc. According to various specific embodiments, the electronic device 100 obtains finger manipulation data from the wearable finger device via the communication interface 190, as will be further described below.

[0037] Figure 2 This is a block diagram of an example of a finger-wearable device 200. The finger-wearable device 200 includes a memory 202 (optionally including one or more computer-readable storage media), a memory controller 222, one or more processing units (CPUs) 220, a peripheral device interface 218, RF circuitry 208, and an input / output (I / O) subsystem 206. These components optionally communicate via one or more communication buses or signal lines 203. Those skilled in the art will understand that... Figure 2 The finger wearable device 200 shown is an example of a finger wearable device, and the finger wearable device 200 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has different configurations or arrangements of components. Figure 2 The various components shown are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application-specific integrated circuits).

[0038] The finger-wearable device 200 includes a power system 262 for powering various components. The power system 262 optionally includes a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in the portable device and / or portable accessories.

[0039] Memory 202 optionally includes high-speed random access memory, and also optionally includes non-volatile memory, such as one or more flash memory devices or other non-volatile solid-state memory devices. Access to memory 202 by other components of the finger wearable device 200 (such as CPU 220 and peripheral interface 218) is optionally controlled by memory controller 222.

[0040] Peripheral interface 218 can be used to couple input and output peripherals of the finger wearable device 200 to CPU 220 and memory 202. One or more processors 220 run or execute various software programs and / or instruction sets stored in memory 202 to perform various functions of the finger wearable device 200 and process data.

[0041] In some implementations, the peripheral interface 218, CPU 220, and memory controller 222 are optionally implemented on a single chip, such as chip 204. In other implementations, they are implemented on separate chips.

[0042] RF (Radio Frequency) circuit 208 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 208 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with electronic device 100 or electronic device 310, communication networks, and / or other communication devices via electromagnetic signals. RF circuit 208 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 208 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). The wireless communication may optionally use any of a variety of communication standards, protocols, and / or technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolved Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE... 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including those not yet developed as of the date of this document submission.

[0043] I / O subsystem 206 couples input / output peripherals, such as other input or control devices 216, on the finger wearable device 200 to peripheral interface 218. I / O subsystem 206 optionally includes one or more position sensor controllers 258, one or more intensity sensor controllers 259, a haptic feedback controller 261, and one or more other input controllers 260 for the other input or control devices. The one or more other input controllers 260 receive electrical signals from / send electrical signals to the other input or control devices 216. Other input or control devices 216 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, click wheels, etc. In some implementations, the other input controllers 260 are optionally coupled to (or not coupled to) any of the following: an infrared port and / or a USB port.

[0044] In some implementations, the finger wearable device 200 includes one or more position sensors 266 that output position data associated with the finger wearable device 200. The position data indicates the position, orientation, or movement of the finger wearable device 200, such as rotational or translational movement. For example, the position sensor 266 includes an inertial measurement unit (IMU) that provides 3D rotational data, such as roll, pitch, and yaw information. For this purpose, the IMU may include a combination of accelerometers, gyroscopes, and magnetometers. As another example, the position sensor 266 includes a magnetic sensor that provides 3D position data, such as the position of the finger wearable device 200. For example, the magnetic sensor measures a weak magnetic field to determine the position of the finger wearable device 200.

[0045] In some implementations, the finger wearable device 200 includes one or more contact strength sensors 268 for detecting the contact intensity (e.g., force or pressure) of a finger wearing the finger wearable device 200 on a physical object. The one or more contact strength sensors 268 output contact strength data associated with the finger wearable device 200. For example, the contact strength data indicates the force or pressure of a tapping gesture associated with a finger wearing the finger wearable device 200 tapping on the surface of a physical table. The one or more contact strength sensors 268 may include an interferometer. The one or more contact strength sensors 268 may include one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors.

[0046] The finger wearable device 200 optionally includes one or more haptic output generators 263 for generating haptic outputs on the finger wearable device 200. In some embodiments, the term "haptic output" refers to a physical displacement of an accessory (e.g., the finger wearable device 200) of an electronic device (e.g., electronic device 100) relative to a previous position of the accessory, a physical displacement of a component of the accessory relative to another component of the accessory, or a displacement of the component relative to the center of mass of the accessory, detected by the user using their tactile senses. For example, when the accessory or a component of the accessory comes into contact with a touch-sensitive surface of the user (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the accessory or a component of the accessory. For example, movement of a component (e.g., the housing of the finger wearable device 200) may optionally be interpreted by the user as a "click" on a physically actuated button. In some cases, users will feel a tactile sensation, such as a “click,” even when a physically actuated button associated with a finger-worn device, which is physically pressed (e.g., displaced) by the user's movement, does not move. While such interpretations of touch by users will be limited by the individualized sensory perceptions of the user, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., a “click”), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the electronic device or its components that will generate the sensory perception of a typical (or common) user.

[0047] Figure 2 A haptic output generator 263 coupled to a haptic feedback controller 261 is shown. The haptic output generator 263 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on electronic devices). The haptic output generator 263 receives haptic feedback generation instructions from a haptic feedback system 234 and generates haptic outputs on a finger wearable device 200 that can be sensed by a user of the finger wearable device 200.

[0048] In some embodiments, the software components stored in memory 202 include an operating system 226, a communication system (or instruction set) 228, a positioning system (or instruction set) 230, a contact intensity system (or instruction set) 232, a haptic feedback system (or instruction set) 234, and a gesture interpretation system (or instruction set) 236. Furthermore, in some embodiments, memory 202 stores device / global internal states associated with the finger wearable device. Device / global internal states include one or more of the following: sensor states, including information obtained from various sensors and other input or control devices 216 of the finger wearable device; position states, including information about the position of the finger wearable device relative to an electronic device (e.g., electronic device 100) (e.g., position, orientation, tilt, rolling, and / or distance); and positional information about the absolute position of the finger wearable device.

[0049] Operating system 226 includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, power management, etc.) and facilitates communication between various hardware and software components.

[0050] The communication system 228 facilitates communication with other devices (e.g., electronic device 100 or electronic device 310) and also includes various software components (e.g., for processing data received by RF circuit 208) suitable for direct coupling to other devices or indirect coupling to other devices via a network (e.g., Internet, wireless LAN, etc.).

[0051] Positioning system 230 optionally detects position information about the finger wearable device 200 by combining position data from one or more position sensors 266. Positioning system 230 optionally includes software components for performing various operations related to detecting the position of the finger wearable device 200 and detecting changes in the position of the finger wearable device 200 in a particular frame of reference. In some embodiments, positioning system 230 detects the positional state of the finger wearable device 200 relative to an electronic device and detects changes in the positional state of the finger wearable device 200 relative to the electronic device. As described above, in some embodiments, electronic device 100 or 310 determines the positional state of the finger wearable device 200 relative to the electronic device and uses information from positioning system 230 to change the positional state of the finger wearable device 200.

[0052] The contact intensity system 232 combines contact intensity data from one or more contact intensity sensors 268 to optionally detect contact intensity information associated with the finger wearable device 200. The contact intensity system 232 includes software components for performing various operations related to contact detection, such as detecting the intensity and / or duration of contact between the finger wearable device 200 and a tabletop. Determining movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact intensity data.

[0053] The haptic feedback system 234 includes various software components for generating instructions used by the haptic output generator 263 to produce haptic output at one or more locations on the finger wearable device 200 in response to user interaction with the finger wearable device 200.

[0054] The finger wearable device 200 optionally includes a gesture interpretation system 236. The gesture interpretation system 236 coordinates with a positioning system 230 and / or a contact strength system 232 to determine the gesture performed by the finger wearable device. For example, gestures include one or more of the following: pinch gesture, pull gesture, pinch and pull gesture, rotation gesture, tap gesture, etc. In some embodiments, the finger wearable device 200 does not include a gesture interpretation system, and an electronic device or system (e.g., a gesture interpretation system integrated within an electronic device) determines the gesture performed by the finger wearable device 200 based on finger manipulation data from the finger wearable device 200. In some embodiments, a portion of the gesture determination is performed at the finger wearable device 200, and another portion is performed at the electronic device / system. In some embodiments, the gesture interpretation system 236 determines the duration associated with the gesture. In some embodiments, the gesture interpretation system 236 determines the contact strength associated with the gesture, such as the amount of pressure associated with a tap of a finger (wearing the finger wearable device 200) on a physical surface.

[0055] Each module and application identified above corresponds to a set of executable instructions for performing one or more of the functions described above, as well as the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These systems (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some specific embodiments, memory 202 optionally stores a subset of the aforementioned systems and data structures. Furthermore, memory 202 optionally stores additional systems and data structures not described above.

[0056] Figures 3A to 3MThis is an example of an electronic device 310 that registers engagement events with a computer-generated object based on corresponding multi-finger gestures according to some specific implementation. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure further relevant aspects of the exemplary embodiments disclosed herein.

[0057] like Figure 3A As shown, electronic device 310 is associated with operating environment 300 (e.g., operates according to operating environment). In various specific embodiments, electronic device 310 and Figure 1 The electronic device 310 is similar to, and modified from, the electronic device 100. In some specific implementations, the electronic device 310 generates one of the aforementioned XR settings.

[0058] Electronic device 310 includes a display 312 associated with a visible area 314 of operating environment 300. For example, in some embodiments, electronic device 310 includes an image sensor associated with a field of view corresponding to visible area 314, and electronic device 310 synthesizes transmitted image data from the image sensor with computer-generated content. As another example, in some embodiments, electronic device 310 includes a perspective display 312 that allows ambient light to enter from a portion of the physical environment associated with visible area 314. Operating environment 300 includes a physical table 302, and visible area 314 includes a portion of the physical table 302.

[0059] The finger wearable device 320 can be worn on the fingers of the user's first hand 52. For example, as Figure 3B As shown, the wearable finger device 320 is worn on the index finger of the first hand 52. In some specific embodiments, the wearable finger device 320 is similar to Figure 2 The finger wearable device 200 shown is modified from that finger wearable device.

[0060] In some implementations, electronic device 310 includes a communication interface configured to communicate with finger-wearable device 320 (e.g., Figure 1The electronic device 310 establishes a communication link with the finger wearable device 320, as indicated by communication link line 322. Establishing a link between the electronic device 310 and the finger wearable device 320 is sometimes referred to as pairing or network sharing. Those skilled in the art will understand that the electronic device 310 can communicate with the finger wearable device 320 according to various communication protocols, such as Bluetooth, IEEE 802.11x, NFC, etc. The electronic device 310 obtains finger manipulation data from the finger wearable device 320 via the communication interface. For example, the electronic device 310 obtains a combination of position data (e.g., output from the IMU sensor and / or magnetic sensor of the finger wearable device 320) and contact strength data (e.g., output from the contact strength sensor of the finger wearable device 320).

[0061] In some specific implementations, such as Figure 3A As shown, the user 50's second hand 54 is holding the electronic device 310. For example, in some embodiments, the electronic device 310 corresponds to one of a smartphone, laptop computer, tablet computer, etc.

[0062] In some embodiments, electronic device 310 corresponds to a head-mounted device (HMD) including an integrated display (e.g., a built-in display) that displays a representation of operating environment 300. In some embodiments, electronic device 310 includes a head-mounted housing. In various embodiments, the head-mounted housing includes an attachment area to which another device having a display can be attached. In various embodiments, the head-mounted housing is shaped to form a receiver for receiving another device (e.g., electronic device 310) including a display. For example, in some embodiments, electronic device 310 slides / snapes into or otherwise attaches to the head-mounted housing. In some embodiments, the display of the device attached to the head-mounted housing presents (e.g., displays) a representation of operating environment 300. For example, in some embodiments, electronic device 310 corresponds to a mobile phone that can be attached to a head-mounted housing.

[0063] In some specific implementations, electronic device 310 includes an image sensor, such as a scene camera. For example, the image sensor acquires image data characterizing the operating environment 300, and electronic device 310 synthesizes the image data with computer-generated content to generate display data for display on display 312. The display data may be characterized by the XR environment. For example, the image sensor acquires image data representing a portion of a physical table 302, and the generated display data displayed on display 312 includes that portion of the physical table 302 (see [link to relevant documentation]). Figure 3B ) represents.

[0064] In some embodiments, display 312 corresponds to a perspective display. A perspective display allows ambient light from the physical environment to pass through it, and the representation of the physical environment is based on this ambient light. For example, a perspective display is a translucent display, such as glass with optical transparency. In some embodiments, a perspective display is an additional display that allows optical transparency of a physical surface, such as an optical HMD (OHMD). For example, unlike pure synthesis using video streams, an additional display is able to reflect projected images from the display while allowing the user's vision to pass through the display. In some embodiments, the perspective display includes a photochromic lens. The HMD adds computer-generated objects to the ambient light entering the perspective display to achieve the display of the operating environment 300. For example, a perspective display allows ambient light from the operating environment 300, which includes a portion of the physical table 302, and therefore the perspective display displays a representation of that portion of the physical table 302 (see [link to documentation]). Figure 3B ).

[0065] like Figure 3B As shown, electronic device 310 displays a representation of this portion of physical table 302 on display 312 (for brevity, it is sometimes referred to as "a portion of physical table 302" or "physical table 302" below). Furthermore, electronic device 310 displays a computer-generated object 330 corresponding to a three-dimensional (3D) cube on display 312. The computer-generated object 330 may be associated with various content types (e.g., representing various content types), such as audio content, video content, image content, document content, text content, metadata content, database content, etc. Moreover, those skilled in the art will understand that in some specific implementations, the computer-generated object 330 is a different object type, such as different 3D objects (e.g., a sphere) or two-dimensional objects (e.g., a tile).

[0066] The wearable finger device 320, worn by the index finger of the first hand portion 52, can be moved into the visible area 314. Therefore, as Figure 3B As shown, the electronic device 310 displays corresponding representations of the first hand 52 and the finger wearable device 320 on the display 312 (for simplicity, they are sometimes referred to as "first hand 52" and "finger wearable device 320" below).

[0067] like Figure 3C As shown, the wearable finger device 320 and the thumb of the first hand 52 together initiate a multi-finger pinch gesture. The multi-finger pinch gesture includes a first gesture associated with the thumb moving toward (e.g., upwards) the wearable finger device 320. The first gesture is... Figure 3CThe first gesture line 334 (shown for illustrative purposes only) indicates this. Furthermore, the multi-finger pinch gesture includes a second gesture associated with the movement of the finger wearable device 320 toward the thumb (e.g., downwards). The second gesture is indicated by... Figure 3C The second gesture line 332 (shown for illustrative purposes only) indicates this.

[0068] Electronic device 310 acquires limb tracking data associated with the first gesture via limb tracking system 340. In some embodiments, limb tracking system 340 is integrated into electronic device 310. Reception of limb tracking data is indicated by tracking line 342 (shown for illustrative purposes only). The limb tracking data indicates a position value associated with the thumb. The position value is indicated by crosshair 344 (shown for illustrative purposes only). For this purpose, in some embodiments, limb tracking system 340 performs computer vision techniques to identify (e.g., track) the thumb. For example, electronic device 310 acquires image data (e.g., via a forward-facing camera) and (optionally with the aid of an integrated neural network) performs semantic segmentation to identify the thumb represented within the image data.

[0069] In some cases, the thumb is partially obscured, such as Figure 3C As shown. Therefore, in some embodiments, the limb tracking system 340 identifies the first hand 52 as the right hand and estimates the position of the thumb in part based on the expected position of the right thumb relative to the index finger. Therefore, in some embodiments, the limb tracking system 340 performs limb tracking on multiple fingers (e.g., thumb and index finger).

[0070] In some specific implementations, limb tracking data indicates multiple location values ​​associated with a limb at corresponding multiple time points. For example, refer to Figure 3C At a first moment (T1) before the thumb initiates the first gesture, limb tracking data indicates a first position value that indicates the thumb is below the computer-generated object 330 on display 312. Continuing this example, refer to... Figure 3D At a second time (T2) while the thumb is performing the first gesture, limb tracking data indicates a second position value indicating that the thumb is positioned above a portion of the computer-generated object 330 on display 312. Continuing with this example, refer to... Figure 3E At the third time (T3), the limb tracking data indicates a third position value, which indicates that the thumb is located at a higher position on the display 312 than the position of the thumb at the second time.

[0071] Furthermore, the electronic device 310 obtains finger manipulation data associated with the second gesture from the finger wearable device 320 via a communication interface. For example, see reference... Figure 3C and Figure 3DBased on IMU data, electronic device 310 determines that the finger wearable device 320 performs a rotational movement, and thus registers the start of the second gesture. For example, see reference... Figure 3D and Figure 3E Based on magnetic sensor data, electronic device 310 determines that wearable finger device 320 is moving downward along a specific axis.

[0072] Based on limb tracking data and finger manipulation data, electronic device 310 determines multi-finger gestures. For example, refer to... Figures 3C to 3E Electronic device 310 determines a multi-finger pinch gesture. In some specific implementations, electronic device 310 determines the multi-finger pinch gesture based on the satisfaction of a proximity threshold. For example, based on limb tracking data, electronic device 310 determines that the thumb... Figure 3C Move the first position in the middle to Figure 3E The second position in the process. Continuing this example, based on finger manipulation data, electronic device 310 determines the finger from wearable device 320. Figure 3C The third position in the middle is moved to Figure 3E The fourth position in the example. Continuing with this example, the electronic device 310 determines that the second position satisfies a proximity threshold relative to the fourth position, and thus determines a multi-finger pinch gesture. In some specific implementations, the second position satisfies the proximity threshold relative to the fourth position when it is less than a threshold distance from the fourth position.

[0073] Electronic device 310 registers engagement events for computer-generated object 330 based on multi-finger gestures. For example, refer to... Figure 3F In response to determining a multi-finger pinch gesture, electronic device 310 selects computer-generated object 330. In some implementations, electronic device 310 alters the appearance of computer-generated object 330 to indicate the selection. For example, as Figure 3E and Figure 3F As shown, the electronic device 310 changes the computer-generated object 330 from having a solid-line boundary to having a dashed-line boundary. The indication of this selection provides feedback to the user 50, thereby reducing erroneous (e.g., unintended) input from the finger wearable device 320 and reducing resource utilization associated with processing finger manipulation data and limb tracking data by the electronic device 310. In some specific implementations, the selection of the computer-generated object 330 is further in response to determining that the multi-finger gesture satisfies a proximity threshold (e.g., less than a threshold distance from the computer-generated object) relative to the computer-generated object 330. For example, when the display 312 includes multiple computer-generated objects, the electronic device 310 selects the specific computer-generated object closest to the multi-finger pinch gesture after completing the multi-finger pinch gesture.

[0074] For example, in some specific implementations, the registration and joining event includes manipulating computer-generated object 330. For instance, such as... Figure 3G As shown, electronic device 310 determines a multi-finger gesture corresponding to an upward movement performed by first hand 52. The upward movement is indicated by movement line 350 (shown for illustrative purposes only). As first hand 52 moves upward, electronic device 310 acquires limb tracking data and finger manipulation data. Based on the limb tracking data and finger manipulation data, electronic device 310 determines the multi-finger movement gesture and thus moves the computer-generated object 330 upward, such as... Figure 3H As shown.

[0075] Therefore, compared to other devices, it achieves more accurate and efficient engagement with computer-generated objects by utilizing multi-finger gestures that vary based on two different datasets (limb tracking data and finger manipulation data).

[0076] like Figures 3I to 3M As shown, electronic device 310 registers engagement events for computer-generated object 330 based on multi-handed multi-finger gestures. Utilizing single-handed multi-finger gestures (e.g., ... Figures 3C to 3H ) and multi-handed, multi-finger gestures (e.g., Figures 3I to 3M By combining computer-generated objects, users gain greater control, resulting in an improved user experience.

[0077] like Figure 3I and Figure 3J As shown, a finger wearable device 320 worn by user 50's first hand 52 performs a tapping gesture associated with the surface of physical table 302. The tapping gesture is indicated by a tapping indicator 352 (shown for illustrative purposes only). As the tapping gesture continues, electronic device 310 acquires finger manipulation data such as contact intensity data. Contact intensity data indicates the tapping force of the pressure associated with the tapping gesture. For example, when a finger touches the surface of physical table 302, finger wearable device 320 senses the deflection of the fingertip (e.g., via a contact intensity sensor). Thus, the various embodiments disclosed herein enable user 50 to feel the physical surface (and texture of that physical surface) with which user 50 is interacting. In some embodiments, electronic device 310 uses IMU data to predict the touch between the finger (wearing finger wearable device 320) and the surface of physical table 302, and uses interferometer data to detect the actual touch between the finger and the surface based on deflection. Based on the detected tapping gesture, electronic device 310 selects a computer-generated object 330, such as Figure 3J As shown.

[0078] like Figure 3KAs shown, after selecting the computer-generated object 330, the user 50's second hand 54 moves into the display 312. The electronic device 310 acquires limb tracking data associated with the second hand 54 via the limb tracking system 340. The reception of the limb tracking data is indicated by tracking lines 360 (shown for illustrative purposes only). The limb tracking data indicates the positional value associated with the middle finger of the second hand 54, as indicated by crosshairs 362 (shown for illustrative purposes only). Those skilled in the art will understand that in some embodiments, the limb tracking data indicates different fingers of the second hand 54, multiple fingers of the second hand 54, or different portions of the second hand 54 (e.g., the palm of the second hand 54).

[0079] like Figure 3L As shown, the middle finger of the second hand 54 begins a leftward movement gesture from the first position to the third position. The leftward movement gesture is indicated by movement line 364 (shown for illustrative purposes only). The first position is indicated by… Figure 3L The first position line 366 (shown for illustrative purposes only) indicates this. Furthermore, the computer-generated object 330 is in a second position before the middle finger moves to the left. The second position is determined by... Figure 3L The second position line 367 (shown for illustrative purposes only) indicates this.

[0080] As the middle finger moves to the left, limb tracking data indicates the change in position value associated with the middle finger. After the leftward movement gesture, the middle finger is positioned at a third position, which is... Figure 3M The third position is indicated by line 368 (shown for illustrative purposes only). Based on limb tracking data, the electronic device 310 correspondingly moves the selected computer-generated object 330 from the second position to the left to the fourth position. The fourth position is indicated by... Figure 3M The fourth position line 369 (shown for illustrative purposes only) indicates this.

[0081] In some implementations, as the second hand 54 performs a leftward movement gesture, the limb tracking data indicates more than one finger of the second hand 54. For example, the limb tracking data indicates corresponding position values ​​associated with the middle and index fingers of the second hand 54. Therefore, the electronic device 310 can register engagement events with respect to the computer-generated object 330 based on finger manipulation data and multi-finger limb tracking data.

[0082] Therefore, refer to Figures 3I to 3MBased on finger manipulation data (indicating a tapping gesture) and limb tracking data (indicating a leftward movement gesture), the electronic device 310 determines a multi-finger gesture. The multi-finger gesture includes a first portion corresponding to a tapping gesture associated with a first hand 52 and a second portion corresponding to a leftward movement gesture associated with a second hand 54. In some embodiments, the portions of the multi-finger gesture satisfy an ordering criterion relative to each other. For example, refer to… Figures 3I to 3M The electronic device 310 only detects the leftward movement gesture after a threshold time has elapsed since the tap gesture was detected.

[0083] Figure 4 This is an example of a flowchart illustrating a method 400 for registering engagement events of a computer-generated object based on multi-finger gestures, according to some specific implementations. In various specific implementations, method 400 or portions thereof are provided by electronic devices (e.g., Figure 1 Electronic devices 100 Figures 3A to 3M The method is executed by an electronic device 310. In various embodiments, method 400 or a portion thereof is executed by a head-mounted device (HMD). In some embodiments, method 400 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 400 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory). In various embodiments, some operations in method 400 are optionally combined, and / or the order of some operations is optionally changed.

[0084] As indicated by box 402, method 400 includes displaying computer-generated objects, such as 2D or 3D objects. For example, see reference... Figure 3B The electronic device 310 displays a computer-generated object 330 corresponding to a 3D cube on the display 312.

[0085] As indicated by box 404, when displaying a computer-generated object, method 400 includes obtaining limb tracking data from a limb tracking system. In some embodiments, the limb tracking system performs computer vision techniques to generate the limb tracking data. For example, the limb tracking system semantically identifies one or more limbs within image data and generates limb tracking data indicating the corresponding positions of the one or more limbs within the image data. In some embodiments, the limb tracking data indicates multiple position values ​​associated with a limb. To this end, in some embodiments, method 400 includes identifying a limb within image data at multiple times and determining multiple position values ​​based on that identification. The multiple position values ​​are associated with multiple times, respectively.

[0086] As indicated by box 406, when displaying a computer-generated object, method 400 includes obtaining finger manipulation data from a wearable finger device via a communication interface. For example, as shown in reference... Figures 3A to 3M As described, electronic device 310 obtains various types of finger manipulation data from finger wearable device 320. The finger manipulation data can indicate positional information (e.g., six degrees of freedom information) and contact intensity information (e.g., force or pressure information) associated with the finger wearable device. In some embodiments, the finger manipulation data indicates gestures performed by the finger wearable device.

[0087] Depending on the specific implementation, the finger manipulation data corresponds to sensor data associated with one or more sensors integrated into the finger wearable device. For example, as indicated by box 408, the sensor data includes position data output from one or more position sensors integrated into the finger wearable device. For instance, the position data indicates rotational movement (e.g., IMU data) and / or translational movement (e.g., magnetic sensor data) of the finger wearable device, such as… Figure 3G and Figure 3H As shown in the diagram. In some specific implementations, magnetic sensor data is output by a magnetic sensor integrated within a wearable finger device, where the magnetic sensor senses a weak magnetic field.

[0088] For example, as shown in box 410, sensor data includes contact strength data output from a contact strength sensor integrated into a finger wearable device. For instance, refer to... Figure 3I and Figure 3J Electronic device 310 obtains contact intensity data from finger wearable device 320. Continuing this example, finger wearable device 320 generates contact intensity data based on a tap of a finger of the first hand 52 (wearing finger wearable device 320) on the surface of physical table 302. For this purpose, when the finger contacts the physical surface, the finger wearable device can sense the deflection of the fingertip (e.g., via a contact intensity sensor). Thus, the various embodiments disclosed herein enable a user to feel the physical surface (and the texture of that physical surface) with which the user is interacting. For example, the contact intensity data includes interferometer data indicating the tapping pressure associated with a gesture performed by the finger wearable device. The interferometer data may come from an interferometer integrated within the finger wearable device. The interferometer data may indicate the pressure level associated with the finger wearing the finger wearable device and contacting the physical object. For example, in some embodiments, the sensor data includes a combination of position data and contact intensity data.

[0089] In some implementations, the finger-wearable device is not visible on the display when acquiring finger manipulation data. For example, a user's hand, including the finger wearing the finger-wearable device, rests on the edge of a sofa. Continuing this example, based on a finger tap, the electronic device acquires contact strength data indicating the tap. Resting the user's hand on a physical object results in a more comfortable user experience, leading to potentially fewer erroneous (e.g., unintended) gestures performed by the finger-wearable device, and therefore less processor and communication link resource utilization by the electronic device.

[0090] As indicated by box 412, in some embodiments, method 400 includes detecting the satisfaction of gesture triggering criteria. In some embodiments, detecting the satisfaction of triggering criteria includes determining that either limb tracking data or finger manipulation data indicates a specific gesture type (e.g., a tapping gesture or a double-tap gesture). For example, detecting the satisfaction of triggering criteria includes determining that contact intensity data from a finger wearable device indicates a tapping gesture on a physical surface. In some embodiments, detecting the satisfaction of triggering criteria includes determining that audio input (e.g., user speech detected by an integrated microphone) indicates a request to begin registering a multi-finger gesture. To this end, in some embodiments, the electronic device includes a speech detection system (e.g., a natural language processing (NLP) system) that generates semantic values ​​associated with the audio input, such as a semantic value for “starting the gesture”.

[0091] As indicated by box 414, method 400 includes determining a multi-finger gesture based on limb tracking data from a limb tracking system and finger manipulation data. In some embodiments, determining the multi-finger gesture is based in part on multiple location values ​​indicated by the limb tracking data as described in reference box 404. In some embodiments, determining the multi-finger gesture is in response to the detection of a satisfaction of a gesture triggering criterion as described in reference box 412.

[0092] In some specific implementations, determining a multi-finger gesture includes determining a first gesture based on limb tracking data and determining a second gesture based on finger manipulation data. The first gesture corresponds to a first part of the multi-finger gesture, and the second gesture corresponds to a second part of the multi-finger gesture that differs from the first part. For example, refer to... Figures 3C to 3E The electronic device 310 determines a first part of a multi-finger pinch gesture based on limb tracking data and a second part of the multi-finger pinch gesture based on finger manipulation data. Therefore, in some implementations, the multi-finger gesture corresponds to a single-handed gesture (e.g., Figures 3C to 3E (See the first hand part, 52). For example, see reference... Figures 3I to 3MThe electronic device 310 determines that the multi-finger gesture corresponds to a first portion of a tap associated with a finger wearable device 320 worn by the first hand 52, and determines that the multi-finger gesture corresponds to a second portion of a leftward movement associated with the second hand 54. Therefore, in some specific embodiments, the multi-finger gesture corresponds to a multi-hand gesture (e.g., Figures 3I to 3M The first hand part 52 and the second hand part 54).

[0093] In some implementations, method 400 includes determining that a multi-finger gesture corresponds to a multi-finger pinch gesture. To this end, method 400 includes determining that limb tracking data indicates movement from a first position to a second position, determining that finger manipulation data indicates movement from a third position to a fourth position, and determining that the second position satisfies a proximity threshold relative to the fourth position. For example, method 400 includes obtaining finger manipulation data from a finger wearable device worn on a user's index finger, and obtaining limb tracking data associated with the user's thumb. Continuing this example, method 400 includes detecting a multi-finger pinch gesture in response to determining that the finger manipulation data and limb tracking data together indicate that the user's thumb and index finger have moved less than a threshold distance relative to each other.

[0094] As indicated in box 416, in some implementations, the first part of a multi-finger gesture satisfies an order criterion relative to the second part of the multi-finger gesture. For example, determining the second part of the multi-finger gesture occurs after a threshold time has elapsed since determining the first part of the multi-finger gesture. For example, refer to... Figures 3I to 3M Electronic device 310 begins to determine a leftward movement gesture associated with second hand 54 after a threshold time (e.g., two seconds) has elapsed since a tapping gesture associated with finger wearable device 320 was determined.

[0095] As indicated in box 418, in some implementations, the first part of a multi-finger gesture satisfies a concurrency criterion relative to the second part of the multi-finger gesture. For example, determining the second part of the multi-finger gesture occurs less than a threshold amount of time after determining the first part of the multi-finger gesture. For example, refer to... Figures 3C to 3E The electronic device 310 determines the first and second parts of the multi-finger pinch gesture (associated with limb tracking data and finger manipulation data, respectively) within a threshold time range between each other.

[0096] As shown in box 420, method 400 includes registering engagement events for a computer-generated object based on a multi-finger gesture. Engagement events may include various different types of engagement with the computer-generated object.

[0097] For example, as shown in box 422, the registration engagement event includes selecting a computer-generated object. For example, selecting a computer-generated object includes determining that a multi-finger gesture corresponds to a multi-finger pinch gesture, such as a reference... Figures 3B to 3FThe multi-finger pinch gesture is described. In some implementations, method 400 includes determining a corresponding location associated with the multi-finger gesture, wherein selecting a computer-generated object includes determining that the corresponding location satisfies a proximity threshold relative to the computer-generated object.

[0098] For example, as indicated in box 424, registration and joining events include manipulating computer-generated objects. Manipulating computer-generated objects may include one or more of the following operations on the computer-generated object: moving, rotating, copying, resizing, changing color / brightness, translating (e.g., along an axis), stopping display, etc. For example, refer to Figures 3I to 3M The electronic device 310 moves the selected computer-generated object 330 to the left based on finger manipulation data (instruction to select computer-generated object 330) and limb tracking data (instruction to move the middle finger of the second hand 54 to the left).

[0099] This disclosure describes various features, none of which alone can achieve the benefits described herein. It should be understood that the various features described herein can be combined, modified, or omitted, as will be apparent to those skilled in the art. Other combinations and sub-combinations beyond those specifically described herein will be apparent to those skilled in the art and are intended to form part of this disclosure. Various methods are described herein in conjunction with various flowchart steps and / or stages. It should be understood that in many cases, certain steps and / or stages can be combined such that multiple steps and / or stages shown in the flowchart can be performed as a single step and / or stage. Additionally, certain steps and / or stages can be divided into additional sub-components to be performed independently. In some cases, the order of steps and / or stages can be rearranged, and certain steps and / or stages can be omitted entirely. Furthermore, the methods described herein should be understood to be broadly interpretable, such that additional steps and / or stages beyond those shown and described herein can also be performed.

[0100] Some or all of the methods and tasks described herein can be performed and fully automated by a computer system. In some cases, the computer system may include multiple different computers or computing devices (e.g., physical servers, workstations, storage arrays, etc.) that communicate and interoperate via a network to perform the functions described herein. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in memory or other non-transitory computer-readable storage media or devices. The various functions disclosed herein may be implemented in such program instructions, but alternatively, some or all of the disclosed functions may be implemented in the computer system's dedicated circuitry (e.g., ASIC, FPGA, or GP-GPU). In cases where the computer system includes multiple computing devices, these devices may be located in the same location or not. The results of the disclosed methods and tasks can be persistently stored by converting physical storage devices such as solid-state memory chips and / or disks into different states.

[0101] The various processes defined herein take into account options for obtaining and using users' personal information. For example, such personal information may be used to provide improved privacy screens on electronic devices. However, the extent to which such personal information is collected should be based on the user's informed consent. As described herein, users should understand and control the use of their personal information.

[0102] Personal information will be used by the appropriate parties only for lawful and reasonable purposes. Parties using such information will comply with privacy policies and practices that are at least in accordance with applicable laws and regulations. Furthermore, such policies should be comprehensive, user-accessible, and considered to meet or exceed government / industry standards. In addition, parties may not distribute, sell, or otherwise share such information except for any reasonable and lawful purpose.

[0103] However, users can limit the extent to which parties can access or otherwise obtain their personal information. For example, settings or other preferences can be adjusted so that users can decide whether their personal information can be accessed by various entities. Furthermore, while some of the characteristics defined herein are described in the context of the use of personal information, aspects of these characteristics can be implemented without the need for such information. For example, if user preferences, account names, and / or location history are collected, this information can be obfuscated or otherwise generalized so that it does not identify the corresponding user.

[0104] This disclosure is not intended to be limited to the specific embodiments shown herein. Various modifications to the specific embodiments described herein will be apparent to those skilled in the art, and the general principles defined herein can be applied to other specific embodiments without departing from the spirit or scope of this disclosure. The teachings of the invention provided herein can be applied to other methods and systems, and are not limited to those described above, and elements and actions of the various specific embodiments described above can be combined to provide further specific embodiments. Therefore, the novel methods and systems described herein can be implemented in many other forms; furthermore, various omissions, substitutions, and changes can be made to the form of the methods and systems described herein without departing from the spirit of this disclosure. The appended claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of this disclosure.

Claims

1. A method to be performed at an electronic device, comprising: In an electronic device having one or more processors, non-transitory memory, a limb tracking system, a display, and a communication interface configured to communicate with a wearable finger device: When displaying a computer-generated object on the monitor: Finger manipulation data is obtained from the wearable finger device via the communication interface, wherein the finger manipulation data indicates a corresponding movement path of the first finger from a first position in the extended reality XR environment to a second position in the XR environment during a first time period; The limb tracking system generates limb tracking data by performing computer vision, wherein performing computer vision includes determining a corresponding movement path of a second finger across multiple images, wherein the corresponding movement path of the second finger is from a third position to a fourth position of the XR environment during a second time period, and wherein the corresponding movement path of the first finger is different from the corresponding movement path of the second finger, and the first time period and the second time period at least partially overlap. Multi-finger gestures are determined based on the limb tracking data from the limb tracking system and the finger manipulation data, wherein the determination of the multi-finger gestures is based on the corresponding movement path of the first finger and the corresponding movement path of the second finger; as well as The multi-finger gesture is used to register engagement events for the computer-generated object.

2. The method of claim 1, wherein determining the multi-finger gesture comprises: A first gesture is determined based on the limb tracking data, wherein the first gesture corresponds to the first part of the multi-finger gesture; as well as A second gesture is determined based on the finger manipulation data, wherein the second gesture corresponds to a second part of the multi-finger gesture that is different from the first part of the multi-finger gesture.

3. The method of claim 2, wherein the first portion of the multi-finger gesture satisfies an order criterion relative to the second portion of the multi-finger gesture.

4. The method of claim 2, wherein the first portion of the multi-finger gesture satisfies a concurrency criterion relative to the second portion of the multi-finger gesture.

5. The method according to any one of claims 1 to 4, wherein the limb tracking data indicates a plurality of position values ​​associated with the plurality of images respectively, wherein the determination of the multi-finger gesture is based in part on the plurality of position values, and wherein the generation of the limb tracking data is independent of the finger manipulation data.

6. The method of claim 5, further comprising: The second finger is identified within the multiple images at multiple times; as well as Based on the identification, the plurality of location values ​​are determined, wherein the plurality of location values ​​are respectively associated with the plurality of times.

7. The method according to any one of claims 1 to 4 and 6, wherein the limb tracking data is associated with a first hand, wherein the finger wearable device is associated with a second hand different from the first hand, and wherein the multi-finger gesture corresponds to a multi-hand gesture.

8. The method according to any one of claims 1 to 4 and 6, wherein the limb tracking data is associated with a first hand, wherein the finger wearable device is also associated with the first hand, and wherein the multi-finger gesture corresponds to a single-hand gesture.

9. The method according to any one of claims 1 to 4 and 6, wherein registering the engagement event includes selecting the computer-generated object.

10. The method of claim 9, wherein selecting the computer-generated object includes determining that the multi-finger gesture corresponds to a pinch gesture.

11. The method of claim 10, wherein determining that the multi-finger gesture corresponds to the pinch gesture comprises: The limb tracking data is determined to indicate movement from a first position to a second position; The finger manipulation data indicates movement from the third position to the fourth position; as well as The second position is determined to satisfy a proximity threshold relative to the fourth position.

12. The method of claim 9, further comprising determining a corresponding position associated with the multi-finger gesture, wherein selecting the computer-generated object includes determining that the corresponding position satisfies a proximity threshold relative to the computer-generated object.

13. The method according to any one of claims 1 to 4, 6 and 10 to 12, wherein registering the engagement event includes manipulating the computer-generated object.

14. The method according to any one of claims 1 to 4, 6 and 10 to 12, wherein the finger wearable device is invisible on the display when the finger manipulation data is obtained.

15. The method according to any one of claims 1 to 4, 6 and 10 to 12, further comprising detecting the satisfaction of a gesture triggering criterion, wherein the multi-finger gesture is determined in response to the detection of the satisfaction of the gesture triggering criterion.

16. The method of claim 15, wherein detecting the satisfaction of the triggering criterion includes determining that one of the limb tracking data or the finger manipulation data indicates a specific gesture type.

17. The method according to any one of claims 1 to 4, 6, 10 to 12 and 16, wherein the finger manipulation data corresponds to sensor data output from one or more sensors integrated within the finger wearable device.

18. The method of claim 17, wherein the sensor data includes position data output from one or more position sensors integrated in the wearable finger device.

19. The method of claim 17, wherein the sensor data includes contact strength data output from a contact strength sensor integrated in the wearable finger device.

20. The method according to any one of claims 1 to 4, 6, 10 to 12, 16 and 18 to 19, wherein the electronic device corresponds to a head-mounted device (HMD).

21. An electronic device, comprising: One or more processors; Non-transitory memory; monitor; Body tracking system; A communication interface configured to communicate with a finger-wearable device; and One or more programs, wherein the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for the following operations: When displaying a computer-generated object on the monitor: Finger manipulation data is obtained from the wearable finger device via the communication interface, wherein the finger manipulation data indicates the corresponding movement path of the first finger from a first position in the extended reality XR environment to a second position in the XR environment within a first time period; The limb tracking system generates limb tracking data by performing computer vision, wherein performing computer vision includes determining a corresponding movement path of a second finger across multiple images, wherein the corresponding movement path of the second finger is from a third position to a fourth position of the XR environment within a second time period, and wherein the corresponding movement path of the first finger is different from the corresponding movement path of the second finger, and the first time period and the second time period at least partially overlap. Multi-finger gestures are determined based on the limb tracking data from the limb tracking system and the finger manipulation data, wherein the determination of the multi-finger gestures is based on the corresponding movement path of the first finger and the corresponding movement path of the second finger; as well as The multi-finger gesture is used to register engagement events for the computer-generated object.

22. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by an electronic device having one or more processors, a display, a limb tracking system, and a communication interface configured to communicate with a wearable finger device, cause said electronic device to perform the following operations: When displaying a computer-generated object on the monitor: Finger manipulation data is obtained from the wearable finger device via the communication interface, wherein the finger manipulation data indicates a corresponding movement path of the first finger from a first position in the extended reality XR environment to a second position in the XR environment during a first time period; The limb tracking system generates limb tracking data by performing computer vision, wherein performing computer vision includes determining a corresponding movement path of a second finger across multiple images, wherein the corresponding movement path of the second finger is from a third position to a fourth position of the XR environment during a second time period, and wherein the corresponding movement path of the first finger is different from the corresponding movement path of the second finger, and the first time period and the second time period at least partially overlap. Multi-finger gestures are determined based on the limb tracking data from the limb tracking system and the finger manipulation data, wherein the determination of the multi-finger gestures is based on the corresponding movement path of the first finger and the corresponding movement path of the second finger; as well as The multi-finger gesture is used to register engagement events for the computer-generated object.

Images