Mapping touch and gesture controls to increase control options

Abstract

Systems and techniques for controlling devices are disclosed. For example, a computing device can detect a first gesture among multiple first gestures of a user based on sensor data from one or more sensors. The computing device can determine a first settings list among multiple settings lists based on the first gesture. The first settings list includes multiple control options. The computing device can detect a control gesture among multiple control gestures of the user. Each control gesture is associated with a different control option among multiple control options. The computing device can determine a first control option among the multiple control options in the first settings list based on the control gesture. The computing device can enable the first control option to control the device.

Description

Technical Field

[0001] This disclosure relates in general to control options for devices. For example, aspects of this disclosure relate to mapping touch and gesture controls to increase control options. Background Technology

[0002] Extended reality (XR) systems or devices (e.g., virtual reality, augmented reality, mixed reality) can provide users with virtual experiences (e.g., XR experiences) by immersing them in a completely virtual environment (composed of virtual content), and / or by combining a real-world or physical environment with a virtual environment to provide extended reality or mixed reality experiences. XR systems or devices may include VR systems that facilitate interaction with virtual reality (VR) environments, AR systems that facilitate interaction with augmented reality (AR) environments, MR systems that facilitate interaction with mixed reality (MR) environments, and / or other XR systems. Examples of XR systems or devices include head-mounted displays (HMDs), such as AR / VR HMDs, which may take the form of smart glasses. XR systems or devices (e.g., HMDs) may be paired with or utilize a touchpad that can be used by the user to control the XR experience from the XR system or device. Summary of the Invention

[0003] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all conceived aspects, nor should it be considered to identify key or decisive elements relating to all conceived aspects or to depict the scope associated with any particular aspect. Thus, the sole purpose of this summary is to present, in a simplified form, certain concepts relating to one or more aspects involving the mechanisms disclosed herein, prior to the detailed description presented below.

[0004] Systems, apparatuses, methods, and computer-readable media for mapping touch and gesture controls to increase control options are disclosed. According to at least one example, an apparatus for enabling one or more control options is provided. The apparatus includes at least one memory and at least one processor coupled to the at least one memory and configured to: detect a first gesture among a plurality of first gestures of a user based on sensor data from one or more sensors; determine a first settings list among a plurality of settings lists based on the first gesture, wherein the first settings list includes a plurality of control options; detect a control gesture among a plurality of control gestures of the user, wherein each of the plurality of control gestures is associated with a different control option among the plurality of control options; determine a first control option among the plurality of control options in the first settings list based on the gesture; and enable the first control option to control the apparatus.

[0005] In another exemplary example, a method for controlling a device such as an XR device is provided. The method includes: using one or more sensors to detect a first gesture among a plurality of first gestures of a user; determining, by one or more processors, a first settings list among a plurality of settings lists based on the first gesture, wherein the first settings list includes a plurality of control options; detecting a control gesture among a plurality of control gestures of the user, wherein each of the plurality of control gestures is associated with a different control option among the plurality of control options; determining, by the one or more processors, a first control option among the plurality of control options in the first settings list based on the control gesture; and enabling, by the one or more processors, the first control option to control the device.

[0006] In another exemplary example, a non-transitory computer-readable medium for a device is provided, the non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: detect a first gesture among a plurality of first gestures of a user based on sensor data from one or more sensors; determine a first settings list among a plurality of settings lists based on the first gesture, wherein the first settings list includes a plurality of control options; detect a control gesture among a plurality of control gestures of the user, wherein each of the plurality of control gestures is associated with a different control option among the plurality of control options; determine a first control option among the plurality of control options in the first settings list based on the control gesture; and enable the first control option to control the device.

[0007] In another exemplary example, an apparatus is provided, comprising: means for detecting a first gesture among a plurality of first gestures of a user based on sensor data from one or more sensors; means for determining a first settings list among a plurality of settings lists based on the first gesture, wherein the first settings list includes a plurality of control options; means for detecting a control gesture among a plurality of control gestures of the user, wherein each of the plurality of control gestures is associated with a different control option among the plurality of control options; means for determining a first control option among the plurality of control options in the first settings list based on the control gesture; and means for enabling the first control option to control the device.

[0008] The aspects generally include, as described substantially with reference to the accompanying drawings and description and illustrated as shown in the drawings and description, methods, apparatus, systems, computer program products, non-transitory computer-readable media, user equipment, user gear, wireless communication equipment, and / or processing systems.

[0009] In some aspects, one or more of the devices described herein may include, or be a subset of, extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), mobile devices (e.g., mobile phones or other mobile devices), wearable devices (e.g., network-connected watches or other wearable devices), personal computers, laptop computers, server computers, televisions, video game consoles, or other devices. In some aspects, the device also includes at least one camera for capturing one or more images or video frames. For example, the device may include one or more cameras (e.g., an RGB camera) for capturing one or more images and / or one or more videos including video frames. In some aspects, the device includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the device includes a transmitter configured to transmit data or information to at least one device via a transmission medium. In some aspects, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or other processing devices or components.

[0010] Some aspects include a device having a processor configured to perform one or more operations of any of the methods outlined above. Further aspects include a processing device for use in the device, configured using processor-executable instructions to perform operations of any of the methods outlined above. Further aspects include a non-transitory processor-readable storage medium storing processor-executable instructions thereon configured to cause the device's processor to perform operations of any of the methods outlined above. Further aspects include a device having components for performing functions of any of the methods outlined above.

[0011] The features and technical advantages of the examples according to this disclosure have been summarized quite extensively above in order to better understand the detailed description below. Additional features and advantages will be described below. The disclosed concepts and specific examples can be readily utilized as the basis for modifying or designing other structures for achieving the same purpose of this disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein (both their organization and operation) and their associated advantages will be better understood from the following description when considered in conjunction with the accompanying drawings. Each figure in the drawings is provided for illustrative and descriptive purposes and not as a definition of limitation of the claims. The foregoing, as well as other features and aspects, will become more apparent upon reference to the following specification, claims, and appended drawings.

[0012] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0013] Based on the accompanying drawings and detailed description, other objects and advantages associated with the aspects disclosed herein will be apparent to those skilled in the art. Attached Figure Description

[0014] The exemplary aspects of this application are described in detail below with reference to the following figures: Figure 1 This is a block diagram illustrating the architecture of an image capture and processing system according to various aspects of this disclosure.

[0015] Figure 2 This is a diagram illustrating the architecture of an example extended reality (XR) system based on some aspects of this disclosure.

[0016] Figure 3A Examples of augmented reality enhancement application engines based on various aspects of this disclosure are illustrated.

[0017] Figure 3B This is a block diagram illustrating an example system for hand tracking according to various aspects of this disclosure.

[0018] Figure 4 This is an illustration of examples of hand markers according to various aspects of this disclosure.

[0019] Figure 5A This is a perspective view of an HMD that illustrates the performance of feature tracking and / or visual simultaneous localization and mapping (VSLAM) based on some examples.

[0020] Figure 5B This is a perspective view illustrating how an HMD is being worn by a user, based on some examples.

[0021] Figure 6A This is a perspective view of the front surface of a mobile device that performs feature tracking and / or VSLAM using one or more front-facing cameras, according to some examples.

[0022] Figure 6B This is a perspective view illustrating the rear surface of a mobile device according to various aspects of this disclosure.

[0023] Figure 7 This is an illustration of an example of a user operating a gesture-controlled device according to various aspects of this disclosure.

[0024] Figure 8The diagram illustrates examples of different gestures that can be used to control the operation of a device according to various aspects of this disclosure.

[0025] Figure 9 The diagram illustrates examples of different combinations of gestures and touch gestures for different operations of a device according to various aspects of this disclosure.

[0026] Figure 10 This is an example of the correspondence between various aspects of this disclosure. Figure 9 A table showing examples of different operations using different combinations of gestures and touch gestures.

[0027] Figure 11 The illustrations are examples of different hand gestures that indicate different numbers according to various aspects of this disclosure.

[0028] Figure 12 This is a diagram illustrating an example of a process for controlling a device according to various aspects of this disclosure, where no numbers are detected and a default settings list is invoked.

[0029] Figure 13 This is a diagram illustrating an example of a process for controlling a device according to various aspects of this disclosure, wherein the number two is detected and an associated second settings list is invoked.

[0030] Figure 14 This is a flowchart illustrating an example of a process for controlling a device, based on some examples.

[0031] Figure 15 This is a diagram illustrating an example of a system used to implement some of the aspects described in this article. Detailed Implementation

[0032] Certain aspects of this disclosure are provided below for illustrative purposes. Alternative aspects may be devised without departing from the scope of this disclosure. Additionally, well-known elements of this disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of this disclosure. Some aspects described herein can be applied independently, and some of them can be combined, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation to provide a thorough understanding of various aspects of this application. However, it will be apparent that various aspects can be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0033] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the exemplary aspects will provide those skilled in the art with a description that can be used to implement the exemplary aspects. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0034] The terms “exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as superior to or better than other aspects. Similarly, the term “aspects of this disclosure” does not require that all aspects of this disclosure include the features, advantages, or modes of operation discussed.

[0035] As previously mentioned, XR (e.g., virtual reality, augmented reality, mixed reality) systems or devices can provide users with virtual experiences (e.g., XR experiences) by immersing them in a completely virtual environment (composed of virtual content), and / or by combining a real-world or physical environment with a virtual environment to provide extended reality or mixed reality experiences. XR systems or devices may include VR systems that facilitate interaction with VR environments, AR systems that facilitate interaction with AR environments, MR systems that facilitate interaction with MR environments, and / or other XR systems. Examples of XR systems or devices may include HMDs, such as AR / VR HMDs, which may take the form of smart glasses. XR systems or devices (e.g., HMDs) may be paired with or utilize a touchpad that can be used by the user to control the XR experience from the XR system or device.

[0036] Currently, many XR devices (e.g., in the form of HMDs) support user control via gestures (e.g., gesture controls such as clenching a fist, directional waving, rapid swiping, pinching, etc.). For example, users can use gestures to control various operations on the XR device, such as scrolling, capturing, zooming, selecting, and / or controlling the device's volume. Gesture recognition may not always be accurate, as some users may perform gestures slightly incorrectly, causing the device to fail to recognize the gesture and thus prevent the associated action from occurring.

[0037] Many XR devices (e.g., HMDs) have touchpads, typically implemented within the left or right hinge of the device. These touchpads are used by the user to control the device. Gesture controls are often used to assist in controlling these devices because a single touchpad can only be assigned a minimal number of functionalities. For example, a single touchpad may only allow a limited number of possible touch gestures (e.g., single tap, double tap, long press, forward swipe, and / or backward swipe), which provide the user with a minimal number of control options (e.g., each touch gesture may be assigned only one control option, and therefore there may only be five possible control options in total) for controlling the device.

[0038] Therefore, it may be beneficial to provide users with a number of control options for controlling devices (e.g., XR devices, such as those in the form of HMD devices) while allowing for accurate detection of user gestures.

[0039] Therefore, this document describes systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") that provide mechanisms for controlling devices (e.g., XR devices, such as those in the form of HMD devices). In one or more aspects, the systems and techniques combine touch gestures with basic gestures to provide users with a number of control options for controlling devices (e.g., XR devices) in a more convenient and efficient manner. The combination of touch and gestures can be used to define a more comprehensive range of control options for controlling the device, while providing more accurate gesture recognition due to the simplicity of the executable gestures.

[0040] In some respects, systems and technologies combine a first gesture (e.g., via the first hand) with a second gesture (e.g., via either the first or second hand) to provide users with a number of control options for controlling devices (e.g., XR devices) in a more convenient and efficient manner. The combination of the first and second gestures can be used to define a more comprehensive range of control options for controlling the device.

[0041] In one or more aspects, for controlling a device (e.g., an XR device, such as in the form of an HMD), one or more sensors can detect gestures among multiple hand gestures of a user. In one or more examples, each of the one or more sensors can be a red-green-blue (RGB) image sensor (e.g., a camera) or a monochrome image sensor (e.g., a camera). In one or more examples, each of the one or more sensors can be implemented within the device itself or within another device. In some examples, the other device can be a mobile device associated with the user, such as a smartphone or wearable device. In one or more examples, the other device (e.g., a smartphone) can communicate with the device (e.g., the XR device) wirelessly (e.g., via Bluetooth) and / or via wired means.

[0042] In one or more examples, the touchpad may be implemented within the device (e.g., an XR device). In some examples, the touchpad may be implemented within the left or right hinge of the device. To control the device, the touchpad may detect a touch gesture among multiple touch gestures of the user. In one or more examples, the multiple touch gestures may include, but are not limited to, single-click, double-click, long-press, forward swipe, and / or backward swipe. In one or more examples, the detection of gestures and the detection of touch gestures occur simultaneously (e.g., at the same time). For example, a user may perform a touch gesture with one hand while simultaneously performing a gesture (e.g., raising several fingers) with the user's other hand.

[0043] In one or more examples, one or more processors may determine a number associated with the gesture. In one or more examples, the gesture may take the form of a user raising a certain number of fingers in front of one or more sensors (e.g., this can be easily recognized by the device) to indicate a specific number (e.g., the same number of fingers). In some examples, each of the one or more processors may be implemented within a device (e.g., an XR device) or within another device (e.g., a mobile device associated with the user, such as a smartphone or wearable device).

[0044] One or more processors may determine a set of settings among multiple set of settings based on a determined number associated with a detected gesture. In one or more examples, each set of settings among the multiple set of settings may be associated with a number. For example, when the number determined from the gesture is one (e.g., the user raises one finger), one or more processors may determine a first set of settings (e.g., set of settings one among three different set of settings in total), when the number determined from the gesture is two (e.g., the user raises two fingers), the one or more processors may determine a second set of settings (e.g., set of settings two among three different set of settings in total), and so on.

[0045] In one or more examples, each different settings list among multiple settings lists may be associated with a different control option. In some examples, control options may include, but are not limited to, pause, select, next, voice assistance, volume up, volume down, brightness up, brightness down, camera launch, zoom in, and / or zoom out. In one or more examples, each control option may be associated with a specific touch gesture.

[0046] One or more processors may determine a control option among multiple control options in a settings list based on a touch gesture. For example, when a gesture associated with the number two is used to indicate a second settings list (e.g., settings list two), a click touch gesture may be used to specifically enable a control option for increasing the brightness of the device's display. The one or more processors may then cause the determined control option (e.g., for increasing the brightness of the display) to occur.

[0047] In one or more examples, after one or more processors determine a set list (e.g., set list two) from a plurality of set lists based on a determined number (e.g., two) associated with a detected gesture, the display of a device (e.g., an XR device) may display the determined set list (e.g., set list two) to the user. In some examples, the device's display may display the determined set list on the display as an overlay image.

[0048] In one or more aspects, the system and technology increase the user input operations (e.g., control options) of a device (e.g., an XR device) to a larger set of operations by using only a small number of different gestures (e.g., only five different gestures in total). Therefore, the system and technology provide a significantly increased number of efficient control options for the user to control the device. In one or more examples, a camera from a mobile phone may be used as the source of input images for the device (e.g., an XR device). The camera may acquire images capturing gestures used by the user to control the device. The device may communicate with the mobile phone via a wireless communication protocol (e.g., via Bluetooth), and the device may receive images from the mobile phone via the wireless communication protocol.

[0049] Additional aspects of this disclosure are described in more detail below.

[0050] Various aspects of this application will be described with reference to the accompanying drawings. Figure 1This is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., an image of scene 110). The image capture and processing system 100 can capture individual images (or photographs) and / or capture video comprising multiple images (or video frames) in a specific sequence. In some cases, a lens 115 and an image sensor 130 may be associated with an optical axis. In one exemplary example, both the photosensitive area of ​​the image sensor 130 (e.g., a photodiode) and the lens 115 may be centered on the optical axis. The lens 115 of the image capture and processing system 100 faces scene 110 and receives light from scene 110. The lens 115 bends the incident light from the scene toward the image sensor 130. The light received by the lens 115 passes through an aperture. In some cases, the aperture (e.g., aperture size) is controlled by one or more control mechanisms 120 and received by the image sensor 130. In some cases, the aperture may have a fixed size.

[0051] One or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or information from image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components; for example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms besides those illustrated, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0052] The focus control mechanism 125B of the control mechanism 120 can obtain the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the positioning of the lens 115 relative to the positioning of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to or further away from the image sensor 130 by actuating a motor or servo system (or other lens mechanism), thereby adjusting the focus. In some cases, an additional lens may be included in the image capture and processing system 100, such as one or more microlenses on each photodiode of the image sensor 130, each of which bends light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings. In some cases, the lens 115 may be fixed relative to the image sensor, and the focus control mechanism 125B may be omitted without departing from the scope of this disclosure.

[0053] The exposure control mechanism 125A of the control mechanism 120 can obtain the exposure settings. In some cases, the exposure control mechanism 125A stores the exposure settings in a memory register. Based on the exposure settings, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or aperture value), the duration of aperture opening (e.g., exposure time or shutter speed), the duration of light collection by the sensor (e.g., exposure time or electronic shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure settings may be referred to as image capture settings and / or image processing settings.

[0054] The zoom control mechanism 125C of the control mechanism 120 can obtain zoom settings. In some examples, the zoom control mechanism 125C stores the zoom settings in a memory register. Based on the zoom settings, the zoom control mechanism 125C can control the focal length of an assembly (lens assembly) of lens elements including lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servo systems (or other lens mechanisms) to move one or more lenses in the lens relative to each other. The zoom settings may be referred to as image capture settings and / or image processing settings. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, this focusing lens may be lens 115) that first receives light from scene 110, where the light then passes through a focusless zoom system between the focusing lens (e.g., lens 115) and image sensor 130 before reaching image sensor 130. In some cases, a focusless zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference between them), with a negative (e.g., diverging, concave) lens between the two positive lenses. In some cases, the zoom control mechanism 125C moves one or more lenses in the focusless zoom system, such as a negative lens and one or both positive lenses. In some cases, the zoom control mechanism 125C can control zoom by capturing images from an image sensor (e.g., including image sensor 130) among a plurality of image sensors at a zoom setting corresponding to the zoom setting. For example, the image processing system 100 may include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a greater zoom. In some cases, the zoom control mechanism 125C may capture images from a corresponding sensor based on the selected zoom setting.

[0055] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered by different filters. In some cases, different photodiodes may be covered in different color filters, and light that matches the color of the filter covering the photodiode can thus be measured. Various color filter arrays can be used, including Bayer color filter arrays, four-color color filter arrays (also known as four-color Bayer color filter arrays or QCFA), and / or any other color filter array. For example, a Bayer color filter includes a red color filter, a blue color filter, and a green color filter, wherein each pixel of the image is generated based on red light data from at least one photodiode covered in the red color filter, blue light data from at least one photodiode covered in the blue color filter, and green light data from at least one photodiode covered in the green color filter.

[0056] Return to Figure 1 Other types of color filters can be used as alternatives to or supplements to red, blue, and / or green filters, such as yellow, magenta, and / or cyan (also known as "emerald green"). In some cases, photodiodes can be configured to measure infrared (IR) light. In some implementations, the photodiode measuring IR light may not be covered by any filter, thus allowing the IR photodiode to measure both visible light (e.g., color) and IR light. In some examples, the IR photodiode may be covered by an IR filter, thus allowing IR light to pass through and blocking light from other parts of the spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 130) may lack filters entirely (e.g., color, IR, or any other part of the spectrum) and may alternatively use different photodiodes (in some cases, vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array can have different spectral sensitivity profiles, thereby responding to light of different wavelengths. Monochrome image sensors may also lack filters and therefore lack color depth.

[0057] In some cases, image sensor 130 may optionally or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, the opaque and / or reflective mask may be used for phase detection autofocus (PDAF). In some cases, the opaque and / or reflective mask may be used to block portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., IR cutoff filters, UV cutoff filters, bandpass filters, low-pass filters, high-pass filters, etc.). Image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output from the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output from the photodiodes (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms in control mechanism 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge-coupled device (CCD) sensor, an electron multiplication CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0058] Image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or related processors. Figure 11The computing system 1100 may include one or more processors of any other type of processor 1110 discussed herein. The host processor 152 may be a digital signal processor (DSP) and / or other types of processor. In some specific implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as Inter-Integrated Circuit 2 (I2C) interface, Inter-Integrated Circuit 3 (I3C) interface, Serial Peripheral Interface (SPI) interface, Serial General Purpose Input / Output (GPIO) interface, Mobile Industrial Processor Interface (MIPI) (such as MIPI CSI-2 physical (PHY) layer port or interface, Advanced High Performance Bus (AHB) bus, any combination thereof and / or other input / output ports). In an exemplary example, host processor 152 may use the I2C port to communicate with image sensor 130, and ISP 154 may use the MIPI port to communicate with image sensor 130.

[0059] Image processor 150 can perform multiple tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 / 1025, read-only memory (ROM) 145 / 1020, cache, memory unit, another storage device, or some combination thereof.

[0060] Various input / output (I / O) devices 160 may be connected to the image processor 150. I / O devices 160 may include displays, keyboards, keypads, touchscreens, touchpads, touch-sensitive surfaces, printers, any other output devices, any other input devices, or some combination thereof. In some cases, text may be input into the image processing device 105B via the physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad on the touchscreen of the I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors that enable wired connections between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 may receive data from and / or send data to one or more peripheral devices. I / O 160 may include one or more wireless transceivers that enable wireless connections between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 may receive data from and / or send data to one or more peripheral devices. Peripheral devices may include any type of I / O device 160 discussed earlier, and they can be considered I / O devices 160 in themselves once they are coupled to ports, jacks, wireless transceivers or other wired and / or wireless connectors.

[0061] In some cases, the image capture and processing system 100 may be a single device. In other cases, the image capture and processing system 100 may be two or more independent devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some embodiments, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0062] like Figure 1 As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, namely, image capture device 105A and image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, certain components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0063] Image capture and processing system 100 may include electronic devices such as mobile or landline phones (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, image capture and processing system 100 may include one or more wireless transceivers for wireless communication (such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof). In some specific implementations, image capture device 105A and image processing device 105B may be different devices. For example, image capture device 105A may include a camera device, and image processing device 105B may include a computing device, such as a mobile phone, desktop computer, or other computing device.

[0064] Although the image capture and processing system 100 is shown to include certain components, those skilled in the art will understand that the image capture and processing system 100 may include more than [other components]. Figure 1 The components shown herein are additional components. Components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the image capture and processing system 100 may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits); and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. Software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 100.

[0065] Figure 2 This is a diagram illustrating the architecture of an example extended reality (XR) system 200 according to some aspects of this disclosure. In some examples, Figure 2The extended reality (XR) system 200 may include an image capture and processing system 100, an image capture device 105A, an image processing device 105B, or a combination thereof. The XR system 200 may run (or execute) XR applications and implement XR operations. In some examples, as part of an XR experience, the XR system 200 may perform tracking and localization, mapping of the environment in the physical world (e.g., a scene), and / or localization and rendering of virtual content on a display 209 (e.g., a screen, a visible plane / area, and / or other display). For example, the XR system 200 may generate a map of the environment in the physical world (e.g., a three-dimensional (3D) map), track the pose (e.g., position and localization) of the XR system 200 relative to the environment (e.g., relative to a 3D map of the environment), localize and / or anchor virtual content at a specific location on the map of the environment, and render the virtual content on the display 209 such that the virtual content appears to be at a location in the environment corresponding to a specific location on the map of the scene, where the virtual content is localized and / or anchored. Display 209 may include glass, screen, lens, projector and / or other display mechanisms that allow users to see a real-world environment and also allow XR content to be overlaid, superimposed, blended or otherwise displayed thereon.

[0066] In this exemplary example, the XR system 200 includes one or more image sensors 202, accelerometers 204, gyroscopes 206, storage devices 207, computing components 210, an XR engine 220, an image processing engine 224, a rendering engine 226, and a communication engine 228. It should be noted that... Figure 2 The components 202-228 shown are non-limiting examples provided for illustrative and explanatory purposes, and other examples may include those with... Figure 2 The components shown may be more numerous, fewer, or different than those shown. For example, in some cases, the XR system 200 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or Figure 2 One or more other software and / or hardware components not shown. While various components of the XR system 200 (such as image sensor 202) may be referred to herein in the singular, it should be understood that the XR system 200 may include multiple components of any of the components discussed herein (e.g., multiple image sensors 202).

[0067] XR system 200 includes an input device 208 or communicates (wired or wirelessly) with that input device. Input device 208 may include any suitable input device, such as a touchscreen, pen or other pointing device, keyboard, mouse, buttons or keys, microphone for receiving voice commands, gesture input device for receiving gesture commands, video game controller, steering wheel, joystick, a set of buttons, trackball, remote control, any other input device discussed herein, or any combination thereof. In some cases, image sensor 202 may capture images that can be processed to interpret gesture commands.

[0068] The XR system 200 can also communicate with one or more other electronic devices (wired or wireless). For example, the communication engine 228 can be configured to manage connections and communicate with one or more electronic devices. In some cases, the communication engine 228 may correspond to... Figure 11 The communication interface is 1140.

[0069] In some embodiments, one or more image sensors 202, accelerometers 204, gyroscopes 206, storage devices 207, computing components 210, XR engines 220, image processing engines 224, and rendering engines 226 may be part of the same computing device. For example, in some cases, one or more image sensors 202, accelerometers 204, gyroscopes 206, storage devices 207, computing components 210, XR engines 220, image processing engines 224, and rendering engines 226 may be integrated into HMDs, extended reality glasses, smartphones, laptops, tablets, gaming systems, and / or any other computing device. However, in some embodiments, one or more image sensors 202, accelerometers 204, gyroscopes 206, storage devices 207, computing components 210, XR engines 220, image processing engines 224, and rendering engines 226 may be part of two or more independent computing devices. For example, in some cases, some of the components in 202-226 may be part of or implemented by a computing device, and the remaining components may be part of or implemented by one or more other computing devices.

[0070] Storage device 207 can be any storage device used for storing data. Furthermore, storage device 207 can store data from any component of the XR system 200. For example, storage device 207 can store data from image sensor 202 (e.g., image or video data), data from accelerometer 204 (e.g., measurements), data from gyroscope 206 (e.g., measurements), data from computing component 210 (e.g., processing parameters, preferences, virtual content, rendered content, scene maps, tracking and positioning data, object detection data, privacy data, XR application data, facial recognition data, occlusion data, etc.), data from XR engine 220, data from image processing engine 224, and / or data from rendering engine 226 (e.g., output frames). In some examples, storage device 207 may include a buffer for storing frames processed by computing component 210.

[0071] One or more computing components 210 may include a central processing unit (CPU) 212, a graphics processing unit (GPU) 214, a digital signal processor (DSP) 216, an image signal processor (ISP) 218, and / or other processors (e.g., a neural processing unit (NPU) implementing one or more trained neural networks). Computing component 210 can perform various operations such as image enhancement, computer vision, graphics rendering, extended reality operations (e.g., tracking, localization, pose estimation, map building, content anchoring, content rendering, etc.), image and / or video processing, sensor processing, recognition (e.g., text recognition, face recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), trained machine learning operations, filtering, and / or any of the various operations described herein. In some examples, computing component 210 may implement (e.g., control, operate, etc.) an XR engine 220, an image processing engine 224, and a rendering engine 226. In other examples, computing component 210 may also implement one or more other processing engines.

[0072] Image sensor 202 may include any image and / or video sensor or capture device. In some examples, image sensor 202 may be part of a multi-camera assembly, such as a dual-camera assembly. Image sensor 202 may capture image and / or video content (e.g., raw image and / or video data), which may then be processed by computing component 210, XR engine 220, image processing engine 224, and / or rendering engine 226, as described herein. In some examples, image sensor 202 may include image capture and processing system 100, image capture device 105A, image processing device 105B, or combinations thereof.

[0073] In some examples, image sensor 202 may capture image data and may generate an image (also referred to as a frame) based on that image data and / or provide the image data or frame to XR engine 220, image processing engine 224, and / or rendering engine 226 for processing. The image or frame may include a video frame in a video sequence or a still image. The image or frame may include an array of pixels representing a scene. For example, the image may be: a red-green-blue (RGB) image with red, green, and blue color components per pixel; a lightness, redness, and blueness (YCbCr) image with a lightness component and two chromaticity (redness and blueness) components per pixel; or any other suitable type of color or monochrome image.

[0074] In some cases, image sensor 202 (and / or other cameras of XR system 200) may also be configured to capture depth information. For example, in some implementations, image sensor 202 (and / or other cameras) may include an RGB depth (RGB-D) camera. In some cases, XR system 200 may include one or more depth sensors (not shown) that are separate from image sensor 202 (and / or other cameras) and capable of capturing depth information. For example, such depth sensors may acquire depth information independently of image sensor 202. In some examples, depth sensors may be physically mounted in the same general location as image sensor 202, but may operate at a different frequency or frame rate than image sensor 202. In some examples, depth sensors may take the form of a light source that can project a structured or textured light pattern (which may include one or more narrowband lights) onto one or more objects in a scene. Depth information can then be obtained by utilizing the geometric deformation of the projected pattern caused by the surface shape of the objects. In one example, depth information may be obtained from a stereo sensor, such as a combination of an infrared structured light projector and an infrared camera registered to a camera (e.g., an RGB camera).

[0075] The XR system 200 may also include other sensors among its one or more sensors. The one or more sensors may include one or more accelerometers (e.g., accelerometer 204), one or more gyroscopes (e.g., gyroscope 206), and / or other sensors. The one or more sensors may provide velocity, orientation, and / or other positioning-related information to the computing component 210. For example, accelerometer 204 may detect the acceleration of the XR system 200 and may generate an acceleration measurement based on the detected acceleration. In some cases, accelerometer 204 may provide one or more translation vectors (e.g., up / down, left / right, forward / backward) that can be used to determine the positioning or pose of the XR system 200. Gyroscope 206 may detect and measure the orientation and angular velocity of the XR system 200. For example, gyroscope 206 may be used to measure the pitch, roll, and yaw of the XR system 200. In some cases, gyroscope 206 may provide one or more rotation vectors (e.g., pitch, yaw, roll). In some examples, the image sensor 202 and / or the XR engine 220 may use measurements obtained by the accelerometer 204 (e.g., one or more translation vectors) and / or measurements obtained by the gyroscope 206 (e.g., one or more rotation vectors) to calculate the pose of the XR system 200. As previously noted, in other examples, the XR system 200 may also include other sensors such as an inertial measurement unit (IMU), a magnetometer, a gaze and / or eye-tracking sensor, a machine vision sensor, a smart scene sensor, a voice recognition sensor, a shock sensor, a vibration sensor, a positioning sensor, a tilt sensor, etc.

[0076] As noted above, in some cases, one or more sensors may include at least one IMU. An IMU is an electronic device that uses a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers to measure specific forces, angular velocities, and / or orientations of the XR system 200. In some examples, one or more sensors may output measured information associated with the capture of images by the image sensor 202 (and / or other cameras of the XR system 200) and / or depth information obtained using one or more depth sensors of the XR system 200.

[0077] XR engine 220 can use the output of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs and / or other sensors) to determine the pose of XR system 200 (also referred to as head pose) and / or the pose of image sensor 202 (or other cameras of XR system 200). In some cases, the pose of XR system 200 and the pose of image sensor 202 (or other cameras) can be the same. The pose of image sensor 202 refers to the positioning and orientation of image sensor 202 relative to a reference frame (e.g., about scene 110). In some specific implementations, camera pose can be determined for 6 degrees of freedom (6DoF), which refers to three translational components (e.g., which can be given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a reference frame such as the image plane) and three angular components (e.g., roll, pitch, and yaw relative to the same reference frame). In some implementations, camera pose can be determined for 3 degrees of freedom (3DoF), which refers to three angular components (e.g., roll, pitch, and yaw).

[0078] In some cases, a device tracker (not shown) may use measurements from one or more sensors and image data from image sensor 202 to track the pose (e.g., 6DoF pose) of the XR system 200. For example, the device tracker may fuse visual data from the image data (e.g., using a visual tracking solution) with inertial data from the measurements to determine the position and motion of the XR system 200 relative to the physical world (e.g., a scene) and a map of the physical world. As described below, in some examples, when tracking the pose of the XR system 200, the device tracker may generate a three-dimensional (3D) map of the scene (e.g., the real world) and / or generate updates to the 3D map for that scene. 3D map updates may include, for example, but not limited to, new or updated features and / or features or landmarks associated with the scene and / or the 3D map of that scene, positioning updates that identify or update the position of the XR system 200 within the scene and the 3D map of that scene, etc. The 3D map provides a digital representation of the scene in the real / physical world. In some examples, 3D maps can anchor location-based objects and / or content to real-world coordinates and / or objects. The XR system 200 can use map-constructed scenes (e.g., scenes in the physical world represented by and / or associated with a 3D map) to merge the physical and virtual worlds and / or merge virtual content or objects with the physical environment.

[0079] In some aspects, computing component 210 may use a visual tracking solution to determine and / or track the pose of image sensor 202 and / or the XR system 200 as a whole, based on images captured by image sensor 202 (and / or other cameras of XR system 200). For example, in some examples, computing component 210 may perform tracking using computer vision-based tracking, model-based tracking, and / or simultaneous localization and mapping (SLAM) techniques. For example, computing component 210 may perform SLAM or may communicate (wired or wirelessly) with a SLAM system (not shown). SLAM refers to a class of techniques that simultaneously track the pose of a camera (e.g., image sensor 202) and / or XR system 200 relative to an environment while creating a map of the environment (e.g., a map of the environment modeled by XR system 200). This map may be called a SLAM map and may be three-dimensional (3D). SLAM technology can be performed using color or grayscale image data captured by image sensor 202 (and / or other cameras of XR system 200) and can be used to generate an estimate of 6DoF pose measurement of image sensor 202 and / or XR system 200. Such SLAM technology configured to perform 6DoF tracking can be referred to as 6DoF SLAM. In some cases, the output of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs and / or other sensors) can be used to estimate, correct, and / or otherwise adjust the estimated pose.

[0080] In some cases, 6DoF SLAM (e.g., 6DoF tracking) can associate features observed from certain input images from image sensor 202 (and / or other cameras) with a SLAM map. For example, 6DoF SLAM can use feature point association from an input image to determine the pose (localization and orientation) of image sensor 202 and / or XR system 200 for that input image. 6DoF map construction can also be performed to update the SLAM map. In some cases, the SLAM map maintained using 6DoF SLAM can contain 3D feature points triangulated from two or more images. For example, keyframes can be selected from an input image or video stream to represent the observed scene. For each keyframe, a corresponding 6DoF camera pose associated with the image can be determined. The pose of image sensor 202 and / or XR system 200 can be determined by projecting features from the 3D SLAM map onto the image or video frame and updating the camera pose according to a verified 2D-3D correspondence.

[0081] In an exemplary example, computational component 210 may extract feature points from some input images (e.g., each input image, a subset of input images, etc.) or from each keyframe. Feature points (also referred to as registration points) as used herein are distinctive or identifiable parts of an image, such as a part of a hand, the edge of a table, etc. Features extracted from a captured image may represent different feature points along three-dimensional space (e.g., coordinates on the X, Y, and Z axes), and each feature point may have an associated feature location. Feature points in a keyframe may match (be identical to or correspond to) feature points in previously captured input images or keyframes, or may not match. Feature detection may be used to detect feature points. Feature detection may include image processing operations that examine one or more pixels of an image to determine if a feature exists at a particular pixel. Feature detection may be used to process the entire captured image or parts of an image. For each image or keyframe, once a feature has been detected, a local image patch around that feature can be extracted. Any suitable technique can be used to extract features, such as Scale Invariant Feature Transform (SIFT) (which localizes features and generates descriptions of them), Learned Invariant Feature Transform (LIFT), Speeded Robust Feature Transform (SURF), Gradient Position Orientation Histogram (GLOH), Orientation Fast Rotation Briefing (ORB), Binary Robust Invariant Scalable Keypoint (BRISK), Fast Retina Keypoint (FREAK), KAZE, Accelerated KAZE (AKAZE), Normalized Cross-Correlation (NCC), Descriptor Matching, another suitable technique, or a combination thereof.

[0082] In some cases, the XR system 200 may also track the user's hands and / or fingers to allow the user to interact with and / or control virtual content in the virtual environment. For example, the XR system 200 may track the pose and / or movement of the user's hands and / or fingertips to identify or translate the user's interaction with the virtual environment. User interaction may include, for example, but not limited to, moving virtual content items, resizing virtual content items, selecting input interface elements in the virtual user interface (e.g., a virtual representation of a mobile phone, a virtual keyboard, and / or other virtual interfaces), and providing input through the virtual user interface.

[0083] Figure 3A Examples of augmented reality enhancement application engines 300 according to various aspects of this disclosure are illustrated. In some cases, the augmented reality enhancement application engine 300 may be implemented as... Figure 2This is part of the XR engine 220. In an exemplary example, the augmented reality enhancement application engine 300 includes a simulation engine 305, a rendering engine 310, a main rendering module 315, and an AR rendering module 360. As illustrated, the main rendering module 315 may include an effects rendering engine 320, a post-processing engine 325, and a user interface (UI) rendering engine 330. The AR rendering module 360 ​​may include an AR effects rendering engine 365 and an AR UI rendering engine 370. It should be noted that... Figure 3A The components 305-370 shown are non-limiting examples provided for illustrative and explanatory purposes, and other examples may include those with... Figure 3A The components shown are compared to those components, which may have more, fewer, or different components.

[0084] In some cases, the augmented reality application engine 300 is included in and / or communicates (wired or wirelessly) with the electronic device 340. In some examples, the augmented reality application engine 300 is included in and / or communicates (wired or wirelessly) with the XR system 350.

[0085] exist Figure 3A In the illustrated example, simulation engine 305 can generate simulations for augmented reality enhancement application engine 300. In some cases, the simulation may include, for example, one or more images, one or more videos, one or more strings (e.g., alphanumeric characters, numbers, text, Unicode characters, symbols, and / or icons), one or more two-dimensional (2D) shapes (e.g., circles, ellipses, squares, rectangles, triangles, other polygons, circular polygons with one or more rounded corners, portions thereof, or combinations thereof), one or more three-dimensional (3D) shapes (e.g., spheres, cylinders, cubes, pyramids, triangular prisms, rectangular prisms, tetrahedrons, other polyhedra, circular polyhedra with one or more rounded edges and / or rounded corners, portions thereof, or combinations thereof), shape textures, shape bump mapping, lighting effects, or combinations thereof. In some examples, the simulation may include at least a portion of an environment. The environment may be a real-world environment, a virtual environment, and / or a hybrid environment including elements of both real-world and virtual environments.

[0086] In some cases, the simulation generated by the simulation engine 305 can be dynamic. For example, the simulation engine 305 may update the simulation based on various triggers, including but not limited to physical contact, sound, gestures, input signals, the passage of time, and / or any combination thereof. As used herein, at a given moment, the application state of the augmented reality application engine 300 may include any information associated with: the simulation engine 305, the rendering engine 310, the main rendering module 315, the effects rendering engine 320, the post-processing engine 325, the UI rendering engine 330, the AR rendering module 360, the AR effects rendering engine 365, the AR UI rendering engine 370, inputs to the augmented reality application engine 300, outputs from the augmented reality application engine 300, and / or any combination thereof.

[0087] As illustrated, the simulation engine 305 may obtain mobile device input 341 from the mobile device 340. In some cases, the simulation engine 305 may obtain XR system input 351 from the XR system 350. Mobile device input 341 and / or XR system input 351 may include, for example, user input via a user interface of an application displayed on the display of the mobile device 340, or input from an input device (e.g., Figure 2 Input device 208), one or more sensors (e.g., Figure 2 User input from the image sensor 202, accelerometer 204, and gyroscope 206. In some cases, the simulation engine 305 can update the application state of the augmented reality application engine 300 based on mobile device input 341, XR system input 351, and / or any combination thereof.

[0088] exist Figure 3A In an exemplary example, rendering engine 310 may obtain application state information from simulation engine 305. In some cases, rendering engine 310 may determine the portion of the application state information that will be rendered by a display usable by augmented reality enhancement application engine 300. For example, rendering engine 310 may determine whether a connection (wired or wireless) has been established between XR system 350 and mobile device 340. In some cases, rendering engine 310 may determine the application state information that will be rendered by main rendering module 315 and AR rendering module 360. In some cases, rendering engine 310 may determine that XR system 350 is not (wired or wirelessly) connected to mobile device 340. In some cases, rendering engine 310 may determine the application state information of main rendering module 315 and abandon the determination that the application state information to be rendered by AR rendering module 360 ​​will not be displayed. Therefore, rendering engine 310 may facilitate adaptive rendering configuration for augmented reality enhancement application engine 300 based on the availability and / or type of available displays. In some specific implementations, such as Figure 3AThe standalone rendering engine 310 is shown. In an exemplary example, the main rendering module 315 and / or the AR rendering module 360 ​​may include at least a portion of the functionality of the rendering engine 310 described above.

[0089] The main rendering module 315 may include an effects rendering engine 320, a post-processing engine 325, and a UI rendering engine 330. In some cases, the main rendering module 315 may render image frames configured for display on the display of the mobile device 340. As illustrated, the main rendering module 315 may output the generated image frames (e.g., media content) for display on the display of the mobile device 340. In some cases, effects rendering information may be used to render application state information generated by the simulation engine 305. For example, the effects rendering engine may generate a 2D projection of a portion of a 3D environment included in the application state information. For example, the effects rendering engine 320 may generate a perspective projection of the 3D environment using a virtual camera. In some cases, the application state information may include the pose of a virtual camera within the environment. In some cases, the effects rendering engine 320 may generate additional visual effects not included in the 3D environment. For example, the effects rendering engine 320 may apply texture mapping to enhance the visual appearance of the effects generated by the effects rendering engine 320. In some cases, the effects rendering engine 320 can exclude portions of the application state information specified by the rendering engine 310 for the AR rendering module 360. For example, the main rendering module 315 can exclude effects present in the simulated environment.

[0090] In some cases, the post-processing engine 325 may provide additional processing to the rendered effects generated by the effects rendering engine 320. For example, the post-processing engine 325 may perform scaling, image smoothing, z-buffering, contrast enhancement, gamma, color mapping, any other image processing, and / or any combination thereof.

[0091] In some implementations, the UI rendering engine 330 can render the UI. In some cases, in addition to rendering effects based on the application environment (e.g., a 3D environment), the user interface can also provide application state information. In some cases, the UI can be generated as an overlay on a portion of the image frame output by the post-processing engine 325.

[0092] AR rendering module 360 ​​may include AR effect rendering engine 365 and AR UI rendering engine 370. In some cases, AR effect rendering engine 365 may render application state information generated by simulation engine 305. For example, AR effect rendering engine 365 may generate a 2D projection of a 3D environment included in the application state information. In some cases, AR effect rendering engine 365 may generate an effect that appears to protrude from the display surface of the display of mobile device 340.

[0093] In some cases, the display of the XR system 350 may have different display parameters than the display of the mobile device 340 (e.g., different resolution, frame rate, aspect ratio, and / or any other display parameters). In some cases, display parameters may also vary between different types of output devices (e.g., different HMD models, other XR systems, etc.). Therefore, rendering display data for the 350 using the AR rendering module 360 ​​can affect the performance of the main rendering module 315 (e.g., by consuming computational resources such as GPU, CPU, and memory). In some cases, including the AR rendering module 360 ​​within the augmented reality enhancement application engine 300 may require periodic updates to provide compatibility with different devices.

[0094] As indicated above, in some cases, XR systems can track physical objects (such as user body parts) to allow users to interact with virtual content (such as virtual objects). As an example, an XR system can use one or more cameras (e.g., ...). Figure 1 Image sensor 130 Figure 2 Image sensors 202, etc., are used to track the user's hand via one or more landmarks (e.g., physical objects) on the hand.

[0095] Figure 3B This is a block diagram illustrating an example system for hand tracking 380 according to various aspects of this disclosure. Figure 3B In this context, the device tracker 382 can receive measurements 388 from the accelerometer 204, measurements 389 from the gyroscope 206, and measurements from the image sensor 202 (e.g., Figure 2 Image data 390 from image sensor 202. In some examples, measurement 388 may include data from accelerometer 204 (e.g., image sensor 202). Figure 2 The motion measurement of the accelerometer 204) and the measurement 389 may include the motion measurement from the gyroscope 206 (e.g., Figure 2 Orientation measurements from the gyroscope 206. For example, measurement 388 may include one or more translation vectors (e.g., up / down, left / right, forward / backward) from the accelerometer 204, and measurement 388 may include one or more rotation vectors (e.g., pitch, yaw, roll) from the gyroscope 206. Furthermore, image data 390 may include data from the image sensor 202 (e.g., [image sensor 202]). Figure 2 The image sensor 202) captures one or more images or frames. The one or more images or frames may capture a scene and / or one or more parts of the scene (e.g., one or more areas, objects, people, etc.) associated with the XR system.

[0096] In some examples, the device tracker 382 can be implemented as the XR engine 385 of an extended reality system (e.g., Figure 2The device tracker 382 is part of the XR engine 220. In other cases, the device tracker 382 may be separate from the XR engine 385 and implemented by one or more computing components on the XR system.

[0097] Device tracker 382 can use measurements 388, 389 and image data 390 to track the pose (e.g., 6DOF pose) of the extended reality system. For example, device tracker 382 can fuse visual data from image data 390 with inertial data (e.g., motion data, orientation data, etc.) from measurements 388, 389 to determine the position and motion of the extended reality system relative to the physical world and a map of the physical world. In some examples, while tracking the pose of the extended reality system, device tracker 382 can generate a three-dimensional (3D) map of the scene (e.g., the real world) and / or generate updates to the 3D map for that scene. 3D map updates may include, for example, but not limited to, new or updated features and / or landmarks associated with the scene and / or the 3D map of that scene, location updates identifying or updating the position of the extended reality system within the scene and the 3D map of that scene, etc. The 3D map provides a digital representation of the scene in the real / physical world. In some examples, the 3D map can anchor location-based objects and / or content to real-world coordinates and / or objects. Extended reality systems can use map-constructed scenes (e.g., scenes in the physical world represented by and / or associated with 3D maps) to merge the physical and virtual worlds and / or merge virtual content or objects with the physical environment.

[0098] Device tracker 382 can provide tracking data 392 and image data 390 generated based on measurement 388 to hand tracker 384 and a set of XR applications 386. Tracking data 392 may include the pose of the XR system and map data calculated by device tracker 382. As described above, map data may include a 3D map of the scene and / or map updates for the 3D map of the scene.

[0099] In some cases, the hand tracker 384 may be included as a component of the XR engine 385. In some cases, the hand tracker 384 may be implemented by the XR system to track the user's hand associated with the XR system (e.g., Figure 4 The hand tracker 384 (426) and / or the fingertips of the user's hand, as explained above. For simplicity and illustrative purposes, this document describes the hand tracker 384 as a component for tracking a hand. However, it should be noted that in other examples, the hand tracker 384 may track other objects and / or body parts. For example, as described above, in addition to tracking the hand itself or alternatively tracking the hand itself, the hand tracker 384 may track the fingers or fingertips of the hand.

[0100] In some examples, the hand tracker 384 may be part of or implemented by the XR engine 385 on the XR system. In other examples, the device tracker 384 may be separate from the XR engine 385 and implemented by one or more computing components on the XR system.

[0101] The hand tracker 384 can also receive image data 390 from the image sensor 202. The hand tracker 384 can use the image data 390 and tracking data 392 to track hand pose 394 (e.g., the pose of the hand and / or the fingers / fingertips). In some examples, the hand tracker 384 can determine the hand pose 394 based on hand landmarks. The hand tracker 384 can then provide the hand pose 394 to one or more XR applications 386. In some examples, the XR application 386 can be an application on an XR system designed and / or configured to provide a specific XR experience. In some cases, an AR engine (such as Augmented Reality Enhancement Application Engine 300) can be the XR application 386. The XR application 386 can also include more advanced applications, such as, for example, AR gaming experiences, AR classroom experiences, and / or any other XR experiences. The XR application 386 can be part of or implemented by the XR engine 385, or it can be separate from the XR engine 385.

[0102] Figure 4 This is a diagram illustrating examples of the landmark points of hand 426 according to various aspects of this disclosure. Figure 4 The landmark points shown correspond to different parts of the hand 426, including landmark point 435 on the palm of the hand, three landmark points on the thumb 430 of the hand 426, three landmark points on the index finger 432 of the hand 426, three landmark points on the middle finger 434 of the hand 426, three landmark points on the ring finger 436 of the hand 426, and three landmark points on the little finger 438 of the hand 426. The palm of the hand 426 can move in three translational directions (e.g., measured in the X, Y, and Z directions relative to a plane (e.g., the image plane)) and three rotational directions (e.g., measured in yaw, pitch, and roll relative to a plane), thus providing six degrees of freedom (6DOF) that can be used for registration and / or tracking. The 6DOF movement of the palm in Figure 4 The center is represented by a square, as indicated in Figure 439.

[0103] Different joints of the fingers in hand 426 allow for varying degrees of movement, as illustrated in Figure 439. For example... Figure 4 As illustrated by the rhombus (e.g., rhombus 433), the base of each finger (corresponding to the metacarpophalangeal joint (MCP) between the proximal phalanges and metacarpals) has two degrees of freedom (2DOF) corresponding to flexion / extension and abduction / adduction. Figure 4As illustrated in the circle (e.g., circle 431), each joint in the upper joint of each finger (corresponding to the interphalangeal joint between the distal phalanx, middle phalanx, and proximal phalanx) has one degree of freedom (1DOF) corresponding to flexion and extension.

[0104] In some cases, the XR system may use one or more landmark points on hand 426 to track hand 426 (e.g., track the pose and / or movement of hand 426) and to track interactions with a virtual interface rendered by the XR system. As noted above, as a result of detecting one or more landmark points on hand 426, the pose of the landmark points (and thus the hand and fingers) relative to the XR system in a relative physical position can be established. For example, landmark points on the palm of hand 426 (e.g., landmark point 435) can be detected in an image, and the positions of the landmark points can be determined relative to the image sensor of the XR system. Points of the virtual interface rendered by the XR system (e.g., center points, such as centroids or other center points) and / or interface elements on the virtual interface selected by hand 426 or interface elements that hand 426 has interacted with can be translated to positions on the display of the XR system (or renderings on the display) relative to positions determined for the landmark points on the palm of hand 426. In some examples, points of a portion of the virtual interface that the hand 426 has already interacted with can be registered relative to the position of one or more landmark points on the hand 426.

[0105] In some examples, the XR system may also register the virtual interface and / or hand 426 to points in the real world (such as those detected in one or more images) and / or other parts of the user. For example, in some implementations, in addition to determining the physical pose of the hand 426 relative to the XR system and / or the virtual interface, the XR system may determine the location of other landmarks, such as unique points on a wall (referred to as keypoints), one or more corners of an object, features on the floor, points on a person's face, points on nearby devices, etc. In some cases, the XR system may place the virtual interface within a specific location relative to keypoints detected in the environment, which may correspond to, for example, objects and / or people detected in the environment.

[0106] Figure 5AThis is a perspective view 500 of a head-mounted display (HMD) 510 illustrating, according to some examples, performing feature tracking and / or visual simultaneous localization and mapping (VSLAM). The HMD 510 can be, for example, an augmented reality (AR) head-mounted device, a virtual reality (VR) head-mounted device, a mixed reality (MR) head-mounted device, an extended reality (XR) head-mounted device, or some combination thereof. The HMD 510 can be an example of an XR system 200. The HMD 510 includes a first camera 530A and a second camera 530B along the front of the HMD 510. In some cases, the HMD 510 may also include a third camera 530C, a fourth camera 530D, a fifth camera (not visible), and a sixth camera (not visible). In some examples, the HMD 510 may have only a single camera. In some examples, in addition to the first camera 530A and the second camera 530B, the HMD 510 may also include one or more additional cameras. In some examples, in addition to the first camera 530A and the second camera 530B, the HMD 510 may also include one or more additional sensors.

[0107] The HMD 510 (e.g., an XR device) includes a touchpad 540. Figure 5A In the diagram, touchpad 540 is shown mounted on the right side of HMD 510. In one or more examples, touchpad 540 may be mounted on the left side of HMD 510. In some examples, HMD 510 may include, for example... Figure 5A The touchpad 540 shown is a multi-touchpad (e.g., these touchpads can be mounted on the right and / or left side of the HMD 510).

[0108] In one or more examples, user 520 may use touch gestures to control the operation of HMD 510 (e.g., an XR device) (e.g., to perform control options). User 520 may perform touch gestures by touching touchpad 540 with user 520's hand (e.g., via one or more fingers of the hand). In one or more examples, touch gestures may include, but are not limited to, a single tap (e.g., where user 520 taps touchpad 540 once with one finger), a double tap (e.g., where user 520 taps touchpad 540 twice with one finger), a long press (e.g., where user 520 touches touchpad 540 with one finger and holds the touch on touchpad 540 for a period of time), a forward swipe (e.g., where user 520 swipes touchpad 540 in a forward direction from behind user's head toward user's face with one finger) and / or a backward swipe (e.g., where user 520 swipes touchpad 540 in a backward direction from user's face toward user's head). In one or more examples, control options (e.g., actions) may include, but are not limited to, pause actions (e.g., for pausing video and / or audio being played by HMD 510), selection actions (e.g., for selecting an option being presented to user 520 by HMD 510), next actions (e.g., for moving to another screen or another object presented to user 520 by HMD 510), voice assistance actions (e.g., for HMD 510 to invoke voice assistance for user 520), volume up actions (e.g., for increasing the volume of audio controlled by HMD 510) and / or volume down actions (e.g., for decreasing the volume of audio controlled by HMD 510).

[0109] Figure 5B This is an example of what is being worn by user 520 based on some examples. Figure 5AA perspective view 530 of a head-mounted display (HMD) 510. A user 520 wears the HMD 510 on their head, above their eyes. The HMD 510 can capture images using a first camera 530A and a second camera 530B. In some examples, the HMD 510 displays one or more display images based on images captured by the first camera 530A and the second camera 530B, facing the user 520's eyes. The display images can provide a stereoscopic view of the environment, and in some cases have overlaid information and / or other modifications. For example, the HMD 510 may display a first display image based on an image captured by the first camera 530A to the user 520's right eye. The HMD 510 may display a second display image based on an image captured by the second camera 530B to the user 520's left eye. For example, the HMD 510 may provide overlay information in the display images that overlays the images captured by the first camera 530A and the second camera 530B. As indicated above, the HMD 510 may also include a fifth camera 530E and a sixth camera 530F. In some cases, the third camera 530C, the fourth camera 530D, the fifth camera 530E, and the sixth camera 530F may be primarily used for tracking and mapping, and the images captured by these cameras may not typically be displayed to the user 520.

[0110] The HMD 510 does not include wheels, propellers, or any other means of transportation of its own. Instead, the HMD 510 relies on the movement of the user 520 to move back and forth in the environment. Therefore, in some cases, when performing SLAM techniques, the HMD 510 may skip path planning using a path planning engine and / or motion actuation using motion actuators. In some cases, the HMD 510 may still perform path planning using a path planning engine and may instruct the user 520 to follow a suggested path to guide the user along the suggested path planned using the path planning engine. In some cases, such as when the HMD 510 is a VR headset, the environment may be fully or partially virtual. If the environment is at least partially virtual, movement through the virtual environment may also be virtual. For example, movement through the virtual environment may be controlled by an input device 508. Motion actuators may include any such input device 508. Movement through the virtual environment may not require wheels, propellers, legs, or any other form of transportation. If the environment is virtual, the HMD 510 can still perform path planning and / or motion actuation using its path planning engine. If the environment is virtual, the HMD 510 can perform motion actuation using motion actuators by performing virtual movement within the virtual environment. Even if the environment is virtual, SLAM technology can still be valuable because the virtual environment may not have been mapped and / or may have been generated by devices other than the HMD 510, such as remote servers or consoles associated with video games or video game platforms. In some cases, feature tracking and / or SLAM can even be performed in a virtual environment by vehicles or other devices with their own physical transport systems that allow them to physically move back and forth in the physical environment. For example, SLAM can be performed in a virtual environment to test if a SLAM system is working correctly without wasting time or energy on movement and without wearing out the physical transport system.

[0111] Figure 6AThis is a perspective view 600 of the front surface 655 of a mobile device 650, illustrating, according to some examples, the use of one or more front-facing cameras 630A to 630B to perform feature tracking and / or visual simultaneous localization and mapping (VSLAM). The mobile device 650 may be, for example, a cellular phone, satellite phone, portable game console, music player, health tracking device, wearable device, wireless communication device, laptop computer, mobile device, any other type of computing device or computing system 1500 discussed herein, or a combination thereof. The front surface 655 of the mobile device 650 includes a display screen 645. The front surface 655 of the mobile device 650 includes a first camera 630A and a second camera 630B. The first camera 630A and the second camera 630B are illustrated in a bezel surrounding the display screen 645 on the front surface 655 of the mobile device 650. In some examples, the first camera 630A and the second camera 630B may be positioned in a notch or cutout cut from the display screen 645 on the front surface 655 of the mobile device 650. In some examples, the first camera 630A and the second camera 630B may be under-display cameras positioned between the display screen 645 and the remainder of the mobile device 650, such that light passes through a portion of the display screen 645 before reaching the first camera 630A and the second camera 630B. In perspective view 600, the first camera 630A and the second camera 630B are front-facing cameras. The first camera 630A and the second camera 630B face a direction perpendicular to the planar surface of the front surface 655 of the mobile device 650. In some examples, the front surface 655 of the mobile device 650 may have only a single camera. In some examples, in addition to the first camera 630A and the second camera 630B, the mobile device 650 may also include one or more additional cameras. In some examples, in addition to the first camera 630A and the second camera 630B, the mobile device 650 may also include one or more additional sensors.

[0112] Figure 6BThis is a perspective view 690 illustrating the rear surface 665 of a mobile device 650. The mobile device 650 includes a third camera 630C and a fourth camera 630D on its rear surface 665. The third camera 630C and fourth camera 630D in perspective view 690 are rear-mounted. The third camera 630C and fourth camera 630D face a direction perpendicular to the planar surface of the rear surface 665 of the mobile device 650. Although the rear surface 665 of the mobile device 650 does not have a display screen 645 as illustrated in perspective view 690, in some examples, the rear surface 665 of the mobile device 650 may have a second display screen. If the rear surface 665 of the mobile device 650 has a display screen 645, any positioning of the third camera 630C and fourth camera 630D relative to the display screen 645 can be used, as discussed regarding the first camera 630A and the second camera 630B at the front surface 655 of the mobile device 650. In some examples, the rear surface 665 of the mobile device 650 may have only a single camera. In some examples, in addition to the first camera 630A, the second camera 630B, the third camera 630C, and the fourth camera 630D, the mobile device 650 may also include one or more additional cameras. In some examples, in addition to the first camera 630A, the second camera 630B, the third camera 630C, and the fourth camera 630D, the mobile device 650 may also include one or more additional sensors.

[0113] Like the HMD 610, the mobile device 650 does not include wheels, propellers, or other means of transport. Instead, the mobile device 650 relies on the movement of the user holding or wearing it to move the mobile device 650 back and forth in the environment. Therefore, in some cases, when performing SLAM techniques, the mobile device 650 may skip path planning using a path planning engine and / or motion actuation using motion actuators. In some cases, the mobile device 650 may still perform path planning using a path planning engine and may instruct the user to follow the suggested path to guide the user along the suggested path planned using the path planning engine. In some cases, such as when the mobile device 650 is used for AR, VR, MR, or XR, the environment may be fully or partially virtual. In some cases, the mobile device 650 may be inserted into a head-mounted device (HMD) (e.g., into the HMD's holder), such that the mobile device 650 acts as the HMD's display, with the mobile device 650's display 645 serving as the HMD's display. If the environment is at least partially virtual, movement through the virtual environment may also be virtual. For example, movement through a virtual environment can be controlled by one or more joysticks, buttons, video game controllers, mice, keyboards, touchpads, and / or other input devices coupled to the mobile device 650 via wired or wireless means. Motion actuators can include any such input devices. Movement through a virtual environment may not require wheels, propellers, legs, or any other form of transport. If the environment is virtual, the mobile device 650 can still perform path planning and / or motion actuation using a path planning engine. If the environment is virtual, the mobile device 650 can perform motion actuation using motion actuators by performing virtual movement within the virtual environment.

[0114] As previously mentioned, many XR devices (e.g., in the form of HMDs, such as...) Figure 5A and Figure 5B The HMD 510 currently supports user control of the XR device using gestures (e.g., gesture controls such as clenching a fist, directional waving, quick swiping, pinching, etc.). For example, users can use gestures to control various operations of the XR device, such as scrolling, capturing, zooming, selecting, and / or controlling the device's volume. Gesture recognition may not be entirely accurate, as some users may perform gestures slightly incorrectly, causing the device to fail to recognize them. Therefore, the device may not be able to trigger the action associated with that gesture.

[0115] Figure 7 This is an illustration of an example of a user 710 using gestures 740a to control the operation of a device 720 (e.g., an XR device). Figure 7 In A, user 710 is shown as wearable device 720, which is in the form of an HMD (e.g., Figure 5A and Figure 5B The device is in the form of an HMD 510. User 710 is in room 700, where screen 730 displays an XR scene being controlled by device 720. User 710 is using gesture 740a (e.g., gesture 740b, shown in more detail) to enable control options (e.g., control operations) on device 720 to control aspects of the XR experience displayed on screen 730. Figure 7 In the image, gestures 740a and 740b are shown as waving gestures (e.g., user 710 waving their hand from one side to the other).

[0116] Figure 8 This is an illustration of different gestures 800 that can be used by a user to enable different control options (e.g., control operations) of the device. Figure 8 In the middle, users (e.g., Figure 7 User 710)'s hand 870 is shown as having multiple axes, including a x Axis, a y axis and a z Axis. These different axes can be used to determine different gestures made by the hand 870. Figure 8 In this context, different gestures 800 include a downward gesture 810 (e.g., waving a hand 870 in a downward direction), an upward gesture 820 (e.g., waving a hand 870 in an upward direction), a leftward gesture 830 (e.g., waving a hand 870 in a leftward direction), a rightward gesture 840 (e.g., waving a hand 870 in a rightward direction), a clockwise gesture 850 (e.g., waving a hand 870 in a clockwise direction), and a counter-clockwise gesture 860 (e.g., waving a hand 870 in a counter-clockwise direction). In one or more examples, the gestures may be different. Figure 8 Gestures other than and / or different from the gestures shown in the different gestures 800. For example, these gestures may include, but are not limited to, gestures that indicate a number by one or more fingers (e.g., presenting one or more fingers of hand 870 to indicate a number corresponding to the number of fingers presented), fist gestures (e.g., hand 870 clenched into a fist), waving gestures (e.g., hand 870 waving from side to side, such as waving gestures 740a, 740b), pinching gestures (e.g., pinching the fingers of hand 870 together), and / or hand movement gestures for forming shapes (e.g., waving to form a shape, such as a circle, square, or triangle).

[0117] Many XR devices (e.g., HMDs) have touchpads (e.g., Figure 5A(Touchpad 540). These touchpads are typically implemented within the left or right hinge of the device. These touchpads are used by the user to control the device. Gesture control is often used to assist the user in controlling these devices because a single touchpad can only be assigned a minimal number of functionalities. For example, a single touchpad may only allow a limited number of possible touch gestures (e.g., such as single click, double click, long press, forward swipe, and / or backward swipe), which can only provide the user with a minimal number of control options (e.g., each touch gesture may be assigned only one control option, and therefore there may only be a total of five possible control options) for controlling the device.

[0118] Therefore, improved systems and technologies that provide users with numerous control options for controlling devices (e.g., XR devices, such as those in the form of HMDs) while allowing for accurate detection of user gestures may be useful.

[0119] In one or more aspects, systems and technologies provide mechanisms for controlling devices (e.g., XR devices, such as those presented as HMD devices). In one or more aspects, systems and technologies combine touch gestures with basic gestures to provide users with a number of control options for controlling devices (e.g., XR devices) in a more easy and efficient manner. The combination of touch and gestures can be used to define a more comprehensive range of control options for controlling the device.

[0120] Figure 9 Examples of different combinations of touch and gestures are shown to enable various control options. Figure 10 The table shows this. Specifically, Figure 9 This is an example used to enable a renderable HMD (e.g., Figure 5A and Figure 5B The illustration of example 900 shows different combinations of gestures and touch gestures for different control options (e.g., for controlling different operations) of a device in the form of an HMD510 (e.g., an XR device), and... Figure 10 This is an example corresponding to Figure 9 Table 1000 provides examples of different control options (e.g., control operations) for different combinations of gestures and touch gestures.

[0121] In one or more examples, the user (e.g., Figure 7 User 710) can use the user's hand to perform gestures (e.g., Figure 9Gesture 910a or gesture 910b can be used to indicate a specific settings list associated with that particular gesture (e.g., settings list 1 930a or settings list 2 930b) among multiple settings lists. In some examples, a user can perform a gesture of one or more fingers indicating a number (e.g., presenting one or more fingers of the user's hand to indicate a number corresponding to the number of fingers presented), which may correspond to a specific settings list (e.g., presenting one finger may indicate the number one, which may correspond to a first settings list). For example, Figure 9 The gesture 910a is shown as presenting one finger of the user's hand to indicate the number one, which may correspond to a first setting list (e.g., setting list 1 930a). For example, Figure 9 Gesture 910b is shown as presenting two fingers of the user's hand to indicate the number two, which may correspond to a second settings list (e.g., settings list 2 930b). As described below, the first settings list and / or the second settings list may be presented (e.g., displayed) by the device to allow the user to perform one or more touch events (e.g., touch gestures) corresponding to one or more commands indicated in the settings list.

[0122] In some examples, the user (e.g., Figure 7 User 710 may use the user's hand to perform a gesture that does not indicate a specific number to indicate a specific setting list associated with that specific gesture among multiple setting lists (e.g., setting list 1930a or setting list 2930b). For example, a gesture that does not specifically indicate a number (e.g., a clenched fist gesture (where the user's hand is clenched into a fist), an open hand, a "C" gesture, etc.) may correspond to a specific setting list (e.g., a clenched fist gesture may correspond to a first setting list, such as setting list 1930a).

[0123] Presented as HMD (e.g., Figure 5A Devices in the form of HMD 510 (e.g., XR devices) typically include a touchpad (e.g., Figure 5A The touchpad 540. A user can perform touch gestures (e.g., touch gesture 920a or touch gesture 920b) to control the operation of the HMD 510 (e.g., an XR device) (e.g., to execute control options). In one or more examples, a user can perform touch gestures to select control options from a selected settings list (e.g., settings list 1 930a or settings list 2 930b). User 520 can touch the touchpad (e.g., via one or more fingers of the hand) using the user's hand (e.g., via one or more fingers of the hand). Figure 5A The touchpad 540 is used to perform touch gestures.

[0124] In one or more examples, touch gestures may include, but are not limited to, a single tap (e.g., where a user taps the touchpad once with one finger), a double tap (e.g., where a user taps the touchpad twice with one finger), a long press (e.g., where a user touches the touchpad with one finger and holds the touch on the touchpad for a period of time), a forward swipe (e.g., where a user swipes the touchpad forward from behind the user's head toward the user's face with one finger), and / or a backward swipe (e.g., where a user swipes the touchpad backward from the user's face toward the user's head).

[0125] In one or more examples, control options (e.g., operations) may include, but are not limited to, pause operations (e.g., for pausing video and / or audio being played by the device), selection operations (e.g., for selecting an option being presented to the user by the device), next operations (e.g., for moving to another screen or another object presented to the user by the device), voice assistance operations (e.g., for the device to invoke voice assistance for the user), volume up operations (e.g., for increasing the volume of audio controlled by the device), volume down operations (e.g., for decreasing the volume of audio controlled by the device), brightness up operations (e.g., for increasing the brightness of a display controlled by the device), brightness down operations (e.g., for decreasing the brightness of a display controlled by the device), camera start operations (e.g., for starting a camera controlled by the device), zoom in operations (e.g., for zooming in on an image displayed on a display controlled by the device), and / or zoom out operations (e.g., for zooming out on an image displayed on a display controlled by the device).

[0126] In one or more examples, a user can perform a combination of gestures (e.g., gesture 910a or gesture 910b) and touch gestures (e.g., touch gesture 920a or touch gesture 920b) to enable specific control options. For example, a user can perform gesture 910a (e.g., to present one finger to indicate the number one) to invoke settings list 1 930a. In conjunction with gesture 910a, the user can also perform touch gesture 920a (e.g., a single tap) to select the pause / select control option in settings list 1 930a. As another example, a user can perform touch gesture 920b (e.g., to present two fingers to indicate the number two) to invoke settings list 2 930b. In conjunction with touch gesture 910b, the user can also perform touch gesture 920b (e.g., a double tap) to select the dimming control option in settings list 2 930b.

[0127] In one or more examples, a user may perform a gesture (e.g., gesture 910a) to invoke a specific settings list (e.g., settings list 1 930a), followed by a touch gesture (e.g., touch gesture 920b) to select a specific control option (e.g., pause / select control option) from the settings list (e.g., settings list 1 930a). In some examples, a user may perform both a gesture (e.g., gesture 910a) and a touch gesture (e.g., touch gesture 920a) simultaneously, allowing the gesture and touch gesture to occur concurrently.

[0128] exist Figure 10 Table 1000 is shown as including: an operation column 1010, which includes different touch gestures; a settings list column 1930a, which includes different control options corresponding to different touch gestures when the user invokes settings list 1930a; and a settings list column 2930a, which includes different control options corresponding to different touch gestures when the user invokes settings list 2930b. In one or more examples, Table 1000 may include, for example... Figure 10 The list shows more or fewer settings.

[0129] Table 1000 shows different touch gestures including single-click, double-click, long-press, forward swipe, and / or backward swipe. In one or more examples, a more... Figure 10 The touch gestures shown in column 1010 of Table 1000 are more or fewer in number and / or different in number. Different control options in column 930a of Settings List 1 are shown as including pause / select, next, voice assistance, volume up, and / or volume down. Different control options in column 930b of Settings List 2 are shown as including brightness up, brightness down, camera start, zoom in, and zoom out. In one or more examples, more than Figure 10 The control options shown in the settings list 1 930a column and / or settings list 2 930b column of Table 1000 include more or fewer control options and / or different control options.

[0130] In some examples, systems and technologies can combine a first gesture (e.g., via the user's first hand) with a second gesture (e.g., via the user's first or second hand) to provide the user with a number of control options to control the device (e.g., an XR device) in a more convenient and efficient manner. The combination of the first and second gestures can be used to define a more comprehensive range of control options for controlling the device.

[0131] As previously mentioned, gesture recognition (e.g., gesture control, such as clenching a fist, directional waving, rapid hand swiping, pinching motions, etc.) may not be entirely accurate because some users may perform gestures slightly incorrectly, causing the device to fail to recognize the gesture and thus prevent the device from inducing the action associated with that gesture. In one or more examples, the system and technology employ gestures that indicate a number using one or more fingers (e.g., presenting one or more fingers of the user's hand to indicate a number corresponding to the number of fingers presented), which may correspond to a specific set of settings (e.g., presenting one finger to indicate the number one, which may correspond to a first set of settings). Currently, efficient hand-tracking cameras exist that can easily and accurately read the number of fingers presented by the user.

[0132] Figure 11 This is an illustration of examples of different gestures (e.g., gesture 1110, gesture 1120, and gesture 1130), each indicating a different number that can be associated with a specific list of settings (e.g., a specific menu). Figure 11 In the gesture, gesture 1110 shows the user's hand with one finger, and therefore gesture 1110 can be associated with the number one (e.g., via a device, such as an XR device) (e.g., the number one can be associated with a first settings list (such as menu 1)). Gesture 1120 shows the user's hand with two fingers, and therefore gesture 1120 can be associated with the number two (e.g., the number two can be associated with a second settings list (such as menu 2)). Gesture 1130 shows the user's hand with three fingers, and therefore gesture 1130 can be associated with the number three (e.g., the number three can be associated with a third settings list (such as menu 3)).

[0133] Figure 12 and Figure 13 This illustrates an example of a process for controlling a device (e.g., an XR device), which may be presented as an HMD (e.g., Figure 5A and Figure 5B In the form of HMD 510. Specifically, Figure 12 This is a diagram illustrating an example of a process 1200 for controlling a device, where no numbers are detected and a default settings list is invoked (e.g., settings list 1 1240).

[0134] exist Figure 12 During operation of process 1200, at box 1210, a touchpad or one or more sensors may be used to detect a user’s control gestures (e.g., touch gestures, or alternatively, gestures) among multiple control gestures (e.g., multiple touch gestures or multiple gestures). In one or more examples, each of the one or more sensors may be a camera, which may be in the form of a red-green-blue (RGB) camera or a monochrome camera.

[0135] For example, the device's touchpad can detect control gestures in the form of touch gestures. In one or more examples, the touch gesture can be a single click, double click, long press, swipe, forward movement, or backward movement.

[0136] For example, one or more sensors (e.g., one or more cameras) can detect control gestures in the form of gestures. In one or more examples, a gesture can be a gesture of one or more fingers indicating a number, a fist gesture, a waving gesture, a pinching gesture, or a hand movement gesture used to form a shape (e.g., a circle, a square, or a triangle).

[0137] In one or more examples, the touchpad (e.g., Figure 5A The touchpad 540 can be used in devices (e.g., XR devices in the form of HMDs, such as...) Figure 5A and Figure 5B The HMD 510 is implemented within the device. In some examples, one or more cameras can be implemented within the device (e.g., HMD 510). Figure 5A and Figure 5B The HMD 510 itself and / or mobile phones (e.g., Figure 6A and Figure 6B The implementation is carried out within another device (e.g., a mobile phone 650). In one or more examples, the device and other devices (e.g., mobile phones) can communicate with each other via a wireless communication protocol (e.g., Bluetooth protocol).

[0138] In one or more examples, each of the multiple control options may be associated with a different control gesture (e.g., a unique control gesture) among the multiple control gestures. In one or more examples, each of the multiple control options may be a pause operation, a select operation, a next operation, a voice-assisted operation, a volume up operation, a volume down operation, a brightness up operation, a brightness down operation, a camera launch operation, a zoom in operation, or a zoom out operation.

[0139] for Figure 12In the example shown, at box 1220, one or more sensors (e.g., one or more cameras) do not detect the first gesture. Since the first gesture is not detected, at box 1230, one or more processors (e.g., implemented within the device and / or another device, such as a mobile phone) determine to invoke a default settings list (e.g., a first settings list, such as settings list 1, 1240) from among multiple settings lists. In one or more examples, the default settings list may include multiple control options (e.g., pause / select operation, next operation, voice assistance operation, volume up operation, and / or volume down operation). One or more processors may determine a first control option among the multiple control options in the default settings list based on the detected control gesture. The one or more processors may then enable the first control option to control the device.

[0140] Figure 13 This is a diagram illustrating an example of a process 1300 for controlling a device, in which the digit two (2) is detected and an associated second settings list is invoked (e.g., settings list 2 1350). Figure 13 During operation of process 1300, at box 1310, a touchpad or one or more sensors (e.g., each sensor may be a camera in the form of an RGB camera or a monochrome camera) may be used to detect control gestures (e.g., multiple touch gestures or multiple gestures) among multiple control gestures of the user (e.g., multiple touch gestures or multiple gestures).

[0141] For example, a device's touchpad can detect control gestures in the form of touch gestures. Touch gestures can be single-click, double-click, long-press, swipe, forward, or backward movements. Alternatively, one or more sensors (e.g., one or more cameras) can detect control gestures in the form of hand gestures. Gestures can be gestures using one or more fingers to indicate numbers, fist gestures, waving gestures, pinching gestures, or hand movement gestures used to form shapes (e.g., circles, squares, or triangles).

[0142] In one or more examples, the touchpad (e.g., Figure 5A The touchpad 540 can be used in devices such as XR devices that can be presented in the form of an HMD, such as Figure 5A and Figure 5B The HMD 510 can be implemented within the device. One or more cameras can be implemented within the device (e.g., HMD 510). Figure 5A and Figure 5B The HMD 510 itself and / or mobile phones (e.g., Figure 6A and Figure 6B This is implemented within another device (such as a mobile phone 650). In some examples, the device and other devices (e.g., mobile phones) can communicate with each other via wireless communication protocols such as Bluetooth.

[0143] In one or more examples, each of the multiple control options may be associated with a different control gesture (e.g., a unique control gesture) among the multiple control gestures. Each of the multiple control options may be a pause operation, a select operation, a next operation, a voice-assisted operation, a volume up operation, a volume down operation, a brightness up operation, a brightness down operation, a camera launch operation, a zoom in operation, or a zoom out operation.

[0144] During the operation of process 1300, at box 1320, one or more sensors (e.g., one or more cameras) may detect a first gesture among a plurality of first gestures of the user. In one or more examples, the detection of control gestures and the detection of first gestures occur simultaneously. In one or more examples, each of the plurality of first gestures may be a gesture of one or more fingers indicating a number, a fist gesture, a waving gesture, a pinching gesture, or a hand movement gesture for forming a shape (e.g., a circle, a square, or a triangle).

[0145] At box 1340, one or more processors (e.g., implemented within a device and / or another device, such as a mobile phone) may determine a first settings list among multiple settings lists based on a detected first gesture. In one or more examples, the first settings list includes multiple control options. Each of the multiple settings lists may be associated with a corresponding gesture.

[0146] In one or more examples, one or more processors may determine a first digit associated with a first gesture. For Figure 13 In the example shown, one or more processors can determine that the first gesture is a gesture 1330 that presents two fingers, and therefore one or more processors can determine that the first number associated with the first gesture is two.

[0147] In one or more examples, each of the multiple setting lists may be associated with a corresponding number. One or more processors may determine a first setting list (e.g., setting list 2 1350) based on a first number (e.g., two), and thus one or more processors may invoke setting list 2 1350 among the multiple setting lists. In one or more examples, setting list 2 1350 may include multiple control options (e.g., brightness increase operation, brightness decrease operation, camera start operation, zoom in operation, and zoom out operation). One or more processors may determine a first control option among the multiple control options in setting list 2 1350 based on a detected control gesture. The one or more processors may then enable the first control option to control the device.

[0148] Figure 14This is a flowchart illustrating an example of a process 1400 for controlling a device. Process 1400 may be executed by a computing device (or apparatus) or by components or systems of a computing device (e.g., chipset, one or more processors such as one or more CPUs, GPUs, NPUs, DSPs, etc.) or other components or systems). In some aspects, the computing device is an extended reality (XR) device (e.g., augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) device). For example, an XR device may be a head-mounted display (HMD) device. The operation of process 1400 may be implemented on one or more processors (e.g., Figure 15 Software components that execute and run on a processor (such as the 1510 or other processors).

[0149] At box 1410, the computing device (or a component thereof) may detect a first gesture among a plurality of first gestures of a user based on sensor data from one or more sensors. In some aspects, each of the one or more sensors is one of a red-green-blue (RGB) camera sensor or a monochrome camera sensor. In some examples, the plurality of first gestures includes gestures of one or more fingers indicating a number, a fist gesture, a waving gesture, a pinching gesture, or hand movement gestures for forming a shape, any combination thereof, and / or other gestures.

[0150] In some cases, each of the one or more sensors is implemented within a computing device or an auxiliary device (e.g., a mobile phone, tablet computer, wearable device such as a network-connected watch, or other device). For example, in some examples, the computing device may include one or more sensors. In some aspects, the computing device may include at least one memory and at least one processor. The auxiliary device may also include at least one processor. In some cases, at least one processor of the auxiliary device is capable of performing one or more operations of process 1400. In some aspects, the computing device and the auxiliary device may be connected via a wireless communication protocol (e.g., Bluetooth). ™ (Protocols or other types of wireless communication protocols) communicate with each other.

[0151] At box 1420, the computing device (or a component thereof) may determine a first set list among multiple set lists based on a first gesture. The first set list includes multiple control options. In some examples, the multiple control options may include pause, select, next, voice assistance, volume up, volume down, brightness up, brightness down, camera start, zoom in, zoom out, any combination thereof, and / or other control options. In some aspects, each set list among the multiple set lists is associated with a corresponding gesture. In some cases, each set list among the multiple set lists is associated with a corresponding number. For example, the computing device (or a component thereof) may determine a first number associated with the first gesture. In such examples, the computing device (or a component thereof) may determine the first set list among the multiple set lists based on the first number.

[0152] At box 1430, the computing device (or a component thereof) may detect a control gesture among a plurality of user control gestures. In some cases, the computing device (or a component thereof) may detect both a control gesture and a first gesture simultaneously. Each of the plurality of control gestures is associated with a different control option among a plurality of control options. In one exemplary example, the control gesture is a touch gesture among a plurality of touch gestures. For example, the plurality of touch gestures may include a single click, a double click, a long press, a forward swipe, a backward swipe, any combination thereof, and / or other touch gestures. In such an example, to detect the control gesture, the computing device (or a component thereof) may detect the user's touch gesture via the computing device's touchpad. In another exemplary example, the control gesture is a second gesture among a plurality of second gestures. In such an example, to detect the control gesture, the computing device (or a component thereof) may detect the second gesture of either the hand from which the user performed the first gesture or the hand from which the user performed the second gesture.

[0153] At box 1440, the computing device (or a component thereof) may determine a first control option among a plurality of control options in a first settings list based on a control gesture. At box 1450, the computing device (or a component thereof) may enable the first control option to control the computing device.

[0154] In some examples, process 1400 may be executed by one or more computing devices or apparatuses. In some exemplary examples, process 1400 may be performed by... Figure 1 Image capture and processing system 100 Figure 2 XR system 200 and / or one or more computing devices or systems (e.g., Figure 15The computing system 1500 executes the process. In some cases, such a computing device or apparatus may include a processor, microprocessor, microcomputer, or other components of a device configured to perform the steps of process 1400. In some examples, such a computing device or apparatus may include one or more sensors configured to capture image data. For example, the computing device may include a smartphone, head-mounted display, mobile device, camera, tablet computer, or other suitable device. In some examples, such a computing device or apparatus may include a camera configured to capture one or more images or videos. In some cases, such a computing device may include a display for displaying images. In some examples, one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such a computing device may further include a network interface configured to transmit data.

[0155] Components that enable the implementation of a computing device in a circuit. For example, a component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein. The computing device may also include a display (as an example of an output device or as a supplement to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0156] Process 1400 is illustrated as a logic flowchart, the operations of which represent a sequence of operations that can be implemented by hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0157] Additionally, process 1400 may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0158] Figure 15 This is a block diagram illustrating an example of a computing system 1500, which can be used by the disclosed system for DRV enhancement for dynamic FPS usage. Specifically, Figure 15 An example of computing system 1500 is illustrated. This computing system can be any computing device, such as constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1505. Connection 1505 can be a physical connection using a bus, or a direct connection to processor 1510, such as in a chipset architecture. Connection 1505 can also be a virtual connection, a networking connection, or a logical connection.

[0159] In some aspects, computing system 1500 is a distributed system in which the functions described herein can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a plurality of such components, each of which performs some or all of the functions described for that component. In some aspects, the components can be physical or virtual devices.

[0160] Example system 1500 includes at least one processing unit (CPU or processor) 1510 and a connection 1505 that communicatively couples various system components, including system memories 1515 such as read-only memory (ROM) 1520 and random access memory (RAM) 1525, to processor 1510. Computing system 1500 may include a cache 1512 of high-speed memory that is directly connected to, closely proximate to, or integrated into processor 1510.

[0161] Processor 1510 may include any general-purpose processor and hardware or software services (such as services 1532, 1534, and 1536 stored in storage device 1530 and configured to control processor 1510), as well as dedicated processors in which software instructions are incorporated into the actual processor design. Processor 1510 may be a substantially completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0162] To enable user interaction, the computing system 1500 includes an input device 1545 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice input, etc. The computing system 1500 may also include an output device 1535 that can be one or more of a plurality of output mechanisms. In some instances, a multi-mode system allows a user to provide multiple types of input / output to communicate with the computing system 1500.

[0163] The computing system 1500 may include a communication interface 1540, which typically controls and manages user input and system output. The communication interface may perform or facilitate the receiving and / or transmitting of wired or wireless communications using wired and / or wireless transceivers, including utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple... ™ Lightning ™ Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, 3G, 4G, 5G and / or other cellular data network wireless signal transmission, Bluetooth ™ Wireless signal transmission, Bluetooth ™ Low-power (BLE) wireless signal transmission, IBEACON ™ Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), microwave access global interoperability (WiMAX), infrared (IR) wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, self-organizing network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or those communications in some combination thereof.

[0164] The communication interface 1540 may also include one or more ranging sensors (e.g., LIDAR sensors, laser rangefinders, RF radars, ultrasonic sensors, and infrared (IR) sensors) configured to collect data and provide measurements to the processor 1510, thereby configuring the processor 1510 to perform determinations and calculations required to obtain various measurements from the one or more ranging sensors. In some examples, measurements may include time of flight, wavelength, azimuth, elevation, distance, linear velocity, and / or angular velocity, or any combination thereof. The communication interface 1540 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the position of the computing system 1500 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S. GPS, the Russian GLONASS, the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS. There are no limitations on operation on any particular hardware arrangement, and therefore the basic features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0165] Storage device 1530 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape, flash memory cards, solid-state storage devices, digital versatile discs, cartridges, floppy disks, hard disks, magnetic tapes, magnetic stripes, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM, rewritable CD, digital video disc (DVD), Blu-ray Disc (BDD), holographic disc, another optical medium, secure digital (SD) card, micro-secure digital (microSD) card, Memory Stick. ®Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (e.g., layer 1 (L1) cache, layer 2 (L2) cache, layer 3 (L3) cache, layer 4 (L4) cache, layer 5 (L5) cache, or other (L#) cache), resistive random access memory (RRAM / ReRAM), phase change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette and / or combinations thereof.

[0166] Storage device 1530 may include software services, servers, services, etc., which enable the system to perform functions when the code defining such software is executed by processor 1510. In some aspects, hardware services performing specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1510, connection 1505, output device 1535, etc. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and which does not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0167] Specific details have been provided in the foregoing description to offer a thorough understanding of the aspects and examples presented herein, but those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative aspects of this application have been described in detail herein, it is to be understood that the various inventive concepts may be embodied and employed in various other ways, and the appended claims are not intended to be construed as including these variations unless limited by prior art. The various features and aspects of the applications described above may be used individually or in combination. Furthermore, without departing from the broader scope of this specification, aspects may be used in any number of environments and applications beyond those described herein. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that, in alternative aspects, the methods may be performed in a different order than described.

[0168] For clarity, in some instances, this technology may be presented as comprising various functional blocks, which include devices, device components, steps, or routines embodied in a method, either in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring these aspects in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the aspects.

[0169] Furthermore, those skilled in the art will understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0170] Various aspects described above can be presented as processes or methods, depicted as flowcharts, diagrams, data flow graphs, structure diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagrams. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0171] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that can be used to store the instructions, the information used, and / or information created during the methods according to the described examples include disks or optical discs, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0172] In some respects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0173] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the above description may, in some cases, be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or light particles, or any combination thereof, depending in part on the specific application, in part on the desired design, in part on the corresponding technology, etc.

[0174] The various exemplary logic blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or performed using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any form factor of various form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Examples of form factors include: laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, self-contained devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0175] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0176] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0177] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0178] Those skilled in the art will understand that, without departing from the scope of this description, the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced with less than or equal to (“>”) respectively. ") and greater than or equal to (" The symbol ) is used instead.

[0179] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0180] The phrase “coupled to” or “communicatively coupled to” means that any component is physically connected directly or indirectly to another component, and / or that any component is in communication with another component directly or indirectly (e.g., connected to that other component via a wired or wireless connection and / or other suitable communication interface).

[0181] Claim language or other languages ​​that state "at least one of" and / or "one or more of" in a set indicate that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repeating information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the language of a claim stating "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0182] Claims using phrases such as "at least one processor, the at least one processor being configured to," "at least one processor being configured to," "one or more processors, the one or more processors being configured to," or "one or more processors being configured to," or other languages, indicate that one or more processors (in any combination) are capable of performing associated operations. For example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks to perform operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" could mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0183] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0184] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0185] The various exemplary logic blocks, modules, engines, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, engines, modules, circuits, and steps have been broadly described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0186] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as an engine, module, or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0187] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0188] The exemplary aspects of this disclosure include: Aspect 1. An apparatus for enabling one or more control options, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: detect a first gesture among a plurality of first gestures of a user based on sensor data from one or more sensors; determine a first settings list among a plurality of settings lists based on the first gesture, wherein the first settings list includes a plurality of control options; detect a control gesture among a plurality of control gestures of the user, wherein each of the plurality of control gestures is associated with a different control option among the plurality of control options; determine a first control option among the plurality of control options in the first settings list based on the control gesture; and enable the first control option to control the apparatus.

[0189] Aspect 2. The apparatus according to aspect 1, wherein: the control gesture is a touch gesture among a plurality of touch gestures; and in order to detect the control gesture, the at least one processor is configured to detect the user's touch gesture via the touchpad of the apparatus.

[0190] Aspect 3. The apparatus according to aspect 2, wherein each of the plurality of touch gestures is one of a single click, a double click, a long press, a forward swipe, or a backward swipe.

[0191] Aspect 4. The apparatus according to any one of Aspects 1 to 3, wherein: the control gesture is a second gesture among a plurality of second gestures; and in order to detect the control gesture, the at least one processor is configured to detect the second gesture of one of the hands or the other hand of the user performing the first gesture.

[0192] Aspect 5. The apparatus according to any one of Aspects 1 to 4, wherein each of the plurality of control options is one of a pause operation, a selection operation, a next operation, a voice-assisted operation, a volume increase operation, a volume decrease operation, a brightness increase operation, a brightness decrease operation, a camera start operation, a zoom in operation, or a zoom out operation.

[0193] Aspect 6. The apparatus according to any one of aspects 1 to 5, wherein each of the plurality of setting lists is associated with a corresponding gesture.

[0194] Aspect 7. The apparatus according to any one of aspects 1 to 6, wherein each of the plurality of first gestures is one of a gesture of one or more fingers indicating a number, a fist gesture, a waving gesture, a pinching gesture, or a hand movement gesture for forming a shape.

[0195] Aspect 8. The apparatus according to any one of Aspects 1 to 7, wherein the at least one processor is configured to determine a first digit associated with the first gesture.

[0196] Aspect 9. The apparatus according to aspect 8, wherein: the at least one processor is configured to determine the first setting list among the plurality of setting lists based on the first number; and each setting list among the plurality of setting lists is associated with a corresponding number.

[0197] Aspect 10. The apparatus according to any one of Aspects 1 to 9, wherein the at least one processor is configured to simultaneously detect the control gesture and the first gesture.

[0198] Aspect 11. The apparatus according to any one of aspects 1 to 10, wherein each of the one or more sensors is a red-green-blue (RGB) camera sensor or a monochrome camera sensor.

[0199] Aspect 12. The apparatus according to any one of aspects 1 to 11, wherein each of the one or more sensors is implemented within one of the apparatus or an auxiliary device.

[0200] Aspect 13. The apparatus according to aspect 12, wherein the apparatus and the additional device communicate with each other via a wireless communication protocol.

[0201] Aspect 14. The apparatus according to aspect 13, wherein the wireless communication protocol is the Bluetooth protocol.

[0202] Aspect 15. The apparatus according to any one of Aspects 12 to 14, wherein the additional device is a mobile phone.

[0203] Aspect 16. The apparatus according to any one of aspects 1 to 15, wherein the apparatus is an extended reality (XR) device.

[0204] Aspect 17. The apparatus according to aspect 16, wherein the XR device is a head-mounted display (HMD) device.

[0205] Aspect 18. The apparatus according to any one of aspects 1 to 17, wherein the apparatus further comprises the one or more sensors.

[0206] Aspect 19. A method for controlling a device, the method comprising: using one or more sensors to detect a first gesture among a plurality of first gestures of a user; determining, by one or more processors, a first settings list among a plurality of settings lists based on the first gesture, wherein the first settings list includes a plurality of control options; detecting a control gesture among a plurality of control gestures of the user, wherein each of the plurality of control gestures is associated with a different control option among the plurality of control options; determining, by the one or more processors, a first control option among the plurality of control options in the first settings list based on the control gesture; and enabling, by the one or more processors, the first control option to control the device.

[0207] Aspect 20. The method according to aspect 19, wherein: the control gesture is a touch gesture among a plurality of touch gestures; and the detection of the control gesture includes detecting the user's touch gesture via the touchpad of the device.

[0208] Aspect 21. The method according to aspect 20, wherein each of the plurality of touch gestures is one of a single click, a double click, a long press, a forward swipe, or a backward swipe.

[0209] Aspect 22. The method according to any one of aspects 19 to 21, wherein: the control gesture is a second gesture among a plurality of second gestures; and the detection of the control gesture includes using the one or more sensors to detect the second gesture of one of the hands or the other hand of the user performing the first gesture.

[0210] Aspect 23. The method according to any one of aspects 19 to 22, wherein each of the plurality of control options is one of a pause operation, a selection operation, a next operation, a voice-assisted operation, a volume increase operation, a volume decrease operation, a brightness increase operation, a brightness decrease operation, a camera start operation, a zoom in operation, or a zoom out operation.

[0211] Aspect 24. The method according to any one of aspects 19 to 23, wherein each of the plurality of setting lists is associated with a corresponding gesture.

[0212] Aspect 25. The method according to any one of aspects 19 to 24, wherein each of the plurality of first gestures is one of a gesture of one or more fingers indicating a number, a fist gesture, a waving gesture, a pinching gesture, or a hand movement gesture for forming a shape.

[0213] Aspect 26. The method according to any one of aspects 19 to 25, the method further comprising: determining, by the one or more processors, a first digit associated with the first gesture.

[0214] Aspect 27. The method according to aspect 26, wherein: the determination of the first setting list among the plurality of setting lists by the one or more processors is based on the first number; and each setting list among the plurality of setting lists is associated with a corresponding number.

[0215] Aspect 28. The method according to any one of Aspects 19 to 27, wherein detecting the control gesture and detecting the first gesture occur simultaneously.

[0216] Aspect 29. The method according to any one of aspects 19 to 28, wherein each of the one or more sensors is a red-green-blue (RGB) camera sensor or a monochrome camera sensor.

[0217] Aspect 30. The method according to any one of aspects 19 to 29, wherein each of the one or more sensors is implemented within one of the device or an auxiliary device.

[0218] Aspect 31. The method according to aspect 30, wherein the device and the additional device communicate with each other via a wireless communication protocol.

[0219] Aspect 32. The method according to aspect 31, wherein the wireless communication protocol is the Bluetooth protocol.

[0220] Aspect 33. The method according to any one of Aspects 30 to 32, wherein the additional device is a mobile phone.

[0221] Aspect 34. The method according to any one of aspects 19 to 33, wherein the device is an extended reality (XR) device.

[0222] Aspect 35. The method according to aspect 34, wherein the XR device is a head-mounted display (HMD) device.

[0223] Aspect 36. The method according to any one of aspects 19 to 35, wherein each of the one or more processors is implemented within the device or another device.

[0224] Aspect 37. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any one of aspects 19 to 36.

[0225] Aspect 38. An apparatus comprising one or more components for performing operations according to any one of aspects 19 to 36.

[0226] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are to be consistent with the full scope of the language claims, wherein an element referred to in the singular is not intended to mean "one and only one," but rather "one or more" unless specifically stated otherwise.

Claims

1. An apparatus for enabling one or more control options, the apparatus comprising: At least one memory; and At least one processor, the at least one processor being coupled to the at least one memory and being configured to: Detecting the first gesture among multiple first gestures of a user based on sensor data from one or more sensors; Based on the first gesture, a first settings list is determined from a plurality of settings lists, wherein the first settings list includes a plurality of control options; Detect a control gesture among a plurality of control gestures of the user, wherein each of the plurality of control gestures is associated with a different control option among a plurality of control options; The first control option in the plurality of control options in the first setting list is determined based on the control gesture; as well as Enable the first control option to control the device.

2. The apparatus according to claim 1, wherein: The control gesture is a touch gesture among multiple touch gestures; and In order to detect the control gesture, the at least one processor is configured to detect the user's touch gesture via the device's touchpad.

3. The apparatus according to claim 2, wherein each of the plurality of touch gestures is one of a single click, a double click, a long press, a forward swipe, or a backward swipe.

4. The apparatus according to claim 1, wherein: The control gesture is the second gesture among a plurality of second gestures; and In order to detect the control gesture, the at least one processor is configured to detect a second gesture performed by one of the user's hands, either the first gesture or the other hand.

5. The apparatus of claim 1, wherein each of the plurality of control options is one of a pause operation, a selection operation, a next operation, a voice-assisted operation, a volume increase operation, a volume decrease operation, a brightness increase operation, a brightness decrease operation, a camera start operation, a zoom in operation, or a zoom out operation.

6. The apparatus of claim 1, wherein each of the plurality of setting lists is associated with a corresponding gesture.

7. The apparatus of claim 1, wherein each of the plurality of first gestures is one of a gesture of one or more fingers indicating a number, a fist gesture, a waving gesture, a pinching gesture, or a hand movement gesture for forming a shape.

8. The apparatus of claim 1, wherein the at least one processor is configured to determine a first digit associated with the first gesture.

9. The apparatus according to claim 8, wherein: The at least one processor is configured to determine the first setting list among the plurality of setting lists based on the first number; and Each of the multiple setting lists is associated with a corresponding number.

10. The apparatus of claim 1, wherein the at least one processor is configured to simultaneously detect the control gesture and the first gesture.

11. The apparatus of claim 1, wherein each of the one or more sensors is an RGB camera sensor or a monochrome camera sensor.

12. The apparatus of claim 1, wherein each of the one or more sensors is implemented within one of the apparatus or an additional device.

13. The apparatus of claim 12, wherein the apparatus and the additional device communicate with each other via a wireless communication protocol.

14. The apparatus of claim 13, wherein the wireless communication protocol is the Bluetooth protocol.

15. The apparatus of claim 12, wherein the additional device is a mobile phone.

16. The apparatus of claim 1, wherein the apparatus is an extended reality (XR) device.

17. The apparatus of claim 16, wherein the XR device is a head-mounted display (HMD) device.

18. The apparatus of claim 1, further comprising the one or more sensors.

19. A method for controlling a device, the method comprising: Use one or more sensors to detect the first gesture among a user's multiple first gestures; One or more processors determine a first settings list from a plurality of settings lists based on the first gesture, wherein the first settings list includes a plurality of control options; Detect a control gesture among a plurality of control gestures of the user, wherein each of the plurality of control gestures is associated with a different control option among a plurality of control options; The one or more processors determine a first control option from the plurality of control options in the first setting list based on the control gesture; as well as The first control option is enabled by one or more processors to control the device.

20. The method of claim 19, wherein: The control gesture is a touch gesture among multiple touch gestures; and The detection of the control gesture includes detecting the user's touch gesture via the device's touchpad.

21. The method of claim 19, wherein: The control gesture is the second gesture among a plurality of second gestures; and The detection of the control gesture includes using the one or more sensors to detect a second gesture by one of the user's hands performing the first gesture or the other hand.

22. The method of claim 19, wherein each of the plurality of control options is one of a pause operation, a selection operation, a next operation, a voice-assisted operation, a volume increase operation, a volume decrease operation, a brightness increase operation, a brightness decrease operation, a camera start operation, a zoom in operation, or a zoom out operation.

23. The method of claim 19, wherein each of the plurality of settings lists is associated with a corresponding gesture.

24. The method of claim 19, wherein each of the plurality of first gestures is one of a gesture of one or more fingers indicating a number, a fist gesture, a waving gesture, a pinching gesture, or a hand movement gesture for forming a shape.

25. The method according to claim 19, further comprising: The first number associated with the first gesture is determined by the one or more processors.

26. The method of claim 25, wherein: The determination of the first setting list among the plurality of setting lists by the one or more processors is based on the first number; and Each of the multiple setting lists is associated with a corresponding number.

27. The method of claim 19, wherein detecting the control gesture and detecting the first gesture occur simultaneously.

28. The method of claim 19, wherein each of the one or more sensors is implemented within one of the device or an auxiliary device.

29. The method of claim 19, wherein the device is an extended reality (XR) device.

30. The method of claim 19, wherein each of the one or more processors is implemented within the device or another device.

Images