XR operation function using a smartwatch

The smartwatch enhances XR interface control by combining head tracking for cursor guidance with smartwatch selection, offering precise object manipulation, addressing the discomfort and imprecision of current XR hand tracking methods.

JP7594619B2Active Publication Date: 2024-12-04NTT DOCOMO INC
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
JP2023042859
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2023-03-09
Filing Date
2023-03-17
Publication Date
2024-12-04
Estimated Expiration
2043-03-17

AI Technical Summary

Technical Problem

Current XR devices rely on hand tracking for user interface manipulation, which is uncomfortable over prolonged use and lacks precision, and smartphones are used as controllers, requiring constant carrying.

Method used

Utilizing a smartwatch to control XR device interfaces through head tracking for cursor guidance and smartwatch selection, enabling precise manipulation of objects via touch inputs, such as moving, rotating, and scaling, with wireless communication between the smartwatch and XR device.

Benefits of technology

Provides a comfortable and precise method for XR interface manipulation, allowing users to control objects with high accuracy using a smartwatch without the need for constant smartphone carrying.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594619000001
    Figure 0007594619000001
  • Figure 0007594619000002
    Figure 0007594619000002
  • Figure 0007594619000003
    Figure 0007594619000003
Patent Text Reader

Abstract

To provide an extended reality (XR) manipulation feature using a smart watch.SOLUTION: A method for controlling objects displayed on a UI in a virtual environment generated by an XR device includes the steps of: receiving a selection of an object; using a wearable device to display a menu having GUI elements on the UI; receiving first touch input data indicating a user selection and translating it, the input data being received in response touching the wearable device; translating additional touch input data from the wearable device into one or more actions associated with the selected GUI element; performing the one or more actions on the object on the UI displayed in the virtual environment; and updating the UI based on results of performing the one or more actions.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 393,182, entitled "XR MANIPULATION FEATURE WITH SMART WATCH," filed on July 28, 2022, which is incorporated by reference in its entirety. [Background technology]

[0002]

[0002] Artificial reality, extended reality, or extra reality (collectively "XR") is a type of reality that is conditioned in some manner before presentation to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or any combination and / or derivative thereof. A variety of XR environments exist that allow users' representations to move around and talk to each other. However, the user interface (UI) and user experience (UX) have limitations. For example, user interfaces often rely on hand tracking, which does not always work effectively and may even be uncomfortable for users. Summary of the Invention

[0003]

[0003] The present disclosure is directed to controlling objects displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device substantially as shown in and / or described in connection with at least one of the figures, and as more fully described in the claims.

[0004]

[0004] The present disclosure will become more fully understood from the detailed description given hereinafter and from the accompanying drawings of various embodiments, which should not be taken as limiting the embodiments described and illustrated in this specification, but are merely for explanation and understanding. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram illustrating an overview of a device in which some embodiments of the disclosed technology can function. [Figure 2A] FIG. 1 is a wiring diagram for a head mounted display (HMD) according to some embodiments. [Figure 2B] FIG. 1 is a wiring diagram of a mixed reality HMD system including a mixed reality HMD and a core processing component. [Figure 3A] FIG. 13 illustrates manipulating an object using a smartwatch with a virtual controller displayed in the user interface of the XR device. [Figure 3B] FIG. 13 illustrates manipulating an object using a smartwatch with a virtual controller displayed in the user interface of the XR device. [Figure 4A] FIG. 1 illustrates moving a selected object using a smartwatch. [Figure 4B] FIG. 1 illustrates moving a selected object using a smartwatch. [Figure 4C] FIG. 1 illustrates rotating a selected object using a smartwatch. [Figure 4D] FIG. 1 illustrates rotating a selected object using a smartwatch. [Figure 4E] FIG. 13 illustrates scaling a selected object using a smartwatch. [Figure 4F]FIG. 13 illustrates scaling a selected object using a smartwatch. [Diagram 5] 1 is a flow diagram of some embodiments of a process for controlling objects displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. [Figure 6] 11 is a flow diagram of several other embodiments of a process for controlling objects displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. [Figure 7] 1 is a flow diagram of some embodiments of a process for moving an object displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. [Figure 8] 1 is a flow diagram of some embodiments of a process for rotating an object displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. [Figure 9] 1 is a flow diagram of some embodiments of a process for scaling objects displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. [Figure 10] FIG. 1 illustrates one embodiment of a computer system that can be used to support the systems and operations discussed herein, including those of an XR device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006]

[0018] In the following description, numerous details are set forth in order to provide a more thorough explanation of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present disclosure.

[0007]

[0019] Methods and apparatus are described for controlling objects displayed in a user interface (UI) in an artificial, extended, or extra reality (collectively "XR") system or virtual environment created by the device using a wearable device. In some embodiments, the XR device receives a selection of an object in the UI displayed in the virtual environment created by the XR device and displays a menu having one or more graphical user interface (GUI) elements selectable by a user using the wearable device. The XR device then receives touch input data from the wearable device indicating a selection of a menu item and / or an action to be performed on the object displayed in the virtual environment created by the XR device. The XR device can be implemented as a head-mounted display (HMD) system, examples of which are described below.

[0008]

[0020] The disclosed embodiments of the technology can include or be implemented in combination with XR, which is a type of reality that is conditioned in some manner before presentation to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or any combination and / or derivative thereof. Extended reality content may include fully generated content or generated content combined with captured content (e.g., real-world photos). Additionally, in some embodiments, extended reality can be associated with, for example, an application, product, accessory, service, or any combination thereof that is used to perform an activity in a synthetic reality. Extended reality systems that provide extended reality content can be implemented on a variety of platforms, including head-mounted displays (HMDs) connected to a host computer system, standalone HMDs, mobile devices or computing systems, "cave" environments or other projection systems, or any other hardware platform capable of providing synthetic reality content to one or more viewers.

[0009]

[0021] "Virtual reality" or "VR" as used herein refers to an immersive experience in which a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user sees images of the real world after they pass through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display them on the tablet's screen opposite the camera. The tablet can process and adjust or "augment" the images as they pass through the system, such as by adding virtual objects. "Mixed reality" or "MR" refers to a system in which some of the light that enters the user's eyes is generated by the computing system and some is light reflected from objects in the real world. For example, an MR headset can be shaped as glasses with a pass-through display that allows light from the real world to pass through a wave guide that simultaneously emits light from a projector in the MR headset to allow the MR headset to present virtual objects mixed with the real objects the user can see. "Artificial reality," "extra reality," or "XR," as used herein, refers to either VR, AR, MR, or any combination or hybrid thereof.

[0010]

[0022] Some embodiments are discussed in more detail below with reference to the figures. FIG. 1 is a block diagram showing an overview of devices in which some embodiments of the disclosed technology can function. The devices can include hardware components of a computing system 100 that can create, manage, and provide interaction modes for an XR environment. In various embodiments, the computing system 100 can include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate over wired or wireless channels to distribute processing and share input data. In some embodiments, the computing system 100 can include a standalone headset that can provide a computer-created or augmented experience to a user without the need for external processing or sensors. In other embodiments, the computing system 100 can include multiple computing devices, such as a headset and a core processing component (such as a console, mobile device, or server system), where some processing operations are performed on the headset and other processing operations are offloaded to the core processing component. Exemplary headsets are described below in connection with Figures 2A and 2B. In some embodiments, location and environmental data may be gathered solely by sensors built into the headset device, while in other embodiments, one or more of the non-headset computing devices may include sensor components that may track environmental or location data.

[0011]

[0023] Computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a holographic processing unit (HPU), etc.). Processor 110 may be a single processing unit, multiple processing units in one device, or distributed across multiple devices (e.g., distributed across two or more of computing devices 101-103).

[0012]

[0024] Computing system 100 may include one or more input devices 120, which provide input to and inform processor 110 of actions. The actions may be mediated by a hardware controller, which interprets signals received from the input devices and communicates the information to processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touch pad, a wearable input device (e.g., a haptic glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, e.g., an infrared sensor), a microphone, or other user input device.

[0013]

[0025] The processor 110 can be coupled to other hardware devices using internal or external buses, such as, for example, a PCI bus, a SCSI bus, or a wireless connection. The processor 110 can communicate with hardware controllers for devices, such as for a display 130. The display 130 can be used to display text and graphics. In some implementations, the display 130 includes an input device as part of the display, such as when the input device is a touch screen or includes a system that monitors eye or head direction. In some implementations, the display is separate from the input device. Examples of display devices include LCD display screens, LED display screens, projected, holographic, or augmented reality displays (such as head-up display devices or head-mounted devices), etc. Other I / O devices 140 can also be coupled to the processor, such as network chips or cards, video chips or cards, audio chips or cards, USB, Firewire or other external devices, cameras, printers, speakers, CD-ROM drives, DVD drives, disk drives, etc.

[0014]

[0026] The computing system 100 may include a communication device capable of wireless or wire-based communication with other local computing devices or network nodes. The communication device may communicate with another device or server over a network, for example using the TCP / IP protocol. The computing system 100 may utilize the communication device to distribute operations across multiple network devices.

[0015]

[0027] The processor 110 can access memory 150, which can be included in one of the computing devices of the computing system 100 or distributed across multiple computing devices of the computing system 100 or other external devices. Memory includes one or more hardware devices for volatile or non-volatile storage and can include both read-only and writeable memory. For example, memory can include one or more of random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writeable non-volatile memory such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, etc. Memory is not a propagating signal separate from the underlying hardware, and therefore memory is non-transitory. Memory 150 can include program memory 160 that stores programs and software, such as an operating system 162, an XR operating system 164, and other application programs 166. Memory 150 may also include data memory 170 , which may contain information to be provided to program memory 160 or any element of computing system 100 .

[0016]

[0028] Some implementations may be operational with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that may be suitable for use with the present technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular telephones, wearable electronics, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0017]

[0029] 2A is a wiring diagram of a head mounted display (HMD) 200 according to some embodiments. The HMD 200 includes a front rigid body 205 and a band 210. The front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computational units 230. The position sensor 220, the IMU 215, and the computational unit 230 may be internal to the HMD 200 and may be invisible to the user. In various implementations, the IMU 215, the position sensor 220, and the locator 225 may track the movement and location of the HMD 200 in the real world and in a virtual environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locator 225 can emit infrared light beams that generate light points on real objects around the HMD 200. As another example, the IMU 215 can include, for example, one or more accelerometers, gyroscopes, magnetometers, other non-camera based position, force, or orientation sensors, or combinations thereof. One or more cameras (not shown) integrated with the HMD 200 can detect the light points. A computation unit 230 in the HMD 200 can use the detected light points to estimate the position and movement of the HMD 200 and to identify the shape and position of real objects surrounding the HMD 200.

[0018]

[0030] The electronic display 245 may be integrated with the front rigid body 205 and may provide image light to the user as directed by the computation unit 230. In various embodiments, the electronic display 245 may be a single electronic display or multiple electronic displays (e.g., a display for each eye of the user). Examples of the electronic display 245 include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a display including one or more quantum dot light emitting diode (QOLED) subpixels, a projector unit (e.g., micro LED, laser, etc.), some other display, or any combination thereof.

[0019]

[0031] In some embodiments, the HMD 200 may be coupled to a core processing component, such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors may monitor the HMD 200 (e.g., via light emitted from the HMD 200), which the PC may use in combination with output from the IMU 215 and position sensor 220 to determine the location and movement of the HMD 200.

[0020]

[0032] 2B is a wiring diagram of a mixed reality HMD system 250 including a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 GHz link) as shown by link 256. In other embodiments, the mixed reality system 250 includes only a headset without an external computing device or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a pass-through display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as a light projector (e.g., laser, LED, etc.), a camera, an eye tracking sensor, MEMS components, networking components, etc.

[0021]

[0033] The projector can be coupled, for example, via optical elements to a pass-through display 258 to display media to a user. The optical elements can include one or more waveguide assemblies, reflectors, lenses, mirrors, collimators, diffraction gratings, etc., to direct light from the projector to the user's eyes. Image data can be transmitted from the core processing component 254 to the HMD 252 via link 256. A controller in the HMD 252 can convert the image data into light pulses from the projector, which can be transmitted via optical elements as output light to the user's eyes. The output light can be mixed with light passing through the display 258, allowing the output light to present virtual objects that appear as if they exist in the real world.

[0022]

[0034] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc. that enable HMD system 250 to, for example, track itself with 3DoF or 6DoF, track parts of the user (e.g., hands, feet, head, or other body parts), map virtual objects to appear stationary as HMD 252 moves, and make virtual objects responsive to gestures and other real-world objects.

[0023]

[0035] In various embodiments, HMD 200 or 250 may also include additional subsystems, such as eye tracking units, head tracking units, audio systems, various network components, etc., to monitor for indications of a user's interaction and intent. For example, in some embodiments, instead of or in addition to a controller, one or more cameras included in HMD 200 or 250 or from an external camera may monitor the position and pose of the user's hands to identify gestures and other hand and body movements.

[0024] XR operation function using a smartwatch

[0036] Currently, in order for a user to operate an object displayed in the user interface of an XR device, the XR device performs hand tracking on the user to observe the user's hands. This requires the user to place his or her hand on a removable area of ​​the hand tracking, which is uncomfortable after long-term use. Also, in other XR devices, a smartphone is used as a controller, which requires the user to be aware of and carry the smartphone with them at all times. In current XR devices, there is no way to use a smartwatch device to operate the user interface displayed by the XR device.

[0025]

[0037] In some embodiments, a user can select an object in a display of a user interface using a combination of head tracking and a smartwatch device. In some embodiments, in this case, the role of the head tracking is to guide a cursor displayed in the user interface, while the role of the smartwatch is to provide a button to select the object at the cursor's location. In this way, the user can use the smartwatch as a controller to manipulate the object, for example, to move, rotate, and / or scale the object. In some embodiments, head tracking is used to identify the object the user is looking at, the smartwatch is used to select the object (e.g., by tapping the center of the smartwatch), and then the watch device is used only to manipulate the object thereafter. This is more precise than current existing UI / UX implementations for XR devices, where manipulation is not precise. In this case, using a smartwatch to manipulate objects is very convenient, because the user only needs to wear the smartwatch, and is provided with the ability to manipulate those objects with high precision due to the physical display of the watch being visible.

[0026]

[0038] In some embodiments, the XR device manipulates an object in two modes: one mode in which one finger is used to manipulate (e.g., move, rotate, scale, etc.) the object, and a second mode in which two or more fingers are used to manipulate the object. In some embodiments, the manipulation includes moving, rotating, or scaling the object.

[0027]

[0039] In one embodiment, the smartwatch and the XR device are communicatively connected, whereby touch input data from the smartwatch is wirelessly transmitted to and acted upon by the XR device. In this case, the touch input data transmitted from the smartwatch is translated into commands executed by the XR device. The wireless communication is a short-range wireless communication (e.g., Bluetooth, etc.).

[0028]

[0040] 3A and 3B show manipulating an object using a smartwatch with a virtual controller displayed in the user interface of the XR device. In this method, the user manipulates the object using the smartwatch as a controller. Referring to FIG. 3A, when the XR device determines via head tracking that the user is looking at an object, such as an object (chair) 301, in the user interface of the virtual environment 302 for a pre-specified period (e.g., 1 second, 2 seconds, etc.), the XR device displays a pop-up menu 305 in the user interface. In some embodiments, the menu 305 includes a select (check mark) button 309 in the center, along with a follow button 310, an edit placement button 311, a product information button 312, and an "X" button 313 displayed around the select button.

[0029]

[0041] The edit placement button 311 allows the user to manipulate the object (e.g., move the object, rotate the object, scale the object, etc.) and is selected by touching a location on the display surface of the smartwatch 330 corresponding to the edit placement button 311. The product information button 312 causes the XR device to show information about the selected object, such as a description of the object (e.g., size, weight, price, etc.), and is selected by touching a location on the display surface of the smartwatch 330 corresponding to the product information button 312. The follow button 310 causes the XR device to make the selected object follow the user's position and is selected by touching a location on the display surface of the smartwatch 330 corresponding to the follow button 310. That is, in some embodiments, after selecting the follow button 310 by touching a corresponding location on the display surface of the smartwatch, the selected object moves along the path the user is looking at in the virtual environment as determined by head tracking. The X button 313 can be selected to close the pop-up menu and make it disappear from the user interface in the virtual environment.

[0030]

[0042] In some embodiments, the pop-up menu 305 is displayed in response to head tracking identifying that the user is looking at an object in the user interface of the virtual environment. In FIG. 3A, head tracking identifies that the user is looking at a chair and highlights the chair as the user is looking at the chair (358). The pop-up menu 305 is then displayed, allowing the user to select an action that can be performed by the XR device on the chair using the smartwatch 330. FIG. 3B shows that the user selects the Edit Placement button 311 using the smartwatch 330, allowing the user to manipulate the chair currently selected based on the head tracking performed by the XR device. In response to sliding the user's finger along the display surface of the smartwatch 330 in the area of ​​the Edit Placement button 311, touch input data captured by the touch sensor of the smartwatch 330 is transmitted to the XR device to indicate that the user wishes to manipulate the chair object in the display.

[0031]

[0043] 4A-4F show an example of manipulating an object using a smartwatch in response to a user selecting an edit placement button (e.g., edit placement button 311 in FIGS. 3A and 3B) in a pop-up menu (e.g., menu 305 in FIGS. 3A and 3B) displayed by an XR device and selected using the smartwatch. With reference to FIGS. 4A-4F, after the user selects the edit placement button, the user can swipe one or more of their fingers across the display interface (surface) of the smartwatch device to move the selected object, rotate the selected object, or scale the selected object.

[0032]

[0044] 4A and 4B illustrate moving a selected object using a smartwatch. In some embodiments, in a first mode, the position of the object is controlled by using the smartwatch display interface, which is performed by swiping across the display surface of the smartwatch using a single finger. In some embodiments, before moving the object, the user indicates two of the XYZ axes 470 that the movement of the finger across the display surface of the smartwatch will control. For example, if the user selects the XY axis mode using button 480 of FIG. 4A, the movement of the user's finger across the display surface of the smartwatch will result in the movement of the object in the X and Y directions, and if the user selects the ZY axis mode using button 481 of FIG. 4B, the movement of the user's finger across the display surface of the smartwatch will result in the movement of the object in the Z and Y directions. In some embodiments, the processing logic detects the selection of the object by head or eye tracking. In some other embodiments, the processing logic detects the selection of the object through tapless selection by determining that the position of the user's hand is held in a position pointed at the object for a pre-specified time. In some other embodiments, the processing logic detects selection of one or more objects in the UI in the virtual environment by tracking the position of the user's hand and thumb or fingers. In some other embodiments, the processing logic detects selection of one or more objects in the UI in the virtual environment by a controller of the XR device.

[0033]

[0045] 4A, in this mode, in some embodiments, the user can move the object to various positions on the XY axis of the XYZ axis 470. For example, the XR device moves the object 461 in the X and Y directions in the user interface of the virtual environment when the user slides his / her finger 463 in the X and Y directions on the display interface of the smartwatch 462. For example, the XR device moves the object 461 along the X axis when the user slides his / her finger left and right, and moves the object 461 along the Y axis when the user slides his / her finger up and down. When any of these actions occur, the touch input data captured by the touch sensor of the display interface of the smartwatch 462 is sent to the XR device, where the touch input data is converted into commands, and the commands are executed by the XR device to move the object 461 accordingly. It should be noted that other single finger movements and directions can be specified to move the object.

[0034]

[0046] Referring to FIG. 4B, in the second mode, the user uses a single finger to place the object in the ZY axis mode. Referring to FIG. 4B, in this mode, in some embodiments, the user can move the object to various positions in the ZY axis of the XYZ axis 470. For example, the XR device moves the object 464 in the Z and Y directions in the user interface of the virtual environment when the user slides his / her finger 466 in the Z and Y directions on the display interface of the smartwatch 465. For example, the XR device moves the object 464 along the Z axis when the user slides his / her finger left and right, and moves the object 464 along the Y axis when the user slides his / her finger up and down. When any of these actions occur, the touch input data captured by the touch sensor of the display interface of the smartwatch 465 is sent to the XR device, where the touch input data is converted into commands, and the commands are executed by the XR device to move the object 466 accordingly. It should be noted that other single finger movements and directions can be specified to move the object.

[0035]

[0047] In FIG. 4C, in the first mode, after selecting the Edit Placement button, the user can swipe a single finger in a circular motion around the display surface of the smartwatch device to rotate the selected object in the user interface displayed by the XR device. For example, the XR device rotates the object 451 in a clockwise direction in the user interface of the virtual environment when the user slides his / her finger 453 in a clockwise direction around the display interface of the smartwatch 452, and rotates the object 451 in a counterclockwise direction in the user interface of the virtual environment when the user slides his / her finger 453 in a counterclockwise direction on the display interface of the smartwatch 452. When any of these circular motions occur, the touch input data captured by the touch sensor of the display interface of the smartwatch 452 is sent to the XR device, where the touch input data is converted into commands, and the commands are executed by the XR device to rotate the object 451 accordingly. It should be noted that other single finger movements and directions can be specified to rotate an object.

[0036]

[0048] 4D illustrates a second mode in which a user can use a single finger or two fingers to rotate a selected object by moving them in a circular direction around the display surface of the smartwatch. For example, the XR device rotates the object 454 in a clockwise direction in the user interface of the virtual environment when the user slides his / her two fingers 456 in a clockwise direction around the display interface of the smartwatch 455, and rotates the object 454 in a counterclockwise direction in the user interface of the virtual environment when the user slides his / her fingers 456 in a counterclockwise direction on the display interface of the smartwatch 455. When any of these circular movements occur, touch input data captured by the touch sensor of the display interface of the smartwatch 455 is sent to the XR device, where the touch input data is converted into commands, which are executed by the XR device to rotate the object 454 accordingly. It should be noted that other finger movements and directions can be specified to rotate an object.

[0037]

[0049] 4E and 4F show an example of performing scaling on an object using a smartwatch. Referring to FIG. 4E, in a first mode, after selecting the edit placement button and selecting the scaling operation, the user moves a single finger up and down to scale the size of the object in the smartwatch interface. For example, the XR device increases the size of the object 450 in the user interface of the virtual environment when the user slides his / her finger 410 upward on the display interface of the smartwatch 400, and decreases the size of the object 450 in the user interface of the virtual environment when the user slides his / her finger 410 downward on the display interface of the smartwatch 400. When either of these upward or downward movements occur, the touch input data captured by the touch sensor of the display interface of the smartwatch 400 is sent to the XR device, where the touch input data is converted into commands, and the commands are executed by the XR device to scale the object 450 accordingly. It should be noted that other single finger movements can be specified to increase the size of an object and to decrease the size of an object. For example, a user can move a single finger left and right to increase and decrease the size of an object, respectively. Note that the movements do not have to be in opposite directions for an object to be enlarged or reduced.

[0038]

[0050] Referring to FIG. 4F, in the second mode, after selecting the Edit Placement button and selecting the Scale operation, the user uses two fingers in a pinching motion (i.e., moving two fingers closer together or apart) to scale the size of the object displayed in the user interface of the virtual environment. For example, the XR device increases the size of the object 460 in the user interface of the virtual environment when the user moves the fingers 420 apart (reverse pinching motion) on the display interface of the smartwatch 401, and decreases the size of the object 460 in the user interface of the virtual environment when the user moves the fingers 420 together in a pinching motion on the display interface of the smartwatch 401. When any of these reverse pinching or pinching motions occur, touch input data captured by the touch sensor of the display interface of the smartwatch 400 is sent to the XR device, where the touch input data is converted into commands and the commands are executed by the XR device to scale the object 450 accordingly. Note that other multiple finger movements can be specified to increase the size of an object and to decrease the size of an object.

[0039]

[0051] 5 is a flow diagram of some embodiments of a process for controlling objects displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. In some embodiments, the process is performed at least in part by processing logic that comprises hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., software running on a chip, software running on a general-purpose computer system or dedicated machine, etc.), firmware, or a combination of the three.

[0040]

[0052] Referring to Figure 5, the process begins with processing logic receiving a selection of an object in a UI displayed in a virtual environment created by an extended reality (XR) device (processing block 501). In some embodiments, processing logic detects selection of the object via tapless selection by determining that a user's hand position is held in a position directed toward the object for a pre-specified amount of time. In some other embodiments, processing logic detects selection of one or more objects in the UI in the virtual environment by tracking the position of the user's hand and thumb or fingers. In some other embodiments, processing logic detects selection of one or more objects in the UI in the virtual environment by a controller of the XR device.

[0041]

[0053] After selection of the object, processing logic displays a menu in the UI having one or more graphical user interface (GUI) elements selectable using the wearable device (processing block 502). In some embodiments, the one or more GUI elements include one or more of a first GUI element for editing the object and a second GUI element for moving the object based on the head tracking data. In some embodiments, the displayed menu is the menu shown in Figures 3A and 3B. In some embodiments, the wearable device includes a wristwatch. In some embodiments, the wristwatch includes a smartwatch device, and the first touch input data and the additional touch input data are from a display interface of the smartwatch.

[0042]

[0054] After displaying the menu, processing logic receives and interprets touch input data indicating a user selection of one of the one or more GUI elements in the menu (processing block 503). In some embodiments, this touch input data is received in response to a user touching the wearable device.

[0043]

[0055] After receiving additional touch input data from the wearable device and translating the additional touch input data into one or more actions associated with the selected GUI element to be performed on the object, processing logic performs the one or more actions on the object in the UI displayed in the virtual environment (processing block 504) and updates the UI based on a result of performing the one or more actions (processing block 505). In some embodiments, the one or more actions include changing a position of the object, rotating the object, and scaling the object. In some embodiments, the additional touch input data for changing a position of the object is from a user moving at least one finger across a display interface of the wearable device. In some embodiments, the additional touch input data includes touch input data generated in response to a user's single finger or multiple finger movement across a display interface of the wearable device.

[0044]

[0056] 6 is a flow diagram of some other embodiments of a process for controlling objects displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. In some embodiments, the process is performed at least in part by processing logic that comprises hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., software running on a chip, software running on a general-purpose computer system or dedicated machine, etc.), firmware, or a combination of the three.

[0045]

[0057] 6, the process begins by processing logic displaying a user interface on a display of an extended reality (XR) device (processing block 601). In some embodiments, the user interface includes one or more objects. Processing logic then determines that the user is looking at an object displayed on the user interface based on head tracking performed by the XR device (processing block 602) and, in response to the determination, highlights the object as a selected object (processing block 603).

[0046]

[0058] While highlighting the object, processing logic displays a menu in the user interface that includes one or more graphical user interface (GUI) elements that are selectable using the device worn by the user (processing block 604). In some embodiments, the device worn by the user is a watch. In some embodiments, the watch includes a smartwatch. In some other embodiments, the device worn by the user is another wearable device.

[0047]

[0059] Thereafter, processing logic receives touch input data indicating a user selection of one or more of the GUI elements in response to the user touching their device (e.g., their smartwatch) (processing block 605). In some embodiments, the user touches the display of the smartwatch to generate touch input data that is then transmitted to an XR device that is communicatively connected to the device worn by the user.

[0048]

[0060] Processing logic then translates the touch input data into an action associated with the selected GUI element to be performed on the object selected by the user in the user interface (process 606). Processing logic then executes the action (processing block 607). In some embodiments, executing the action includes updating the displayed user interface based on the results of executing the action. Such updates may include changing the position of the selected object, rotating the object, and / or scaling the object. Other actions may also be performed.

[0049]

[0061] 7 is a flow diagram of some embodiments of a process for moving an object displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. In some embodiments, the process is performed at least in part by processing logic that comprises hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., software running on a chip, software running on a general-purpose computer system or dedicated machine, etc.), firmware, or a combination of the three.

[0050]

[0062] Referring to FIG. 7, the process begins with processing logic receiving a selection of an object in a UI in the virtual environment (processing block 701). In some embodiments, the object is selected by performing head tracking and the XR device determining that the user is looking at the object for a pre-specified period of time (e.g., 2-5 seconds, etc.). In some other embodiments, the processing logic detects the selection of the object through tapless selection by determining that the position of the user's hand is held in a position directed toward the object for a pre-specified period of time. In some other embodiments, the processing logic detects the selection of one or more objects in the UI in the virtual environment by tracking the position of the user's hand and thumb or finger. In some other embodiments, the processing logic detects the selection of one or more objects in the UI in the virtual environment by a controller of the XR device.

[0051]

[0063] In response to selecting the object, processing logic displays a menu of user-selectable actions that the user can perform in a UI in the virtual environment generated by the XR device (processing block 702). The actions in the menu are selectable using the user's wearable device. Examples of such menus are shown in Figures 3A and 3B.

[0052]

[0064] Thereafter, processing logic receives touch input data from a user interacting with the wearable device indicating that the user is selecting an Edit Placement menu item (processing block 703). In some embodiments, the wearable device includes a smartwatch. In some embodiments, the touch input data corresponds to the user touching a display interface of the wearable device (e.g., smartwatch) at a location that corresponds to an Edit Placement menu item of a menu displayed in a UI in the virtual environment generated by the XR device.

[0053]

[0065] Processing logic determines that the user desires to move the selected object (processing logic 704). In some embodiments, processing logic receives a selection of one of the placement editing operations to be performed on the object.

[0054]

[0066] In some embodiments, processing logic displays a multiple axis mode in a UI displayed in the virtual environment in response to identifying a user's desire to move an object (processing block 705). In some embodiments, the multiple axis mode includes an XY axis mode for indicating to the XR device that touch input data received from moving one or more fingers across a display interface of the wearable device corresponds to movement in the X and Y axes, and a ZY axis mode for indicating to the XR device that touch input data received from moving one or more fingers across a display interface of the wearable device corresponds to movement in the Z and Y axes. In such a case, processing logic may also receive a selection of the XY axis mode or the ZY axis mode (processing block 706).

[0055]

[0067] In response to the user moving one or more fingers across a surface (e.g., a display surface) of the wearable device, processing logic receives and converts additional touch input data from the wearable device specifying the direction and amount to move the object (processing block 707) and moves the object according to the additional touch input data and the selection of the XY axis mode or the ZY axis mode (processing block 708). In some embodiments, the touch input data is created by the user moving one finger across the surface of the wearable device. In some other embodiments, the touch input data is created by the user moving two fingers across the surface of the wearable device. As the user repeats the finger movements to indicate additional object movements he or she wishes to make, processing logic may receive more touch input data from the wearable device.

[0056]

[0068] After the user moves the object, processing logic receives touch input data from the wearable device indicating that the user has completed moving the object (processing block 709). In some embodiments, the touch input data is the result of the user pressing the surface of the wearable device at a location designated to indicate that the placement editing operation has been completed by the user (e.g., pressing the center of the wearable device corresponding to the location of a checkmark (e.g., see the checkmark in FIGS. 3A and 3B )).

[0057]

[0069] 8 is a flow diagram of some embodiments of a process for rotating an object displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. In some embodiments, the process is performed at least in part by processing logic comprising hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., software running on a chip, software running on a general-purpose computer system or dedicated machine, etc.), firmware, or a combination of the three.

[0058]

[0070] Referring to FIG. 8, the process begins with processing logic receiving a selection of an object in a UI in the virtual environment (processing block 801). In some embodiments, the object is selected by performing head tracking and the XR device determining that the user is looking at the object for a pre-specified period of time (e.g., 2-5 seconds, etc.). In some other embodiments, the processing logic detects the selection of the object through tapless selection by determining that the position of the user's hand is held in a position directed toward the object for a pre-specified period of time. In some other embodiments, the processing logic detects the selection of one or more objects in the UI in the virtual environment by tracking the position of the user's hand and thumb or finger. In some other embodiments, the processing logic detects the selection of one or more objects in the UI in the virtual environment by a controller of the XR device.

[0059]

[0071] In response to selecting the object, processing logic displays a menu of user-selectable actions that the user can perform in a UI in the virtual environment generated by the XR device (processing block 802). The actions in the menu are selectable using the user's wearable device. Examples of such menus are shown in Figures 3A and 3B.

[0060]

[0072] Thereafter, processing logic receives touch input data from a user interacting with the wearable device indicating that the user is selecting an Edit Placement menu item (processing block 803). In some embodiments, the wearable device includes a smartwatch. In some embodiments, the touch input data corresponds to the user touching a display interface of the wearable device (e.g., smartwatch) at a location that corresponds to an Edit Placement menu item of a menu displayed in a UI in the virtual environment generated by the XR device.

[0061]

[0073] Processing logic determines that the user desires to rotate the selected object (processing logic 804). In some embodiments, processing logic receives a selection of one of the placement editing operations to be performed on the object.

[0062]

[0074] In some embodiments, processing logic displays multiple axis modes in a UI displayed in the virtual environment in response to identifying a user's desire to rotate an object (processing block 805). In some embodiments, the multiple axis modes include an X-axis mode for indicating to the XR device that touch input data received from moving one or more fingers across a display interface of the wearable device corresponds to a rotation in an X-axis, a Y-axis mode for indicating to the XR device that touch input data received from moving one or more fingers across a display interface of the wearable device corresponds to a rotation in a Y-axis, and a Z-axis mode for indicating to the XR device that touch input data received from moving one or more fingers across a display interface of the wearable device corresponds to a rotation in a Z-axis. In such a case, processing logic may also receive a selection of the X-axis mode, the Y-axis mode, or the Z-axis mode (processing block 806).

[0063]

[0075] In response to the user moving one or more fingers in a circular direction across a surface (e.g., a display surface) of the wearable device, processing logic receives and converts additional touch input data from the wearable device specifying a direction and amount of rotation to rotate the object (processing block 807) and rotates the object according to the additional touch input data and the selection of X, Y, or Z axis mode (processing block 808). In some embodiments, the touch input data is created by the user moving one finger across the surface of the wearable device in a circular direction around an edge of the display surface of the wearable device. In some other embodiments, the touch input data is created by the user moving two fingers across the surface of the wearable device in a circular direction around an edge of the display surface of the wearable device. As the user repeats the finger movement to indicate additional rotations of the object he or she wishes to make, processing logic may receive more touch input data from the wearable device.

[0064]

[0076] After the user finishes rotating the object, processing logic receives touch input data from the wearable device indicating that the user has completed rotating the object (processing block 809). In some embodiments, the touch input data is the result of the user pressing a surface of the wearable device at a location designated to indicate that a placement editing operation has been completed by the user (e.g., pressing the center of the wearable device corresponding to the location of a checkmark (e.g., see the checkmark in FIGS. 3A and 3B )).

[0065]

[0077] 9 is a flow diagram of some embodiments of a process for scaling an object displayed in a user interface (UI) in a virtual environment generated by an XR device using a wearable device. In some embodiments, the process is performed at least in part by processing logic comprising hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., software running on a chip, software running on a general-purpose computer system or dedicated machine, etc.), firmware, or a combination of the three.

[0066]

[0078] Referring to FIG. 9 , the process begins with processing logic receiving a selection of an object in a UI in the virtual environment (processing block 901). In some embodiments, the object is selected by performing head tracking and the XR device determining that the user is looking at the object for a pre-specified period of time (e.g., 2-5 seconds, etc.). In some other embodiments, the processing logic detects the selection of the object through tapless selection by determining that the position of the user's hand is held in a position directed toward the object for a pre-specified time. In some other embodiments, the processing logic detects the selection of one or more objects in the UI in the virtual environment by tracking the position of the user's hand and thumb or finger. In some other embodiments, the processing logic detects the selection of one or more objects in the UI in the virtual environment by a controller of the XR device.

[0067]

[0079] In response to selecting the object, processing logic displays a menu of user-selectable actions that the user can perform in a UI in the virtual environment generated by the XR device (processing block 902). The actions in the menu are selectable using the user's wearable device. Examples of such menus are shown in Figures 3A and 3B.

[0068]

[0080] Thereafter, processing logic receives touch input data from a user interacting with the wearable device indicating that the user is selecting an Edit Placement menu item (processing block 903). In some embodiments, the wearable device includes a smartwatch. In some embodiments, the touch input data corresponds to the user touching a display interface of the wearable device (e.g., smartwatch) at a location that corresponds to an Edit Placement menu item of a menu displayed in a UI in the virtual environment generated by the XR device.

[0069]

[0081] Processing logic determines that a user desires to scale the size of a selected object (processing logic 904). In some embodiments, processing logic receives a selection of one of the placement editing operations to be performed on the object.

[0070]

[0082] In response to the user moving one or more fingers in a circular direction across a surface (e.g., a display surface) of the wearable device, processing logic receives and converts additional touch input data from the wearable device specifying a scaling amount to scale the size of the object (processing block 905) and scales the object according to the additional touch input data and the selection of X, Y, or Z axis mode (processing block 906). In some embodiments, the touch input data is created by the user moving one finger across the surface of the wearable device, where moving the finger in one direction (e.g., up, down, left, right, etc.) increases the size of the object and moving the finger in another direction (e.g., the opposite direction, down, up, right, left, etc.) decreases the size of the object. In some other embodiments, the touch input data is created by the user moving two fingers together or apart across the surface of the wearable device to indicate whether to scale the object larger or smaller. As the user repeats the finger movement to indicate additional scaling of the object that he or she would like to perform, the processing logic may receive more touch input data from the wearable device.

[0071]

[0083] After the user finishes scaling the object, processing logic receives touch input data from the wearable device indicating that the user has completed scaling the object (processing block 907). In some embodiments, the touch input data is the result of the user pressing the surface of the wearable device at a location designated to indicate that a placement editing operation has been completed by the user (e.g., pressing the center of the wearable device corresponding to the location of a checkmark (e.g., see the checkmark in FIGS. 3A and 3B )).

[0072]

[0084] As discussed above, in some embodiments, the user interface (UI) / user experience (UX) of the XR device includes a tapless selection feature for selecting an object, a graphical user interface (GUI) element (e.g., a button, a menu, etc.), or other item in a virtual environment displayed in the UI of the XR device. In some embodiments, when performing a tapless selection, the user does not have to do anything other than hold their hand in place for a pre-specified period of time. That is, as long as the person can keep their hand position toward the object the user wants to select for a specific period of time, the XR device can identify the object the user wants to select and makes the selection. As long as the user's hand and / or fingers are paused at a specific position for a pre-specified amount of time, the user can use the tapless selection feature to select a menu or icon displayed in the user interface of the XR device. In some embodiments, the pre-specified time can be set at a variable time. For example, in some embodiments, the pre-specified period during which the user must hold or pause their hand and / or fingers at a specific position to select a menu or icon in the user interface of the XR device is 1 or 2 seconds. In some embodiments, the pre-specified time can be set and / or optimized based on the application. In some embodiments, the user interface displays an indicator when the hand and / or finger is paused or held at a particular position. The indicator shows the amount of time remaining to hold the position to perform a tapless selection. In some embodiments, the indicator is a closing circle in which a circle is formed from a single point in the user interface that continues to be drawn in a clockwise (or counterclockwise) direction until it reaches the starting point. The time to draw the circle is the time a user needs to hold their hand and / or finger(s) in place to select an item in the user interface.It should be noted that other indicators may be used. Examples of XR devices that perform tapless selection are described, for example, in U.S. Provisional Patent Application No. 63 / 393,172, entitled "XR Tapless Selection," filed July 28, 2022, which is incorporated herein by reference.

[0073]

[0085] 10 is one embodiment of a computer system that may be used to support the systems and operations discussed herein, including those of an XR device. However, it will be apparent to one of ordinary skill in the art that other alternative systems of various system architectures may also be used.

[0074]

[0086] The data processing system shown in Figure 10 includes a bus or other internal communication means 1015 for communicating information, and one or more processors 1010 coupled to the bus 1015 for processing information. The system further includes a random access memory (RAM) or other volatile storage device 1050 (referred to as memory) coupled to the bus 1015 for storing information and instructions to be executed by the processor(s) 1010. The main memory 1050 may also be used to store temporary variables or other intermediate information during execution of instructions by the processor(s) 1010. The system also includes a read only memory (ROM) and / or static storage device 1020 coupled to the bus 1015 for storing static information and instructions for the processor(s) 1010, and a data storage device 1025, such as a magnetic or optical disk and its corresponding disk drive. The data storage device 1025 is coupled to the bus 1015 for storing information and instructions.

[0075]

[0087] The system may further be coupled to a display device 1070, such as a light emitting diode (LED) display or liquid crystal display (LCD), coupled to the bus 1015 through the bus 1065 for displaying information to a computer user. An alphanumeric input device 1075, including alphanumeric and other keys, may also be coupled to the bus 1015 through the bus 1065 for communicating information and command selections to the processor(s) 1010. An additional user input device is a cursor control device 1080, such as a touch pad, mouse, trackball, stylus, or cursor direction keys, coupled to the bus 1015 through the bus 1065 for communicating directional information and command selections to the processor 1010, as well as for controlling cursor movement on the display device 1070.

[0076]

[0088] Another device that may be optionally coupled to computer system 1000 is a communication device 1090 for accessing other nodes of a distributed system over a network. Communication device 1090 may include any of several commercially available networking peripheral devices such as those used to couple to Ethernet, Token Ring, Internet, or wide area networks. Communication device 1090 may also be a null modem connection or any other mechanism that provides connectivity between computer system 1000 and the outside world. It should be noted that any or all of the components of this system and associated hardware shown in FIG. 10 may be used in the various embodiments discussed herein.

[0077]

[0089] There are several exemplary embodiments described herein.

[0078]

[0090] Example 1 is a method for controlling a user interface (UI) including receiving a selection of an object in a UI displayed in a virtual environment created by an extended reality (XR) device; displaying a menu in the UI having one or more graphical user interface (GUI) elements selectable using a wearable device; receiving and converting first touch input data indicating a user selection of one of the one or more GUI elements, the first touch input data being received in response to a user touching the wearable device; receiving additional touch input data from the wearable device and converting the additional touch input data into one or more actions associated with the selected GUI element to be performed on the object; and performing one or more actions on the object in the UI displayed in the virtual environment, including updating the UI based on a result of performing the one or more actions.

[0079]

[0091] Example 2 is the method of Example 1 which may optionally include that the one or more GUI elements include one or more of a first GUI element for editing the object and a second GUI element for moving the object based on head tracking data.

[0080]

[0092] Example 3 is the method of example 1 that can optionally include where the one or more actions include changing a position of the object, rotating the object, and scaling the object.

[0081]

[0093] Example 4 is the method of Example 3 that can optionally include that the additional touch input data for changing a position of the object is from a user moving at least one finger across a display interface of the wearable device.

[0082]

[0094] Example 5 is the method of Example 4 that may optionally include displaying multiple axis modes in a UI displayed in the virtual environment, the multiple axis modes including an XY axis mode for indicating to the XR device that touch input data received from moving a single finger across a display interface of the wearable device corresponds to movement in the X and Y axes, and a ZY axis mode for indicating to the XR device that touch input data received from moving a single finger across a display interface of the wearable device corresponds to movement in the Z and Y axes; receiving a selection of the XY axis mode or the ZY axis mode; moving the object according to the additional touch input data and the selection of the XY axis mode or the ZY axis mode; and receiving touch input data from the wearable device indicating that the user has completed moving the object.

[0083]

[0095] Example 6 is the method of Example 3 which can optionally include that the additional touch input data for rotating the object is from the user moving at least one finger in a circular direction on a display interface of the wearable device, the method further including receiving touch input data from the wearable device indicating that the user has completed rotating the object.

[0084]

[0096] Example 7 is the method of Example 3 which can optionally include that the additional touch input data for scaling the object is from a user moving at least one finger on a display interface of the wearable device, the method further including receiving touch input data from the wearable device indicating that the user has completed scaling the object.

[0085]

[0097] Example 8 is the method of Example 1 which can optionally include that the additional touch input data includes touch input data generated in response to a movement of a single finger or multiple fingers of a user across a display interface of the wearable device.

[0086]

[0098] Example 9 is the method of example 1 which can optionally include that the wearable device comprises a wristwatch.

[0087]

[0099] Example 10 is the method of Example 1, optionally including, where the wristwatch includes a smartwatch device, and the first touch input data and the additional touch input data are from a display interface of the smartwatch.

[0088]

[0100] Example 11 is the method of Example 1, which may optionally include the steps of displaying a user interface on a display of an extended reality (XR) device, the user interface (UI) including one or more objects, determining that a user is viewing an object displayed on the UI based on head tracking by the XR device, and in response to determining that the user is viewing the object, highlighting the object as a selected object.

[0089]

[0101] Example 12 is a computing system for controlling a user interface, the computing system comprising one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process including receiving a selection of an object in a UI displayed in a virtual environment created by an extended reality (XR) device; displaying in the UI a menu having one or more graphical user interface (GUI) elements selectable using the wearable device; receiving and converting first touch input data indicating a user selection of one of the one or more GUI elements, the first touch input data being received in response to a user touching the wearable device; receiving additional touch input data from the wearable device and converting the additional touch input data into one or more actions associated with the selected GUI element to be performed on the object; and updating the UI based on a result of performing the one or more actions.

[0090]

[0102] Example 13 is the computing system of Example 12, wherein the wearable device includes a smartwatch device, the first touch input data and the additional touch input data are from a display interface of the smartwatch, and the one or more actions can optionally include changing a position of an object, rotating an object, and scaling an object.

[0091]

[0103] Example 14 is the computing system of Example 13, which can optionally include that the additional touch input data for changing a position of the object is from a user moving at least one finger across a display interface of the wearable device.

[0092]

[0104] Example 15 is the computing system of Example 14, which may optionally include displaying multiple axis modes in a UI displayed in the virtual environment, the multiple axis modes including an XY axis mode for indicating to the XR device that touch input data received from moving a single finger across a display interface of the wearable device corresponds to movement in the X and Y axes, and a ZY axis mode for indicating to the XR device that touch input data received from moving a single finger across a display interface of the wearable device corresponds to movement in the Z and Y axes; receiving a selection of the XY axis mode or the ZY axis mode; moving the object according to the additional touch input data and the selection of the XY axis mode or the ZY axis mode; and receiving touch input data from the wearable device indicating that the user has completed moving the object.

[0093]

[0105] Example 16 is the computing system of Example 13 which can optionally include that the additional touch input data for rotating the object is from the user moving at least one finger in a circular direction on a display interface of the wearable device, the computing system further including receiving touch input data from the wearable device indicating that the user has completed rotating the object.

[0094]

[0106] Example 17 is the computing system of Example 13 which may optionally include that the additional touch input data for scaling the object is from a user moving at least one finger on a display interface of the wearable device, the computing system further including receiving touch input data from the wearable device indicating that the user has completed scaling the object.

[0095]

[0107] Example 18 is the computing system of Example 13, which can optionally include that the additional touch input data includes touch input data generated in response to a movement of a single finger or multiple fingers of a user across a display interface of the wearable device.

[0096]

[0108] Example 19 is one or more non-transitory computer-readable storage media having instructions stored thereon that, when executed by a system having at least a processor and memory therein, cause the system to perform operations including receiving a selection of an object in a UI displayed in a virtual environment created by an extended reality (XR) device; displaying a menu in the UI having one or more graphical user interface (GUI) elements selectable using the wearable device; receiving and converting first touch input data indicating a user selection of one of the one or more GUI elements, where the first touch input data is received in response to a user touching the wearable device; receiving additional touch input data from the wearable device and converting the additional touch input data into one or more actions associated with the selected GUI element to be performed on the object; and updating the UI based on a result of performing the one or more actions.

[0097]

[0109] Example 20 is the one or more non-transitory computer-readable storage media of Example 19, optionally including: the wearable device includes a smartwatch device; the first touch input data and the additional touch input data are from a display interface of the smartwatch; and the one or more actions include changing a position of an object, rotating an object, and scaling an object.

[0098]

[0110] It will be apparent to one of ordinary skill in the art that the systems, methods, and processes described herein can be implemented as software stored in a main memory or a read-only memory and executed by a processor. This control logic or software can also be present in an article of manufacture that includes a non-transitory computer-readable medium having computer-readable program code embodied therein and readable by a mass storage device for operating a processor according to the methods and teachings herein.

[0099]

[0111] The embodiments discussed herein may also be embodied in a handheld or portable device that includes a subset of the computer hardware components described above. For example, a handheld device may be a mobile phone, a tablet computer, a dedicated computing device, etc., configured to include only a bus, a processor, and a memory. A handheld device may also be configured to include a set of buttons or input signaling components with which a user can select from a set of available options. A handheld device may also be configured to include an output device, such as a liquid crystal display (LCD) or a matrix of display elements for displaying information to a user of the handheld device. Conventional methods may be used to implement such handheld devices. The implementation of the embodiments for such devices will be apparent to one of ordinary skill in the art given the disclosure provided herein.

[0100]

[0112] The embodiments discussed herein may also be embodied in a special-purpose appliance that includes a subset of the computer hardware components described above. For example, the appliance may include a processor, a data storage device, a bus, and memory, and only a rudimentary communication mechanism, such as a small touch screen that allows a user to communicate in a basic manner with the device. In general, the more special-purpose the device, the fewer elements that need to be present for the device to function.

[0101]

[0113] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0102]

[0114] However, keep in mind that all of these and similar words are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically noted, as will be apparent from the discussion that follows, discussions utilizing words such as "processing," "calculating," "computing," "determining," "displaying," and the like throughout this description refer to the actions and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of the computer system into other data that are also represented as physical quantities in the memory or registers of the computer system, or other such information storage, transmission, or display device.

[0103]

[0115] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes or may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer-readable storage medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0104]

[0116] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages ​​can be used to implement the teachings of the present disclosure as described herein.

[0105]

[0117] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, machine-readable media includes read-only memory ("ROM"), random-access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, electrically propagated signals, optically propagated signals, acoustically propagated signals, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), etc.

[0106]

[0118] Although many variations and modifications of the present disclosure will no doubt become apparent to those of ordinary skill in the art after reading the foregoing description, it should be understood that any particular embodiments shown and described by way of illustration are in no way intended to be considered limiting. Thus, references to details of various embodiments are not intended to limit the scope of the claims, which inherently recite only those features regarded as essential to the present disclosure.

Claims

1. 1. A method for controlling a user interface (UI), comprising: receiving a selection of an object in the UI being displayed in a virtual environment created by an extended reality (XR) device; Displaying a menu in the UI having one or more Graphical User Interface (GUI) elements selectable using the wearable device; receiving first touch input data indicating a user selection of one of the one or more GUI elements and transforming the first touch input data, the first touch input data being received in response to a user touching the wearable device; receiving additional touch input data from the wearable device and translating the additional touch input data into one or more actions associated with the selected GUI element to be performed on the object in the UI displayed in the virtual environment; performing the one or more actions on the object in the UI displayed in the virtual environment, the one or more actions including updating the UI based on a result of performing the one or more actions; The method includes:

2. The menu, The method of claim 1 , comprising one or more of: a first GUI element for editing the object; and a second GUI element for moving the object based on head tracking data.

3. The one or more actions are: changing the position of the object; Rotating the object; and Scaling the object; The method of claim 1 , comprising:

4. The method of claim 3 , wherein the additional touch input data for changing a position of the object results from the user moving at least one finger across a display interface of the wearable device.

5. The method comprises: Displaying a plurality of axis modes in the UI displayed in the virtual environment, the plurality of axis modes comprising: an XY axis mode for indicating to the XR device that touch input data received from moving a single finger across the display interface of the wearable device corresponds to movement in an X axis and a Y axis; and the displaying step including a ZY axis mode for indicating to the XR device that touch input data received from moving the single finger across the display interface of the wearable device corresponds to movement in a Z axis and a Y axis; receiving a selection of the XY axis mode or the ZY axis mode; moving the object according to the additional touch input data and a selection of the XY axis mode or the ZY axis mode; receiving touch input data from the wearable device indicating that the user has completed moving the object; The method of claim 4 further comprising:

6. the additional touch input data for rotating the object results from the user moving at least one finger in a circular direction on a display interface of the wearable device; The method further comprising: receiving touch input data from the wearable device indicating that the user has completed rotating the object; The method of claim 3 further comprising:

7. the additional touch input data for scaling the object results from the user moving at least one finger on a display interface of the wearable device; The method further comprising: receiving touch input data from the wearable device indicating that the user has completed scaling the object; The method of claim 3 further comprising:

8. The method of claim 1 , wherein the additional touch input data comprises touch input data generated in response to a movement of a single finger or multiple fingers of the user across a display interface of the wearable device.

9. The method of claim 1 , wherein the wearable device comprises a watch.

10. The wristwatch includes a smart watch, The method of claim 9 , wherein the first touch input data and the additional touch input data originate from a display interface of the smartwatch.

11. The method comprises: displaying the user interface on a display of an extended reality (XR) device, the user interface (UI) including one or more objects; determining that the user is looking at an object displayed on the UI based on head tracking by the XR device; in response to identifying the user as viewing the object, highlighting the object as a selected object; The method of claim 1 further comprising:

12. 1. A computing system for controlling a user interface (UI), comprising: one or more processors; one or more memories storing instructions; Equipped with The instructions, when executed by the one or more processors, receiving a selection of an object in the UI being displayed in a virtual environment created by an extended reality (XR) device; Displaying a menu in the UI having one or more graphical user interface (GUI) elements selectable using the wearable device; receiving and converting first touch input data indicating a user selection of one of the one or more GUI elements, the first touch input data being received in response to a user touching the wearable device; receiving additional touch input data from the wearable device and converting the additional touch input data into one or more actions associated with the selected GUI element to be performed on the object in the UI displayed in the virtual environment; performing the one or more actions on the object in the UI displayed in the virtual environment, the one or more actions including updating the UI based on a result of performing the one or more actions; causing said computing system to execute a process comprising: Computing system.

13. the wearable device comprises a smart watch; the first touch input data and the additional touch input data originate from a display interface of the smartwatch; the one or more actions include changing a position of the object, rotating the object, and scaling the object. The computing system of claim 12.

14. 14. The computing system of claim 13, wherein the additional touch input data for changing the position of the object results from the user moving at least one finger across a display interface of the wearable device.

15. The process comprises: Displaying a plurality of axis modes in the UI displayed in the virtual environment, the plurality of axis modes comprising: an XY axis mode for indicating to the XR device that touch input data received from moving a single finger across the display interface of the wearable device corresponds to movement in an X-axis and a Y-axis; and a ZY axis mode for indicating to the XR device that touch input data received from moving the single finger across the display interface of the wearable device corresponds to movement in a Z axis and a Y axis. Such indication, and receiving a selection of the XY axis mode or the ZY axis mode; moving the object according to the additional touch input data and the selection of the XY axis mode or the ZY axis mode; receiving touch input data from the wearable device indicating that the user has completed moving the object; The computing system of claim 14 further comprising:

16. the additional touch input data for rotating the object results from the user moving at least one finger in a circular direction on a display interface of the wearable device; The process comprises: receiving, by the computing system, touch input data from the wearable device indicating that the user has completed rotating the object; The computing system of claim 13 further comprising:

17. the additional touch input data for scaling the object results from the user moving at least one finger on a display interface of the wearable device; The process comprises: receiving, by the computing system, touch input data from the wearable device indicating that the user has completed scaling the object; The computing system of claim 13 further comprising:

18. 14. The computing system of claim 13, wherein the additional touch input data comprises touch input data generated in response to a movement of a single finger or multiple fingers of the user across a display interface of the wearable device.

19. One or more non-transitory computer-readable storage media having instructions stored thereon, The instructions, when executed by a system having at least a processor and a memory therein, Receiving a selection of an object in a user interface (UI) displayed in a virtual environment created by an extended reality (XR) device; Displaying a menu in the UI having one or more graphical user interface (GUI) elements selectable using the wearable device; receiving and converting first touch input data indicating a user selection of one of the one or more GUI elements, the first touch input data being received in response to a user touching the wearable device; receiving additional touch input data from the wearable device and converting the additional touch input data into one or more actions associated with the selected GUI element to be performed on the object in the UI displayed in the virtual environment; performing the one or more actions on the object in the UI displayed in the virtual environment, the one or more actions including updating the UI based on a result of performing the one or more actions; causing the system to perform operations including One or more non-transitory computer-readable storage media.

20. the wearable device comprises a smart watch; the first touch input data and the additional touch input data originate from a display interface of the smartwatch; the one or more actions include changing a position of the object, rotating the object, and scaling the object.

20. One or more non-transitory computer readable storage media as recited in claim 19.

Citation Information

Patent Citations

  • Pellet filling vessel

    JP1988090799A

  • Image display correction based on sensor input for transmissive myopia displays.

    JP2015504616A

  • Intelligent interaction methods, devices and systems

    JP2018508909A

  • Dynamically configurable application control elements

    JP2021523426A

  • Glasses-type information appliance, method for the same, and program

    JP2022081342A