Triggering field transitions of artificial reality objects
By defining near-field and far-field areas in an artificial reality environment, and using touch and light interaction mechanisms to dynamically manage the display of virtual objects, the problems of space congestion and inconvenience in virtual object interaction in traditional technology are solved, and efficient Virtual object interaction and management.
Patent Information
- Application Number
- CN202380032730.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-08
- Filing Date
- 2023-04-09
- Publication Date
- 2024-11-15
AI Technical Summary
Traditional virtual object interactions have problems of spatial congestion and inconvenience in artificial real environments, especially in the interaction between users and far-field virtual objects, it is difficult to achieve efficient fine-grained selection and precise control.
By defining near-field and far-field areas in an artificial reality environment, and using touch interaction and light projection interaction mechanisms, combining engagement gestures and motion detection, dynamically manage the display position and size of virtual objects to achieve the area between virtual objects Convert to meet the user's interaction needs.
It effectively reduces the mess in the near field area, improves the distance between the user and the virtual object, enhances the fineness and accuracy of interaction, and supports efficient virtual object interaction and management.
Smart Images

Figure CN118974688A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to triggering object transitions between near-field regions and far-field regions in an artificial reality environment. Background Art
[0002] Artificial reality devices provide users with the opportunity to experience interactions in artificial reality environments. However, there are still challenges in user interaction with virtual objects in these environments. Touch-based interactions usually require virtual objects to be displayed near the user, which sometimes creates a cluttered space near the user. For virtual objects that are displayed far away from the user, the distance between the user and the virtual object can create its own set of interaction challenges. Summary of the invention
[0003] According to a first aspect of the present invention, a method for triggering an object conversion between a near field area and a far field area in an artificial reality environment is provided, the method comprising: displaying a virtual object comprising at least a two-dimensional surface to a user in the artificial reality environment, wherein the virtual object is displayed in the near field area, and the near field area is located within an arm's length from the user; detecting an engagement gesture of the user, which engages the virtual object in the near field area until the user releases the engagement; after engaging the virtual object, detecting a hand movement of the user that meets a trigger criterion, wherein the trigger criterion comprises: a) a threshold radial distance from the user; b) a speed threshold; c) a speed change threshold; or d) any combination of the above; and based on the detected hand movement that meets the trigger criterion, converting the virtual object from the near field area to the far field area, wherein the virtual object is displayed in the far field area, and the far field area is located at a distance greater than an arm's length from the user.
[0004] Optionally, when the virtual object in the artificial reality environment is displayed in the near field area, the user interacts with the virtual object using touch interaction, and when the virtual object in the artificial reality environment is displayed in the far field area, the user interacts with the virtual object using light projection interaction.
[0005] Optionally, the user interacts with the virtual object in the near-field area using touch interaction by touching the virtual object with the user's hand in the artificial reality environment, and the user interacts with the virtual object in the far-field area using light projection interaction using light extending from the user's body in the artificial reality environment.
[0006] Optionally, the engage gesture comprises a pinch gesture aimed at the virtual object.
[0007] Optionally, the far-field area includes: a) a radial distance greater than 1 meter from the user; b) a radial distance from the user that meets the convergence criteria of the artificial reality system to achieve an artificial reality environment; or c) any combination of the above.
[0008] Optionally, when the virtual object is converted from the near-field area to the far-field area, the virtual object is fixed to a position at the far-field area intersecting with a ray extending from the user's body.
[0009] Optionally, when the virtual object is transformed from the near field area to the far field area, the size of the virtual object is enlarged.
[0010] Optionally, the near-field region comprises a three-dimensional freeform space configured such that user interactions with the virtual object position the virtual object at different locations within the three-dimensional freeform space.
[0011] Optionally, the trigger criteria includes the threshold radial distance, and the threshold radial distance includes an estimate of when the user's arm is fully extended.
[0012] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores instructions, which when executed by a computing system, cause the computing system to trigger an object conversion between a near-field area and a far-field area in an artificial reality environment, the process comprising: displaying a virtual object including at least a two-dimensional surface to a user in the artificial reality environment, wherein the virtual object is displayed in a near-field area, and the near-field area is located within an arm's length from the user; detecting a user's engagement gesture, which engages the virtual object in the near-field area until the user releases the engagement; after engaging the virtual object, detecting a hand movement of the user that meets a trigger criterion, wherein the trigger criterion comprises: a) a threshold radial distance from the user; b) a speed threshold; c) a speed change threshold; or d) any combination of the above; and based on the detected hand movement that meets the trigger criterion, converting the virtual object from the near-field area to the far-field area, wherein the virtual object is displayed in the far-field area, and the far-field area is located at a distance greater than an arm's length from the user.
[0013] Optionally, when the virtual object in the artificial reality environment is displayed in the near field area, the user interacts with the virtual object using touch interaction, and when the virtual object in the artificial reality environment is displayed in the far field area, the user interacts with the virtual object using light projection interaction.
[0014] Optionally, the user interacts with the virtual object in a near-field area using touch interaction by touching the virtual object with the user's hand in the artificial reality environment, and the user interacts with the virtual object in a far-field area using light projection interaction using light extending from the user's body in the artificial reality environment.
[0015] Optionally, the engage gesture comprises a pinch gesture aimed at the virtual object.
[0016] Optionally, the far field area includes: a) a radial distance greater than 1 meter from the user; b) a radial distance from the user that meets the convergence criteria of the artificial reality system to achieve an artificial reality environment; or c) any combination of the above.
[0017] Optionally, when display of the virtual object is switched from the near field area to the far field area, the virtual object is fixed to a position at the far field area intersecting with a ray extending from the user's body.
[0018] Optionally, when the virtual object is transformed from the near field area to the far field area, the size of the virtual object is enlarged.
[0019] Optionally, the near-field region comprises a three-dimensional freeform space configured such that user interactions with the virtual object position the virtual object at different locations within the three-dimensional freeform space.
[0020] Optionally, the trigger criteria comprises a threshold radial distance, and the threshold radial distance comprises an estimate of the user's arm when fully extended.
[0021] According to another aspect of the present invention, a computing system for triggering object conversion between a near field area and a far field area in an artificial reality environment is provided, the computing system comprising: one or more processors; and one or more memories storing instructions, which, when executed by the one or more processors, cause the computing system to execute a process, the process comprising: displaying a virtual object including at least a two-dimensional surface to a user in the artificial reality environment, wherein the virtual object is displayed in a near field area, and the near field area is located within an arm's length from the user; detecting a user's engagement gesture, which engages the virtual object in the near field area until the user releases the engagement; after engaging the virtual object, detecting a hand movement of the user that meets a trigger criterion, wherein the trigger criterion is met when the hand movement is in the following items: a) extending beyond a threshold radial distance from the user; b) meeting or exceeding a speed threshold; c) meeting or exceeding a speed change threshold; or d) any combination of the above; and based on the detected hand movement that meets the trigger criterion, converting the virtual object from the near field area to the far field area, wherein the virtual object is displayed in the far field area, and the far field area is located at a distance greater than an arm's length from the user.
[0022] Optionally, when the virtual object in the artificial reality environment is displayed in the near field area, the user interacts with the virtual object using touch interaction, and when the virtual object in the artificial reality environment is displayed in the far field area, the user interacts with the virtual object using light projection interaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a block diagram showing an overview of a number of devices on which some embodiments of the present technology may operate.
[0024] Figure 2A is a line diagram illustrating a virtual reality headset that may be used in some embodiments of the present technology.
[0025] Figure 2B is a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.
[0026] Figure 2C is a line diagram illustrating a plurality of controllers that, in some implementations, a user may hold with one or both hands to interact with an artificial reality environment.
[0027] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.
[0028] Figure 4 is a block diagram illustrating various components that, in some implementations, may be used in a system employing the disclosed techniques.
[0029] Figure 5A , Figure 5B and Figure 5C is a schematic diagram showing near-field object display and far-field object display in an artificial reality environment.
[0030] Fig. 6A and Figure 6B is a schematic diagram showing field transitions for object display in an artificial reality environment.
[0031] Figure 7 is a flow chart illustrating a process for triggering a field transition for an object in an artificial reality environment in some embodiments of the present technology.
[0032] Figure 8 is a flow chart illustrating a process for performing field transformations of an object in an artificial reality environment in some embodiments of the present technology.
[0033] The technology described herein may be better understood by referring to the following detailed description in conjunction with the drawings, in which like reference numbers refer to identical or functionally similar elements. DETAILED DESCRIPTION
[0034] Various aspects of the present disclosure relate to triggering object conversion between near-field areas and far-field areas in an artificial reality environment. Implementations of the artificial reality system may project a user into an artificial reality environment including a virtual object. For example, a virtual object may be a two-dimensional panel, a three-dimensional volume, or any other suitable structure. Within the artificial reality environment, the user may interact with the virtual object using different interaction mechanisms. In some embodiments, the artificial reality environment may include areas related to the user. The near-field area may be within the user's arm's length, while the far-field area may be greater than the user's arm's length. Objects may be displayed in these areas so that the user may interact with objects in the artificial reality environment in different modes (e.g., using light for the far field and direct touch in the near field).
[0035] Implementations of the field manager may manage the display of objects in near-field and far-field regions. For example, the field manager may cause a displayed object to transition from a near-field region to a far-field region (or cause a transition from a far-field region to a near-field region). In the near-field region, a user may interact with a displayed object using touch interaction (e.g., with a user's hand). For example, a user may use a join gesture (e.g., a pinch gesture) to join a displayed object, move the display position of the object in the near-field region, and otherwise interact with the object by touching an element of the object. In the far-field region, a user may interact with a displayed object using light interaction. For example, a light may extend from the user's body (e.g., from the user's hand or wrist) to a target object in the far-field region. A user may also interact with an object displayed in the far field using a join gesture, however, some interactions may pose challenges.
[0036] For example, light-based interactions with objects displayed in the far field may pose challenges for fine-grained selection. If an object displays text elements to the user (e.g., a news object), the user may find it difficult to use light interactions to select news articles, scroll through articles, and navigate different areas of the news interface displayed at the object, etc. In addition, due to the distance between the far field area and the user, the text displayed by the object may be difficult to read. Alternatively, the field manager can detect user movements that trigger the transition of the object from the far field area to the near field area. The benefits of displaying objects in the near field area can alleviate these challenges, such as a smaller distance between the object and the user and the availability of touch interactions with the object, which can achieve improved accuracy of fine-grained selection.
[0037] Embodiments of the field manager compare sensed user movement to one or more trigger criteria that trigger a transition of an object between field regions. For example, detection of an engagement gesture (e.g., a pinch gesture) can initiate engagement with a displayed object (either in a far field region or a near field region). When engaging an object, the user movement can move the position of the object within its current field region. When engaging an object, the field manager can compare the sensed user movement to a trigger criteria, such as a distance threshold (e.g., radial distance from the user), a speed threshold (e.g., total speed and / or speed change), any combination of these, or any other suitable trigger criteria. When the sensed user movement meets the trigger criteria, the field manager can trigger a transition between field regions (e.g., a transition from a near field region to a far field region or a transition from a far field region to a near field region).
[0038] In some embodiments, when an object is joined, a user movement that does not meet the trigger criteria can move the object within its current field area, and a user movement that meets the trigger criteria can cause a conversion between the field areas. For example, the near field area can be a free-form three-dimensional space, and the user can join the object in the near field and move with the object anywhere in the free-form three-dimensional space (by means of user movement that does not meet the trigger criteria). In order to convert the object to the far field area, for example, to eliminate the clutter of the near field area, the user can perform a movement that meets the trigger criteria when joining the object (for example, a movement that exceeds a speed threshold and / or exceeds a radial distance threshold can trigger a conversion). The field manager can sense user movement and cause the object to be converted to the far field area. In this example, the user can use various user movements (for example, any movement that does not meet the trigger criteria) to manipulate the object in the near field area, thereby allowing the user to manage the near field area with a high degree of customization. In other words, because the field area conversion is limited to the user movement that meets the trigger criteria, the user can freely perform other user movements to design and / or organize the display of objects in a given field area.
[0039] In some embodiments, the radial distance between the far field area and the user is greater than 1 meter (e.g., 1.3 meters), the far field area meets the convergence criteria of the artificial reality system that implements the artificial reality environment, the far field area meets both of the above, or the far field area can have any other suitable distance from the user. When an object is displayed in the far field area, the user can engage the object and move the object, but the movement of the object may be limited. For example, an object can be freely moved along a flat or curved two-dimensional surface defined around the user in an artificial reality environment (e.g., the radial distance from the object to the user can be fixed, or can have a limited range of movement, such as 0.1 meters). In some embodiments, an object displayed in the far field area can be fixed at a predetermined radial distance of the far field area.
[0040] When an object is converted, for example, from a near field area to a far field area or from a far field area to a near field area, the field manager can dynamically change the size of the object, dynamically position the object in the new area, and maintain the display at the object. For example, the object may include at least a two-dimensional surface (e.g., a panel), and when the object is converted from a near field area to a far field area, the object may be enlarged to a larger size. In some embodiments, the object may be maintained relative to the user's field of view. For example, at a distance closer to the user (e.g., a near field area), the object may occupy a first portion of the user's field of view. When the object is converted to a distance farther from the user (e.g., a far field area), in order to occupy the first portion of the user's field of view, the field manager may enlarge the object to a larger size. On the other hand, when the object is converted from a far field area to a near field area, the object may be reduced to a smaller size. In this example, when the object is converted to a distance closer to the user (e.g., a near field area), in order to occupy the same portion of the user's field of view, the field manager may reduce the object to a smaller size.
[0041] The field manager may also dynamically position the object, for example, as the object transitions from a near field region to a far field region. For example, a ray projected from the user's body may intersect the far field region at an intersection point. The field manager may determine a display position of the engaged object based on the intersection point where the ray projected from the user's body meets the far field region (e.g., when the field manager detects that the tracked movement meets a trigger criterion). The field manager may fix the object at the determined transitioned position.
[0042] In some embodiments, the field manager can also dynamically position objects when the objects transition from the far field area to the near field area. For example, a ray of light projected from the user's body that is aimed at an object in the far field area can intersect the near field area. In some embodiments, the intersection can be represented by a line passing through the three-dimensional space that defines the near field area. The field manager can determine the display position of the engaged object based on the intersection line of the ray of light projected from the user's body passing through the near field area (e.g., at a point on the intersection line). For example, the point on the intersection line can be a midpoint, a point on a line at a predetermined radial distance from the user, or any other suitable point.
[0043] Embodiments of the field manager may also maintain the display at the object during the transition. For example, the transition may include moving the object from a pre-transition position to a post-transition position, and in some examples, the transition may include dynamic resizing of the object. During the execution of this transition, the content at the object may continue. For example, during the transition, the field manager may continue to play a video at the object. In some embodiments, the video may be dynamically resized based on the dynamic resizing of the entire object. In another example, during the transition, the field manager may continue to display images and / or text, and may also dynamically resize these elements based on the dynamic resizing of the entire object.
[0044] Embodiments of the disclosed technology may include an artificial reality system, or be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and artificial reality or XR may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with collected content (e.g., real-world photos). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality, and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave-like" environment or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
[0045] As used herein, "virtual reality" or "VR" refers to an immersive experience in which a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to systems in which a user views images of the real world after they have passed through a computing system. For example, a tablet with a camera on the back can capture images of the real world, which can then be displayed on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images as they pass through the system, such as by adding virtual objects. "Mixed reality" or "MR" refers to systems in which the light entering the user's eyes is generated in part by a computing system and in part from light reflected from objects in the real world. For example, an MR head-mounted viewer can be formed as a pair of glasses with a transparent display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, thereby allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of the following: VR, AR, MR, or any combination or hybrid thereof.
[0046] Traditional virtual object interactions cannot provide an experience that meets user expectations, may lead to crowded spaces, and cannot support efficient interaction mechanisms for users. For example, traditional XR environments can display virtual objects near users, but this may create crowded spaces around users. As another example, traditional XR environments can allow users to position virtual objects around the user in a free-form three-dimensional space. However, these XR environments may also become crowded, and these systems cannot effectively utilize spaces in the XR environment that are farther than the user's arm's length.
[0047] Embodiments of the disclosed technology are expected to overcome these defects in conventional systems by providing near-field regions, far-field regions, and efficient conversion mechanisms for converting virtual objects between these field regions. For example, a user can use touch-based interactions to interact with virtual objects in the near-field region to obtain fine control and interaction capabilities. Virtual objects can be converted to the far-field region to eliminate the clutter of the near-field region. Light-based interactions allow users to interact with virtual objects in the far-field region, and users can effectively convert virtual objects from the far field to the near field to obtain fine control and interaction. Each embodiment further improves the conventional system by comparing user movements with trigger criteria that trigger the conversion of virtual objects between the near-field region and the far-field region. For example, a user can perform a joining gesture to interact with a virtual object. Once a virtual object is joined, it can move within its current field region according to user movements that do not meet the trigger criteria. When a user movement that meets the trigger criteria is detected, the joined virtual object can be converted to a new field region. Therefore, the user is able to manipulate virtual objects in the near-field region using a variety of user movements (e.g., any movement that does not meet the trigger criteria), thereby allowing the user to manage the near-field region with a high degree of customization. In other words, because the field area transitions are limited to user movements that meet the triggering criteria, the user is free to perform other user movements to design and / or organize the display of objects within a given field area.
[0048] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1 is a block diagram showing an overview of multiple devices on which some embodiments of the disclosed technology may run. These devices may include hardware components of a computing system 100 that can trigger object conversions between near-field areas and far-field areas in an artificial reality environment. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted viewer that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted viewer and a core processing component (e.g., a console, a mobile device, or a server system), where some processing operations are performed on the head-mounted viewer and other processing operations are transferred to the core processing component. The following is about Figure 2A and Figure 2BAn example head mounted view device is described. In some implementations, the location data and environmental data may be collected solely by sensors included in the head mounted view device, while in other implementations, one or more of the plurality of non-head mounted view computing devices may include sensor components that may track environmental data or location data.
[0049] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among the computing devices 101 to 103).
[0050] The computing system 100 may include one or more input devices 120 that provide input to the processor 110, thereby informing the processor of actions. These actions may be communicated by a hardware controller that interprets the signals received from the input devices and transmits information to the processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.
[0051] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus (e.g., a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus) or a wireless connection. The processor 110 can communicate with a hardware controller of a device (e.g., a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separate from the input device. Examples of display devices are: a liquid crystal display (LCD) display screen; a light emitting diode (LED) display screen; a projection, holographic or augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or card, a video chip or card, an audio chip or card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.
[0052] In some embodiments, the computing system 100 can use input from the I / O device 140 (e.g., a camera, a depth sensor, an inertial motion unit (IMU) sensor, a global positioning system (GPS) unit, a laser radar (LiDAR) or other time-of-flight sensor, etc.) to identify and map the user's physical environment while tracking the user's position in the environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., a topology, a grid, etc.) of an area (which can be a room, a building, an outdoor space, etc.) and / or obtain a map previously generated by the computing system 100 or another computing system that has mapped the area. The SLAM system can track users in the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other position changes, etc.
[0053] The computing system 100 may include a communication device that can communicate wirelessly or based on a wire with other local computing devices or network nodes. The communication device can communicate with another device or server over a network, for example, using a transmission control protocol / internet protocol (TCP / IP). The computing system 100 can use the communication device to distribute operations on multiple network devices.
[0054] Processor 110 may access memory 150, which may be included on one of the computing devices of computing system 100, or may be distributed across the computing devices of computing system 100, or other external devices. Memory includes one or more hardware devices for volatile storage or non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include one or more of the following: random access memory (RAM), various cache memories, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, tape drive, etc. Memory is not a propagating signal that is separated from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160, which stores programs and software, such as operating system 162, farm manager 164, and other applications 166. Memory 150 may also include data storage 170 , which may include information such as object data, standard data, regional configuration data, social graph data, configuration data, settings, user options or preferences, etc. that may be provided to program storage 160 or any element of computing system 100 .
[0055] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with the present technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices, etc.
[0056] Figure 2A2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a strap 210. Front rigid body 205 includes: one or more electronic display elements of electronic display 245; an inertial motion unit (IMU) 215; one or more position sensors 220; a locator 225; and one or more computing units 230. Position sensors 220, IMU 215, and computing units 230 may be located inside HMD 200 and may not be visible to the user. In various embodiments, IMU 215, position sensors 220, and locators 225 may track the movement and position of HMD 200 in the real world and in an artificial reality environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locator 225 may emit infrared beams that produce light spots on real objects around HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; other non-camera-based position, force, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect light points. The computing unit 230 in the HMD 200 may use the detected light points to infer the position and movement of the HMD 200 and to recognize the shapes and positions of real objects around the HMD 200.
[0057] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user according to the instructions of the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (for example, one display for each user's eye). Examples of the electronic display 245 include: a liquid crystal display (LCD); an organic light-emitting diode (OLED) display; an active-matrix organic light-emitting diode (AMOLED) display; a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels; a projector unit (for example, micro-LED, laser (LASER)), etc.); some other display; or some combination thereof.
[0058] In some embodiments, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., via light emitted from the HMD 200), which the PC can use in conjunction with outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.
[0059] Figure 2B 2 is a line diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 GHz link) as indicated by link 256. In other embodiments, the mixed reality system 250 includes only a head-mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a transparent display 258 and a frame 260. The frame 260 can accommodate various electronic components (not shown), such as a light projector (e.g., LASER, LED, etc.), a camera, an eye tracking sensor, a micro-electromechanical system (MEMS) component, a network component, etc.
[0060] The projector can be coupled to the transparent display 258, for example, via various optical elements, to display media to the user. These optical elements can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc., for guiding the light from the projector to the user's eyes. Image data can be transmitted to the HMD 252 via the link 256 from the core processing component 254. The controller in the HMD 252 can convert the image data into multiple light pulses from the projector, which can be transmitted to the user's eyes as output light via the optical element. The output light can be mixed with the light passing through the display 258, so that the output light can present the following virtual objects: these virtual objects look like they exist in the real world.
[0061] Similar to the HMD 200, the HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow the HMD system 250 to track itself, for example, in 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects to appear stationary when the HMD 252 moves, and make virtual objects react to gestures and other real-world objects.
[0062] Figure 2C A plurality of controllers 270 (including controllers 276A and 276B) are shown, which in some embodiments can be held by a user with one or both hands to interact with an artificial reality environment presented by the HMD 200 and / or the HMD system 250. The controllers 270 can communicate directly with the HMD or via an external device (e.g., the core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit more light points. The HMD 200 or HMD 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., tracking the controllers with 3DoF or 6DoF). The computing unit 230 or the core processing component 254 in the HMD 200 can combine the IMU output and the position output and use this tracking to monitor the user's hand position and movement. The controllers may also include various buttons (eg, buttons 272A-272F) and / or levers (eg, levers 274A and 274B) that a user may actuate to provide input and interact with objects.
[0063] In various embodiments, HMD 200 or HMD 250 may also include a number of additional subsystems for monitoring user interactions and indications of intent, such as an eye tracking unit, an audio system, various network components, and the like. For example, in some embodiments, one or more cameras included in HMD 200 or HMD 250 instead of or in addition to a controller, or one or more cameras from a plurality of external cameras, may monitor the position and posture of a user's hands to determine posture and other hand and body movements. As another example, one or more light sources may illuminate one or both eyes of a user, and HMD 200 or HMD 250 may use an eye-facing camera to capture reflections of such light to determine eye position (e.g., based on a set of reflections around the user's cornea), model the user's eyes, and determine gaze direction.
[0064] Figure 3305B may be a HMD 200 or a HMD system 250. The client computing device 305 may be a network environment, using a logical connection to one or more remote computers (e.g., a server computing device) through a network 330.
[0065] In some implementations, server 310 may be an edge server that receives multiple client requests and coordinates the fulfillment of these requests through other servers (e.g., servers 320A to 320C). Server computing devices 310 and 320 may include a computing system, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, multiple server computing devices may each be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.
[0066] Client computing device 305 and server computing devices 310 and 320 can all act as servers or clients of one or more other servers / client devices. Server 310 can be connected to database 315. Servers 320A to 320C can each be connected to corresponding databases 325A to 325C. As discussed above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share a database, or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can all be distributed computing environments that include multiple computing devices, which can be located in their corresponding servers, or can be located at the same physical location or at geographically different physical locations.
[0067] The network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. The network 330 may be the Internet or some other public or private network. The client computing device 305 may be connected to the network 330 via a network interface, for example, by wired or wireless communication. Although the connections between the server 310 and the plurality of servers 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including the network 330 or a separate public or private network.
[0068] Figure 4 4 is a block diagram showing a plurality of components 400, which in some embodiments may be used in a system employing the disclosed technology. The plurality of components 400 may be included in one device of the computing system 100, or may be distributed across multiple devices of the computing system 100. The plurality of components 400 include hardware 410, middleware 420, and dedicated components 430. As discussed above, a system implementing the disclosed technology may use various hardware including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.), and a storage memory 418. In various embodiments, the storage memory 418 may be one or more of the following: a local device; an interface to a remote storage device; or a combination thereof. For example, the storage memory 418 may be one or more hard drives or flash drives accessible via a system bus, or may be a cloud storage provider (e.g., in a storage device 315 or 325) or other network storage device accessible via one or more communication networks. In various implementations, the plurality of components 400 may be implemented in a client computing device (eg, client computing device 305 ) or on a server computing device (eg, server computing devices 310 or 320 ).
[0069] Middleware 420 may include components that transfer resources between hardware 410 and dedicated components 430. For example, middleware 420 may include an operating system, services, drivers, a basic input output system (BIOS), a controller circuit, or other hardware or software systems.
[0070] The dedicated components 430 may include software or hardware configured to perform operations for triggering the conversion of objects between near-field areas and far-field areas in an artificial reality environment. Each dedicated component 430 may include: a conversion manager 434; a near-field manager 436; a far-field manager 438; a motion tracker 440; and components and APIs that can be used to provide a user interface, transmit data, and control the dedicated components, such as interface 432. In some embodiments, multiple components 400 may be located in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more of the dedicated components 430. Although each dedicated component 430 is depicted as a separate component, these dedicated components may be logical or other non-physical distinctions of functionality, and / or may be sub-modules or code blocks of one or more applications.
[0071] The near field region can be a three-dimensional space near a user in the XR environment (e.g., within a threshold radial distance of the user), such as within arm's length of the user. The near field manager 436 can manage objects (e.g., two-dimensional panels, three-dimensional volumes, etc.) and user interactions in the near field region. For example, a user (e.g., a user projected in the XR environment) can engage objects in the near field region, such as by aiming at the object with hand movements (e.g., touch interactions) and performing an engagement gesture (e.g., a pinch gesture). The near field manager 436 can initiate engagement between the user and the engagement object and maintain engagement between the user and the engagement object until the user releases the engagement gesture.
[0072] When the user engages with the object, the near field manager 436 can dynamically move the display of the object according to the user's movement. The near field area can be a three-dimensional free-form space, and the user can dynamically move the engaged object anywhere in the three-dimensional free-form space according to the user's hand movement. When the user releases the engagement gesture, the near field manager 436 can position the object (e.g., lock or fix the object) at the position in the three-dimensional free-form space. The user can use touch interaction to interact with the object in the near field area, such as interaction of scrolling the content displayed by the object, interaction of starting a button that performs a function (e.g., playing a video, causing an application-specific function), interaction of selecting a display element at the object, and other suitable interactions.
[0073] In some embodiments, the far field region can be located at a distance greater than arm's length from the user in the XR environment, such as greater than 1 meter, 1.3 meters, the convergence distance of the XR system to implement the XR environment, any combination thereof, or any other suitable distance. The far field manager 438 can manage objects and user interactions in the far field region. For example, a user can engage an object in the far field region by, for example, aiming at the object with hand movement (e.g., light interaction) and performing an engagement gesture (e.g., a pinch gesture). The far field manager 438 can initiate engagement between a user and an engaged object, and maintain engagement between the user and the engaged object until the user releases the engagement gesture.
[0074] When the user engages with the object, the far field manager 438 can dynamically move the display of the object according to the user's movement. The far field area can be a two-dimensional space at a fixed radial distance from the user or a two-dimensional space with a limited third dimension (e.g., 0.1 meters, etc.). The user can dynamically move the engagement object in the two-dimensional space at a fixed radial distance, or move the engagement object in the two-dimensional space and the limited third dimension. When the user releases the engagement gesture, the far field manager 438 can position the object (e.g., lock or fix the object) at a position in the two-dimensional space (or limited three-dimensional space). The user can use light interaction to interact with objects in the far field area, such as interaction of aiming at an object, interaction of display elements at the selection object, and other suitable interactions. In some embodiments, the light-based interaction that can be used for objects in the far field area can be similar to the touch-based interaction that can be used for objects in the near field area, but the light-based interaction may not be as accurate as the touch-based interaction.
[0075] Ray-based interactions may include projection of rays (i.e., straight lines) from a control point along a projection direction. For example, a control point may be a palm, fingertip, fist, wrist, etc., and the projection direction may be along a line passing through a control point and an origin (e.g., shoulder, eye, or hip). In other embodiments, the control point may be based on other tracked body parts, such as the user's eyes, head, or chest. For example, a control point may be an estimated location of the center of a user's pupil, and an origin may be an estimated location of the center of a user's retina. In some cases, a graphical representation of the ray projection (the entire line or only the point where the ray hits an object) may be displayed in an artificial reality environment, while in other cases, the ray projection is tracked by the XR system without displaying the ray projection. In various embodiments, the ray projection may extend from a control point until it intersects a first object, or the ray projection may extend through multiple objects. In some embodiments, the direction of the ray projection may be adjusted to "catch" an object that it is close to intersecting, or the ray projection may be bent by a threshold amount to maintain intersection with such an object.
[0076] The transition manager 434 can manage transitions between near-field regions and far-field regions within the XR environment. In some embodiments, when a user engages with an object (e.g., after the user has aimed at the object, performed an engagement gesture, and held the engagement gesture), the transition manager 434 can compare the tracked user's movements (e.g., hand movements tracked by the movement tracker 440) to trigger criteria. When the user's movements meet the trigger criteria, the engaged object can transition from its current field region to a new field region (e.g., from a near-field region to a far-field region or from a far-field region to a near-field region).
[0077] The trigger criteria may include one or more of the following: a) a threshold radial distance from the user; b) a speed threshold; c) a speed change threshold; or d) any combination thereof. For example, when the tracked user movement meets the speed threshold, the transition manager 434 may perform a field region transition for the joined object. In another example, when the tracked user movement exceeds a radial distance threshold (or falls below a radial distance threshold), the transition manager 434 may perform a field region transition for the joined object.
[0078] In one example, the conversion manager 434 can convert the joined object in the near field area to the far field area. In this example, the object management for the joined object can be transferred from the near field manager 436 to the far field manager 438. The conversion manager 434 can dynamically position the joined object in the far field area. For example, the light projected from the user's body can intersect with the far field area at an intersection. The displayed position of the joined object after conversion can be determined as the intersection where the light projected from the user's body meets the far field area (for example, when the conversion manager 434 detects that the tracked movement meets the trigger criteria). When the user releases the gesture for the joined object, the conversion manager 434 can fix the object at the determined converted position in the far field area.
[0079] When the joined object is transformed from the near field area to the far field area, the transformation manager 434 can dynamically enlarge the joined object to a larger size. In some embodiments, the transformation manager 434 can maintain the field of view of the joined object during the transformation. For example, at a distance closer to the user (e.g., the near field area), the joined object can occupy a first portion of the user's field of view. When the joined object is transformed to a distance farther away from the user (e.g., the far field area), in order to occupy the first portion of the user's field of view, the transformation manager 434 can enlarge the joined object display to a larger size.
[0080] In another example, the conversion manager 434 can convert the joint object in the far field area to the near field area. In this example, the object management for the joint object can be converted from the far field manager 438 to the near field manager 436. The conversion manager 434 can dynamically position the joint object in the near field area. For example, a light ray projected from the user's body aimed at the joint object in the far field can intersect with the near field area. In some embodiments, the intersection can be represented by a line passing through the three-dimensional space defining the near field area. The display position of the joint object after conversion can be determined as a point on the intersection line where the light ray projected from the user's body passes through the near field area. For example, the point on the intersection line can be a midpoint, a point on a line at a predetermined radial distance from the user, or any other suitable point.
[0081] When the joined object is transformed from the far field area to the near field area, the transformation manager 434 can dynamically reduce the joined object to a smaller size. In some embodiments, the transformation manager 434 can maintain the field of view of the joined object during the transformation. For example, at a distance farther from the user (e.g., the far field area), the joined object can occupy a first portion of the user's field of view. When the joined object is transformed to a distance closer to the user (e.g., the near field area), in order to occupy the first portion of the user's field of view, the transformation manager 434 can reduce the joined object display to a smaller size.
[0082] Embodiments of the conversion manager 434 may also maintain display at a joined object during a conversion. For example, a conversion may include movement of a joined object from a pre-conversion position to a post-conversion position. Content displayed by the joined object may continue to be displayed while the conversion manager 434 converts an object. For example, a video displayed by the joined object may continue to be played, display of images and / or text may be maintained, and other suitable content may continue to be displayed. In some embodiments, the size of the video, image, and / or text, as well as other suitable content, may be dynamically adjusted during a conversion, for example, based on dynamic resizing of the joined object.
[0083] The motion tracker 440 can track real-world user movements sensed by the XR system. For example, the XR system may include one or more cameras, one or more handheld controllers, one or more wearable devices, or one or more any other suitable sensors. These sensors can sense the movement of the user in the real world. The motion tracker 440 can track the sensed real-world movement so that the user movement can be projected in the XR environment. For example, a user projected in the XR environment can interact with objects in the XR environment based on the sensed and tracked movement. The tracked user hand movements may include the distance traveled, the speed (e.g., the rate and direction), and any other suitable tracking. In some embodiments, the transformation manager 434 may compare the tracked movement (e.g., the tracked user hand movement) with one or more trigger criteria that trigger a field transition of an engaged object in the XR environment.
[0084] Embodiments may use any suitable technique to estimate the maximum hand / arm reach of a single user. For example, the XR system may track the user's movements over time. The maximum reach (e.g., radial distance from the user) of the user's hand / arm across multiple XR environment interactions may be tracked and updated. Based on these tracked values, a maximum distance of arm length may be estimated for the user.
[0085] Figure 5A , Figure 5B and Figure 5C Schematic diagrams showing near-field object display and far-field object display in an artificial reality environment. Schematic diagram 500A includes object 502, user 504, and user hand 506. Schematic diagrams 500B and 500C include object 502, user 504, user hand 506, and thresholds 508 and 510. Fig. 6A and Figure 6B Schematic diagrams 600A and 600B include an object 502 , a user 504 , a user hand 506 , and thresholds 508 and 510 .
[0086] Schematic diagrams 500A, 500B, 500C, 600A, and 600B illustrate an XR environment implemented by an XR system. The XR environment may include a near field area and a far field area. User 504 may be a projection of a user in the XR environment. User hand 506 in the XR environment may simulate user hand movements sensed in the real world. Object 502 may be a virtual object, such as a two-dimensional object (e.g., a panel), a three-dimensional volume, or any other suitable virtual object.
[0087] In schematic 500A, object 502 is located in the far field region of the illustrated XR environment. For example, user 504 can interact with object 504 at the far field region based on a light-based interaction. In some embodiments, light projected from user hand 506 can be aimed at object 502. In schematic 500B, user hand 506 has performed a join gesture (e.g., a pinch gesture) to join object 502 in the far field region. For example, light projected from user hand 506 has been aimed at object 502 and the join gesture has joined object 502. In schematic 500C, object 502 is located in the near field region of the illustrated XR environment, and user hand 506 has performed a join gesture to join object 502 in the near field region. In this example, user 504 can interact with object 502 at the near field region based on a touch-based interaction.
[0088] Schematic diagrams 500B and 500C also show thresholds 508 and 510. Thresholds 508 and 510 can be example triggering criteria, such as radial distance thresholds from user 504. In some embodiments, the radial distance can be measured from a predetermined position on user 504 (e.g., a point on the shoulder, chest, or other suitable point). Threshold 508 can correspond to a radial distance for triggering a conversion from a near-field region to a far-field region, and threshold 510 can correspond to a radial distance for triggering a conversion from a far-field region to a near-field region. For example, threshold 508 can be set to a radial distance at or near the maximum extension of the user's arm length, and threshold 510 can be set to a radial distance that deviates from the maximum retraction. Thresholds 508 and 510 can be set to any other suitable distances, and can represent radial distances or any other suitable distance types.
[0089] Schematic diagrams 500B and 600A show an example conversion of object 502 from a far field region to a near field region based on user hand 506 satisfying threshold 510. For example, schematic diagram 500B shows that object 502 is located at a pre-conversion position, and schematic diagram 600A shows that object 502 is located at a post-conversion position. Schematic diagram 500B shows that object 502 is engaged. When the tracked user hand movement (e.g., represented by the movement of user hand 506) meets threshold 510, the conversion can be triggered. Schematic diagram 600A shows that user hand 506 has moved beyond threshold 510. For example, threshold 510 can be a minimum radial distance threshold, and user hand 506 can move to a radial distance less than or equal to threshold 510 from user 504. Based on the tracked user movement having met threshold 510 (e.g., triggering criteria), object 502 can be converted from a far field region to a near field region. Accordingly, schematic diagram 600A shows that object 502 is located at a post-conversion position in a near field region.
[0090] Schematic diagrams 600A and 600B illustrate an example transition of an object 502 from a near-field region to a far-field region based on a user's hand 506 satisfying a threshold 508. For example, schematic diagram 600A illustrates that the object 502 is in a pre-transition position, and schematic diagram 600B illustrates that the object 502 is in a post-transition position. In some implementations, the user may first transition the object 502 from the far-field region to the near-field region, as illustrated in the sequence of schematic diagrams 500B and 600A, and may then transition the object 502 from the near-field region to the far-field region, as illustrated in the sequence of schematic diagrams 600A and 600B.
[0091] Schematic diagram 600A shows that object 502 is engaged. When the tracked user hand movement (e.g., represented by movement from user hand 506) satisfies threshold 508, a transition can be triggered. Schematic diagram 600B shows that user hand 506 has moved beyond threshold 508. For example, threshold 508 can be a maximum radial distance threshold, and user hand 506 can move to a radial distance from user 504 that is greater than or equal to threshold 508. Based on the tracked user movement having satisfied threshold 508 (e.g., a triggering criterion), object 502 can be transitioned from the near-field region to the far-field region. Accordingly, schematic diagram 600B shows that object 502 is in a transitioned position in the far-field region.
[0092] In some embodiments, thresholds 508 and 510 can be dynamic based on the position of the user's hand 506 performing the engagement gesture and establishing engagement with the object 502. For example, if the user's hand 506 initially engages the object 502 near the threshold 508 (e.g., corresponding to a radial distance at or near the user's maximum extension), a slight movement of the user's hand 506 may inadvertently trigger a transition. In some embodiments, when the user's hand 506 engages the object 502 within a proximity threshold of the threshold 508, an incremental distance (e.g., a predetermined radial distance) can be added to the threshold 508. In this example, the threshold 508 can be extended beyond the original threshold to reduce inadvertent triggering of a transition of the object 502. In some cases, when the user's hand passes the threshold distance, hysteresis can be applied to prevent continuous triggering of field transitions when the user's hand approaches the trigger distance. In some embodiments, the incremental distance can be temporarily added to the threshold 508, and the threshold can be restored to the original radial distance threshold after a period of time (e.g., 1 second, 2 seconds, 5 seconds, etc.).
[0093] In another example, if the user's hand 506 initially engages the object 502 near the threshold 510 (e.g., corresponding to a radial distance at or near the user's maximum retraction), a slight movement of the user's hand 506 may inadvertently trigger a transition. In some implementations, when the user's hand 506 engages the object 502 within a proximity threshold of the threshold 510, a delta distance (e.g., a predetermined radial distance) may be subtracted from the threshold 510. In this example, the threshold 510 may be moved back from the original threshold to reduce inadvertently triggering a transition of the object 502. In some implementations, the delta distance may be temporarily subtracted from the threshold 510, and the threshold may be restored to the original radial distance threshold after a period of time.
[0094] It will be understood by those skilled in the art that the above Figures 1 to 4 , Figure 5A , Figure 5B , Figure 5C , Fig. 6A and Figure 6B The components shown in the flowcharts and in each of the flowcharts discussed below may be changed in various ways. For example, the order of the logic may be rearranged, the sub-steps may be performed in parallel, the logic shown may be omitted, other logic may be included, etc. In some embodiments, one or more of the above components may perform one or more of the following processes.
[0095] Figure 7 700 is a flow chart illustrating a process for triggering a field transition of an object in an artificial reality environment in some embodiments of the present technology. In some embodiments, process 700 may be performed by an application or operating system of an XR system. Process 700 may be performed while a user is interacting with an XR environment.
[0096] At block 702, process 700 may display a virtual object to a user in an XR environment. For example, the virtual object may be a virtual object having a two-dimensional surface (e.g., a panel or interface), a three-dimensional volume, or any other suitable virtual object. The user may be an XR representation of the user projected by the XR system into a VR environment, or may be viewing a live version of themselves in an MR environment or an AR environment.
[0097] In some embodiments, the XR environment may include a near field area and a far field area. For example, the near field area may be a three-dimensional free-form space at a distance within arm's length from the user. In another example, the far field area may be a two-dimensional space at a fixed distance from the user, or a two-dimensional space with a third dimension (from the user) having a finite span (0.1 meters, etc.). The far field area may have: a) a radial distance greater than 1 meter from the user; b) a radial distance from the user that meets the convergence criteria of the XR system for implementing the XR environment; or c) any combination thereof. Virtual objects may be located in the near field area or the far field area.
[0098] In some embodiments, when a virtual object in an XR environment is displayed in a near field area, the user interacts with the virtual object using touch interaction, and when a virtual object in an XR environment is displayed in a far field area, the user interacts with the virtual object using ray casting interaction. For example, the user interacts with the virtual object using touch interaction by touching the object with the user's hand in the XR environment. In another example, the user interacts with the virtual object in the XR environment using ray casting interaction, which uses light extending from the user's body.
[0099] At box 704, process 700 can detect an engagement gesture from the user. For example, the XR system can track user movement and detect an engagement gesture from the user (e.g., a pinch gesture, an air click gesture, a grab gesture, etc.). In some embodiments, the engagement gesture can initiate engagement with the virtual object. For example, the user can aim at the virtual object, for example, using touch-based interaction and / or light-based interaction. When aiming at the virtual object, the user can perform an engagement gesture to initiate engagement with the virtual object. In some embodiments, the engagement between the user and the virtual object can be maintained until the user releases the engagement.
[0100] At block 706, process 700 can detect user movement. In some cases (e.g., for some VR embodiments), the XR system can convert real-world user movement (e.g., hand movement) into user movement projected in the XR environment.
[0101] At block 708, process 700 may move the virtual object in accordance with the user movement. For example, the detected user movement may be performed while maintaining engagement with the virtual object, and the virtual object may be moved in accordance with the user movement. For example, the user movement may dynamically move the engaged virtual object in the near field anywhere within the three-dimensional free-form space defined for the near field (e.g., in accordance with the user's hand movement). In another example, the user movement may dynamically move the engaged virtual object in the far field in a two-dimensional space (or a two-dimensional space and a limited third dimension of the far field) at a fixed distance from the far field.
[0102] At block 710, process 700 may determine whether the user movement satisfies the distance criteria. When the virtual object is in the near field region, the user movement may satisfy the distance criteria when the radial distance of the user's hand position (e.g., relative to the user's body in the XR system) meets or exceeds a radial distance threshold corresponding to the user's maximum arm extension (e.g., an estimate of the user's arms when fully extended) when the virtual object is in the far field region, the user movement may satisfy the distance criteria when the radial distance of the user's hand position (e.g., relative to the user's body in the XR system) meets or is below a radial distance threshold corresponding to the user's maximum arm retraction. When the user movement satisfies the distance criteria, process 700 may proceed to block 716. When the user movement does not satisfy the distance criteria, process 700 may proceed to block 712.
[0103] At block 712, process 700 may determine whether the user movement meets the speed criteria. For example, the speed criteria may include a speed threshold, a speed change threshold, or any other suitable speed or rate-based threshold. In some embodiments, the speed criteria may be met when the speed / rate of the determined user movement meets or exceeds one or more speed thresholds. When the user movement meets the speed criteria, process 700 may proceed to block 716. When the user movement does not meet the speed criteria, process 700 may proceed to block 714.
[0104] In some embodiments, blocks 710 and 712 may be combined such that a field transition for engaging virtual objects is triggered by detected user movement when a distance criterion is met and a speed criterion is met. For example, a field region transition may be triggered when the detected user movement meets a distance criterion (e.g., meets or exceeds a first radial distance threshold, meets or falls below a second radial distance threshold, etc.) and meets a speed criterion (e.g., meets or exceeds a speed threshold and / or meets or exceeds a speed change threshold).
[0105] At block 716, process 700 may transform the field region of the joined virtual object. For example, the joined virtual object may transform from a near field region to a far field region or from a far field region to a near field region. Figure 8 Process 800 further describes the execution of a field region conversion.
[0106] At block 714, process 700 can determine whether the user is still maintaining an engagement gesture. For example, an engagement gesture (e.g., a pinch gesture) can be maintained until a disengagement gesture (e.g., releasing the pinch gesture) is detected. Embodiments of the XR system sense real-world user hand movements to detect whether the user performs a disengagement gesture.
[0107] In some embodiments, the engage gesture and / or the disengage gesture can be detected by a model (e.g., a trained machine learning model) configured to detect gestures. For example, the model can output a confidence value (e.g., a value between 1 and 100) that the user has performed a disengage gesture (or that the user has performed or continues to perform an engage gesture). In some embodiments, a disengage gesture can be detected when the confidence value satisfies a threshold confidence value (e.g., 60, 70, or any other suitable threshold).
[0108] In some embodiments, the threshold confidence value can be dynamically increased based on the determined speed / rate of the detected user movement. For example, the higher the value of the determined speed / rate of the detected user movement, the higher the threshold confidence value (e.g., the greater the dynamic increase). The dynamic increase in the threshold confidence value can reduce the risk of accidental release by the user when moving at high speeds. In particular, gesture predictions generated by some models may reduce accuracy at higher rates of user movement. Therefore, some embodiments may increase the threshold confidence value to offset the reduced accuracy. In some embodiments, the amount by which the user gesture changes to release the gesture can also be based on the speed of movement. For example, if the engagement gesture is a pinch, the faster the user's hand moves, the farther the thumb and index finger tips must move before the system can recognize the release of the gesture.
[0109] When the user does not maintain the engage gesture, process 700 may proceed to block 718. When the user maintains the engage gesture, process 700 may loop back to block 706 where additional user movements may be detected.
[0110] At block 718, process 700 may disengage the virtual object. For example, the virtual object may be positioned at a location in the current field area where the virtual object is located. The positioned location may correspond to the location of the virtual object when the disengagement is detected. In some embodiments, when disengagement is performed in a near field area, the virtual object may be positioned (e.g., locked or fixed) at a location in a three-dimensional freeform space. In some embodiments, when disengagement is performed in a far field area, the virtual object may be positioned (e.g., locked or fixed) at a location in a two-dimensional space (e.g., located at a fixed distance from the user).
[0111] Figure 8 800 is a flow chart illustrating a process for performing field transformation of an object in an artificial reality environment in some embodiments of the present technology. In some embodiments, process 800 may be performed by an application or operating system of an XR system. Process 800 may be performed while a user is interacting with the XR environment. In some embodiments, process 800 may reflect a transformation process of a virtual object, such as Figure 7 Frame 714.
[0112] At block 802, process 800 may determine a transformed position for a virtual object. In an example where a virtual object is transitioned from a near field region to a far field region, a ray projected from a user's body may intersect the far field region at an intersection point. Upon detecting that the user's movement satisfies a triggering criterion, the determined display position of the virtual object at the far field region after the transition may be the intersection point where the ray projected from the user's body meets the far field region.
[0113] In another example of engaging a virtual object to transition from a far field region to a near field region, a ray projected from the user's body aimed at a virtual object in the far field region may intersect the near field region. In some embodiments, the intersection may be represented by a line passing through a three-dimensional space defining the near field region. Upon detecting that the user's movement meets a triggering criterion, the determined display position of the virtual object at the near field region after the transition may be a point on the intersection line, such as a midpoint, a point on the line at a predetermined radial distance from the user, or any other suitable point.
[0114] Although any box can be removed or rearranged in various embodiments, box 804 is shown in dotted lines to indicate that there is a specific instance of skipping box 804. At box 804, process 800 can adjust the size of the display of the virtual object according to the trigger conversion. For example, when converting from a near field area to a far field area, the virtual object can be enlarged to a larger size. In some embodiments, the interface display can be kept relative to the user's field of view. For example, at a distance closer to the user (e.g., a near field area), the virtual object can occupy the first part of the user's field of view. When the virtual object is converted to a distance farther away from the user (e.g., a far field area), in order to occupy the first part of the user's field of view, the virtual object can be dynamically enlarged to a larger size. On the other hand, when converting from a far field area to a near field area, the virtual object can be reduced to a smaller size. In this example, when the virtual object is converted to a distance closer to the user (e.g., a near field area), in order to occupy the same part of the user's field of view, the virtual object can be dynamically reduced to a smaller size.
[0115] At block 806, process 800 can position the display of the virtual object at the determined position to trigger the transition. For example, the virtual object can be fixed or locked to the determined post-transition position. Engagement with the virtual object (e.g., continued engagement or engagement via a newly detected engagement gesture) and detected user movement can move the virtual object away from the post-transition position.
[0116] At box 808, process 800 can maintain the dynamic display at the virtual object during the execution of the triggered conversion. For example, the conversion can include the movement of the virtual object from a pre-conversion position to a post-conversion position, and in some examples, includes dynamic resizing of the virtual object. During the execution of the conversion, the content at the virtual object can continue. For example, a video playing at the virtual object can continue to be played during the conversion. In some embodiments, the size of the video can be dynamically adjusted according to the dynamic resizing of the entire virtual object. In another example, images and / or text can continue to be displayed during the conversion, and the size of these elements can also be dynamically adjusted according to the dynamic resizing of the entire virtual object.
[0117] In this specification, references to "embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described with respect to that embodiment is included in at least one embodiment of the present disclosure. The appearances of these phrases in different places in the specification do not necessarily all refer to the same embodiment, nor do they necessarily refer to separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not others.
[0118] As used herein, being higher than a threshold value means that the value of a comparison item is higher than other values specified, the comparison item is located in a certain specified number of items with a maximum value, or the comparison item has a value within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of a comparison item is lower than other values specified, the comparison item is located in a certain specified number of items with a minimum value, or the comparison item has a value within a specified minimum percentage value. As used herein, being within a threshold value means that the value of a comparison item is between two other values specified, the comparison item is located in a middle specified number of items, or the comparison item has a value within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting a fast connection" may be understood to select a connection with a value corresponding to the connection speed assigned thereto that is higher than a threshold value.
[0119] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B, or C" refers to at least one of A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B, and C; or multiples of any item, such as A and A; B, B, and C; A, A, B, C, and C; etc.
[0120] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except as limited by the appended claims.
[0121] Any patent, patent application and other references mentioned above are incorporated herein by reference. If necessary, various aspects may be modified to provide further embodiments using the systems, functions and concepts of the various references mentioned above. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.
Claims
1. A method for triggering object conversion between a near field area and a far field area in an artificial reality environment, the method comprising: displaying a virtual object comprising at least a two-dimensional surface to a user within an artificial reality environment, wherein the virtual object is displayed in a near field area, the near field area being within arm's length of the user; detecting an engagement gesture of the user, the engagement gesture engaging the virtual object in the near field area until the user releases the engagement; After engaging the virtual object, detecting a hand movement of the user that satisfies a triggering criterion, wherein the triggering criterion comprises: a) a threshold radial distance from the user; b) a velocity threshold; c) a velocity change threshold; or d) any combination of the above; and Based on the detected hand movement satisfying the trigger criteria, the virtual object is transitioned from the near-field area to a far-field area, wherein the virtual object is displayed in the far-field area, the far-field area being located at a distance greater than an arm's length from the user.
2. The method according to claim 1, wherein: When the virtual object in the artificial reality environment is displayed in the near-field area, the user interacts with the virtual object using touch interaction, and when the virtual object in the artificial reality environment is displayed in the far-field area, the user interacts with the virtual object using light projection interaction.
3. The method according to claim 2, wherein: The user interacts with the virtual object in the near-field area using touch interaction by touching the virtual object with the user's hand in the artificial reality environment, and the user interacts with the virtual object in the far-field area using light projection interaction using light extending from the user's body in the artificial reality environment.
4. A method according to any preceding claim, wherein: The engage gesture includes a pinch gesture aimed at the virtual object.
5. A method according to any preceding claim, wherein: The far-field area includes: a) a radial distance greater than 1 meter from the user; b) a radial distance from the user that meets the convergence criteria of the artificial reality system to achieve the artificial reality environment; or c) any combination of the above.
6. The method according to claim 5, wherein: When the virtual object is converted from the near-field area to the far-field area, the virtual object is fixed to a position at the far-field area intersecting a ray extending from the body of the user.
7. The method according to claim 5, wherein: When the virtual object is transferred from the near field area to the far field area, the size of the virtual object is enlarged.
8. A method according to any preceding claim, wherein: The near-field region includes a three-dimensional freeform space configured such that user interactions with the virtual object position the virtual object at different locations within the three-dimensional freeform space.
9. A method according to any preceding claim, wherein: The trigger criteria includes the threshold radial distance, and the threshold radial distance includes an estimate of when the user's arm is fully extended.
10. A computer-readable storage medium storing instructions, which, when executed by a computing system, causes the computing system to trigger object conversion between a near-field region and a far-field region in an artificial reality environment, the process comprising: displaying a virtual object comprising at least a two-dimensional surface to a user within an artificial reality environment, wherein the virtual object is displayed in a near field area, the near field area being within arm's length of the user; detecting an engagement gesture of the user, the engagement gesture engaging the virtual object in the near field area until the user releases the engagement; After engaging the virtual object, detecting a hand movement of the user that satisfies a triggering criterion, wherein the triggering criterion comprises: a) a threshold radial distance from the user; b) a velocity threshold; c) a velocity change threshold; or d) any combination of the above; and Based on the detected hand movement satisfying the trigger criteria, the virtual object is transitioned from the near-field area to a far-field area, wherein the virtual object is displayed in the far-field area, the far-field area being located at a distance greater than an arm's length from the user.
11. The computer-readable storage medium of claim 10, wherein: When the virtual object in the artificial reality environment is displayed in the near-field area, the user interacts with the virtual object using touch interaction, and when the virtual object in the artificial reality environment is displayed in the far-field area, the user interacts with the virtual object using light projection interaction. In these cases, optionally, the user interacts with the virtual object in the near-field area using touch interaction by touching the virtual object with the user's hand in the artificial reality environment, and the user interacts with the virtual object in the far-field area using light projection interaction using light extending from the user's body in the artificial reality environment.
12. The computer-readable storage medium of claim 10 or 11, and one or more of the following: a) wherein the engagement gesture comprises a pinch gesture aimed at the virtual object; or b) Among them, The far field area includes: a) a radial distance greater than 1 meter from the user; b) a radial distance from the user that satisfies the convergence criteria of the artificial reality system to implement the artificial reality environment; or c) any combination of the above, in which cases, optionally, one or more of the following: i) wherein, when the display of the virtual object is transitioned from the near field area to the far field area, the virtual object is fixed to a position at the far field area where a light ray extending from the user's body intersects; or ii) wherein, when the virtual object is transitioned from the near field area to the far field area, the size of the virtual object is enlarged; or c) wherein the near-field region comprises a three-dimensional free-form space configured such that user interaction with the virtual object positions the virtual object at different locations within the three-dimensional free-form space; or d) wherein the trigger criteria comprises the threshold radial distance, and the threshold radial distance comprises an estimate of the user's arm when fully extended.
13. A computing system for triggering object conversion between a near field region and a far field region in an artificial reality environment, the computing system comprising: one or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: displaying a virtual object comprising at least a two-dimensional surface to a user within an artificial reality environment, wherein the virtual object is displayed in a near field area, the near field area being within arm's length of the user; detecting an engagement gesture of the user, the engagement gesture engaging the virtual object in the near field area until the user releases the engagement; After engaging the virtual object, detecting a hand movement of the user that satisfies a trigger criterion, wherein the trigger criterion is satisfied when the hand movement: a) extends beyond a threshold radial distance from the user; b) meets or exceeds a velocity threshold; c) meets or exceeds a velocity change threshold; or d) any combination of the above; and Based on the detected hand movement satisfying the trigger criteria, the virtual object is transitioned from the near-field area to a far-field area, wherein the virtual object is displayed in the far-field area, the far-field area being located at a distance greater than an arm's length from the user.
14. The computing system of claim 13, wherein: When the virtual object in the artificial reality environment is displayed in the near-field area, the user interacts with the virtual object using touch interaction, and when the virtual object in the artificial reality environment is displayed in the far-field area, the user interacts with the virtual object using light projection interaction.
Citation Information
Cited By
Display module control method, display module control device and near-eye display device
WO2025103267A1
