Converting interactions on two-dimensional interface into artificial reality experience

Abstract

Many users access an artificial reality (XR) experience through their mobile phones. However, it is difficult to convert the XR experience into a two-dimensional (2D) screen in an intuitive and natural manner. Thus, the present technique can map an interaction plane in a three-dimensional (3D) scene to a 2D screen with as many as possible affordance representations, even if the plane is not parallel to the mobile phone. The plane may be a fixed surface or a dynamically variable surface through which the user sends input through the 2D screen. The plane-to-2D screen mapping may control interaction with virtual objects on an interaction plane in the XR environment, thereby enabling peering between the same experiences on the XR interface and the non-XR interface.

Description

Technical Field

[0001] The present disclosure relates to converting interactions on a two-dimensional (2D) interface into an artificial reality (hyper-reality (XR)) experience via a dynamic virtual plane. Background Art

[0002] A user interacting with an artificial reality (XR) device can view content in an artificial reality environment that includes real-world objects and / or two-dimensional (2D) virtual objects and / or three-dimensional (3D) virtual objects. For example, an artificial reality environment can be a virtual environment depicted by a virtual reality (VR) device that shows a set of virtual objects. As another example, an artificial reality environment can be a mixed reality environment that has real-world objects and virtual objects that are supplemented on top of the real-world objects. Users can view objects in the artificial reality environment and modify content in the artificial reality environment.

[0003] There may be applications that can operate on both an XR interface and a two-dimensional (2D) interface. A 2D interface can be a flat surface that can display 2D content (e.g., objects, graphics, text, etc.). For example, a 2D interface can be part of a laptop, mobile device, TV, etc. Compared to a fully immersive XR experience, due to the limitations of the 2D interface, the XR content rendered and interacted with on the 2D interface may be different from the content rendered on the XR interface. Summary of the invention

[0004] According to one aspect of the present invention, a method for converting dynamic planar interactions on a two-dimensional interface into an artificial reality experience is provided, the method comprising: projecting an interaction plane in the artificial reality experience onto the two-dimensional interface, the interaction plane being a virtual surface through which a user can send input to the artificial reality experience through the two-dimensional interface mapped to the interaction plane; receiving instructions for manipulating the interaction plane in the artificial reality experience; in response to the instructions, manipulating the interaction plane in the artificial reality experience by applying at least one of yaw, roll, pitch, or scaling or any combination thereof to the interaction plane; receiving, via the two-dimensional interface, a request to interact with a virtual object through mapping the two-dimensional interface to the interaction plane, the virtual object intersecting with the interaction plane in the artificial reality experience; and in response to the request, causing virtual objects related to the interaction plane to interact.

[0005] Optionally, the instruction is generated by a virtual joystick displayed on the two-dimensional interface.

[0006] Optionally, the instructions are generated in response to data collected by a gyroscope that is in operative communication with the two-dimensional interface.

[0007] Optionally, the instructions are generated by a script that performs a time-based continuous rotation of the interactive plane.

[0008] Optionally, the interaction plane is not parallel to the two-dimensional interface.

[0009] Optionally, interacting with the virtual object comprises moving the virtual object along the interaction plane via a hand of an avatar associated with the user.

[0010] Optionally, the interaction with the virtual object comprises selecting the virtual object, and wherein the method further comprises: in response to selecting the virtual object, performing a second action predefined for the virtual object.

[0011] Optionally, the shape of the interaction plane is capable of deforming in response to user commands.

[0012] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores instructions, which, when executed by a computing system, cause the computing system to execute a process for converting dynamic plane interactions on a two-dimensional interface into an artificial reality experience, the process comprising: projecting an interaction plane in the artificial reality experience onto the two-dimensional interface, the interaction plane being a virtual surface through which a user can send input to the artificial reality experience through the two-dimensional interface mapped to the interaction plane; receiving a command for manipulating the interaction plane in the artificial reality experience; in response to the command, manipulating the interaction plane in the artificial reality experience by applying at least one of left and right pan, roll, pitch, or zoom, or any combination thereof to the interaction plane; receiving, via the two-dimensional interface, a request to interact with a virtual object through a mapping of the two-dimensional interface to the interaction plane, the virtual object intersecting the interaction plane in the artificial reality experience; and in response to the request, causing the virtual object associated with the interaction plane to interact.

[0013] Optionally, the interaction plane is manipulated in the artificial reality experience by applying at least one of yaw, roll, pitch, or zoom, or any combination thereof, to the interaction plane.

[0014] Optionally, the command is generated by a virtual joystick displayed on the two-dimensional interface.

[0015] Optionally, the command is generated in response to data collected by a gyroscope that is in operative communication with the two-dimensional interface.

[0016] Optionally, the command is generated by a script that performs a continuous time-based rotation of the interaction plane.

[0017] Optionally, the interaction plane is not parallel to the two-dimensional interface.

[0018] Optionally, interacting with the virtual object comprises moving the virtual object along the interaction plane via a hand of an avatar associated with the user.

[0019] According to another aspect of the present invention, a computing system for converting dynamic planar interactions on a two-dimensional interface into an artificial reality experience is provided, the computing system comprising: one or more processors; and one or more memories storing instructions, which, when executed by the one or more processors, cause the computing system to execute a process, the process comprising: projecting an interaction plane in the artificial reality experience onto the two-dimensional interface, the interaction plane being a virtual surface through which a user can send input to the artificial reality experience through the two-dimensional interface mapped to the interaction plane; receiving a command for manipulating the interaction plane in the artificial reality experience; in response to the command, manipulating the interaction plane in the artificial reality experience by applying at least one of left and right sway, roll, pitch, or zoom, or any combination thereof to the interaction plane; receiving, via the two-dimensional interface, a request to interact with a virtual object through a mapping of the two-dimensional interface to the interaction plane, the virtual object intersecting with the interaction plane in the artificial reality experience; and in response to the request, causing the virtual object associated with the interaction plane to interact.

[0020] Optionally, the interaction plane is manipulated in the artificial reality experience by applying at least one of yaw, roll, pitch, or zoom, or any combination thereof, to the interaction plane.

[0021] Optionally, the command is generated by a virtual joystick displayed on the two-dimensional interface.

[0022] Optionally, the command is generated in response to data collected by a gyroscope that is in operative communication with the two-dimensional interface.

[0023] Optionally, the command is generated by a script that performs a continuous time-based rotation of the interaction plane. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a block diagram showing an overview of various devices on which some embodiments of the present technology may operate.

[0025] Figure 2A is a line diagram illustrating a virtual reality headset that may be used with some embodiments of the present technology.

[0026] Figure 2B is a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.

[0027] Figure 2C is a line diagram illustrating a controller that a user may hold with one or both hands to interact with an artificial reality environment in some embodiments.

[0028] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0029] Figure 4 is a block diagram illustrating various components that may be used in a system employing the disclosed techniques in some implementations.

[0030] Figure 5 A flowchart illustrating a process for converting dynamic planar interactions on a two-dimensional interface into an artificial reality experience in some embodiments of the present technology.

[0031] Fig. 6A A conceptual diagram illustrating an example artificial reality experience with a projected interaction plane that intersects virtual objects for interaction with an avatar's hands.

[0032] Figure 6B A conceptual diagram illustrating an example artificial reality experience with a manipulated interaction plane that intersects a virtual object, with an avatar's hand extending along the interaction plane to grasp the virtual object.

[0033] Figure 7 A conceptual diagram illustrating an example artificial reality experience with an interactive plane projected onto a fixed surface.

[0034] Figure 8 A conceptual diagram illustrating an example artificial reality experience with an interaction plane projected onto virtual objects for interaction.

[0035] The technology described herein may be better understood by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numbers refer to identical or functionally similar elements. DETAILED DESCRIPTION

[0036] Various aspects of the present disclosure relate to converting dynamic planar interactions on a two-dimensional (2D) interface into an artificial reality (XR) experience. Many users access XR experiences through a 2D interface (e.g., a mobile phone, tablet, or laptop computer). However, it is difficult to convert an XR experience into a 2D interface in a way that feels intuitive and natural. Therefore, the disclosed technology can map an interactive plane in a three-dimensional (3D) scene to a 2D interface, even if the interactive plane may not be parallel to the viewing window of the 2D interface. In some embodiments, the interactive plane can be fixed to a surface in the XR experience, and the user sends input to the surface through the 2D interface. However, in other embodiments, the interactive plane can be dynamically variable, for example, by: a joystick on the screen, a gyroscope of a mobile phone, or a continuous rotation of the plane based on time. In some embodiments, the mapping of the interactive plane to the 2D interface can control the avatar's hand to extend into the interactive plane to interact with the XR environment. In some embodiments, the mapping of the interactive plane to the 2D interface can allow the user to select virtual objects in the XR environment and / or interact with these virtual objects without moving their avatar. Therefore, the present technology can achieve parity between the same experience on XR interfaces and non-XR interfaces.

[0037] For example, a user may load an XR baseball experience on a mobile phone in which the user's avatar may swing a virtual bat at a virtual ball projected into the XR environment. Some embodiments may project an interaction plane that intersects the virtual ball at a predetermined position and at a predetermined angle, for example, the interaction plane is parallel to a 2D surface with the virtual ball at the center of the interaction plane. The user may use the mobile phone's gyroscope to change the angle of the interaction plane through which the user wishes to cause their avatar to swing the virtual bat. When the interaction plane is at the desired angle, the user may tap the screen of the mobile device (e.g., tap the virtual ball) to cause the virtual bat to swing along the interaction plane and hit the virtual ball.

[0038] In another example, a user may load an XR construction experience on a laptop computer in which the user's avatar may build a virtual house in an XR environment. Some embodiments may project an interaction plane in the XR construction experience. The user may manipulate a virtual joystick to change the position of the interaction plane so that the interaction plane intersects a virtual nail. The user may also manipulate the virtual joystick to change the angle of the interaction plane relative to the virtual nail. For example, in order for the user's avatar to hit the virtual nail, the interaction plane may be manipulated to be vertical. The user may then click on the virtual nail to cause a hammer to swing along the interaction plane and hit the virtual nail.

[0039] In yet another example, a user may load an XR outdoor experience on a tablet in which the user's avatar may walk through a virtual neighborhood and park. Some embodiments may project an interactive plane in a virtual park to allow the user to select and / or interact with virtual objects in the virtual park. For example, some embodiments may project an interactive plane through an input mechanism (e.g., a virtual button) corresponding to suggested actions, such as customizing the appearance of the virtual park or traveling to another virtual world.

[0040] As used herein, an "XR interface" may be a device capable of displaying a fully immersive XR experience, such as a head-mounted display within an XR system. In some embodiments, an XR system may include devices and components that support the XR experience in addition to the XR interface (e.g., processing components, input / output devices (e.g., controllers), etc.). FIG. 2A to FIG. 2C Such components are described further below.

[0041] A "2D interface" may be an application or device that can render an XR environment on a 2D surface. For example, a 2D interface may be a computer screen, a television display, a mobile device (e.g., a cell phone), a mobile application, a web browser, etc. A 2D interface may be part of a 2D system that includes other devices and components (e.g., processing components, input / output devices, etc.).

[0042] Embodiments of the disclosed technology may include an artificial reality system, or may be implemented in conjunction with an artificial reality system. Artificial reality or hyperreality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and artificial reality or hyperreality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with collected content (e.g., real-world photos). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in an artificial reality, and / or to be used in an artificial reality (e.g., to perform activities in an artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave" environment projection system or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0043] As used herein, "virtual reality" or "VR" refers to an immersive experience in which a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views real-world images after they have passed through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display the images on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images (e.g., by adding virtual objects) as they pass through the system. "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is partially generated by a computing system and partially comes from light reflected from objects in the real world. For example, an MR head-mounted viewer can be shaped as a pair of glasses with a pass-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of: VR, AR, MR, or any combination or hybrid thereof.

[0044] These embodiments improve upon the prior art of rendering XR experiences on 2D interfaces in that they simulate 3D experiences on 2D interfaces. Some embodiments may convert input on a 2D interface into the six degrees of freedom inherent to the XR interface without converting the input to be parallel to the 2D interface. Some embodiments may reproduce fairly complex 3D behaviors on a 2D interface by adjusting how user input is applied to the XR environment via an interaction plane. The embodiments described herein must be rooted in computing technology (e.g., presenting an XR experience on a 2D interface and interacting with the XR experience) and overcome problems that traditionally arise from accessing XR experiences on a 2D interface. For example, these embodiments may convert input received on a 2D interface into three dimensions to better simulate a full 3D experience that traditionally requires an XR interface.

[0045] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 11 is a block diagram showing an overview of devices on which some embodiments of the disclosed technology may operate. These devices may include hardware components of a computing system 100 that can transform dynamic planar interactions on a two-dimensional interface into an artificial reality experience. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted viewer that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted viewer and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head-mounted viewer and other processing operations are offloaded to the core processing component. The following is a description of the present invention in conjunction with the present invention. Figure 2A and Figure 2B In some implementations, the location data and environmental data may be collected solely by sensors integrated into the head mounted device, while in other implementations, one or more of the plurality of non-head mounted device computing devices may include sensor components that may track environmental data or location data.

[0046] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed in multiple devices (e.g., distributed in two or more computing devices among the computing devices 101 to 103).

[0047] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be communicated by a hardware controller that interprets the signals received from the input devices and transmits information to the processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0048] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus (e.g., a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus), or a wireless connection. The processor 110 can communicate with a hardware controller of a device (e.g., a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or the input device is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices include: liquid crystal display (LCD) display screen; light emitting diode (LED) display screen; projection, holographic, or augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or network card, a video chip or video card, an audio chip or audio card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0049] In some embodiments, input from I / O device 140 (e.g., a camera, a depth sensor, an inertial motion unit (IMU) sensor, a GPS unit, a LiDAR or other time-of-flight sensor, etc.) can be used by computing system 100 to identify and map the user's physical environment and track the user's position within the environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., a topology, a grid, etc.) of an area (which can be a room, a building, an outdoor space, etc.) and / or obtain a map previously generated by computing system 100 or another computing system that has mapped the area. The SLAM system can track users within the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other position changes, etc.

[0050] The computing system 100 may include a communication device capable of wireless or wire-based communication with other local computing devices or network nodes. The communication device may communicate with another device or server over a network using, for example, the Transmission Control Protocol / Internet Protocol (TCP / IP protocol). The computing system 100 may utilize the communication device to distribute operations across multiple network devices.

[0051] Processor 110 can access memory 150, which can be included in one of the multiple computing devices of computing system 100, or can be distributed on multiple computing devices of computing system 100, or multiple other external devices. Memory includes one or more hardware devices for volatile or non-volatile storage, and can include both read-only memory and writable memory. For example, memory can include one or more of the following: random access memory (RAM), various cache memories, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, tape drive, etc. Memory is not a propagation signal separated from the underlying hardware; therefore, memory is non-transient. Memory 150 can include program memory 160, which stores programs and software, such as operating system 162, dynamic plane interaction system 164, and other applications 166. The memory 150 may also include a data memory 170, which may include, for example, interactive plane rendering data, interactive plane manipulation data, virtual object rendering data, virtual object manipulation data, avatar rendering data, user interaction data, configuration data, settings, user options or preferences, etc., which may be provided to the program memory 160 or any element of the computing system 100.

[0052] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that may be suitable for use with the present technology include, but are not limited to, XR head-mounted viewers, personal computers, server computers, handheld devices or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices, etc.

[0053] Figure 2A2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a strap 210. Front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a plurality of locators 225, and one or more computing units 230. Position sensors 220, IMU 215, and computing units 230 may be located inside HMD 200 and may not be visible to the user. In various embodiments, IMU 215, position sensors 220, and locators 225 may track the movement and position of HMD 200 in the real world and artificial reality environments with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locators 225 may emit infrared beams that produce light spots on real objects around HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; other non-camera-based position sensors, force sensors, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect light points. The computing unit 230 in the HMD 200 may use the detected light points to infer the position and movement of the HMD 200, and to recognize the shapes and positions of real objects around the HMD 200.

[0054] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user as indicated by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., micro-LED, laser (LASER)), some other display, or some combination thereof.

[0055] In some embodiments, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., via light emitted from the HMD 200), and the PC can use the external sensors in combination with the outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0056] Figure 2B 2 is a line diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 Gigahertz (GHz) link) as indicated by link 256. In other embodiments, the mixed reality system 250 includes only a head-mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a transparent display 258 and a frame 260. The frame 260 can house various electronic components (not shown), such as a light projector (e.g., lasers, LEDs, etc.), a camera, an eye tracking sensor, a micro-electromechanical system (MEMS) component, a network component, etc.

[0057] The projector can be coupled to the transparent display 258, for example, via an optical element to display media to the user. The optical element may include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. for guiding the light from the projector to the user's eyes. The image data can be transmitted from the core processing component 254 to the mixed reality HMD 252 via the link 256. The controller in the HMD 252 can convert the image data into light pulses from the projector, which can be transmitted to the user's eyes as output light via the optical element. The output light can be mixed with the light passing through the display 258, thereby allowing the output light to present virtual objects as follows: these virtual objects appear to exist in the real world.

[0058] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example, with 3DoF or 6DoF, track parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects to appear stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0059] Figure 2C A plurality of controllers 270 (including controllers 276A and 276B) are shown, which in some embodiments can be held by a user with one or both hands to interact with an artificial reality environment presented by HMD 200 and / or HMD 250. Controllers 270 can communicate directly with the HMD or can communicate with the HMD via an external device (e.g., core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit additional light points. The HMD 200 or HMD 250, external sensors, or sensors in the controllers can track these controller light points (e.g., tracking the controllers with 3DoF or 6DoF) to determine the controller position and / or orientation. The computing unit 230 or core processing component 254 in HMD 200 can use this tracking to monitor the user's hand position and hand movement in combination with the IMU output and the position output. The controller may also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A and 274B) that a user may actuate to provide input and interact with objects.

[0060] In various embodiments, HMD 200 or HMD 250 may also include additional subsystems (e.g., an eye tracking unit, an audio system, various network components, etc.) to monitor the user's interactions and indications of intent. For example, in some embodiments, instead of or in addition to a controller, one or more cameras included in HMD 200 or HMD 250, or one or more cameras from a plurality of external cameras, may monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources may illuminate either or both of the user's eyes, and HMD 200 or HMD 250 may use eye-facing cameras to capture reflections of that light to determine eye position (e.g., based on a set of reflections around the user's cornea), model the user's eyes, and determine gaze direction.

[0061] Figure 3 305B may be an HMD 200 or an HMD system 250. The client computing device 305 may be a logical connection to one or more remote computers (e.g., a server computing device) through a network 330.

[0062] In some implementations, server 310 may be an edge server that receives multiple client requests and coordinates fulfillment of these requests through other servers (e.g., servers 320A to 320C). Server computing devices 310 and 320 may include computing systems, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, the server computing devices may each be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.

[0063] Client computing device 305 and server computing devices 310 and 320 can each act as a server or client of one or more other server / client devices. Server 310 can be connected to database 315. Servers 320A to 320C can each be connected to corresponding databases 325A to 325C. As described above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share a database or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can each be a distributed computing environment containing multiple computing devices, which can be located in its corresponding server, or can be located at the same physical location or geographically different physical locations.

[0064] The network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. The network 330 may be the Internet or some other public or private network. The client computing device 305 may be connected to the network 330 via a network interface, for example, by wired or wireless communication. Although the connections between the server 310 and the plurality of servers 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including the network 330 or a separate public or private network.

[0065] Figure 41 is a block diagram showing a plurality of components 400 that can be used in a system that employs the disclosed technology in some embodiments. The plurality of components 400 can be included in one device of the computing system 100, or can be distributed on a plurality of devices in the computing system 100. These components 400 include hardware 410, middleware 420, and a plurality of dedicated components 430. As described above, the system implementing the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.), and a storage memory 418. In various embodiments, the storage memory 418 can be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, the storage memory 418 can be one or more hard drives or flash drives accessible via a system bus, or can be a cloud storage provider (e.g., in a storage device 315 or 325) or other network storage device accessible via one or more communication networks. In various implementations, the plurality of components 400 may be implemented in a client computing device (eg, client computing device 305 ), or on a server computing device (eg, server computing devices 310 or 320 ).

[0066] The middleware 420 may include components that transfer resources between the hardware 410 and the plurality of specialized components 430. For example, the middleware 420 may include an operating system, a service system, a driver, a basic input output system (BIOS), a controller circuit, or other hardware or software systems.

[0067] The specialized components 430 may include software or hardware configured to perform operations for converting interactions on a two-dimensional (2D) interface into an artificial reality (XR) experience via a dynamic virtual plane. The multiple specialized components 430 may include an interactive plane projection module 434; a virtual joystick module 436; a sensor data conversion module 438; a continuous rotation module 440; an interactive plane manipulation module 442; a virtual object interaction module 444; and components and application programming interfaces (APIs) (e.g., interface 432) that may be used to provide a user interface, transmit data, and control each specialized component. In some embodiments, the multiple components 400 may be located in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more of the multiple specialized components 430. Although the multiple specialized components 430 are depicted as separate components, these specialized components may be logical functions or other non-physically distinguished functions, and / or may be sub-modules or code blocks of one or more applications. In addition, it is contemplated that one or more of the multiple specialized components 430 may be omitted in some embodiments. For example, when one or more virtual joysticks are used to receive input, it is contemplated that the sensor data conversion module 438 and / or the continuous rotation module 440 may be omitted.

[0068] The interactive plane projection module 434 can project the interactive plane in the XR experience to a 2D interface. The interactive plane can be a virtual three-dimensional (3D) surface (e.g., a virtual shape or grid), and the user can send input through the virtual 3D surface through the 2D interface. The interactive plane projection module 434 can project the interactive plane in the XR experience so that the user can select virtual objects that intersect with the plane and / or interact with these virtual objects. The interactive plane projection module 434 can project the interactive plane at any angle (pitch, roll, and left and right) and position (x-axis, y-axis, z-axis) relative to the 2D interface. Therefore, some embodiments can convert the input on the 2D interface into the 6 degrees of freedom of the XR interface itself, so as to better simulate the control of the XR experience through the XR interface.

[0069] The interaction plane projection module 434 can initially project the interaction plane at any point, position, and angle in the XR experience. For example, the interaction plane projection module 434 can project the interaction plane at any or default point and angle in the XR experience, such as projecting the interaction plane at an area in the middle of a 2D interface in the XR experience, where the interaction plane is parallel to the 2D interface. In another example, the interaction plane projection module 434 can project the interaction plane at a predicted or suggested position and / or angle so that the interaction plane can intersect with one or more interactive virtual objects visible to the user in the XR experience. Therefore, in some embodiments, the user can interact with virtual objects in the XR experience without further manipulating the interaction plane, i.e., making the interaction plane manipulation module 442 unnecessary. References herein Figure 5 Block 502 of the accompanying drawings describes more details about projecting the interactive plane in the XR experience into the 2D interface.

[0070] The virtual joystick module 436 can receive instructions for manipulating the interactive plane in the XR experience through user input to the virtual joystick. In some embodiments, the virtual joystick can control the angle and / or position of the interactive plane in response to selections (e.g., taps) and / or movements (e.g., slides, gestures, etc.) made by the user on the touch screen of the 2D interface. In one example, the virtual joystick may include a virtual left controller and a virtual right controller displayed on the 2D interface, which can change the position of the interactive plane and the angle of the interactive plane, respectively, such as Fig. 6A and Figure 6B As shown. In another example, the virtual joystick can be controlled by the user performing a specific action on the 2D interface (e.g., selecting and holding an action) in a specific area on the 2D interface without displaying the virtual controller. In yet another example, the virtual joystick can be controlled by an I / O device (e.g., a mouse, a keyboard, etc.) included in the plurality of I / O devices 416. When the virtual joystick module 436 is included in the dedicated component 430, it is contemplated that in some embodiments, the sensor data conversion module 438 and / or the continuous rotation module 440 can be omitted from the dedicated component 430. However, in some embodiments, the virtual joystick module 436, the sensor data conversion module 438, and the continuous rotation module 440 can all be included in the dedicated component 430. Reference is made to Figure 5 Block 504 of the present invention describes further details regarding receiving instructions for manipulating the interactive plane in the XR experience via the virtual joystick.

[0071] The sensor data conversion module 438 can receive instructions for manipulating the interactive plane in response to data collected by one or more sensors (e.g., gyroscopes, compasses, accelerometers, etc.) of an inertial measurement unit (IMU) that is in operative communication or integrated with the 2D interface (not shown). In some embodiments, the IMU can be included in the hardware 410. For example, the 2D interface can collect gyroscope data, and the sensor data conversion module 438 can convert the gyroscope data into instructions for controlling the pitch, roll, and / or yaw of the interactive plane. When the sensor data conversion module 438 is included in the dedicated component 430, it is contemplated that in some embodiments, the virtual joystick module 436 and / or the continuous rotation module 440 can be omitted from the dedicated component 430. However, in some embodiments, the virtual joystick module 436, the sensor data conversion module 438, and the continuous rotation module 440 can all be included in the dedicated component 430. Reference is made to Figure 5 Block 504 describes further details regarding receiving instructions for manipulating the interaction plane in response to data collected by one or more sensors.

[0072] The continuous rotation module 440 can receive instructions for automatically manipulating the interactive plane by executing a script (e.g., in response to detecting an interactive virtual object in the XR experience) and perform time-based continuous rotation of the interactive plane. In some embodiments, the interactive plane can be moved continuously in time so that the interactive plane rotates in the field of view of the user of the 2D interface.

[0073] For example, the interaction plane projection module 434 may project the interaction plane to intersect with the virtual object. The continuous rotation module 440 may continuously rotate and / or move the interaction plane around and through the virtual object through various angles and / or positions until the desired position of the interaction plane is reached. Then, in some embodiments, the continuous rotation module 440 may pause the interaction plane at that point through user input. However, in some embodiments, the continuous rotation module 440 may continuously rotate and / or move the interaction plane without accepting user input for pausing the movement of the interaction plane. In such an embodiment, a user of the 2D interface may wait until the interaction plane intersects with the virtual object and / or is at a desired angle relative to the virtual object, and then time their actions based on the timing of the intersection. When the continuous rotation module 440 is included in the dedicated component 430, it is contemplated that in some embodiments, the virtual joystick module 436 and / or the sensor data conversion module 438 may be omitted from the dedicated component 430. However, in some embodiments, the virtual joystick module 436, the sensor data conversion module 438, and the continuous rotation module 440 may all be included in the dedicated component 430. Reference herein Figure 5Block 504 of the accompanying drawings describes further details regarding manipulating the interactive plane by executing a script that applies time-based continuous rotation of the interactive plane.

[0074] The interaction plane manipulation module 442 can manipulate the interaction plane in the XR experience in response to one or more instructions received through the virtual joystick module 436, the sensor data conversion module 438, and / or the continuous rotation module 440. The interaction plane manipulation module 442 can manipulate the interaction plane in accordance with the one or more instructions received by moving the interaction plane to a different location in the XR experience and / or adjusting the angle of the interaction plane (e.g., by applying at least one of yaw, roll, pitch, or zoom, or any combination thereof to adjust the angle of the interaction plane).

[0075] In some embodiments, the interaction plane manipulation module 442 can deform or change the shape of the interaction plane. For example, a user can change the default shape of the interaction plane (e.g., a circle) to a different shape (e.g., a rectangle, a triangle, etc.). In another example, the interaction plane manipulation module 442 can change the shape of the interaction plane based on the shape of the surface onto which the interaction plane is projected, such as changing a circular interaction plane to a square based on a square table onto which the interaction plane is projected. In some embodiments, the user can change the interaction plane to be three-dimensional (3D), textured, and / or non-planar. Although the interaction plane is described herein as being manipulable, it is contemplated that in some embodiments, the interaction plane may be a fixed surface to which a user can send input via a 2D interface. Reference is made herein to Figure 5 Block 506 of the accompanying description describes more details about manipulating the interactive plane in the XR experience.

[0076] The virtual object interaction module 444 can receive requests to interact with virtual objects in the XR experience along the interaction plane via the 2D interface and I / O device 416. The interaction can be movement of a virtual object, movement of an avatar relative to a virtual object, selection of a virtual object, etc. In some embodiments, a virtual object can be a selectable element and / or a controllable element, such as a notification, a button, a text box, a graphic, etc. For example, a virtual object can be a text box with a suggested action. When the text box is selected through the interaction plane, the virtual object interaction module 444 can perform the suggested action, such as traveling to a different virtual world, interacting with another virtual object, etc.

[0077] In some embodiments, the interaction plane can define the position and / or movement angle of the avatar's hand associated with the user of the 2D interface. In other words, the mapping of the interaction plane to the 2D interface can control the avatar's hand to reach into the interaction plane to interact with virtual objects in the XR environment. Figure 5Block 508 of the accompanying drawings describes further details regarding receiving a request to interact with a virtual object along an interaction plane.

[0078] The virtual object interaction module 444 can interact with the virtual object along the interaction plane in the XR experience in response to the request. For example, the virtual object interaction module 444 can move the virtual object to another location on the interaction plane, for example, in response to the user tapping the virtual object that intersects the interaction plane and dragging the virtual object along the interaction plane. In another example, the virtual object interaction module 444 can move the virtual object that intersects the interaction plane to a location outside the interaction plane in the XR experience. For example, a user can use the projected interaction plane to select a virtual object (possibly from multiple virtual objects) that intersects the interaction plane, and then tap a location outside the interaction plane to move the virtual object to that location. In yet another example, the virtual object interaction module 444 can select controllable elements that intersect the interaction plane, such as buttons, text boxes, graphics, etc., which can enable actions to be taken within the XR experience. References herein Figure 5 Block 510 of the accompanying drawings describes more details about interacting with virtual objects along the interaction plane.

[0079] Those skilled in the art will appreciate that the above described Figures 1 to 4 The components shown in the flowcharts and in each of the flowcharts discussed below may be changed in various ways. For example, the order of the logic may be rearranged, multiple sub-steps may be performed in parallel, the logic shown may be omitted, other logic may be included, etc. In some embodiments, one or more of the above-mentioned multiple components may perform one or more of the multiple processes described below.

[0080] Figure 5 Flowchart showing process 500 for converting dynamic plane interactions on a two-dimensional (2D) interface into an artificial reality (XR) experience in some embodiments. In some embodiments, process 500 can be performed in response to a user request to project an interactive plane in an XR experience onto a 2D interface. In some embodiments, process 500 can be automatically performed when interactive elements such as virtual objects (e.g., virtual balls, virtual game pieces, virtual buttons, notifications, virtual menus, etc.) are displayed. In some embodiments, process 500 can be performed by a 2D interface. In some embodiments, process 500 can be performed by a server that supports such a 2D interface. In some embodiments, some boxes of process 500 can be performed locally by the 2D interface, while other boxes of process 500 can be performed remotely by the server.

[0081] At box 502, process 500 can project an interaction plane in the XR experience to a 2D interface. The interaction plane can be a virtual surface through which a user can send input through the 2D interface. For example, process 500 can project an interaction plane in the XR experience so that the user can interact with virtual objects that intersect the plane. In some embodiments, the interaction plane does not have to be parallel to the 2D interface. For example, the interaction plane can be projected partially perpendicular to the 2D interface and / or at any angle (pitch, roll, and yaw) and position (x-axis, y-axis, z-axis) relative to the 2D interface.

[0082] In some embodiments, process 500 may project the interaction plane at any point in the XR experience, for example, projecting the interaction plane at an area in the middle of a 2D interface in the XR experience. In some embodiments, process 500 may project the interaction plane at a specific point in the XR experience. For example, process 500 may project the interaction plane at a position and / or angle in the XR experience that intersects one or more virtual objects in the user's field of view. In another example, process 500 may project the interaction plane at a position and / or angle corresponding to the predicted movement of the user's avatar within the XR experience, such as projecting the interaction plane at a point, range, and / or angle within the virtual reach of the avatar's arm.

[0083] At block 504, process 500 can receive instructions for manipulating the interaction plane in the XR experience. In some implementations, process 500 can receive instructions for manipulating the interaction plane from user input. For example, in some implementations, user input through a virtual joystick displayed on a 2D interface can generate instructions for manipulating the interaction plane. The virtual joystick can control the angle and position of the interaction plane. For example, the virtual joystick can include a virtual left controller and a virtual right controller, one of which can change the position of the interaction plane and the other can change the angle of the interaction plane, as described herein with reference to FIG. Figure 6B Further described.

[0084] In another example, instructions for manipulating the interactive plane may be generated in response to data collected by a gyroscope and / or other sensor of an inertial measurement unit (IMU) that is in operative communication with the 2D interface. For example, when the 2D interface is a mobile phone, the mobile phone may include one or more sensors of an IMU that can be used to detect movement of the mobile phone. Process 500 may convert the movement of the mobile phone into one or more instructions for manipulating the interactive plane. For example, motion data associated with the movement of the 2D interface may be collected by one or more sensors (e.g., accelerometers, gyroscopes, GPS, and / or magnetometers). The measurements may include non-gravitational acceleration of the device in the x-, y-, and z-directions; gravitational acceleration of the device in the x-, y-, and z-directions; left and right sway, roll, and pitch of the device; derivatives of these measurements; gravity difference angle of the device; and changes in the normed gravitational acceleration of the device.

[0085] In some embodiments, process 500 can receive instructions from a script for automatically manipulating the interaction plane, for example in response to detecting an interactive virtual object in the XR experience. For example, a script that performs time-based continuous rotation of the interaction plane can generate instructions for manipulating the interaction plane. In some embodiments, the interaction plane can move continuously in time so that the interaction plane rotates in the field of view of the user of the 2D interface. In some embodiments, process 500 can pause the interaction plane at a certain angle through user input. In some embodiments, the user can wait until the interaction plane intersects with the virtual object and then time its actions based on the timing of the intersection. Although the interaction plane is described as manipulable herein, it is contemplated that in some embodiments, the interaction plane can be a fixed surface to which the user can send input through a 2D interface.

[0086] At block 506, the process 500 may manipulate the interactive plane in the XR experience in response to the instruction. In some embodiments, the process 500 may manipulate the interactive plane by moving the interactive plane to a different position in the XR experience, for example, based on user input received from a virtual joystick or movement detected by a sensor of the IMU. In some embodiments, the process 500 may manipulate the interactive plane in the XR experience by applying at least one of sway, roll, pitch, or scale, or any combination thereof, to the interactive plane. For example, the process 500 may rotate the interactive plane around a vertical axis (i.e., sway), around a side-to-side axis (i.e., pitch), and / or around a front-to-back axis (i.e., roll) by one or more instructions received to manipulate the interactive plane. Additionally or alternatively, the process 500 may scale the interactive plane so that the interactive plane becomes larger or smaller within the XR experience. In some embodiments, the shape of the interactive plane may deform in response to a user command. For example, a user may change the default shape of the interactive plane (e.g., a circle) to a different shape (e.g., a rectangle, a triangle, etc.). In some implementations, the user can change the interaction plane to be three-dimensional (3D).

[0087] At block 508, process 500 may receive, via the 2D interface, a request to interact with a virtual object along an interaction plane in the XR experience. In some embodiments, the interaction may be to select a virtual object and / or to move the virtual object along the interaction plane. For example, process 500 may manipulate the interaction plane to intersect the virtual object. Process 500 may select a virtual object and / or move the virtual object along the manipulated interaction plane based on user input. For example, the user input may be a selection of an area on the interaction plane (e.g., a tap or click), a drag motion of an object along the interaction plane, and the like.

[0088] In some embodiments, the hand of an avatar associated with a user of the 2D interface can be moved along the interaction plane to interact with a virtual object that intersects the interaction plane. In such an embodiment, the interaction plane can define the angle at which the avatar's arm rotates from the shoulder joint. The mapping of the interaction plane to the 2D interface can control the avatar's hand to extend into the interaction plane to interact with the virtual object in the XR experience. In some embodiments, the virtual object can be another interactive element, such as a notification, a button, or other controllable element.

[0089] At box 510, process 500 may interact with the virtual object along the interaction plane in the XR experience in response to the request. For example, process 500 may move a virtual object that intersects the interaction plane to a location in the XR experience that corresponds to a location where a tap or click is made via a 2D interface. In another example, process 500 may receive a tap or click on a virtual object that intersects the interaction plane, followed by a drag motion along the interaction plane. In such an example, process 500 may move the virtual object to a location in the XR experience where the drag motion ends within the interaction plane via the 2D interface. In yet another example, process 500 may select a virtual object (e.g., a controllable element and / or a selectable element) to perform a specific action on the virtual object (e.g., changing the color of a virtual car that intersects the interaction plane) or a specific action specified by the virtual object (e.g., traveling to a different world suggested by a text box that intersects the interaction plane).

[0090] Fig. 6A 6 is a conceptual diagram of an example XR experience 600A with a projected interaction plane 606 displayed on a 2D interface that intersects a virtual object 608 for interaction by a hand 604 of an avatar 602. In this example, virtual object 608 is a virtual ball. Some embodiments may project interaction plane 606 in XR experience 600A in response to a user request to display interaction plane 606, while some embodiments may automatically project interaction plane 606 in XR experience 600A in response to detecting a virtual object 608 with which avatar 602 can interact. Fig. 6A In FIG. 6 , the projected interaction plane 606 is initially parallel to the 2D interface. Therefore, a user manipulating the hand 604 of the avatar 602 will move the hand 604 vertically along the interaction plane 606 at the initial position and angle.

[0091] XR experience 600A also includes virtual joysticks 610A, 610B. A user of the 2D interface can manipulate virtual joysticks 610A, 610B to change the position and / or angle of interaction plane 606. For example, manipulation of joystick 610A can control the position of interaction plane 606, while manipulation of joystick 610B can control the angle of interaction plane 606.

[0092] Figure 6B6 is a conceptual diagram of an example XR experience 600B with a manipulated interaction plane 606 intersecting a virtual object 608, with a hand 604 of an avatar 602 extending along the interaction plane 606 to grasp the virtual object 608. Some embodiments may receive instructions for manipulating the interaction plane 606 in the XR experience 600B via virtual joysticks 610A, 610B. For example, a user may manipulate the virtual joysticks 610A, 610B to change the position and angle of the interaction plane 606 in the XR experience 600B, as compared to the XR experience 600A, such that the interaction plane 606 is angled at approximately 45 degrees and extends away from the avatar 602, while still intersecting the virtual object 608.

[0093] Some embodiments may receive a request to move hand 604 along manipulated interaction plane 606 in XR experience 600B via a 2D interface. Figure 6B In the example, the interaction plane 606 can define the angle at which the arm 612 of the avatar 602 rotates from the shoulder 614. The mapping of the interaction plane 606 to the 2D interface can control the hand 604 of the avatar 602 to extend into the interaction plane 606 to interact with the virtual object 608, i.e., grasp the ball.

[0094] Figure 7 is a conceptual diagram showing an example XR experience 700 with an interaction plane 706 projected onto a fixed surface 708. Figure 7 In the example, an avatar of a first user (not shown) is participating in an XR experience 700 (in this example, a virtual checkers game 702) with an avatar of a second user 704 through a 2D interface. Some embodiments may project an interaction plane 706 onto a fixed surface 708 (in this example, a desktop) so that the avatar of the first user (not shown) and the avatar of the second user 704 can move virtual checkers pieces 710 along the interaction plane 706 to play the virtual checkers game 702.

[0095] Some embodiments may project the interactive plane 706 in the XR experience 700 in response to a user request to display the interactive plane 706. Some embodiments may automatically project the interactive plane 706 in the XR experience 700 in response to detecting a fixed surface 708 and / or a virtual checkers pawn 710 with which an avatar (not shown) of a first user and an avatar 704 of a second user may interact. Some embodiments may manipulate the shape of the interactive plane 708 to match the shape of the fixed surface 708 so that virtual objects (e.g., virtual checkers pawn 710) that intersect the interactive plane 706 may be selected, moved along the interactive plane 706 projected onto the fixed surface 708, or otherwise interacted with.

[0096] Figure 88 is a conceptual diagram showing an example XR experience 800 having an interaction plane 816 projected onto a virtual object 810 for interaction. Figure 8 In the example, XR experience 800 is a virtual community experience in which an avatar 802 of a first user on a 2D interface can explore the virtual community and interact with virtual objects and other avatars in the virtual community. The virtual objects may include, for example, a virtual dog 804, a virtual car 806, and an avatar 808 of a second user, as well as selectable elements 810, 812, and 814.

[0097] In XR experience 800, a first user associated with avatar 802 can select selectable elements 810, 812, and 814 without having to touch selectable elements 810, 812, and 814. For example, some embodiments can project interaction plane 816 so that interaction plane 816 intersects selectable element 810, and / or the first user associated with avatar 802 can manipulate interaction plane 816 so that interaction plane 816 intersects selectable element 810. The first user associated with avatar 802 can then tap or click on a location in the 2D interface corresponding to selectable element 810, for example, to select selectable element 810. In this example, selectable elements 810-814 can be selected to perform actions specified by selectable elements 810-814, respectively. For example, avatar 802 associated with the first user can move to pet virtual dog 804 in response to selecting selectable element 810 via interaction plane 816.

[0098] References in this specification to "embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described with respect to that embodiment is included in at least one embodiment of the present disclosure. The appearances of these phrases in different places in the specification do not necessarily all refer to the same embodiment, nor are they necessarily separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be present in some embodiments but not in other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not in other embodiments.

[0099] As used herein, being higher than a threshold value means that the value of the item being compared is higher than other values ​​specified, the item being compared is located in an item with a maximum value, a certain specified number, or the item being compared has a value within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of the item being compared is lower than other values ​​specified, the item being compared is located in an item with a minimum value, a certain specified number, or the item being compared has a value within a specified minimum percentage value. As used herein, being within a threshold value means that the value of the item being compared is between two other values ​​specified, the item being compared is located in an item with a middle specified number, or the item being compared has a value within a middle specified percentage range. When not defined otherwise, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting fast connection" may be understood to mean selecting a connection with a value corresponding to the connection speed assigned thereto that is higher than a threshold value.

[0100] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items, such as A and A; B, B and C; A, A, B, C and C; etc.

[0101] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except in the appended claims.

[0102] Any patent, patent application and other references mentioned above are incorporated herein by reference. If necessary, various aspects may be modified to provide additional embodiments using the systems, functions and concepts of the various references mentioned above. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.

Claims

1. A method for converting dynamic planar interaction on a two-dimensional interface into an artificial reality experience, the method comprising: projecting an interaction plane in the artificial reality experience onto the two-dimensional interface, the interaction plane being a virtual surface through which a user can send input to the artificial reality experience via the two-dimensional interface mapped to the interaction plane; receiving instructions for manipulating the interaction plane in the artificial reality experience; In response to the instructions, manipulating the interactive plane in the artificial reality experience by applying at least one of yaw, roll, pitch, or zoom, or any combination thereof, to the interactive plane; receiving, via the two-dimensional interface, a request to interact with a virtual object through a mapping of the two-dimensional interface to the interaction plane, the virtual object intersecting the interaction plane in the artificial reality experience; as well as In response to the request, the virtual object associated with the interaction plane is enabled to interact.

2. The method according to claim 1, wherein: The instructions are generated by a virtual joystick displayed on the two-dimensional interface.

3. The method according to claim 1 or 2, wherein: The instructions are generated in response to data collected by a gyroscope that is in operable communication with the two-dimensional interface.

4. A method according to any preceding claim, wherein: The instructions are generated by a script that performs a continuous, time-based rotation of the interactive plane.

5. A method according to any preceding claim, wherein: The interaction plane is not parallel to the two-dimensional interface.

6. A method according to any preceding claim, wherein: Interacting with the virtual object includes moving the virtual object along the interaction plane via a hand of an avatar associated with the user.

7. A method according to any preceding claim, wherein: The interaction with the virtual object comprises selecting the virtual object, and wherein the method further comprises: In response to the virtual object being selected, a second action predefined for the virtual object is performed.

8. A method according to any preceding claim, wherein: The shape of the interaction plane is deformable in response to user commands.

9. A computer-readable storage medium storing instructions, which, when executed by a computing system, cause the computing system to perform a process for converting a dynamic planar interaction on a two-dimensional interface into an artificial reality experience, the process comprising: projecting an interaction plane in the artificial reality experience onto the two-dimensional interface, the interaction plane being a virtual surface through which a user can send input to the artificial reality experience via the two-dimensional interface mapped to the interaction plane; receiving commands for manipulating the interaction plane in the artificial reality experience; In response to the command, manipulating the interaction plane in the artificial reality experience by applying at least one of yaw, roll, pitch, or zoom, or any combination thereof, to the interaction plane; receiving, via the two-dimensional interface, a request to interact with a virtual object through a mapping of the two-dimensional interface to the interaction plane, the virtual object intersecting the interaction plane in the artificial reality experience; as well as In response to the request, the virtual object associated with the interaction plane is enabled to interact.

10. The computer-readable storage medium of claim 9, wherein: The interaction plane is manipulated in the artificial reality experience by applying at least one of yaw, roll, pitch, or zoom, or any combination thereof, to the interaction plane.

11. The computer-readable storage medium according to claim 9 or 10, wherein: The commands are generated by a virtual joystick displayed on the two-dimensional interface.

12. The computer-readable storage medium according to any one of claims 9 to 11, wherein: The commands are generated in response to data collected by a gyroscope that is in operable communication with the two-dimensional interface.

13. The computer-readable storage medium according to any one of claims 9 to 12, and any one or more of the following: a) wherein the commands are generated by a script that performs a continuous, time-based rotation of the interactive plane; or b) Among them, The interaction plane is not parallel to the two-dimensional interface; or c) wherein the interaction with the virtual object comprises moving the virtual object along the interaction plane via a hand of an avatar associated with the user.

14. A computing system for converting dynamic planar interactions on a two-dimensional interface into artificial reality experiences, the computing system comprising: one or more processors; as well as one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: projecting an interaction plane in the artificial reality experience onto the two-dimensional interface, the interaction plane being a virtual surface through which a user can send input to the artificial reality experience via the two-dimensional interface mapped to the interaction plane; receiving commands for manipulating the interaction plane in the artificial reality experience; In response to the command, manipulating the interaction plane in the artificial reality experience by applying at least one of yaw, roll, pitch, or zoom, or any combination thereof, to the interaction plane; receiving, via the two-dimensional interface, a request to interact with a virtual object through a mapping of the two-dimensional interface to the interaction plane, the virtual object intersecting the interaction plane in the artificial reality experience; and In response to the request, the virtual object associated with the interaction plane is enabled to interact.

15. The computing system of claim 14, and any one or more of the following: a) wherein the interaction plane is manipulated in the artificial reality experience by applying at least one of yaw, roll, pitch, or zoom, or any combination thereof, to the interaction plane; or b) Among them, The command is generated by a virtual joystick displayed on the two-dimensional interface; or c) Among them, The commands are generated in response to data collected by a gyroscope in operable communication with the two-dimensional interface; or d) wherein the commands are generated by a script that performs a time-based continuous rotation of the interactive plane.

Images