Metaverse Content Modality Mapping

Abstract

Implementations generally relate to metaverse content modality mapping. In some implementations, a method includes obtaining functionality developed for a first modality of a virtual environment. The method further includes mapping the functionality to a second modality of the virtual environment. The method further includes executing the functionality developed for the first modality based on user interactions associated with the second modality.

Description

[Technical field]

[0001] The present invention relates to metaverse content modality mapping. [Background technology]

[0002] This application claims priority from U.S. Provisional Patent Application No. 63 / 277,163, entitled "REALITY ENGINE," filed November 8, 2021, which is incorporated by reference herein for all purposes as if fully set forth in this application.

[0003] Metaverse content is an interactive experience in which the system uses computer-generated objects to present objects in a virtual or real-world environment. Metaverse content can be presented in different modalities, such as mobile phone augmented reality (AR), headset AR or virtual reality (VR), and two-dimensional (2D) displays (e.g., on a desktop computer). Metaverse content can be displayed in an app or a mobile browser. A barrier to metaverse content development is the need to handle multiple modalities. Summary of the Invention

[0004] Implementations generally relate to metaverse content modality mapping. In some implementations, a system includes one or more processors and includes logic encoded in one or more non-transitory computer-readable storage media for execution by the one or more processors. When executed, the logic is operable to cause the one or more processors to perform operations including obtaining functionality developed for a first modality of a virtual environment, mapping the functionality to a second modality of the virtual environment, and executing the functionality developed for the first modality based on user interactions associated with the second modality.

[0005] Further with respect to the system, in some implementations, the first modality is associated with a first device type of augmented reality, the first device type being a mobile device, and the second modality is associated with a second device type, the second device type being one of an augmented reality headset, a virtual reality headset, or a desktop computer. In some implementations, the logic, when executed, is further operable to cause the one or more processors to perform operations including determining a device associated with the second modality, launching a software module associated with the second modality, and adapting user interactions with the device to the functionality developed for the first modality. In some implementations, the logic, when executed, is further operable to cause the one or more processors to perform operations including mapping one or more user gestures within a three-dimensional scene associated with the second modality to one or more two-dimensional user interface elements associated with the first modality. In some implementations, the logic, when executed, is further operable to cause the one or more processors to perform operations including mapping user interactions with one or more input devices associated with the second modality to one or more two-dimensional user interface elements associated with the first modality. In some implementations, the logic, when executed, is further operable to cause the one or more processors to perform operations including adapting one or more background elements in a three-dimensional scene associated with the second modality based on a device type associated with the second modality. In some implementations, the logic, when executed, is further operable to cause the one or more processors to perform operations including adapting at least one target object in a three-dimensional scene associated with the second modality based on a device type associated with the second modality.

[0006] In some implementations, a non-transitory computer-readable storage medium having program instructions is provided that, when executed by one or more processors, are operable to cause the one or more processors to perform operations including obtaining functionality developed for a first modality of a virtual environment, mapping the functionality to a second modality of the virtual environment, and executing the functionality developed for the first modality based on user interactions associated with the second modality.

[0007] Further with respect to the computer-readable storage medium, in some implementations, the first modality is associated with a first device type of augmented reality, the first device type being a mobile device, and the second modality is associated with a second device type, the second device type being one of an augmented reality headset, a virtual reality headset, or a desktop computer. In some implementations, the logic, when executed, is further operable to cause the one or more processors to perform operations including determining a device associated with the second modality, launching a software module associated with the second modality, and adapting user interactions with the device to the functionality developed for the first modality. In some implementations, the instructions, when executed, are further operable to cause the one or more processors to perform operations including mapping one or more user gestures in a three-dimensional scene associated with the second modality to one or more two-dimensional user interface elements associated with the first modality. In some implementations, the instructions, when executed, are further operable to cause the one or more processors to perform an operation including mapping a user interaction with one or more input devices associated with the second modality to one or more two-dimensional user interface elements associated with the first modality. In some implementations, the instructions, when executed, are further operable to cause the one or more processors to perform an operation including adapting one or more background elements in a three-dimensional scene associated with the second modality based on a device type associated with the second modality. In some implementations, the instructions, when executed, are further operable to cause the one or more processors to perform an operation including adapting at least one target object in a three-dimensional scene associated with the second modality based on a device type associated with the second modality.

[0008] In some implementations, a computer-implemented method includes obtaining functionality developed for a first modality of a virtual environment, mapping the functionality to a second modality of the virtual environment, and executing the functionality developed for the first modality based on user interactions associated with the second modality.

[0009] Further with respect to the method, in some implementations, the first modality is associated with a first device type of augmented reality, the first device type being a mobile device, and the second modality is associated with a second device type, the second device type being one of an augmented reality headset, a virtual reality headset, or a desktop computer. In some implementations, the method further includes determining a device associated with the second modality, launching a software module associated with the second modality, and adapting user interactions with the device to the functionality developed for the first modality. In some implementations, the method further includes mapping one or more user gestures in a three-dimensional scene associated with the second modality to one or more two-dimensional user interface elements associated with the first modality. In some implementations, the method further includes mapping user interactions with one or more input devices associated with the second modality to one or more two-dimensional user interface elements associated with the first modality. In some implementations, the method further includes adapting one or more background elements in a three-dimensional scene associated with the second modality based on a device type associated with the second modality.

[0010] A further understanding of the nature and advantages of specific implementations disclosed herein may be realized by reference to the remaining portions of the specification and the attached drawings. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of an example environment associated with a system for metaverse content modality mapping that may be used in the implementations described herein. [Diagram 2] FIG. 2 is an exemplary flow diagram for adapting web-based augmented reality applications for deployment in a metaverse environment that includes mobile phones, augmented reality headsets, virtual reality headsets, and desktop computers, according to some implementations. [Diagram 3] FIG. 3 is an example flow diagram for deploying metaverse content or applications across different modalities within a metaverse environment, according to some implementations. [Figure 4] FIG. 4 is an example flow diagram for providing a spatialized user interface (UI) when deploying augmented reality applications across different modalities in a metaverse environment, according to some implementations. [Diagram 5] FIG. 5 is an example flow diagram for deploying augmented reality applications in a metaverse environment that includes interaction mapping of desktop computers to mobile devices, according to some implementations. [Figure 6] FIG. 6 is an example flow diagram for deploying an augmented reality application in a metaverse environment including headset to mobile device interaction mapping, according to some implementations. [Figure 7] FIG. 7 is a block diagram illustrating a side view of a three-dimensional (3D) ray intersecting an object, such as a table 702, in accordance with some implementations. [Figure 8] FIG. 8 is a block diagram illustrating a perspective view of a 3D ray intersection intersecting an object such as a table, according to some implementations. [Figure 9]FIG. 9 is an example flow diagram for matching appropriate background elements in a 3D scene based on a device type associated with a modality, according to some implementations. [Figure 10] FIG. 10 is an exemplary flow diagram for providing responsive scale to virtual content within a 3D scene of a metaverse environment based on a device type associated with a modality, according to some implementations. [Figure 11] FIG. 11 is a block diagram illustrating a side view of a user in a standing position using an augmented reality (AR) headset looking at a target object on the ground, in accordance with some implementations. [Figure 12] FIG. 12 is a block diagram illustrating a side view of a user in a seated position using an AR headset looking at a target object on the ground, according to some implementations. [Figure 13] FIG. 13 is a block diagram illustrating a side view of a user in a standing position using a mobile device to view a target object on a table, in accordance with some implementations. [Figure 14] FIG. 14 is a block diagram illustrating a perspective view through a viewer 1400 representing views of a target object in different modalities, according to some implementations. [Figure 15] FIG. 15 is a block diagram of an example network environment that may be used in some implementations described herein. [Figure 16] FIG. 16 is a block diagram of an exemplary computer system that may be used in some implementations described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The implementations generally relate to metaverse content modality mapping. In various implementations, the system obtains functionality developed for a primary or first modality of a metaverse environment. The first modality may include a mobile device, such as a smartphone or tablet. The system maps the functionality to an auxiliary or second modality of the metaverse environment. The second modality may include an augmented reality (AR) headset, a virtual reality (VR) headset, or a desktop computer. The system executes the functionality developed for the first modality based on user interactions associated with the second modality.

[0013] The Metaverse Content Modality Mapping implementation provides developers with the tools to create content and applications for the next iteration of the Internet: the Metaverse. The implementation provides a powerful platform that enables developers to take advantage of metaverse deployments, fully optimized to fit a myriad of devices to deliver the right immersive experience every time. This will be beneficial as the Web evolves and becomes more spatial and immersive.

[0014] The implementation enables developers to build web-based augmented reality (WebAR) projects once and deploy them everywhere, including iOS and Android smartphones, tablets, desktop and laptop computers, and virtual reality and augmented reality head-worn devices. The implementation enables WebAR projects to be instantly accessible without additional development time on the devices that are essential to people's daily lives today, as well as the devices that will facilitate life in the metaverse tomorrow.

[0015] The implementations described herein allow end users accessing WebAR world effect experiences to engage with them on mobile devices, AR headsets, VR headsets, and desktop computers. The implementations ensure that the user receives the appropriate experience based on the device they are using, and manage all the mappings necessary for the user to properly view, interact with, and engage with the immersive content no matter what device the user is on.

[0016] The implementation provides the beginning of a new responsive web. Just as 2D websites needed to adapt from desktop to mobile devices, immersive websites also need to react to the different devices used to experience them. The implementation enables developers to build WebAR experiences that respond instantly to the most popular mobile devices, head-worn devices, and desktop computers.

[0017] 1 is a block diagram of an example environment 100 associated with a system for metaverse content modality mapping that may be used in the implementations described herein. In various implementations, the environment 100 includes a system 102, a mobile device 104, an AR headset 106, a VR headset 108, and a desktop computer 110.

[0018] In various implementations, the mobile device 104 may be any suitable smart device having a touch screen user interface (UI). For example, the mobile device 104 may be a smartphone, a tablet, etc. The AR headset 106 and the VR headset 108 may be any suitable headset system having a head-mounted display, such as goggles, glasses, etc., having one or more display screens in front of the user's eyes.

[0019] In various implementations, a desktop computer can be any type of computer system that is typically used on a desk. For example, a desktop computer can be a laptop computer or a traditional desktop computer having a computer chassis, a monitor, a keyboard, and a mouse and / or a trackpad. The actual configuration of a desktop computer can vary depending on the particular implementation. For example, a desktop computer can be a monitor with an integrated computer, keyboard, and mouse and / or a trackpad.

[0020] Mobile device 104, AR headset 106, VR headset 108, and desktop computer 110 may communicate with system 102 and / or with each other directly or via system 102. Network environment 100 also includes a network 112 through which system 102 and client devices 104, 106, 108, and 110 communicate. Network 112 may be any suitable communications network, such as a Bluetooth network, a Wi-Fi network, the Internet, or the like, or a combination thereof.

[0021] As described in more detail herein, the system 102 obtains functionality developed for a primary or first modality of the metaverse environment. In various implementations, the first modality is associated with a web-based AR of a first device type. The first device type may be a mobile device, such as the mobile device 104 (e.g., a smartphone, a tablet, etc.). Although various implementations are described herein in the context of web-based AR, the implementations may also apply to native AR applications on various client devices. The system maps functionality to a secondary or second modality of the metaverse environment. The system executes functionality developed for the first modality based on user interactions associated with the second modality. In various implementations, the second modality is associated with a second device type that is different from the first device type. As described in more detail herein, the second device type may be one of several types of devices. For example, in some implementations, the second device type may be an AR headset, such as the AR headset 106. In some implementations, the second device type is a VR headset, such as VR headset 108. In some implementations, the second device type is a desktop computer, such as desktop computer 110.

[0022] The first and second modalities may also be referred to as interaction modalities in that they provide different modalities of interaction to users of different types of devices. For example, a user may interact with a mobile device 104 via a touch screen. A user may interact with an AR headset 106 or a VR headset 108 via a handheld controller. A user may interact with a desktop computer 110 via a keyboard, trackpad, and / or mouse (not shown). Each of these devices may be considered a different modality or interaction modality.

[0023] For ease of illustration, a first modality refers to a primary modality that includes a mobile device such as a smartphone or tablet, and a second modality refers to a secondary modality that includes one or more AR headsets, one or more VR headsets, or one or more desktop computers, or a combination thereof. Such interoperability between the first and second modalities enables implementation of the universal or metaverse deployments described herein.

[0024] For ease of illustration, Figure 1 shows one block for each of the system 102, the mobile device 104, the AR headset 106, the VR headset 108, and the desktop computer 110. Blocks 102-110 may represent multiple systems, server devices, databases, mobile devices, AR headsets, VR headsets, and desktop computers. In other implementations, the environment 100 may not have all of the components shown and / or may have other elements, including other types of elements, instead of or in addition to the elements shown herein.

[0025] Although the system 102 performs the implementations described herein, in other implementations, any suitable component or combination of components associated with the system 102, or any suitable processor or processors associated with the system 102, may facilitate performing the implementations described herein.

[0026] 2 is an exemplary flow diagram for adapting a web-based augmented reality application for deployment in a metaverse environment including mobile phones, augmented reality headsets, virtual reality headsets, and desktop computers, according to some implementations. With reference to FIGS. 1 and 2, the method begins at block 202, where a system such as system 102 obtains functionality developed for a primary or first modality of the metaverse environment. As indicated above, in various implementations, the first modality is associated with a web-based AR for a first device type, which may be a mobile device such as mobile device 104 (e.g., a smartphone, tablet, etc.). Also, as noted above, although various implementations are described herein in the context of web-based AR, the implementations may also be applied to native AR applications on various client devices.

[0027] In block 204, the system maps the functionality to an auxiliary or second modality of the metaverse environment. As indicated above, in various implementations, the second modality is associated with a second device type that is different from the first device type. As described in more detail herein, the second device type may be one of several types of devices. For example, in various implementations, the second device type may be an AR headset, such as AR headset 106, a VR headset, such as VR headset 108, or a desktop computer, such as desktop computer 110. The implementations described herein may be applied to any of these types of client devices, or a combination thereof.

[0028] At block 206, the system executes functionality developed for the first modality based on user interactions associated with the second modality. Various example implementations including a system that executes functionality developed for the first modality based on user interactions associated with the second modality are described in more detail herein.

[0029] Although steps, operations, or computations may be presented in a particular order, the order may be changed in a particular implementation. Other orderings of steps are possible, depending on the particular implementation. In some particular implementations, multiple steps shown in succession herein may be performed simultaneously. Also, some implementations may not have all of the steps shown and / or may have other steps instead of or in addition to the steps shown herein.

[0030] FIG. 3 is an example flow diagram for deploying metaverse content or applications across different modalities in a metaverse environment, according to some implementations. With reference to FIGS. 1 and 3, the method begins at block 302, where a system, such as system 102, determines a device associated with a second modality. The system makes this determination at run-time, when a connection between the system and a client device is established. Once a device is detected, the system determines the type of device and its associated runtime environment or second modality (e.g., AR headset, VR headset, desktop computer, etc.). Based on the runtime environment, the system loads the correct interaction mapping library for a given modality. The interaction mapping library includes software modules available for the modality.

[0031] In block 304, the system launches the appropriate software module associated with the second modality. The system also verifies that the software module to be executed is activated by default. For example, if an immersive modality including an AR headset or a VR headset is used, the system ensures that the necessary software modules, as well as associated drivers and application programming interfaces (APIs), are available in the browser. In various implementations, the system determines whether each software module and associated driver can power and / or run on a particular modality based on the device type (e.g., AR headset, VR headset, desktop computer, etc.).

[0032] The system also ensures that software modules that are not supported by the device of the second modality are disabled. For example, some code or tangential code, such as face effects, may not execute properly on a particular type of device, such as a device that does not have a camera. Thus, the system may disable the software module or part of the software module associated with the code. The system may suggest using a particular modality instead if necessary.

[0033] In block 306, the system adapts user interactions with the second modality device (e.g., AR headset, VR headset, desktop computer, etc.) to the capabilities developed for the first modality mobile device (e.g., smartphone, tablet device, etc.). In various implementations, appropriate activated software modules and associated drivers provide startup procedures, frame-by-frame updates, and shutdown procedures for the second modality device. For example, in a mobile phone AR, the system uses appropriate software modules to start the camera, provide frame updates, and stop the camera at the end of the session. In a VR or AR headset, the system uses appropriate software modules to start a VR or AR session, provide and update head pose and controller data, and stop the VR or AR session when finished. In a desktop computer, the system uses appropriate software modules to display 3D content, map the second modality keyboard and / or trackpad and / or mouse to the first modality 2D touch, and hide the 3D content at the end of the session.

[0034] In various implementations, the system utilizes software modules developed using WebAssembly and Web Graphics Library (WebGL), and Javascript APIs adapted at runtime for each unique device type. The implementations provide an intelligent wrapper that uses a camera application framework to provide a best-in-class mobile WebAR experience, but properly integrates with WebXR APIs to optimize WebAR projects for non-mobile devices, such as AR or VR headsets, or computers. As described in various implementations herein, the implementations optimize WebAR projects by selecting the appropriate combination of techniques to execute the experience, provide UX compatibility mapping via modality-specific mechanisms, build or hide virtual environments, and spatialize 2D interfaces.

[0035] Although steps, operations, or computations may be presented in a particular order, the order may be changed in a particular implementation. Other orderings of steps are possible, depending on the particular implementation. In some particular implementations, multiple steps shown in succession herein may be performed simultaneously. Also, some implementations may not have all of the steps shown and / or may have other steps instead of or in addition to the steps shown herein.

[0036] 4 is an example flow diagram for providing a spatialized UI when deploying an augmented reality application across different modalities in a metaverse environment, according to some implementations. In various implementations, the system maps one or more user gestures associated with the spatialized UI in the 3D scene of the second modality to one or more 2D UI elements associated with the first modality. As described in more detail below, the system translates interactive 2D UI elements originally designed for a mobile device of the first modality onto a spatial control panel of the second modality when the virtual experience is accessed on an AR or VR headset.

[0037] 1 and 4, the method begins at block 402, where a system, such as system 102, displays mobile 2D UI elements of a first modality in a 3D scene of a second modality. In some implementations, the system may render 2D UI elements of the first modality in a second UI of the second modality. For example, the system may determine which 2D UI elements (e.g., buttons, etc.) are drawn on a 2D screen. The system may then capture the 2D UI elements and convert the 2D UI elements into corresponding 3D UI element versions that the system displays in the 3D scene. Such 3D UI elements are functional in that a user may select and interact with 3D UI elements in the second modality just as if the user were selecting and interacting with 2D UI elements in the first modality.

[0038] In some implementations, the system may recreate the 2D UI elements as 3D UI elements directly in the 3D scene of the second modality. In some scenarios, converting from the intended 2D UI elements may result in the corresponding 3D UI elements being distorted in the 3D environment. Therefore, in some implementations, the system may also create an optimized layout of the corresponding 3D UI elements in the 3D scene of the second modality.

[0039] In block 404, the system determines user interactions with 3D UI elements in the 3D scene. These user interactions may be referred to as trigger events. In various implementations, the system determines the location of the 3D UI elements in the 3D scene with which the user interacts. The system also determines the type of user interaction with each 3D UI element and the information (e.g., vectors, values, etc.) associated with each user interaction. For example, the system may detect virtual control selection and dragging of virtual objects, such as when a user selects and manipulates a given virtual object. In another example, the system may determine when a user selects a given virtual object and modifies the virtual object (e.g., changes size, color, etc.). The system may also determine user interactions with various other types of virtual controls (e.g., clicking a button to exit an AR or VR environment, etc.).

[0040] In block 406, the system maps the location of the 3D UI element that the user interacts with in the second modality to the corresponding location of the 2D UI element of the first modality, or directly to the 2D UI element. For example, if the user clicks on a given virtual control (e.g., a button, etc.) on a screen in the 3D scene of the second modality, the system maps the triggering event to a "tap" at the corresponding location of the virtual control (e.g., a button, etc.) on the 2D screen of the first modality (e.g., on the 2D screen of a mobile device), or directly to the rendered corresponding control object on the 2D screen. In other words, the system converts a triggering event gesture, such as a button click of the second modality, into a button tap of the first modality. In this exemplary scenario, the system may detect and identify a user click on a virtual control based on the user physically pressing a button on a handheld controller in an AR or VR headset scenario, where the position of the handheld controller is mapped to a rendered virtual button in the 3D scene. The system then traces or propagates the trigger event of the second modality to the corresponding 2D UI element of the first modality.

[0041] In block 408, the system executes the corresponding command associated with the 2D UI element of the first modality.

[0042] At block 410, the system updates frames in the 3D scene of the second modality based on updates to the 2D scene of the first modality. As a result, the user seamlessly experiences interactions in the 3D scene of the second modality while actions are being performed in the first modality.

[0043] While 3D content is present in many AR experiences, the majority of WebAR projects also contain numerous 2D UI elements. As exemplified above, 2D elements such as buttons or text help facilitate user interaction. These 2D elements are ideal for flat screens such as smartphones, tablets, and desktop computers. The implementations described herein provide special consideration when available for AR and VR headsets where the experience becomes spatial. The 3D elements on the virtual space control panel with the 3D UI elements in the 3D scene may be referred to as a Document Object Model (DOM) tablet. In various implementations, the DOM tablet facilitates user interaction, such as engaging with buttons that affect the 3D scene. In some implementations, the system may allow the DOM tablet to be repositioned by the user or minimized on the user's wrist when not needed so as not to interfere with the user's immersive experience.

[0044] Although steps, operations, or computations may be presented in a particular order, the order may be changed in a particular implementation. Other orderings of steps are possible, depending on the particular implementation. In some particular implementations, multiple steps shown in succession herein may be performed simultaneously. Also, some implementations may not have all of the steps shown and / or may have other steps instead of or in addition to the steps shown herein.

[0045] 5 is an example flow diagram for deploying an augmented reality application in a metaverse environment including desktop computer to mobile device interaction mapping according to some implementations. As described in more detail below, in various implementations, the system maps user interactions with one or more input devices on a desktop computer associated with a second modality to one or more 2D UI elements on a mobile device associated with a first modality.

[0046] 1 and 5, the method begins at block 502, where a system, such as system 102, receives input signals from one or more input devices to a desktop computer. As indicated above, in various implementations, the desktop computer may be any type of computer system typically used on a desk, such as a laptop computer or a traditional desktop computer having a computer chassis, a monitor, a keyboard, and a mouse and / or a trackpad, etc. In various implementations, the input device is a mechanical input device, and the particular input device type may vary depending on the particular implementation. For example, the input device may include a keyboard, a mouse, a trackpad, a joystick, a game controller, etc.

[0047] At block 504, the system maps each input signal of the desktop computer to a corresponding 2D UI element of the mobile device. As shown herein, the mobile device is the primary or first modality in the metaverse environment, and the desktop computer is the secondary or second modality in the metaverse environment. An exemplary input signal to 2D UI element mapping may be a point, click, and drag gesture or scroll gesture on an object shown on the monitor of the desktop computer that is mapped to a pinch gesture or other multi-touch gesture to the same object on the mobile phone. The specific combination of input signals and the mapping to the corresponding 2D UI elements may vary depending on the specific implementation. In another example, the input signal may be based on a scroll gesture to change the scale of an object shown on the monitor of the desktop computer, and the system maps the scroll gesture to a multi-touch pinch gesture to the same object on the mobile phone. In another example, the input signal may be based on an option selected via a keyboard and a click-and-drag gesture to drag an object shown on the monitor of the desktop computer, and the system maps this set of signals to a two-finger touch gesture to the same object on the mobile phone.

[0048] Although some implementations are described in the context of 2D UI elements shown on a mobile device screen, these implementations may also be applied to the movement of a mobile phone. For example, the input signals may be based on right-clicking and dragging an object shown on a desktop computer monitor to rotate the object, and the system maps this set of signals to the movement of the mobile phone, where the movement of the phone corresponds to the rotation of the object or the movement around the object. In another example, the input signals may be based on pressing arrow keys on a keyboard for an xy translation of an object shown on a desktop computer monitor, and the system maps this set of signals to an xy translation of the object on the screen of the mobile phone.

[0049] At block 506, the system executes a command associated with each 2D UI element of the mobile device. In various implementations, the execution of the command results in various manipulations of one or more target objects on the screen of the mobile device, such as those described in the previous example. As a result, the system converts an input signal from an input device of the second modality (e.g., a desktop computer) into a 2D UI element on the first modality (e.g., a mobile device).

[0050] Although steps, operations, or computations may be presented in a particular order, the order may be changed in a particular implementation. Other orderings of steps are possible, depending on the particular implementation. In some particular implementations, multiple steps shown in succession herein may be performed simultaneously. Also, some implementations may not have all of the steps shown and / or may have other steps instead of or in addition to the steps shown herein.

[0051] 6 is an example flow diagram for deploying an augmented reality application in a metaverse environment including headset to mobile device interaction mapping according to some implementations. As described in more detail below, in various implementations, the system maps user interactions with one or more input devices on a headset associated with a second modality to one or more 2D UI elements on a mobile device associated with a first modality.

[0052] 1 and 6, the method begins at block 602, where a system, such as system 102, receives an input signal from one or more input devices to a headset. In various implementations, the headset may be an AR or VR headset, such as AR headset 106 or VR headset 108 of FIG. 1. As indicated above, the AR or VR headset may be any suitable headset system having a head-mounted display, such as goggles, glasses, etc., having one or more display screens in front of the user's eyes. In various implementations, the input device may include a mechanical input device, and the particular input device type may vary depending on the particular implementation. For example, the input device may include a game controller, an eye tracking device, etc. Exemplary input signals may include detection and tracking of ray intersections associated with user interaction of a headset and a handheld controller.

[0053] At block 604, the system maps each input signal to the headset to a corresponding 2D UI element on the mobile device. The input signals to the headset may include, for example, control signals from a handheld controller, input from an eye gaze tracker, 3D ray intersection data, etc. As illustrated herein, the mobile device is a first modality in the metaverse environment and the headset is a second modality in the metaverse environment. In some implementations, the system may map detection and tracking of 3D ray intersections associated with the headset and user interactions with the handheld controller to 2D touch coordinates on the mobile device.

[0054] At block 606, the system executes a command associated with each 2D UI element of the mobile device. In various implementations, the execution of the command results in various manipulations of one or more target objects on the screen of the mobile device, such as those described in the previous example. As a result, the system converts an input signal from an input device of the second modality (e.g., a headset) into a 2D UI element on the first modality (e.g., the mobile device).

[0055] Although steps, operations, or computations may be presented in a particular order, the order may be changed in a particular implementation. Other orderings of steps are possible, depending on the particular implementation. In some particular implementations, multiple steps shown in succession herein may be performed simultaneously. Also, some implementations may not have all of the steps shown and / or may have other steps instead of or in addition to the steps shown herein.

[0056] FIG. 7 is a block diagram illustrating a side view 700 of a 3D ray intersecting an object, such as a table 702, according to some implementations. Shown are the table 702, a 3D ray 704 associated with an AR headset 706, and a handheld controller 708. In some implementations, in the context of the AR headset 706, the system may determine and track the 3D ray 704 based on a camera or pair of cameras of the AR headset 706 that captures a view of the real world. The camera or cameras correspond to the user's eyes (not shown). In some implementations, the 3D ray 704 intersects the table 702 at a 3D ray intersection 710. In some implementations, the 3D ray 704 may also be based on a camera or pair of cameras of the AR headset 706 that captures the line of sight (e.g., line of sight) of the user's eyes at the table 702. The actual techniques for determining the direction of the 3D ray 704 and the location of the 3D ray intersection 710 may vary depending on the particular implementation. In some implementations, the system may display a 3D ray intersection 710 of the AR headset 706, as described in more detail below in connection with FIG. 8.

[0057] 8 is a block diagram illustrating a perspective view 800 of a 3D ray intersection 710 intersecting an object, such as the table 702 of FIG. 7, according to some implementations. Shown are the table 702, the 3D ray intersection 710, and the 3D ray intersection 802. In various implementations, the perspective view 800 may represent the visual view a user sees from an AR headset 706 (shown in FIG. 7). As shown, the system may display the 3D ray intersection 710 in the visual view of a viewer in the AR headset 706.

[0058] In a scenario where either the user's AR headset and / or line of sight (e.g., gaze) moves laterally such that the 3D ray moves laterally, the 3D ray intersection shifts from 3D ray intersection 710 on the table 702 to 3D ray intersection 802, which is on the ground beside the table. In various implementations, the system smooths the movement of the 3D ray intersection to maintain smooth continuity from the transition of the 3D ray intersection from the table to the floor or ground. In some implementations, the system determines a reference or anchor point at 3D ray intersection 802 where the 3D ray first intersects the table 702. The system continues to smoothly update the 3D ray intersection as it is dragged laterally, thereby preventing the 3D ray intersection from rapidly moving from the table 702 to the floor.

[0059] In various implementations, the system determines a user selection of a table 702 in the 3D scene of the second modality in combination with user interaction of the handheld controller, and the system maps the combination of these input signals to a 2D UI element (e.g., 2D coordinates) of the first modality. For example, the system may allow a user to select an object such as the table 702 by clicking the handheld controller 708 when the 3D ray intersection is on the table 702. This allows a user to point to a particular object such as the table 702 based on the 3D ray intersection 710. The system may lock the 3D ray intersection 710 on the table 702 as long as the user maintains the selection (e.g., holds down a button on the handheld controller 708). If the table 702 is a virtual object, the system may allow the user to move and drag around the table 702 in the 3D scene while the table 702 is selected.

[0060] In some implementations, in the context of a VR headset (not shown), the 3D ray intersection may be based on a camera or pair of cameras in the VR headset that captures the line of sight of the user's eyes (e.g., gaze) of an object in the virtual 3D scene. The camera or cameras that track the user's gaze also correspond to the user's eyes.

[0061] In both scenarios, whether the headset is an AR headset 707 or a VR headset (not shown), the system maps 3D ray intersections of the 3D scene in the second modality to 2D touches on the screen of the mobile phone in the first modality. The system also allows the user to select a given object and manipulate the object based on user interaction with the handheld controller. In other words, user interaction with the headset and controller results in a "tap" on a target point in 3D space (e.g., 3D ray intersection 710), which is translated into a tap on the target point on the touch screen of the mobile device.

[0062] Referring again to FIG. 6, in block 606, the system executes commands associated with each 2D UI element of the mobile device. In various implementations, the execution of the commands results in various manipulations (e.g., touching, tapping, dragging, etc.) of one or more target objects on the screen of the mobile device, such as those described in the previous examples. As a result, the system translates input signals from an input device of the second modality (e.g., an AR or VR headset) into 2D UI elements on the first modality (e.g., the mobile device). Thus, the system translates functions (e.g., button selection, etc.) of the headset and handheld controller into touch and drag gestures on the mobile device.

[0063] Traditionally, to engage with AR on a mobile device, a user is often required to perform several gestures, such as taps, pinches, and swipes, on the screen of the mobile device. An advantage of the implementations described herein, including metaverse deployment, is that they make AR experiences available on non-mobile devices, based in part on these new device category or second modality interaction mappings described herein. To achieve these advantages, the implementations spatialize mobile WebAR touch inputs by mapping them to a multitude of input options available across AR and VR headsets, desktop computers, and associated input devices, including keyboards, touchpads, mice, controllers, hand tracking, and the like. As the device the user is on is identified at runtime, the implementations handle the interaction mapping to provide the appropriate interaction for the user, allowing the user to intuitively interact with the 3D content.

[0064] 9 is an example flow diagram for adapting appropriate background elements in a 3D scene based on a device type associated with a modality, according to some implementations. As described in more detail below, the system detects the modality used and whether the experience has an appropriate environment in the 3D scene. The system performs environment mapping to enhance the metaverse experience across different modalities. In various implementations, the system adapts one or more background elements in a 3D scene associated with a second modality based on a device type associated with the second modality.

[0065] In various implementations, the method begins at block 902, where a system, such as system 102 of FIG. 1, determines whether a given modality displays a real environment or a virtual scene environment. For example, an AR headset and a mobile device with a camera displays a real environment but not a virtual scene environment. A VR headset and a desktop computer displays a virtual scene environment but not a real environment.

[0066] In block 904, the system displays an appropriate background based on the deployed or used target device. As shown above, the AR headset and the mobile device with the camera display the real environment, but do not display the virtual scene environment. In the scenario where the AR headset or the mobile device is used, the system may simply display the real environment captured by the camera of the respective device. If the AR headset or the mobile device displays the scene environment, the system may simply remove the virtual scene environment so that the real environment is fully visible, since the virtual scene environment is not needed.

[0067] As shown above, the VR headset and desktop computer display the virtual scene environment, but not the real environment. In a scenario where a VR headset or desktop computer is used, the system may continue to display the virtual environment. If the VR headset or desktop computer is not yet displaying the virtual environment, the system adds appropriate background elements to the 3D scene. For example, the system may generate and display a floor or ground to provide a sense of relative vertical depth between a given virtual object (e.g., a virtual car) and the floor or ground. Without a floor, a virtual object such as a materialized car has no sense of scale and appears to float in a blank space. In another example, the system may generate and display fog on the horizon to provide a sense of horizontal or lateral depth between the virtual object and the horizon.

[0068] In various implementations, the system may add patterns to the floor. This allows the user to see the movement of an object, such as a car, as the object moves in the 3D scene. A floor pattern that appears to shift would indicate to the user that an object is moving across the floor. The system may provide a pattern or background, such as fog on the horizon, as a default to any added floor. The fog may prevent the user from seeing infinite distances and provide a sense of distance. In some implementations, the system may also provide scene elements, such as trees or mountains, by default to provide a sense of scale and distance. In some implementations, the system may add color variations to further provide a sense of depth. For example, the system may make more distant parts of the floor or more distant objects a different color (e.g., a more blue-gray hue, etc.).

[0069] In various implementations, the system allows the developer or user to add a custom background with any desired elements with colors, patterns, and various shapes, including ground or terrain, buildings, trees, clouds, etc., to enhance the overall experience. The system may also allow the developer or user to provide walls to place the 3D scene indoors. Thus, the 3D scene canvas may be outdoor or indoor, or may include both outdoor and indoor environments.

[0070] Although steps, operations, or computations may be presented in a particular order, the order may be changed in a particular implementation. Other orderings of steps are possible, depending on the particular implementation. In some particular implementations, multiple steps shown in succession herein may be performed simultaneously. Also, some implementations may not have all of the steps shown and / or may have other steps instead of or in addition to the steps shown herein.

[0071] 10 is an example flow diagram for providing responsive scaling to virtual content in a 3D scene of a metaverse environment based on a device type associated with a modality, according to some implementations. As described in more detail below, the system adapts target objects in the 3D scene to accommodate the user's location by making the 3D content comfortable and accessible, while ensuring that the developer's vision of the 3D scene is consistent across devices of different modalities.

[0072] In various implementations, the method begins at block 1002, where a system, such as system 102 of FIG. 1, determines a second modality device type being used in the metaverse environment. For example, the system may determine whether the device is an AR headset, a VR headset, a desktop computer, etc.

[0073] In block 1004, the system identifies a target object within the 3D scene displayed by the device.

[0074] In block 1006, the system determines the starting height of the viewer of the device. In some implementations, the starting height may be the height of the camera relative to the ground. The starting height represents the user having a visual view of the target object when the user begins to look at the target object through the viewer.

[0075] In block 1008, the system adapts one target object in the 3D scene associated with the second modality based on the device type associated with the second modality. For example, the system scales the target object up or down to meet consistent visual angles and visual comfort across devices of different modalities. This allows the starting position of the target object in the 3D scene to maintain the same or nearly the same size or scale across headsets, desktops, and mobile phones, and scenarios. In some implementations, as an alternative to scaling the object in the scene, other implementations may change the height of the virtual camera and scale subsequent camera movements accordingly. In various implementations, the system may also adjust the viewing angle of the target object to make it appear the same or similar on different devices or modalities or scenarios. Thus, the implementation ensures that the content of the 3D scene, including the target object, is visually comfortable and accessible across devices of different modalities and user scenarios. Also, the user does not need to navigate the scene to have a comfortable view.

[0076] Although steps, operations, or computations may be presented in a particular order, the order may be changed in a particular implementation. Other orderings of steps are possible, depending on the particular implementation. In some particular implementations, multiple steps shown in succession herein may be performed simultaneously. Also, some implementations may not have all of the steps shown and / or may have other steps instead of or in addition to the steps shown herein.

[0077] Figures 11, 12, and 13 show different views of a target object viewed at different heights and with different modalities.

[0078] 11 is a block diagram illustrating a side view 1100 of a user standing and using an augmented reality headset to view a target object on a ground surface, according to some implementations. Shown are an AR headset 1102, a target object 1104, and a ground surface 1106. In this scenario, the user is standing and looking toward the ground surface 1106, and the target object 1104 is a virtual car. In various implementations, the system adapts the target object 1104 so that the user views the entire target object 1104 through the AR headset 1102.

[0079] 12 is a block diagram illustrating a side view 1200 of a user in a seated position using an AR headset and looking at a target object on a ground surface, according to some implementations. Shown is an AR headset 1202, a target object 1204, which is a car, and a ground surface 1206. In this scenario, the user is seated and looking toward the ground surface 1206. The user may be seated in a chair, a wheelchair, etc. Also, the target object 1204 is a virtual car. In various implementations, the system adapts the target object 1204 so that the user sees the entire target object 1204 through the AR headset 1202. This provides accessibility to a fully immersive experience for users with limited mobility.

[0080] 13 is a block diagram illustrating a side view 1300 of a user in a standing position looking at a target object on a table using a mobile device, according to some implementations. Shown are a mobile device 1302, a target object 1304, which is a car, and a table 1306. In this scenario, the user is standing and looking towards the table 1306, and the target object 1304 is a virtual car. In various implementations, the system adapts the target object 1304 so that the user sees the entire target object 1304 through the mobile device 1302.

[0081] FIG. 14 is a block diagram showing a perspective view through a viewer 1400 representing a view of a target object in different modalities according to some implementations. A target object 1402, which is a car, is shown as being viewed within a display 1404 of the viewer. In various implementations, the target object 1402 is shown to be in a particular portion of the display where the entire target object 1402 is visible. As shown above, in various implementations, the system may also adjust the viewing angle of the target object to make it appear the same or similar in different devices or modalities or scenarios. This view of the target object 1402 may represent the view of the target object in any of the scenarios of FIG. 11, FIG. 12 and FIG. 13 based on an adjustment to the scale of the target object 1402 for the type of device being used. As a result, the target object covers the same amount of the field of view in different scenarios and modalities.

[0082] The implementation is therefore beneficial in that it takes into account not only the type of device, but also the user's position while engaged in the 3D experience. This applies whether the user is standing or sitting in the virtual reality experience. The implementation dynamically adjusts the viewing height to ensure that all content seen is comfortable and accessible regardless of the device the user is using. The implementation achieves these benefits while respecting the perspective of the initial view so as to increase confidence that the user is seeing the content as intended by the developer.

[0083] The implementation has various other benefits. For example, the implementation eliminates or minimizes much of the work from cross-platform development, but the implementation includes powerful mechanisms as part of the platform that aid in customization of 3D experiences per device category. The implementation fully supports WebAR world effects created using three.js and A-Frame. The implementation's metaverse deployment capabilities are also optimized for iOS and Android smartphones and tablets, desktops and laptops, and various AR and VR headset systems. The implementation enables developers to create a variety of WebAR world effects, face effects, and image targeting experiences specifically for mobile.

[0084] The implementation makes the web a powerful home for smartphone-based AR reality, giving developers access to billions of smartphones across iOS and Android devices, the broadest reach of any augmented reality platform. The implementation unlocks more places to access and engage with immersive content, greatly expanding this reach without extending development time. The implementation of metaverse deployment allows developers to create WebAR projects that automatically adapt from mobile devices to computers and headsets.

[0085] 15 is a block diagram of an exemplary network environment 1500, which may be used in some implementations described herein. In some implementations, the network environment 1500 includes a system 1502 including a server device 1504 and a database 1506. For example, the system 1502 may be used to implement the system 102 of FIG. 1 and to perform the implementations described herein. The network environment 1500 also includes client devices 1510, 1520, 1530, and 1540 that may communicate with the system 1502 and / or communicate directly or via the system 1502 with each other. The network environment 1500 also includes a network 1550 over which the system 1502 and the client devices 1510, 1520, 1530, and 1540 communicate. The network 1550 may be any suitable communication network, such as a Wi-Fi network, a Bluetooth network, the Internet, etc.

[0086] For ease of illustration, FIG. 15 shows one block for each of the system 1502, server device 1504, and network database 1506, and four blocks for client devices 1510, 1520, 1530, and 1540. Blocks 1502, 1504, and 1506 may represent multiple systems, server devices, and network databases. Also, there may be any number of client devices. In other implementations, the environment 1500 may have other elements, including other types of elements, instead of or in addition to the elements shown herein.

[0087] The server device 1504 of the system 1502 performs the implementations described herein, although in other implementations, any suitable component or combination of components associated with the system 1502, or any suitable processor or processors associated with the system 1502, may facilitate performing the implementations described herein.

[0088] In various implementations described herein, the processor of the system 1502 and / or the processor of any of the client devices 1510, 1520, 1530, and 1540 causes the elements (e.g., information, etc.) described herein to be displayed in a user interface on one or more display screens.

[0089] FIG. 16 is a block diagram of an exemplary computer system 1600, which may be used in some implementations described herein. For example, computer system 1600 may be used to implement server device 1504 of FIG. 15 and / or system 102 of FIG. 1, as well as to execute implementations described herein. In some implementations, computer system 1600 may include a processor 1602, an operating system 1604, a memory 1606, and an input / output (I / O) interface 1608. In various implementations, processor 1602 may be used to implement various functions and features described herein, as well as to execute implementations of methods described herein. Although processor 1602 is described as executing implementations described herein, any suitable component or combination of components associated with computer system 1600, or any suitable processor or processors associated with computer system 1600 or any suitable system, may execute the steps described. The implementations described herein may be executed on a user device, a server, or a combination of both.

[0090] Computer system 1600 also includes software applications 1610, which may be stored in memory 1606 or any other suitable storage location or computer-readable medium. The software applications 1610 provide instructions that enable processor 1602 to perform the implementations and other functions described herein. The software applications may also include engines, such as a network engine, for performing various functions associated with one or more networks and network communications. The components of computer system 1600 may be implemented by one or more processors or any combination of hardware devices, as well as any combination of hardware, software, firmware, etc.

[0091] For ease of illustration, Figure 16 shows one block for each of processor 1602, operating system 1604, memory 1606, I / O interfaces 1608, and software applications 1610. These blocks 1602, 1604, 1606, 1608, and 1610 may represent multiple processors, operating systems, memories, I / O interfaces, and software applications. In various implementations, computer system 1600 may have other elements, including other types of components, instead of or in addition to the components shown herein.

[0092] Although the description has been given with respect to specific implementations thereof, these specific implementations are merely exemplary and not limiting, and the concepts illustrated in the embodiments may be applied to other embodiments and implementations.

[0093] In various implementations, the software is encoded on one or more non-transitory computer-readable media for execution by one or more processors, the software being operable to perform the implementations and other functions described herein when executed by the one or more processors.

[0094] Any suitable programming language may be used to implement the routines of a particular implementation, including C, C++, C#, Java, JavaScript, assembly language, etc. Different programming techniques may be used, such as procedural or object-oriented. The routines may be executed on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a particular order, this order may be changed in different particular implementations. In some particular implementations, multiple steps shown as successive in this specification may be executed simultaneously.

[0095] Certain implementations may be implemented in a non-transitory computer-readable storage medium (also referred to as a machine-readable storage medium) for use by or in connection with an instruction execution system, apparatus, or device. Certain implementations may be implemented in the form of control logic in software or hardware or a combination of both. The control logic, when executed by one or more processors, is operable to perform the implementations and other functions described herein. For example, a tangible medium such as a hardware storage device may be used to store the control logic, which may include executable instructions.

[0096] A "processor" may include any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor may include a general-purpose central processing unit, multiple processing units, dedicated circuits to accomplish a function, or other systems. Processing need not be limited to a geographic location, nor need it be time-limited. For example, a processor may perform its functions in "real-time," "offline," "batch mode," etc. Portions of the processing may be performed by different (or the same) processing systems at different times and in different locations. A computer may be any processor in communication with a memory. Memory may be any suitable data storage device, memory, and / or non-transitory computer-readable storage medium, including electronic storage such as random access memory (RAM), read-only memory (ROM), magnetic storage (such as hard disk drives), flash, optical storage (such as CDs, DVDs), magnetic or optical disks, or other tangible media suitable for storing instructions (e.g., program or software instructions) for execution by a processor. For example, tangible media such as hardware storage devices may be used to store control logic that may include executable instructions. The instructions may also be contained in or provided as electronic signals, for example, in the form of Software as a Service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system).

[0097] It will also be understood that one or more of the elements shown in the drawings / figures may be implemented in a more separated or integrated manner, or may be removed or rendered inoperative in certain cases, as may be useful depending on the particular application. It is also within the spirit and scope to implement a program or code that may be stored on a machine-readable medium to enable a computer to perform any of the methods described above.

[0098] As used throughout this description and the claims that follow, "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Also, as used throughout this description and the claims that follow, the meaning of "in" includes "in" and "on," unless the context clearly dictates otherwise.

[0099] Thus, while specific implementations have been described herein, it will be understood that a breadth of modification, various changes, and substitutions are contemplated in the foregoing disclosure, and that in some instances, some features of a specific implementation may be employed without a corresponding use of other features without departing from the scope and spirit of what is described. Therefore, many modifications may be made to adapt a particular situation or material to the essential scope and spirit.

Claims

1. one or more processors; Logic encoded in one or more non-transitory computer-readable storage media for execution by the one or more processors, when executed, Obtaining functionality developed for a first modality of a virtual environment, the first modality including displaying the virtual environment on a two-dimensional (2D) display device, the functionality including 2D control of the 2D display device for interacting with the virtual environment; mapping the functionality to a second modality of the virtual environment, the second modality comprising a display of the virtual environment on a three-dimensional (3D) display device, the mapping comprising linking the 2D controls of the first modality to 3D input gestures of the second modality; receiving an indication from a client device that a user of the client device has made the 3D input gesture while interacting with the virtual environment via the second modality; executing the functionality developed for the first modality in response to the indication that the user has made the 3D input gesture while interacting with the virtual environment via the second modality; logic operable to perform operations including: A system comprising:

2. the first modality is associated with augmented reality for a first device type, the first device type being a mobile device, and the second modality is associated with a second device type, the second device type being one of an augmented reality headset, a virtual reality headset, or a desktop computer; The system of claim 1 .

3. The logic, when executed, causes the one or more processors to: determining a second client device associated with the second modality; launching a software module associated with the second modality; Adapting user interactions with the second client device to the functionality developed for the first modality; and further operable to cause the device to perform operations including: The system of claim 1 .

4. The 3D input gestures include one or more user gestures within a three-dimensional scene associated with the second modality. The system of claim 1 .

5. the logic, when executed, is further operable to cause the one or more processors to perform operations including mapping user interactions with one or more input devices associated with the second modality to one or more two-dimensional user interface elements associated with the first modality; The system of claim 1 .

6. the logic, when executed, is further operable to cause the one or more processors to perform operations including adapting one or more background elements in a three-dimensional scene associated with the second modality based on a device type associated with the second modality; The system of claim 1 .

7. the logic, when executed, is further operable to cause the one or more processors to perform operations including adapting at least one target object in a three-dimensional scene associated with the second modality based on a device type associated with the second modality; The system of claim 1 .

8. A non-transitory computer-readable storage medium bearing program instructions, comprising: The program instructions, when executed by one or more processors, Obtaining functionality developed for a first modality of a virtual environment, the first modality including displaying the virtual environment on a two-dimensional (2D) display device, the functionality including 2D control of the 2D display device for interacting with the virtual environment; mapping the functionality to a second modality of the virtual environment, the second modality comprising a display of the virtual environment on a three-dimensional (3D) display device, the mapping comprising linking the 2D controls of the first modality to 3D input gestures of the second modality; receiving an indication from a client device that a user of the client device has made the 3D input gesture while interacting with the virtual environment via the second modality; executing the functionality developed for the first modality in response to the indication that the user has made the 3D input gesture while interacting with the virtual environment via the second modality; and operable to perform operations including: A non-transitory computer-readable storage medium.

9. the first modality is associated with augmented reality for a first device type, the first device type being a mobile device, and the second modality is associated with a second device type, the second device type being one of an augmented reality headset, a virtual reality headset, or a desktop computer; The non-transitory computer-readable storage medium of claim 8.

10. The program instructions, when executed, cause the one or more processors to: determining a second client device associated with the second modality; launching a software module associated with the second modality; Adapting user interactions with the second client device to the functionality developed for the first modality; and further operable to cause the device to perform operations including: The non-transitory computer-readable storage medium of claim 8.

11. The program instructions, when executed, are further operable to cause the one or more processors to perform operations including mapping one or more user gestures within a three-dimensional scene associated with the second modality to one or more two-dimensional user interface elements associated with the first modality. The non-transitory computer-readable storage medium of claim 8.

12. The program instructions, when executed, are further operable to cause the one or more processors to perform operations including mapping user interactions with one or more input devices associated with the second modality to one or more two-dimensional user interface elements associated with the first modality. The non-transitory computer-readable storage medium of claim 8.

13. The program instructions, when executed, are further operable to cause the one or more processors to perform operations including adapting one or more background elements in a three-dimensional scene associated with the second modality based on a device type associated with the second modality. The non-transitory computer-readable storage medium of claim 8.

14. The program instructions, when executed, are further operable to cause the one or more processors to perform operations including adapting at least one target object in a three-dimensional scene associated with the second modality based on a device type associated with the second modality. The non-transitory computer-readable storage medium of claim 8.

15. Obtaining functionality developed for a first modality of a virtual environment, the first modality including displaying the virtual environment on a two-dimensional (2D) display device, the functionality including 2D control of the 2D display device for interacting with the virtual environment; mapping the functionality to a second modality of the virtual environment, the second modality comprising a display of the virtual environment on a three-dimensional (3D) display device, the mapping comprising linking the 2D controls of the first modality to 3D input gestures of the second modality; receiving an indication from a client device that a user of the client device has made the 3D input gesture while interacting with the virtual environment via the second modality; executing the functionality developed for the first modality in response to the indication that the user has made the 3D input gesture while interacting with the virtual environment via the second modality; A method for providing

16. the first modality is associated with augmented reality for a first device type, the first device type being a mobile device, and the second modality is associated with a second device type, the second device type being one of an augmented reality headset, a virtual reality headset, or a desktop computer; 16. The method of claim 15.

17. determining a second client device associated with the second modality; launching a software module associated with the second modality; Adapting user interactions with the second client device to the functionality developed for the first modality; 16. The method of claim 15, further comprising:

18. and further comprising mapping one or more user gestures within the three-dimensional scene associated with the second modality to one or more two-dimensional user interface elements associated with the first modality.

16. The method of claim 15.

19. and further comprising mapping user interactions with one or more input devices associated with the second modality to one or more two-dimensional user interface elements associated with the first modality.

16. The method of claim 15.

20. and further comprising: adapting one or more background elements within the three-dimensional scene associated with the second modality based on a device type associated with the second modality.

16. The method of claim 15.

Images