Application subspaces for 3D environments

Abstract

Described techniques enable the use of a subspace of a three-dimensional (3D) space of a device by an application executing on the device, where the 3D space is defined with respect to the device. In response to a request from the application to render application content within the subspace, including a 3D location of the application content within the subspace, the application content may be rendered at the 3D location.

Description

Atty Docket No. 0120-962BW01APPLICATION SUBSPACES FOR 3DENVIRONMENTSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority- to U.S. Provisional Patent Application No. 63 / 727,073, filed on December 2, 2024, entitled “SEARCH IN RESPONSE TO SELECTION OF VISUAL CONTENT”, the disclosure of which is incorporated by reference herein in its entirety-.BACKGROUND

[0002] Conventional approaches to displaying an application(s) in a two-dimensional (2D) context, such as for a smartphone or a tablet, include dedicating one or more 2D panels or windows for each application. For example, to conserve screen space and conform to screen boundaries, a single application panel may be displayed, while other panels are maintained in the background. A user may then toggle through open applications to choose to view a different application (e.g., application panel). Upon selection, the different application replaces the original application within the display as the primary application being used, while the original application is relegated to the background. When more screen space is available, such as when using a tablet as compared to using a smartphone, two or more panels may be sized for simultaneous display.SUMMARY

[0003] As described herein, a computing device may be configured to enable three- dimensional (3D) presentation of application content, even when the underlying application is also designed for, and usable in, 2D contexts. Specifically, as described herein, a computing device such as an extended reality (XR) device may be configured to associate an application with a subspace within a 3D space of the XR device. The application may be provided with control of application content within the designated subspace, including, e.g., controlling a size, number, spacing, and 3D positioning of application panels into which application content is rendered. Meanwhile, the XR device itself may maintain (e.g., may provide a user with) control of the subspace, including maintaining control of multiple subspaces of a corresponding plurality of applications within the larger 3D space of the XR device. Thus,Atty Docket No. 0120-962BW01XR devices may be provided with an ability to provide one or more 2D applications in a 3D context, thereby providing a native 3D feel for users of the applications and of the XR devices. Moreover, application providers may be enabled to provide and develop applications that can easily be deployed in both 2D and 3D environments. Although at least some of the description is related to an XR device, the concepts described herein can be applied to any type of computing device, such as, e.g., a mobile phone, laptop computing device, smart device, and / or so forth.

[0004] In a general aspect, a method includes providing, to an application executing on a device having a three-dimensional (3D) space defined with respect thereto, a subspace of the 3D space, receiving a request from the application to render application content within the subspace, including a 3D location of the application content within the subspace, and rendering, in response to the request, the application content at the 3D location.

[0005] In another general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and comprises instructions. When executed by at least one computing device (e.g.. by at least one processor of the at least one computing device), the instructions are configured to cause the at least one computing device to provide, to an application executing on a device having a three-dimensional (3D) space defined wi th respect thereto, a subspace of the 3D space. When executed by at least one computing device (e.g., by at least one processor of the at least one computing device), the instructions are configured to cause the at least one computing device to receive a request from the application to render application content within the subspace, including a 3D location of the application content within the subspace. When executed by at least one computing device (e.g., by at least one processor of the at least one computing device), the instructions are configured to cause the at least one computing device to render, in response to the request, the application content at the 3D location.

[0006] In another general aspect, a wearable device includes at least one display, at least one processor, and at least one memory storing instructions. When executed, the instructions cause the at least one processor to provide, to an application executing on a device having a three-dimensional (3D) space defined with respect thereto, a subspace of the 3D space. When executed, the instructions cause the at least one processor to receive a request from the application to render application content within the subspace, including a 3D location of the application content within the subspace. When executed, the instructions cause the at least one processor to render, in response to the request, the application content at the 3D location.Atty Docket No. 0120-962BW01

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 illustrates an example implementation of a system for application subspaces for 3D environments

[0009] FIG. 2 illustrates an example user interface of the system of FIG. 1.

[0010] FIG. 3 A illustrates a first example use case scenario of a 3D environment with included application subspaces.

[0011] FIG. 3B illustrates a second example use case scenario of the 3D environment of FIG. 3 A.

[0012] FIG. 4 is a block diagram of a more detailed example system for implementing the system of FIG. 1.

[0013] FIG. 5 is a flowchart illustrating example operations of the systems of FIGS. 1 and 4.

[0014] FIG. 6A illustrates a first example relationship between 2D application content and corresponding application content rendered in an application subspace.

[0015] FIG. 6B illustrates an example relationship between application content rendered in an application subspace and a corresponding subspace graph.

[0016] FIG. 7 is a third person view of a user in an ambient computing environment.

[0017] FIGS. 8 A and 8B illustrate front and rear views of an example implementation of a pair of smartglasses.

[0018] FIG. 9 is a perspective view of a head-wom device according to a possible implementation of the present disclosure.DETAILED DESCRIPTION

[0019] Described systems and techniques enable use of 2D applications in 3D environments, while providing a native feel and 3D features not otherwise available when using 2D applications in 3D environments. For example, the systems and techniques described herein enable use of an application subspace (e.g., a 3D subspace) for each application running in the 3D environment, while providing each corresponding application with control over a 3D location, e.g., a size, spacing, and / or 3D positioning, of application content within its assigned 3D subspace.Atty Docket No. 0120-962BW01

[0020] At least one technical problem solved by the described techniques, therefore, includes providing 2D applications in 3D environments, while maintaining compatibility and interoperabi lily between different instances of the 2D applications that run in both 2D and 3D environments. For example, a single installable file of an application may be executed in both a 2D and a 3D environment, while providing a native feel in both environments when executing.

[0021] At least one technical problem solved by the described techniques includes enabling an XR device to render 3D application content within 3D subspaces of a 3D space of the XR device, with each of the 3D subspaces used by a corresponding application. At least one technical problem solved by the described techniques includes application multitasking within a 3D space of an XR device, including transitioning between multiple applications being used in conjunction with corresponding 3D subspaces of a 3D space of the XR device and a single application being provided with use of an entirety of the 3D space.

[0022] At least one technical solution to the above and other technical problems includes assigning an application a 3D application subspace within a 3D space of an XR device, and providing the application with control over application content and other application aspects within its corresponding 3D application subspace. For example, the at least one technical solution may include providing an application with control of sizing, positioning, rotation, and spacing of application content rendered within a corresponding 3D application subspace, while an underlying operating system controls the sizing, positioning, rotation, and spacing of the 3D application subspace itself.

[0023] At least one technical solution to the above and other technical problems includes providing an XR device with control over one or more such 3D application subspaces. Such control may include, e.g.. assigning each 3D application subspace to a corresponding application, maintaining a sizing, spacing, and positioning of each 3D application subspace, and maintaining relative sizing / positioning / spacing of application content within a 3D application subspace when the 3D application subspace is resized or repositioned. Additional example technical solutions include transitioning a selected application between use of its corresponding 3D application subspace and use of a full or entire 3D space of the XR device.

[0024] In more detail, example embodiments are directed to techniques for decomposing one or more standard 2D applications (e.g., designed for use on a smartphones or tablets) into corresponding 3D subspaces (e.g., volumes, containers, or spatial regions) within 3D environments or spaces. For example, in a 3D environment, such as an XRAtty Docket No. 0120-962BW01 environment, a display space may include a 360° view around a user, and items may be displayed along an x, y, or z axis in any direction. Many applications are designed for such 3D environments and implemented in a manner(s) that takes advantage of available features of the 3D environment, while also using available computing resources in an efficient manner. It is difficult, infeasible, or impossible to run such applications in standard 2D environments, and. conversely, conventional techniques provide only rudimentary use of standard 2D applications within 3D XR environments.

[0025] For example, it is possible to simply cast, project, or reproduce a panel of a 2D application within a 3D XR environment. For example, a 2D representation of an application running on a tablet or other 2D device may be reproduced within a 3D XR environment, and a user of the 3D XR environment may interact with the 2D representation. For example, just as in a 2D environment, it may be possible to use system (e.g., operating system) features to select, (re)size, or position a 2D projection(s) of one or more applications, under control of either the operating system or the user.

[0026] In such approaches, users are unable to seamlessly use applications across both 2D and 3D environments, in a manner(s) that utilizes available features of both environments and provides a native feel in each environment. Further, application developers may be forced to develop multiple versions of a single application for different environments, or may provide an application in only a 2D or 3D environment.

[0027] Described techniques, in contrast, enable use of a single application (e.g.. single installable file) across both 2D and 3D environments, while taking advantage of features in both environments and ensuring cross-compatibility between environments. For example, an application executing on an HMD may be provided with a 3D subspace (e.g., a cube, or a rectangular prism). Then, the application may be provided with control over where and how application content may be rendered within the provided subspace.

[0028] In this way, e.g., the application may provide panels at different depths, sizes, visibility states, and positions within its assigned subspace, to provide convenient user access to application content while also providing a native feel and other advantages within the 3D environment. Moreover, the approach may be repeated for multiple applications (e.g., multiple such subspaces may be provided), so that a user may be provided with the described benefits, while also multitasking across multiple applications.

[0029] The subspaces themselves may be sized and positioned by the XR operating system or by the user. When moving or resizing a subspace, all application content may be moved or sized together and cohesively. Users may also expand an application’s subspace toAtty Docket No. 0120-962BW01 a full space mode in which the application is provided with a subspace that encompasses all or almost all of the 3D environment of the HMD, thereby providing a fully native feel to using the application, while maintaining full backwards compatibility with 2D environments.

[0030] FIG. 1 illustrates an example implementation of a system for application subspaces for 3D environments. In the example of FIG. 1, a wearable device 102 is illustrated as being worn by a user 104. For example, the wearable device 102 may represent a headmounted device (HMD), such as smartglasses or smart goggles. Although this example is directed to an HMD device, the concepts described herein can be applied to any type of computing device such as a mobile phone, laptop computing device, smart device, and / or so forth, including any type of XR device(s).

[0031] The wearable device 102 may be understood to provide a user interface, display, or other view that provides, and / or is defined with respect to, a 3D space 106 that partially or completely surrounds the user 104. For example, from a point of view of the user 104 when wearing the wearable device, a user interface may include, for example, a Virtual Reality (VR) interface in which the user 104 is immersed in a 360° field of view. In other examples, a user interface may include an Augmented Reality (AR) interface in which the user 104 has visibility with respect to a surrounding physical environment, and various virtual user interface elements or aspects are superimposed on, or otherwise added to or shown in the context of, the physical environment. Any other suitable type of user interface that may be provided by the wearable device 102 may be used, as well.

[0032] In the example of FIG. 1, the wearable device 102 is illustrated as executing an application 108. For example, the application 108 may be configured to execute in both the context of the wearable device 102, and in the context of various 2D devices (not illustrated separately in FIG. 1), such as smartphones, tablets, or laptop computers. As such, the application 108 may be referred to as a 2D application, although, as referenced in included examples, the application 108 may also provide various types of 3D content.

[0033] To execute the application 108 using the wearable device 102 while taking advantage of the available 3D space 106, including providing a native feel to the user 104 in the context of the various types of VR, AR, or other XR environments implementable by the wearable device 102, the wearable device 102 may provide a 3D application subspace 110 within the 3D space 106. For example, the 3D space may be generated by the wearable device 102 to be viewable by the user 104, and to provide at least a part of a user interface with which the user 104 may interact. The 3D application subspace 110 may be designated for the application 108, and may be alternately referred to as, e.g., a 3D subspace, anAtty Docket No. 0120-962BW01 application subspace, or a subspace.

[0034] The 3D application subspace 110 may include, or provide, a 3D volume within the 3D space 106, with corresponding length, width, and depth (e.g., x, y, z) dimensions, as well as rotational, opacity, or scaling parameters. The 3D application subspace 110 may thus be understood to provide a container or spatial region for use in providing various types of application content 112 of the application 108.

[0035] For example, application content 112 should be understood to include or represent, e.g., any images, text, or other information that may be rendered. The application content 112 may also include any application presentation, user interface elements, and / or control elements. For example, the application content 112 may include one or more panels, surfaces, or windows in which additional application content may be rendered.

[0036] In example embodiments, all of the application content 112 within the 3D application subspace 110 may be controlled by the application 108. For example, as described and illustrated with respect to FIG. 2, the application 108 may control a 3D location of the application content 112 within the 3D application subspace 110, including, e.g., defining the 3D location of the application content 112 using 3D (e.g., x, y, z) coordinates that may be defined with respect to a perimeter of the 3D application subspace 110 and / or with respect to the 3D space 106. In more specific examples, the application 108 may control an absolute and / or relative sizing, positioning, or spacing of multiple application windows, including assigning values for such parameters in three dimensions within the 3D application subspace 110. For example, the application 108 may cause displayed panels to be rotated so as to all be facing the user 104. The application 108 may also control other display aspects, such as, e.g., transparency of displayed application content 112.

[0037] When controlling the application content 112 within the 3D application subspace 110, the application 108 may define the size, position, orientation, or spacing of various instances of the application content 112 using three dimensional coordinates that are defined with respect to either the 3D application subspace 110 and / or the 3D space 106. For example, the application 108 may assign an application window of the application content 112 to be positioned at the center of the 3D application subspace 110. In another example, the application 108 may assign an application window of the application content 112 to be rotated to be facing in a direction of the user 104 within both the 3D application subspace 110 and the 3D space 106. More generally, one or more coordinate systems may be defined with respect to the 3D application subspace 110 and / or the 3D space 106 and provided to the application 108 for use by the application 108 in providing the application content 112.Atty Docket No. 0120-962BW01

[0038] In example embodiments, when the application 108 is initially executed by the wearable device 102, the wearable device 102 may provide the application 108 with a default size, shape, position, orientation, and other characteristics of the 3D application subspace 110. The application 108 may thus proceed to render, display, or otherwise provide any number of instances or types of the application content 112 defined by the application 108, including various configurations and adjustments requested by the user 104, using the provided characteristics of the 3D application subspace 110.

[0039] For example, the application 108 may provide one or more application windows displaying rendered instances of the application content 112. The application 108 may provide such windows in default positions, e.g., side-by-side, based on pre-configured defaults and / or on received characteristics of the 3D application subspace 110. The user 104 may adjust such defaults or otherwise configure the application content 112, such as by, for example, re-positioning or re-sizing one or more of the application windows.

[0040] The wearable device 102, e.g., an operating system (OS) of the wearable device 102, may be configured to implement commands or control of the application 108, or of the user 104, including implementing the various types of examples of re-sizing or repositioning just provided. Additionally, the wearable device 102 may be configured to control the 3D application subspace 110 itself.

[0041] For example, as referenced above, the wearable device 102 may be configured to provide an initial or default size, position, and orientation of the 3D application subspace 110, in conjunction with a launch ofthe application 108. Further, the wearable device 102 may be configured to re-size, re-position, or re-orient the 3D application subspace 110, based, for example, on selections of the user 104 and / or on other operations of the wearable device 102 (such as. e.g., launching a second application and associated second 3D application subspace).

[0042] For example, a perimeter or boundaries of the 3D application subspace 110 may be partially or completely visible to the user 104, along with associated control elements. In other examples, the user 104 may select (e.g., toggle) between a visible or not visible status of a perimeter of the 3D application subspace 110. Accordingly, the user 104 may be provided with an ability, using various resizing techniques (e.g., drag-and-drop), to set any relative or absolute size, position, or orientation of the 3D application subspace 110.

[0043] For example, as illustrated and described with respect to FIG. 3A, each of two or more applications may each be assigned its own 3D application subspace, so that the user 104 may select a desired one of the applications / subspaces for current focus and interaction.Atty Docket No. 0120-962BW01The wearable device 102 may be configured to position the multiple 3D application subspaces relative to one another and within the 3D space 106, either in default sizes / positions / orientations or at sizes / positions / orientations assigned by the user 104.

[0044] In some example implementations, as described and illustrated with respect to FIGS. 3A and 3B, the 3D application subspace 110 may be increased in size so as to completely (or almost completely) coincide with a boundary or perimeter of the entire 3D space 106. Put another way, in some examples, the 3D application subspace 110 may be enlarged to consume a full space of the 3D space 106, in which case the corresponding application 108 may be the only application controlling content of the 3D space 106. In such a mode, the application 108 may be provided with additional levels of control commensurate with occupying the entire 3D space, such as, e.g., controlling a pass-through functionality of the wearable device 102 when in a VR context.

[0045] The wearable device 102 may be configured to provide various other types of control of the one or more 3D application subspace(s) 110 and associated application(s) 108. For example, and again as described with respect to FIGS. 3 A and 3B, the wearable device 102 may be configured to transition a selected application between a multi-application mode and a single-application mode, where the single-application mode may allocate an entirety of the 3D space 106 to the application 108, as referenced above.

[0046] In other examples, the wearable device 102 may be configured to monitor, control, and enforce compliance by the application 108 with rules defined with respect to the 3D application subspace 110. For example, the wearable device 102 may be configured to monitor whether the application 108 requests a rendering of any of the application content 112 that would, if rendered, extend beyond the boundaries of the 3D application subspace 110. If so, the wearable device 102 may be configured to clip portions of the application content 112 that would extend beyond the boundaries of the 3D application subspace 110, or otherwise prevent such violations.

[0047] The application 108 may be configured to enforce similar rules, but within the 3D application subspace 110 in which the application 108 maintains control of the application content 112. For example, the application content 112 may include an application window, and the application may be configured to clip any of the application content 112 (e g., a displayed image) that extends outside of the application window.

[0048] In specific examples, the application 108 may be configured to generate its request to render the application content 112 within the 3D application subspace 110 through a set of defined Application Program Interfaces (APIs) provided by the wearable device 102Atty Docket No. 0120-962BW01(e.g., by the OS 412 and subspace manager 414 of FIG. 4, as discussed below). The request from the application 108 may be implemented as a structured data object or API call that includes or references the application content 112 to be rendered, along with the required 3D location. The rendering request may include, but is not limited to, the following elements:• Content Reference: A pointer or handle to the content data (e.g., text strings, image data, 2D UI buffer 410 of FIG. 4, or a panel description) to be displayed;• Panel Type: An identifier specifying the type of UI element (e.g., a simple surface, a fully interactive window, or a control element);• 3D Location Data: The precise location, orientation, and scale for the content within the application's assigned 3D application subspace 110;• View Properties: Additional rendering properties such as opacity / transparency. z- order layer (if applicable for overlapping content within the subspace), and texture / lighting information.

[0049] The 3D location of the application content 112 may be formatted as a set of coordinates and vectors, e.g., defined relative to the perimeter or origin of the 3D application subspace 110. For example, the application 108 may utilize a coordinate system (e.g., Cartesian x, y, z coordinates) provided by the wearable device 102 and defined with respect to the 3D application subspace 110. This allows the application to define depth (z-axis) and rotation, achieving a native 3D feel.

[0050] In example implementations, when the rendering request is received by the wearable device 102 (e.g., by the subspace manager 414 of FIG. 4), the device first translates the application's relative 3D location (relative to the subspace's origin) into absolute 3D coordinates within the larger 3D space 106. In such embodiments, this translation is achieved by combining the application’s requested relative coordinates with the current absolute 3D location (position and orientation) of the 3D application subspace 110 as maintained by the OS. The rendering engine 422 then uses these absolute coordinates to draw the application content 112 at the specified size, position, and orientation within the overall scene presented to the user 104.

[0051] For the application 108 to generate the request with the correct 3D location, it may be made aware of the dimensions and characteristics of its assigned 3D application subspace 110. For example, in some example embodiments, upon launching the application 108, the wearable device 102 (e.g., via the subspace manager 414 of FIG. 4) provides the application with the initial parameters of the allocated 3D application subspace 110, including its size (length, width, depth), initial position, and orientation within the 3D space 106.Atty Docket No. 0120-962BW01

[0052] If the OS or the user 104 changes the size, position, or orientation of the 3D application subspace 110 (e.g.. dragging it to a new position, as referenced in FIG. 3A), the wearable device 102 may communicate this change to the application 108. This enables the application to adjust the rendering of its internal content, or to respond to a re-size request, to maintain a desired visual appearance and cohesiveness. The application's ability to maintain the relative relationships of its internal content may thus be maintained when the OS moves or resizes the subspace as a whole.

[0053] As described above and illustrated and described in more detail below, the application 108 may be provided with three-dimensional control of the application content 112 within the 3D application subspace 110, while the wearable device 102 may be provided with control of the 3D application subspace 110 itself. The wearable device 102 may also be configured to maintain and enforce application control of the application 108 when controlling aspects of the 3D application subspace 110. For example, when the 3D application subspace 110 is changed in size or location either by the user 104 or the wearable device 102, the wearable device 102 may be configured to maintain relative sizes, spacings, positionings, and orientations of all instances of the application content 112 within the 3D application subspace 110.

[0054] FIG. 1 illustrates various example aspects of described implementations, where such example aspects generally illustrate that the application 108 may be provided with control of the application content 112 within the 3D application subspace 110, while the wearable device 102 controls the 3D application subspace 110 (and any other 3D application subspaces provided for other executing applications). The examples of FIG. 1, and throughout the present description, are not intended to be limiting or exhaustive, and many other implementations are possible, some of which are described below in further detail. For example, although not illustrated in FIG. 1, the application 108 may be configured to initiate a launch of a second application within the 3D application subspace 110, so that the second application is embedded within the 3D application subspace 110, either within its own new / second subspace and under the control of the second application, or directly within the 3D application subspace 110 and under the control of the application 108.

[0055] FIG. 2 illustrates an example user interface of the system of FIG. 1. In the example of FIG. 2, application content 202 is illustrated as being displayed within a 3D application subspace 204 of a larger 3D space 206. For example, the application content 202 may represent application content of a browser application, so that, as illustrated in FIG. 2 and discussed in more detail, below, the application content 202 includes a plurality ofAtty Docket No. 0120-962BW01 application windows, panels, or other surfaces displayed for a user.

[0056] In the example of FIG. 2. and consistent with the example of FIG. 1, the 3D space 206 may be provided by, defined with respect to, or otherwise associated with, a wearable device corresponding to the wearable device 102 of FIG. 1. The 3D application subspace 204 therefore may be understood to represent the type of volume or spatial region within the 3D space 206 that is described above with respect to the 3D application subspace 110 of FIG. 1.

[0057] In FIG. 2, the 3D application subspace 204 may be provided with a visible border or perimeter within the large 3D space 206, or the application content 202 may simply be visibly separated from other application content of other applications, or other displays, provided using the relevant wearable device. Various real-world aspects 208a-208c (e.g.. a table surface 208a, room comer 208b, and various objects 208c) are illustrated in FIG. 2 for the sake of providing perspective and relative positioning / orientation of the application content 202). From the description of FIG. 1, it may be appreciated that a user, not visible in FIG. 2, may be assumed to be facing a center of the application content 202.

[0058] Further in FIG. 2, a first window 210, control bar 212, second window 214, and third window 21 may be displayed and provided in a first plane and at a first orientation with respect to the user. Meanwhile, a fourth window 218 may be positioned in a second plane and at a second orientation. A sub-window 220 may be positioned at a horizontal distance from the user in a fourth plane, while a control panel 222 may be positioned in a fifth plane that is parallel to the first plane.

[0059] In some implementations, the control panel 222 may provide control to one or more of the various windows of FIG. 2. In other w ords, for example, rather than provide a particular set of control elements that may be common to each window within a context of each window the control panel 222 may be provided separately. Then, any selected window may be associated with, e.g., controlled by, the control panel 222. In this way, space and computing resources may be conserved.

[0060] The various examples referenced above of the application content 202 are illustrated and described for the purpose of providing non-limiting and non-exhaustive examples of approaches that may be used to take advantage of the 3D application subspace 204 when rendering the application content 202. For example, it may occur that the first window' 210 and the control bar 212 are initially provided upon launch of the application, directly facing the user in a horizontally spaced plane from the user and within the 3D application subspace. Upon selection of a particular link within, e.g., the first window 210,Atty Docket No. 0120-962BW01 the application may spawn the second window- 214, and the third window- 216 may similarly be spawned after selection of a link within the second window 214. Initially, the first window 210, the control bar 212, the second window' 214, and the third window7216 may all be positioned w'ithin the same plane used to provide the first window 210 and the control bar 212 at launch.

[0061] Then, upon selection of content in the third window 216, the first window 210, the control bar 212, the second window 214, and the third window 216 may be rotated into a new plane, while the fourth window 218 is spawned in a second plane that is at an angle with the new' plane. In other w ords, in the example, rather than continuing to spawn new' application content in a single 2D plane in front of the user, the 3D application subspace 204 may be used to position different portions of the application content 202 at different angles and different depths with respect to the user, thereby taking advantage of the 3D space 206.

[0062] For example, in FIG. 2, the first window' 210, control bar 212, second window 214, and third window 216 may be positioned parallel to an edge of the table surface 208a, while the fourth window 218 may be positioned over the objects 208c. By placing the application content 202 in the described and illustrated manner, the user may be provided with larger window sizes and / or more w indow's, without simply consuming more 2D space in a viewing field of the user. Such an approach is more convenient for the user with respect to viewing the application content 202. while also leaving room for other applications to render other application content within the 3D space (as shown, e.g.. in FIG. 3A).

[0063] It will be appreciated from the descriptions and illustrations of FIGS. 1 and 2 that the underlying browser application may maintain control over all such sizing, spacing, and positioning decisions with respect to the application content 202 within the 3D application subspace 204. Meanwhile, the relevant wearable device may be configured to control content in other portions of the 3D space 206, w hile also controlling a size, position, and orientation of the 3D application subspace 204 within the 3D space 206.

[0064] For example, the browser application may determine a total or maximum space consumed by the first window 210, control bar 212, second window 214, and third window 216 before shifting their plane and orientation relative to, and in conjunction with the launching of, the fourth window' 218. More generally, each time new application content is rendered, the browser application may be provided with the control to determine, in response to the request for rendering of the new application content, the size, position, spacing, and orientation of the new application content, along with any corresponding updates to the size, position, spacing, and orientation of any existing application content. The application isAtty Docket No. 0120-962BW01 thereby provided with both static and dynamic control over all of the application content 202.

[0065] Thus, the application of FIG. 2 may be rendered in the 3D environment of the 3D application subspace 204, while the application (or a user) is provided with control over a placement (including depth) of multiple rendered panels of the application. For example, a user clicking on a link in the first window 210 in a conventional (e.g., 2D) setting might normally advance to a selected page that includes multiple sub-parts. In described techniques, however, as in FIG. 2, the browser application may decompose a panel of a selected link / page into one or more resulting windows (e.g., the second window 214 and the third window 216). The resulting windows themselves may be separately displayed (e.g., positioned, sized, spaced, and oriented) by the application within the 3D application subspace 204 and / or separately controlled (e.g.. used or positioned) by the user.

[0066] A number of references are made herein with respect to an application controlling a size, spacing, positioning, and orientation of the application content 202. Such examples of application control are not limiting, and are not exhaustive. For example, the application may also determine a level of transparency of each element of the application content 202 For example, the control panel 222 may be partially transparent to enable viewing of covered portions of the second window 214, while the sub-window 220 may be opaque.

[0067] Many other implementation aspects may be provided, as well. For example, some user actions taken with respect to a first application deployed within the 3D application subspace 204 may cause a second application to be launched. In such cases, the second application may be launched within the same 3D application subspace 204, e.g., may be embedded within the same 3D application subspace 204 as the first application.

[0068] In such examples, control of sizing / spacing / positioning / orienting application content of the second application may be provided to either the first application or the second application. In other examples, the second application may be provided with its own 3D application subspace, in which the second application may be provided with control for rendering application content of the second application.

[0069] FIG. 3A illustrates a first example use case scenario of a 3D environment with included application subspaces. In the example of FIG. 3, a physical environment 302 is occupied by a user 304. The user 304 should be understood to be wearing a w earable device (analogous to the w earable device 102 of FIG. 1), which is not visible in the example perspective of FIG. 3 A.

[0070] A 3D space 306 is provided by, defined by, or associated with the wearableAtty Docket No. 0120-962BW01 device, and multiple applications may be understood to be executed by the wearable device. As described with respect to FIGS. 1 and 2, each such application may be provided with its own 3D application subspace.

[0071] For example, a first 3D application subspace 308, associated with (and controlled by) a first application, is illustrated as including rendered application content 310. A second 3D application subspace 312, associated with (and controlled by) a second application, is illustrated as including rendered application content 314. A third 3D application subspace 316, associated with (and controlled by) a third application, is illustrated as including rendered application content 318.

[0072] FIG. 3 A thus illustrates that the 3D space 306 may be used to provide application multitasking in a convenient and intuitive manner. Each 3D application subspace 308, 312, 316 may be positioned within the 3D space 306 relative to the user 304 (and to the wearable device) by the wearable device. Meanwhile, each application may be provided with control of its corresponding application content 310, 314, 318 within each corresponding and respective 3D application subspace 312. 316, 320.

[0073] The wearable device may provide needed or helpful information to each application to facilitate desired types of control by each application. For example, the wearable device may provide each application with 3D coordinates of its corresponding 3D application subspace, so that each application may position its application content in a desired fashion with respect to the user 304 or the wearable device. In other examples, an application may be provided with a direction, angle, and distance between its 3D application subspace and the user 304.

[0074] FIG. 3 A illustrates the provision of multitasking among multiple applications to the user 304, while providing, for each such application, a native feel within the 3D space 306 of the wearable device. As described with respect to FIGS. 1 and 2, each application may be configured to enforce rules related to its own application content, including, e.g., clipping any application content that extends beyond a border or perimeter of an application content window or panel.

[0075] Meanwhile, the wearable device may be configured to enforce rules that apply to each and all of the 3D application subspaces 308, 312, 316. For example, the wearable device may be configured to enforce clipping of any application content of any application that extends beyond a border of a corresponding 3D application subspace.

[0076] Further, the wearable device may be configured to enable movement, resizing, repositioning, or reorienting of each 3D application subspace(s) 308, 312, 316, while alsoAtty Docket No. 0120-962BW01 maintaining relative sizes, positions, spaces, and orientations of all included application content 310, 314, 318. For example, the user 304 may be provided with an ability’ to drag- and-drop the 3D application subspace 316 to any desired location within the 3D space 306.

[0077] In other examples, the user 304 may be provided with an ability to make, e.g., the 3D application subspace 312 larger or smaller. Then, the wearable device may be configured to inform the application of the 3D application subspace 312 of a new size of the 3D application subspace 312, whereupon that application may update the application content 314 to maintain a desired appearance. In other examples, the wearable device itself may be configured to directly update the application content 314, and report the resulting updates to the relevant application.

[0078] In a more specific example of related scenarios, FIG. 3B illustrates a second example use case scenario in which the 3D application subspace 312 is provided with access to, and control of, the application content 314 within an entirety’ of the 3D space 306. That is, in FIG. 3A, it will be appreciated that the user 304 may be provided with an ability to select any desired 3D application subspace 308, 312, 316 as being in a current state of focus / use, to interact with selected, included application content 310, 314, or 318. Such a state of focus may be indicated using any suitable means, such as, e.g., changing or increasing a color of a selected 3D application subspace, or changing a relative or absolute size of a selected 3D application subspace.

[0079] In some cases, for example, as in FIG. 3B. the user 304 may wish to devote all, or almost all of the 3D space 306 to the use of a single application, such as the application providing the application content 314 within the 3D application subspace 312. Thus, in FIG. 3B, the 3D space 306 may be understood to provide the 3D application subspace 312, and the application content 314 is expanded into more detailed and more inclusive application content 314a, 314b, 314c, and 314d.

[0080] That is, the application content 314 may be expanded to take advantage of availability of the increased space within the 3D space 306. Such expansion may include, e.g., decomposing one or more application windows into multiple application windows, or spreading multiple application windows 314a, 314b, 314c that may have been obscuring one another within the smaller 3D application subspace 312 to each be fully visible, as shown in FIG. 3B.

[0081] In additional examples, a control element 314d may be included. Similar to the control panel 222 of FIG. 2, the control element 314d may provide control of any selected one of the various application windows 314a, 314b, 314c.Atty Docket No. 0120-962BW01

[0082] Further, additional aspects of control or use of the wearable device and of the 3D space 306 may be provided in the example of FIG. 3B, that may not be feasible or desired to provide in the example of FIG. 3A. For example, in FIG. 3B, the application providing the application content 314a, 314b, 314c, 314d may also be provided with levels or types of control normally reserved for the wearable device itself.

[0083] For example, in a VR environment or context, the user 304 may be immersed in a virtual setting in which the surrounding physical space 302 is not visible to the user 304, and the user 304 sees only or primarily, virtual content. The wearable device may thus provide a pass-through function in which the user 304 is provided with an ability' to see through the virtual setting, such as by providing a window within the VR setting through which the physical space 302 is visible. In the example of FIG. 3B. since the application providing the application content 314a, 314b, 314c, 314d is provided with control within an entirety' of the 3D space 306, the application may also be provided with control of the pass- through function, including, e.g., a size or position of a pass-through window.

[0084] FIG. 4 is a block diagram of a more detailed example system for implementing the system of FIG. 1. In the example of FIG. 4, an HMD 402 is illustrated as executing an application 404. The application 404 may represent 2 or more applications, as described in more detail, below.

[0085] The HMD 402 may include a processor 406 (which may represent one or more processors), as well as a memory' 408 (which may represent one or more memories (e.g.. non- transitory computer readable storage media)). A buffer 410 represents a particular memory or memory' resource used in providing the types of subspaces and included content described herein. More detailed examples of the HMD 402 and various associated hardware / software resources are provided below, e.g., with respect to FIGS. 7, 8A. and 8B.

[0086] The processor 406 may execute instructions stored using the memory 408 to provide an operating system (OS) 412, as well as to execute the application 404. In FIG. 4, the OS 412 includes a subspace manager 414 that is configured to provide one or more subspaces for the application 404 and other applications being executed by the HMD 402.

[0087] For example, in operation, the subspace manager 414 may receive a request from the application 404 to provide content, such as the content shown in FIG. 2. A subspace generator 416 may be configured to generate an application subspace for the application 404, such as the application subspace 204 of FIG. 2.

[0088] For example, upon launch of the application 404, the subspace manager 414 may first determine whether the application 404 is configured to utilize the ty pes ofAtty Docket No. 0120-962BW01 application subspaces described herein. If not. the OS 412 may proceed to execute the application 404 in conventional fashion (e.g., within a single panel or surface).

[0089] If the application 404 is configured to use application subspaces, the subspace generator 416 may allocate an initial application subspace. For example, the subspace generator 416 may provide a standard or default subspace size and shape, such as a cube of pre-determined proportions.

[0090] According to an implemented rule or algorithm an allocated subspace size, shape, or orientation is determined. For example, such a rule may assign a default size / shape / orientation, where the default size may depend on other factors, such as how many applications are currently active and assigned a 3D subspace. For example, such a rule may assign an equally sized 3D subspace to all requesting applications, or may assign size / shape / orientation based on expressed user preferences. Such rule(s) may depend on a ty pe or other characteristic of the application 404. Such rule(s) may be embedded or included in the request from the application and / or may be implemented by the wearable device with respect to any / all requesting applications. In other examples, a new subspace may be generated based on a current size and positioning of existing subspaces within a total 3D space provided by the HMD 402.

[0091] A subspace handler 424 of the application 404 may thus be provided with an initial application subspace and associated parameters. As described herein, the application 404 may thereafter be provided with control over generating and placing / positioning content within the assigned application subspace.

[0092] For example, as described with respect to FIGS. 1-3, a user of the application 404 may request desired content to be shown or provided by the application 404. A panel controller 426 may then determine one or more corresponding panel(s) and associated controls to be rendered within the assigned application subspace. A content generator 428 may then render the content into the panel(s).

[0093] To facilitate these outcomes, the subspace manager 414 includes a buffer provider 418. In response to a request from the application 404 (e.g., from the panel controller 426) to create and render into a panel, the buffer provider 418 may allocate a corresponding portion of the buffer 410.

[0094] A panel manager 420 may be configured to track relative positions and relationships of two or more panels of the application 404, as dictated by the panel controller 426. For example, as referenced above and described in detail below with respect to FIGS. 6A and 6B, multiple panels and application content may be configured in a hierarchy. In thisAtty Docket No. 0120-962BW01 way, characteristics of panels and relationships between panels may be maintained in a desired fashion.

[0095] For example, from a perspective of the application 404, the application 404 may assign new panels or other content in desired positions within its assigned application subspace. For example, in FIG. 2, as described above, the first window 210 may be displayed and provided in a first plane and at a first orientation, while the fourth window 218 may be positioned in a second plane and at a second orientation, e.g., in response to a selection of content from within the first window 210. The sub-window 220 may be positioned at a horizontal distance from the user in response to a selection of content from the fourth window 218. By maintaining the windows 210, 218, 220 (and other content of FIG. 2) in a defined hierarchy, the application 404 may track relative positions of the windows and other relationships between the windows, even as new windows are generated and / or as the user opens new windows, or closes, re-sizes, or re-positions windows.

[0096] Moreover, the hierarchy may easily be shared with the panel manager 420 of the subspace manager 414, to facilitate management of the application subspace. For example, as described herein, either the OS 412 and / or a user may decide to re-size the applications subspace, such as when another application subspace is added / removed, or when the application 404 is transitioned to a full space mode in which the application 404 has control of an entire 3D space of the HMD 402.

[0097] In these and similar examples, the subspace manager 414 is configured to maintain relative sizes and positions of content within the application subspace. For example, if a user decides to perform a drag-and-drop operation to increase a size of the application subspace 204 in FIG. 2 within the 3D space 206, the panel manager 420 may be configured to use the corresponding hierarchy from the panel controller 426 to automatically enlarge all of the content of the application subspace 204 in a uniform and consistent manner.

[0098] A rendering engine 422 may be configured to render content from the content generator 428 into any designated panel within the application subspace. The rendering engine 422 is illustrated separately from the subspace manager 414 in FIG. 4 and may represent or include various standard rendering functionalities, which are not described here in detail. In other examples, the rendering engine 422 may be partially or completely included as part of the subspace manager 414, e.g., may be part of a rendering pipeline that provides described functionalities in conjunction with other rendering operations of the OS 412.

[0099] FIG. 5 is a flowchart illustrating example operations of the systems of FIGS. 1 and 4. In the example of FIG. 5, operations 502-506 are illustrated as separate, sequentialAtty Docket No. 0120-962BW01 operations. However, in various example implementations, the operations 502-506 may be implemented in a different order than illustrated, in an overlapping or parallel manner, and / or in a nested, iterative, looped, or branched fashion. Further, various operations or suboperations may be included, omitted, or substituted.

[0100] In the example of FIG. 5, an application executing on a device having a three- dimensional (3D) space defined with respect thereto is provided with a subspace of the 3D space (502). For example, as described with respect to FIG. 4, the application 404 may be assigned a 3D application subspace by the subspace generator 41 of the subspace manager 414.

[0101] A request may be received from the application to render application content within the subspace, including a 3D location of the application content within the subspace (504). For example, the content generator 428 may designate content to be included in an application window, surface, or other panel, as part of the functionality of the application 404. The subspace handler 424, having received coordinates and characteristics of the subspace from the subspace generator 416, may be configured to position the content at any configured position within the subspace.

[0102] As described herein, the application may be provided with full control over how such positioning occurs within the subspace. For example, the application may have a set of rules that dictate that initial application content is positioned at a center of the subspace, and subsequent content surrounds the initial content, as the subsequent content is generated. In other examples, the initial content may itself be re-positioned as new content is added. In these and many other examples, the application in question maintains full control over the positioning, sizing, spacing, and orientation of its own content within its designated subspace.

[0103] In response to the request, the application content may be rendered at the 3D location (506). That is, as just described, the device (e.g., the OS of the device) may be configured to follow the instructions of the application with respect to where and how any and all application content should be rendered. For example, as described with respect to FIG. 4, the subspace manager 414, e.g., the buffer provider 418, may provide a buffer for each requested panel, so that the content generator 428 of the application 404 may render into the assigned buffer(s).

[0104] Thus, the techniques of FIGS. 1-5 enable a straightforward transition between 2D and 3D environments when rendering application content. For example, conventional applications in 2D environments may request, using 2D application program interfacesAtty Docket No. 0120-962BW01(APIs), 2D buffers of specified size(s). Once filled with content (rendered into) by the application, the application may request the operating system to render the content from the buffer in a specified location.

[0105] In described techniques, similar techniques may be used and expanded by specifying 3D coordinates within an assigned application subspace, at which the application content may be positioned within the application subspace. For example, any available 2D UI toolkit may be used to render into an assigned buffer and corresponding panel.

[0106] Then, additional application content (e.g., additional panels or windows) may be similarly generated and positioned at different locations within the application subspace. More specifically, the application itself may be provided with control over where to position and orient the application content within the application subspace.

[0107] FIG. 6A illustrates a first example relationship between 2D application content and corresponding application content rendered in an application subspace. In FIG. 6A, a boundary 602 defines an outer boundary of a space in which an application may provide content.

[0108] For example, in a traditional 2D screen, such as a tablet or laptop, an application may be constrained by a size of the physical screen of the relevant device. In a 3D context, there is no physical screen size and an application may theoretically expand to an entirety of an available 3D space, or may be constrained to a defined portion of the 3D space.

[0109] Further in FIG. 6A. a subspace boundary 604a defines bounds or limits of an application subspace assigned to an application, illustrated in the 3D context as subspace boundary' 604b. A subspace 606a, as described above in FIGS. 1-5, is illustrated in FIG. 6A as subspace 604b.

[0110] Thus, boundary 602 represents a total screen space available for one or more applications. Subspace boundary’ 604a / 604b represents a maximum allocated subspace for one or more subspaces of one or more applications, and subspace 606a / 606b represents a current size of a subspace assigned to, and used by, an application.

[0111] In FIG. 6A. the boundary 602 is not illustrated in the 3D context. For example, as mentioned above, in a 3D context there may not be a corresponding boundary.Alternatively', it may be the case that the subspace boundary 604b is maximally-sized so as to coincide with an entirety' of a 3D space, or w ith any defined maximum bound.

[0112] As further illustrated, the subspace 606a / 606b is sized to consume only a portion of the space defined by the subspace boundary 604a / 604b. For example, the subspace boundary’ 604a / 604b may be defined by default by a HMD, and the relevant application mayAtty Docket No. 0120-962BW01 use only a portion of the assigned space.

[0113] In other examples, a user may size the subspace 606a / 606b to be smaller than the total available space defined by the subspace boundary 604a / 604b. Conversely, if a user attempts to size the subspace 608b to be larger than the subspace boundary 606b, then content of the subspace 608b may be clipped or resized. Further, as show n in FIG. 6B, an application controlling application content within the subspace 606a / 606b may be configured to launch a second application, which may be assigned its own subspace (not shown in FIG. 6A) within the subspace 606b.

[0114] A surface 608a in 2D space corresponds to a surface 608b within the 3D space of the subspace 606b. A panel 610a / 610b may be stacked on, or in front of, the surface 608a, and content 612a / 612b may be stacked on, or in front of, the panel 610a / 610b. Similar comments apply to surface 614a / 614b and stacked content 616a / 616b.

[0115] In other words, in a 2D context, one or more panels or other content may be stacked in a z-direction, and a user may select content to be displayed by rendering the content at a front-most position within the stack. In the 3D context, and as shown in FIG. 2, similar presentations of stacked content are possible, and, additionally, as also shown in FIG. 2, individual pieces of application content (e.g., panels or surfaces) may be rendered at any desired size / location within the subspace 606b.

[0116] Finally in FIG. 6 A, a subspace control element 618 represents an example of an external control element that may be designated for the subspace boundary 604b. For example, the subspace control element 618 may enable selection elements for minimizing, maximizing, or closing the relevant application(s) and associated application content.

[0117] FIG. 6B illustrates an example relationship between application content rendered in an application subspace and a corresponding subspace graph. In FIG. 6B, an application subspace 620 is illustrated with a subspace control element 621, analogously to the subspace 608b and subspace control element 618 of FIG. 6A.

[0118] Within the application subspace 620, a surface 622 has a panel 624 stacked thereon. Further, the application associated with the application subspace 620 is illustrated as having launched a second application subspace 626 for a second application, within the application subspace 620. As shown, the second application subspace 626 includes a surface 628 with a panel 630 stacked thereon.

[0119] A subspace graph 632 represents and illustrates relationships between the various elements of the application subspace 620. For example, a boundary 634 represents the type of subspace boundary or other boundary described with respect to FIG. 6A. A nodeAtty Docket No. 0120-962BW01620a corresponds to the application subspace 620. and the subspace graph 632 indicates that a first application, associated with the application subspace 620, maintains full control over the surface 622 and panel 624, as indicated by nodes 622a, 624a, as well as over rendered content from an associated buffer as illustrated by a node 636.

[0120] The first application also maintains control over a size and position of the second application subspace 626, represented by node 626a. However, within the second application subspace 626, the second application may be provided with full control over descendants of the node 626a, including the surface 628 and the panel 630, as shown by the nodes 628a, 630a, as well as over rendered content from an associated buffer as illustrated by a node 638.

[0121] Meanwhile, the second application may be provided with read access to the second application subspace 626 (for use as a parent for descendant nodes), but may not be provided with an ability to modify a size or position of the second application subspace 626. The second application may be provided with full control of content within the second application subspace 626, but may have no ability to access or modify the application subspace 620a or any of its descendants.

[0122] Thus, as described above, the subspace graph 632 enables tracking and control of application subspaces, including nested application subspaces, so that relationships between subspace elements may be maintained at an OS level. For example, the application subspace 620 may be moved or resized by the OS or by a user (if permitted by the OS), and all of the above-described relationships, including relative sizes and positions, may be maintained.

[0123] FIG. 7 is a third person view of a user 702 (analogous to the user 104 of FIG. 1) in an ambient environment 7000, with one or more external computing systems shown as additional resources 752 that are accessible to the user 702 via a netw ork 7200. FIG. 7 illustrates numerous different wearable devices that are operable by the user 702 on one or more body parts of the user 702, including a first wearable device 750 in the form of glasses worn on the head of the user, a second wearable device 754 in the form of ear buds worn in one or both ears of the user 702, a third wearable device 756 in the form of a watch worn on the waist of the user, and a computing device 706 held by the user 702. In FIG. 7, the computing device 706 is illustrated as a handheld computing device but may also be understood to represent any personal computing device, such as a table or personal computer.

[0124] In some examples, the first wearable device 750 is in the form of a pair of smart glasses including, for example, a display, one or more images sensors that can captureAtty Docket No. 0120-962BW01 images of the ambient environment, audio input / output devices, user input capability, computing / processing capability and the like. Additional examples of the first wearable device 750 are provided below, with respect to FIGS. 8A and 8B.

[0125] In some examples, the second wearable device 754 is in the form of an ear worn computing device such as headphones, or earbuds, that can include audio input / output capability, an image sensor that can capture images of the ambient environment 7000, computing / processing capability, user input capability and the like. In some examples, the third wearable device 756 is in the form of a smart watch or smart band that includes, for example, a display, an image sensor that can capture images of the ambient environment, audio input / output capability, computing / processing capability, user input capability and the like. In some examples, the handheld computing device 706 can include a display, one or more image sensors that can capture images of the ambient environment, audio input / output capability, computing / processing capability, user input capability, and the like, such as in a smartphone. In some examples, the example wearable devices 750, 754, 756 and the example handheld computing device 706 can communicate with each other and / or with external computing system(s) 752 to exchange information, to receive and transmit input and / or output, and the like. The principles to be described herein may be applied to other types of wearable devices not specifically shown in FIG. 7 or described herein.

[0126] The user 702 may choose to use any one or more of the devices 706, 750, 754, or 756. perhaps in conjunction with the external resources 752, to implement any of the implementations described above with respect to FIGS. 1 -6B.

[0127] The device 706 may access the additional resources 752 to facilitate the aspects of the various techniques described herein, or related techniques. In some examples, the additional resources 752 may be partially or completely available locally on the device 706. In some examples, some of the additional resources 752 may be available locally on the device 706, and some of the additional resources 752 may be available to the device 706 via the network 7200. As show n, the additional resources 752 may include, for example, server computer systems, processors, databases, memory storage, and the like. In some examples, the processor(s) may include training engine(s), transcription engine(s), translation engine(s), rendering engine(s), and other such processors.

[0128] The device 706 may operate under the control of a control system 760. The device 706 can communicate with one or more external devices, either directly (via wired and / or wireless communication), or via the network 7200. In some examples, the one or more external devices may include various ones of the illustrated wearable computing devices 750,Atty Docket No. 0120-962BW01754, 756, another mobile computing device similar to the device 706, and the like. In some implementations, the device 706 includes a communication module 762 to facilitate external communication. In some implementations, the device 706 includes a sensing system 764 including various sensing system components. The sensing system components may include, for example, one or more image sensors 765, one or more position / orientation sensor(s) 764 (including for example, an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer and other such sensors), one or more audio sensors 766 that can detect audio input, one or more image sensors 767 that can detect visual input, one or more touch input sensors 768 that can detect touch inputs, and other such sensors. The device 706 can include more, or fewer, sensing devices and / or combinations of sensing devices.

[0129] Captured still and / or moving images may be displayed by a display device of an output system 772, and / or transmitted externally via a communication module 762 and the network 7200, and / or stored in a memory 770 of the device 706. The device 706 may include one or more processor(s) 774. The processors 774 may include various modules or engines configured to perform various functions. In some examples, the processor(s) 774 may include, e.g., training engine(s), transcription engine(s), translation engine(s), rendering engine(s), and other such processors. The processor(s) 774 may be formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processor(s) 774 can be semiconductor-based including semiconductor material that can perform digital logic. The memory 770 may include any type of storage device or non-transitory computer-readable storage medium that stores information in a format that can be read and / or executed by the processor(s) 774. The memory' 770 may store applications and modules that, when executed by the processor(s) 774, perform certain operations. In some examples, the applications and modules may be stored in an external storage device and loaded into the memory' 770.

[0130] Although not shown separately in FIG. 7, it will be appreciated that the various resources of the computing device 706 may be implemented in whole or in part within one or more of various wearable devices, including the illustrated smartglasses 750, earbuds 754, and smartwatch 756, which may be in communication with one another to provide the various features and functions described herein.

[0131] An example head mounted wearable device 800 in the form of a pair of smart glasses is shown in FIGS. 8A and 8B, for purposes of discussion and illustration. The example head mounted wearable device 800 includes a frame 802 having rim portions 803 surrounding glass portion, or lenses 807, and arm portions 830 coupled to a respective rimAtty Docket No. 0120-962BW01 portion 803. In some examples, the lenses 807 may be corrective / prescription lenses. In some examples, the lenses 807 may be glass portions that do not necessarily incorporate corrective / prescription parameters. A bridge portion 809 may connect the rim portions 803 of the frame 802. In the example shown in FIGS. 8A and 8B, the wearable device 800 is in the form of a pair of smart glasses, or augmented reality glasses, simply for purposes of discussion and illustration.

[0132] In some examples, the wearable device 800 includes a display device 804 that can output visual content, for example, at an output coupler providing a visual display area 805, so that the visual content is visible to the user. In the example shown in FIGS. 8A and 8B, the display device 804 is provided in one of the two arm portions 830, simply for purposes of discussion and illustration. Display devices 804 may be provided in each of the two arm portions 830 to provide for binocular output of content. In some examples, the display device 804 may be a see through near eye display. In some examples, the display device 804 may be configured to project light from a display source onto a portion of teleprompter glass functioning as a beamsplitter seated at an angle (e.g., 30-45 degrees). The beamsplitter may allow for reflection and transmission values that allow the light from the display source to be partially reflected while the remaining light is transmitted through. Such an optic design may allow a user to see both physical items in the w orld, for example, through the lenses 807, next to content (for example, digital images, user interface elements, virtual content, and the like) output by the display device 804. In some implementations, waveguide optics may be used to depict content on the display device 804.

[0133] The example w earable device 800, in the form of smart glasses as shown in FIGS. 8A and 8B, includes one or more of an audio output device 806 (such as, for example, one or more speakers), an illumination device 808, a sensing system 810. a control system 812, at least one processor 814, and an outw ard facing image sensor 816 (for example, a camera). In some examples, the sensing system 810 may include various sensing devices and the control system 812 may include various control system devices including, for example, the at least one processor 814 operably coupled to the components of the control system 812. In some examples, the control system 812 may include a communication module providing for communication and exchange of information betw een the w earable device 800 and other external devices. In some examples, the head mounted w earable device 800 includes a gaze tracking device 815 to detect and track eye gaze direction and movement. Data captured by the gaze tracking device 815 may be processed to detect and track gaze direction and movement as a user input. In the example shown in FIGS. 8A and 8B, the gaze trackingAtty Docket No. 0120-962BW01 device 815 is provided in one of two arm portions 830, simply for purposes of discussion and illustration. In the example arrangement shown in FIGS. 8A and 8B, the gaze tracking device 815 is provided in the same arm portion 830 as the display device 804, so that user eye gaze can be tracked not only with respect to objects in the physical environment, but also with respect to the content output for display by the display device 804. In some examples, gaze tracking devices 815 may be provided in each of the two arm portions 830 to provide for gaze tracking of each of the two eyes of the user. In some examples, display devices 804 may be provided in each of the two arm portions 830 to provide for binocular display of visual content.

[0134] The wearable device 800 is illustrated as glasses, such as smartglasses, augmented reality (AR) glasses, or virtual reality (VR) glasses. More generally, the wearable device 800 may represent any head-mounted device (HMD), including, e.g., goggles, helmet, or headband, or another device (e.g., smartphone) mounted to a user’s head using an external frame for use as an HMD. Even more generally, the wearable device 800 and the computing device 706 may represent any wearable device(s). handheld computing device(s), or combinations thereof.

[0135] Use of the wearable device 800, and similar wearable or handheld devices such as those shown in FIG. 7, enables useful and convenient use case scenarios of implementations of FIGS. 1-6. For example, a portion of the frame 802 enclosing the lenses 807 may be used to control a cursor or other control aspect of user interface in a horizontal plane (x, y coordinates), while one or both of the arm portions 830 may be used to control a cursor or other control aspect of the same user interface in a depth direction (z coordinates).

[0136] FIG. 9 is a perspective view of a head-worn device 900 according to a possible implementation of the present disclosure. The head-worn device 900 includes a headengagement portion 920 (e.g., head-strap) configured to mechanically couple the head-wom device 900 to the head of a user. The head-wom device 900 also includes a face-engagement portion 930 that can make contact with a face of the user, especially when the headengagement portion 920 is in position on the head of the user. The face-engagement portion 930 may encircle the eyes of a user so that the user can view light reflected from, or passed through, a curved window-element 910 of the face-engagement portion 930.

[0137] In a possible implementation, the curved window-element 910 may have a radius of curvature in a horizontal direction (e.g., relative to a user) to better match the curvature of the face of the user. The curved window-element 910 may be defined variously and is not limited to any number of curvatures, any combination of curvatures, any directionAtty Docket No. 0120-962BW01 of curvatures, and / or any orientations of curvatures. Although not illustrated explicitly, it will be appreciated that the device 900 of FIG. 9 may implement or utilize any one or more of the components described above with respect to the examples of FIGS. 7, 8 A, and 8B.

[0138] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0139] These computer programs (also known as modules, programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine- readable medium” “computer-readable medium” refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0140] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or LED (light emitting diode)) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback (e.g.. visual feedback, auditory’ feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0141] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which aAtty Docket No. 0120-962BW01 user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN’'), a wide area network (“WAN”), and the Internet.

[0142] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0143] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the description and claims.

[0144] In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

[0145] Further to the descriptions above, a user is provided with controls allowing the user to make an election as to both if and when systems, programs, devices, networks, or features described herein may enable collection of user information (e.g.. information about a user’s social network, social actions, or activities, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that user information is removed. For example, a user's identity may be treated so that no user information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0146] The computer system (e.g., computing device) may be configured to wirelessly communicate with a network server over a network via a communication link established with the netw ork server using any known wireless communications technologies and protocols including radio frequency (RF). microw ave frequency (MWF). and / or infraredAtty Docket No. 0120-962BW01 frequency (IRF) wireless communications technologies and protocols adapted for communication over the network.

[0147] In accordance with aspects of the disclosure, implementations of various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium), for processing by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that when executed cause a processor to perform a process. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0148] Specific structural and functional details disclosed herein are merely representative for purposes of describing example implementations. Example implementations, however, may be embodied in many alternate forms and should not be construed as limited to only the implementations set forth herein.

[0149] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the implementations. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including," when used in this specification, specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0150] It will be understood that when an element is referred to as being "coupled," "connected," or "responsive" to, or "on," another element, it can be directly coupled, connected, or responsive to, or on, the other element, or intervening elements may also be present. In contrast, when an element is referred to as being "directly coupled," "directly connected," or "directly responsive" to, or "directly on," another element, there are noAtty Docket No. 0120-962BW01 intervening elements present. As used herein the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0151] Spatially relative terms, such as "beneath," "below," "lower," "above," "upper," and the like, may be used herein for ease of description to describe one element or feature in relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the term "below" can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 130 degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

[0152] Example implementations of the concepts are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized implementations (and intermediate structures) of example implementations. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and / or tolerances, are to be expected. Thus, example implementations of the described concepts should not be construed as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing. Accordingly, the regions illustrated in the figures are schematic in nature and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to limit the scope of example implementations.

[0153] It will be understood that although the terms "first." "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a "first" element could be termed a "second" element without departing from the teachings of the present implementations.

[0154] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these concepts belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and / or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.Atty Docket No. 0120-962BW01

[0155] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover such modifications and changes as fall within the scope of the implementations. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described.

Claims

Atty Docket No. 0120-962BW01WHAT IS CLAIMED IS:

1. A computer-implemented method comprising: providing, to an application executing on a device generating a three-dimensional (3D) space in an extended reality environment defined with respect thereto, a subspace of the 3D space; receiving a request from the application to render application content within the subspace, including a 3D location of the application content within the subspace; and rendering, in response to the request, the application content at the 3D location.

2. The method of claim 1. further comprising: positioning the subspace at 3D subspace coordinates of the 3D location within the 3D space; and rendering the application content based on the 3D subspace coordinates.

3. The method of claim 1 or 2, further comprising: receiving panel coordinates from the application for a panel to be rendered in the subspace; and rendering the application content in the panel.

4. The method of any one of claims 1 to 3, further comprising: communicating a change in size of the subspace to the application; receiving a re-size request for the application content, based on the change in size of the subspace; and re-sizing the application content within the subspace, based on the re-size request.

5. The method of any one of the preceding claims, further comprising: receiving a request from the application to expand the subspace to fill the 3D space; and expanding the subspace to fill the 3D space, in response to the request.

6. The method of any of the preceding claims, further comprising: receiving a request from the application to launch a second application; and providing the second application with a second subspace that is within the subspace.Atty Docket No. 0120-962BW017. The method of any one of the preceding claims, further comprising: providing a separate subspace of the 3D space to a second application; receiving a request from the second application to render second application content within the separate subspace, including a second 3D location of the second application content within the separate subspace; and rendering, in response to the request, the second application content at the second 3D location.

8. The method of any one of the preceding claims, wherein the device includes a head-mounted device.

9. The method of any one of the preceding claims, wherein the subspace includes a 3D volume and the 3D location is specified using 3D coordinates that are defined relative to a perimeter of the 3D volume.

10. The method of any one of the preceding claims, further comprising: receiving a second request from the application to render second application content within the subspace, including a second 3D location of the second application content within the subspace; and rendering, in response to the request, the second application content at the second 3D location.

11. A computer program product, the computer program product being tangibly- embodied on anon-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: provide, to an application executing on the computing device generating a three- dimensional (3D) space in an extended reality environment defined with respect thereto, a subspace of the 3D space; receive a request from the application to render application content within the subspace, including a 3D location of the application content within the subspace; and render, in response to the request, the application content at the 3D location.Atty Docket No. 0120-962BW0112. The computer program product of claim 11, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: position the subspace at 3D subspace coordinates within the 3D space; and render the application content based on the 3D subspace coordinates.

13. The computer program product of claim 11 or 12, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: receive panel coordinates from the application for a panel to be rendered in the subspace; and render the application content in the panel.

14. The computer program product of any of claims 11 to 13, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: communicate a change in size of the subspace to the application; receive a re-size request for the application content, based on the change in size of the subspace; and re-size the application content within the subspace, based on the re-size request.

15. The computer program product of any of claims 11 to 14, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: receive a request from the application to launch a second application; and provide the second application with a second subspace that is within the subspace.

16. The computer program product of any of claims 11 to 15, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: provide a separate subspace of the 3D space to a second application; receive a request from the second application to render second application content within the separate subspace, including a second 3D location of the second application content within the separate subspace; andAtty Docket No. 0120-962BW01 render, in response to the request, the second application content at the second 3D location.

17. The computer program product of any of claims 11 to 16, wherein the subspace includes a 3D volume and the 3D location is specified using 3D coordinates that are defined relative to a perimeter of the 3D volume.

18. A wearable device comprising: at least one display; at least one processor; and at least one memory, the at least one memory storing a set of instructions, which, when executed, cause the at least one processor to: provide, to an application executing on the wearable device generating a three- dimensional (3D) space in an extended reality environment defined with respect thereto, a subspace of the 3D space; receive a request from the application to render application content within the subspace, including a 3D location of the application content within the subspace; and render, in response to the request, the application content at the 3D location.

19. The wearable device of claim 18, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the wearable device to: communicate a change in size of the subspace to the application; receive a re-size request for the application content, based on the change in size of the subspace; and re-size the application content within the subspace, based on the re-size request.

20. The wearable device of claim 18 or 19, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the wearable device to: receive a request from the application to launch a second application; and provide the second application with a second subspace that is within the subspace.

Images