Selection of Virtual Location for Virtual Content

Abstract

To provide a method for installing content in an appropriate augmented reality system.SOLUTION: A notification about to availability of new content for display in an expansion reality system is received. A confirmation indicating reception of the new content is received. Three-dimensional information describing a physical environment is provided to an external computing device, and allows the external computing device to be used for selecting an allocated location in the physical environment for the new content. Location information indicating the allocated location is received from the external computing device. A display location on a display system of the expansion reality system for displaying the new content so that the new content seems to a user as if displayed as an overlay in the allocation location in the physical environment is determined on the basis of the location information. The new content is displayed in the display location on the display system.SELECTED DRAWING: Figure 1B

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Application No. 62 / 865,756, filed Jun. 24, 2019, which is hereby incorporated by reference in its entirety.

[0002] (Field) This specification generally relates to mixed - reality (MR) devices.

Background Art

[0003] (Background) A typical method for viewing content on a conventional device is to open an application that can display the content on a display screen (e.g., a monitor of a computer, smartphone, tablet, etc.). The user can navigate to the application and view the content. Usually, when the user is looking at the display screen of the display, there is a fixed format regarding how the content is displayed within the application and on the display screen of the display device.

Summary of the Invention

Means for Solving the Problems

[0004] An innovative aspect of the subject matter described in this specification relates to the placement of content within an augmented - reality (AR) system.

[0005] Using a virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) system, an application can display content within a spatial three-dimensional (3D) environment. In particular, the location of a content item can be selected in response to receiving a notification of the availability of a new content stream. Thus, according to one general implementation, a user can use a mobile device as a controller for selecting a location for the virtual display of a new content item. A representation of the user's environment can be displayed on the mobile device, and the user can select a location within the representation corresponding to a location within the user's physical environment for the virtual display of the new content. An indication of the user's selected location can be provided to the AR system by the mobile device. In response to receiving location information from the mobile device, the AR system can virtually display a new content stream such that the new content stream appears to the user as an overlay at the selected location within the user's physical environment.

[0006] According to the implementation described, a notification regarding the availability of new content for display within the augmented reality system is received. The augmented reality system is configured to present the content on a display system such that the content appears to the user as if it is attached at an assigned location within the user's physical environment. A confirmation input indicating acceptance of the new content is received. In response to receiving the confirmation input, three-dimensional information describing the physical environment is provided to an external computing device external to the augmented reality system, enabling the external computing device to be used to select an assigned location within the physical environment for the new content. Location information indicating the assigned location within the physical environment for the new content is received from the external computing device. A display location on the display system for displaying the new content is determined based on the location information such that the new content appears to the user as if it is displayed as an overlay at the assigned location within the physical environment. The new content is displayed on the display system at the display location.

[0007] The described content placement approach is advantageous in that the user can use a mobile device as a controller for the AR system. The mobile device can receive 3D information from the AR system and display a representation of the user's environment that enables selection of virtual display locations for new and existing content. The user can use the representation of the user's environment to select a location for virtual display of the new content, and the AR system can virtually display the new content as an overlay at the physical location corresponding to the location selected using the representation of the environment.

[0008] Other implementations of this aspect include a corresponding system, an apparatus, and a computer program recorded on a computer storage device, each configured to perform the operations of the method. This specification also provides the following items. (Item 1) A computer-implemented method, comprising: Receiving a notification regarding the availability of new content for display within an extended reality system, the extended reality system being configured to present the content on a display system such that the content appears to the user as being attached at an assigned location within the user's physical environment; Receiving a confirmation input indicating acceptance of the new content; In response to receiving the confirmation input, providing three-dimensional information describing the physical environment to an external computing device external to the extended reality system, enabling the external computing device to be used to select an assigned location within the physical environment for the new content; Receiving, from the external computing device, location information indicating the assigned location within the physical environment for the new content; Based on the location information, determining a display location on the display system for the new content such that the new content appears to the user as being overlaid at the assigned location within the physical environment; Displaying the new content at the display location on the display system. A computer-implemented method as described above. (Item 2) The computer-implemented method according to Item 1, wherein the external computing device is a mobile device. (Item 3) The computer-implemented method according to Item 1, wherein the notification is received from an external source external to the extended reality system. (Item 4) The computer-implemented method according to Item 1, wherein the notification is received as a result of an internal determination by the extended reality system that new content is available. (Item 5) The computer-implemented method according to Item 1, further comprising displaying the notification on the display system. (Item 6) The computer-implemented method according to Item 5, further comprising removing the notification from the display system in response to receiving the confirmation input. (Item 7) The computer-implemented method according to Item 5, further comprising determining candidate assigned locations within the physical environment for the new content. The computer-implemented method according to item 1, wherein the three-dimensional information provided to the external computing device includes three-dimensional information for the candidate allocation location. (Item 8) Determining the three-dimensional information includes determining the current viewing point of the user as seen through the display system, and generating the three-dimensional information based on the current viewing point. The computer-implemented method according to item 1. (Item 9) Determining that the current viewing point of the user has changed to a new viewing point, determining updated three-dimensional information based on the new viewing point, and providing the updated three-dimensional information to the external computing device. The computer-implemented method according to item 8, further comprising. (Item 10) The computer-implemented method according to item 1, wherein the three-dimensional information corresponds to a static representation of the physical environment of the user. (Item 11) Determining existing content currently being displayed by the display system, and determining an allocation location for the existing content. Further comprising, The computer-implemented method according to item 1, wherein the three-dimensional information provided to the external computing device includes allocation location information for the existing content. (Item 12) Receiving an updated allocation location for a first existing content item from the external computing device, determining an updated display location on the display system for displaying the first existing content item such that the first existing content item appears to the user as an updated overlay at the updated allocation location within the physical environment based on the updated allocation location, and displaying the first existing content item at the updated display location on the display system. The computer-implemented method according to item 11, further comprising. (Item 13) An augmented reality system, a frame structure comprising a display system configured to display content such that the content appears to the user to be attached at an allocation location within the user's physical environment, a processor coupled to the frame structure, wherein the processor Receiving a notification regarding the availability of new content for display on the display system; Receiving a confirmation input indicating acceptance of the new content; In response to receiving the confirmation input, providing three-dimensional information describing the physical environment to an external computing device outside the augmented reality system, enabling the external computing device to be used to select an assigned location within the physical environment for the new content; Receiving location information indicating the assigned location within the physical environment for the new content from the external computing device; Based on the location information, determining a display location on the display system such that the new content appears to the user as if it is being overlaid at the assigned location within the physical environment; Displaying the new content at the display location on the display system; A processor configured to perform the above; An augmented reality system comprising the above. (Item 14) The augmented reality system according to item 13, wherein the external computing device is a mobile device. (Item 15) The augmented reality system according to item 13, wherein the notification is received from an external source outside the augmented reality system. (Item 16) The augmented reality system according to item 13, wherein the notification is received as a result of an internal determination by the augmented reality system that new content is available. (Item 17) A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more computer processors, cause the one or more computer processors to: Receive a notification regarding the availability of new content for display within an augmented reality system, the augmented reality system being configured to present the content on a display system such that the content appears to the user as if it is attached at an assigned location within the user's physical environment; Receive a confirmation input indicating acceptance of the new content; In response to receiving the confirmation input, provide three-dimensional information that describes the physical environment to an external computing device outside the augmented reality system, and enable the external computing device to be used to select an assigned location within the physical environment for the new content. Receive location information indicating the assigned location within the physical environment for the new content from the external computing device. Based on the location information, determine a display location on the display system such that the new content appears to the user as if it is being displayed as an overlay at the assigned location within the physical environment. Display the new content at the display location on the display system. A non-transitory computer-readable storage medium that causes operations including the above to be performed. (Item 18) The non-transitory computer-readable storage medium according to item 17, wherein the external computing device is a mobile device. (Item 19) The non-transitory computer-readable storage medium according to item 17, wherein the notification is received from an external source outside the augmented reality system. (Item 20) The non-transitory computer-readable storage medium according to item 17, wherein the notification is received as a result of an internal determination by the augmented reality system that new content is available.

[0009] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

Brief Description of the Drawings

[0010]

Figure 1A

[0011]

Figure 1B

[0012]

Figure 1C

[0013]

Figure 2

[0014]

Figure 3

[0015] Like reference numerals and symbols in the various drawings indicate like elements.

Modes for Carrying Out the Invention

[0016] (Detailed Description) FIG. 1A illustrates a scene where a room 100 in which a head-mounted AR system 101 is used. The head-mounted AR system 101 may be a mixed reality device that presents an interface to a user for interacting with and experiencing a mixed reality world. The mixed reality world can include computer-generated content and physical objects in the user's physical environment. The head-mounted AR system 101 can provide, for example, an image of a virtual object mixed with a physical object within the user's field of view.

[0017] The mixed reality world can be viewed by the user through the eyepiece(s) of the head-mounted AR system 101. For example, a monocular view 102 as seen through the right eyepiece 104 of the head-mounted AR system 101 is shown. The eyepiece(s) of the head-mounted AR system 101 (including the right eyepiece 104) is at least partially transparent and can serve as an "optical see-through" display through which the user can visually recognize real objects within the user's physical environment. The user's physical environment is the user's physical surrounding environment when the user moves around and visually recognizes real-world objects through the head-mounted AR system 101. For example, the user's physical environment includes a chair 106, a table 108, and a bookshelf 110 as seen within the monocular view 102. The bookshelf 110 is a partitioned bookshelf that includes a grid of individual shelves including shelf 111.

[0018] The head-mounted AR system 101 can be configured to present virtual content to a user that can be perceived as an augmentation to the physical reality through the eyepieces of the head-mounted AR system 101. For example, the head-mounted AR system 101 can produce an image of a virtual object that is transposed onto a partially transparent physical surface. For example, when the head-mounted AR system 101 determines that the table 108 is within the field of view of the head-mounted AR system 101, the virtual content 112 can be virtually displayed on the upper surface 114 of the table 108 through the eyepieces of the head-mounted AR system 101. "Virtual display of content" as described herein refers to displaying content on a display system or a device (such as the right eyepiece 104) such that the content appears to the user as if it is being displayed as an overlay at a specific three-dimensional location within the physical environment. The head-mounted AR system 101 can be configured to continue rendering the virtual content 112 such that when the user turns their head up or down, the display device within the head-mounted AR system 101 appears to remain attached to the upper surface 114 of the table 108.

[0019] Virtual content can include, or correspond to, for example, web pages, blogs, digital photos, videos, news articles, newsletters, or music. Virtual digital content can correspond to content stored on a memory device accessible by the head-mounted AR system 101, or the virtual content can be a representation of streaming content such as a live video feed. Multiple items of virtual content can be virtually displayed on multiple different physical surfaces within the room 100, expanding the overall work space area for the user, and the virtual content can be displayed on the work space area. Various planes can be designated, for example, as virtual displays.

[0020] The head-mounted AR system 101 can determine that new content items are available for virtual display by the head-mounted AR system 101. The head-mounted AR system 101 can make an internal determination that new content is available and / or receive notification from an external system that new content is available. The head-mounted AR system 101 may, for example, subscribe to a bot or another content source.

[0021] The new content may be various types of push-type content. Push-type content can be content that can be rendered into the user's environment without the user having to search for or select the content. For example, push-type content can include (a) notifications from various applications such as stock price notifications, news feeds, etc., (b) prioritized content (e.g., updates and notifications from social media applications, email updates, or messages from contacts), (c) messages targeting a broad target group and / or a specific target group, or (d) other types of content or content streams.

[0022] In response to determining that a new content item is available for virtual display, a notification 116 can be displayed to the user. The notification 116 indicates that a new stock price chart content item is available. The notification 116 also prompts the user to select a location within the physical environment for virtually displaying the new stock price chart.

[0023] Notification 116 can be virtually displayed to the user by the head-mounted AR system 101. Notification 116 can be shown on a specific physical surface within the room 100 or can be displayed as if it is "appearing in the space". As another example, Notification 116 can be displayed within the monocular view 102 with respect to one or more edges of the eyepiece 104. Other examples include Notification 116 being displayed on the physical display of a computing device (not shown) or as an audio notification to the user (such as being played through the speakers of the head-mounted AR system 101 or another connected computing device).

[0024] The user can configure various settings for controlling the display of Notification 116 and other new content notifications. For example, the user can configure a specific location (such as a physical surface, physical computing device display) for virtually displaying new content notifications. As another example, the user can configure the notification frequency (such as display immediately, display periodically) and the content type for which the notification is to be displayed.

[0025] The user can interact with Notification 116, indicate acceptance / confirmation of Notification 116, and / or initiate a selection of a location (such as within the physical environment of the room 100) for virtually displaying new content or a content stream. The user can select a specific location in various ways. One approach for selecting a location is by using the user interface displayed on the user's mobile device.

[0026] FIG. 1B illustrates an exemplary mobile device user interface 140. The user interface 140 can be displayed on the user's user device 142 in response to the user receiving a notification regarding a new content item that can be displayed within the AR environment (such as Notification 116 described above with respect to FIG. 1A).

[0027] The user interface 140 displays a spatial three-dimensional (3D) environment that includes a 3D model 144 of the room 100 as described above with respect to FIG. 1A. The 3D model 144 can be a simplified geometric model of the room 100. For example, the 3D model 144 can be a digital 3D floor plan of the room 100 (e.g., a "minimap").

[0028] The 3D model 144 can be a static pre-constructed representation of the room 100. As another example, the 3D model 144 can be generated in near real-time based on information generated by the head-mounted AR system 101 of FIG. 1A. For example, the head-mounted AR system 101 can generate and transmit 3D model information that can be used by the user device 142 to display the 3D model 144. In some implementations, the head-mounted AR system 101 can respond to changes in the user's head pose and / or eye movements and transmit updated 3D model information to the user device 142 so that an updated 3D model 144 can be displayed within the user interface 140 (e.g., the updated 3D model 144 represents a view of the room 100 from the user's current perspective).

[0029] The user interface 140, which can present a simplified and processed view of the room 100, can include representations of physical objects within the room 100. For example, the user interface 400 includes a chair object 146, a table object 148, and a bookshelf object 150 corresponding to the chair 106, the table 108, and the bookshelf 110, respectively. The user can zoom in and pan the user interface 140 to focus on a particular area of the 3D model 144.

[0030] The user device 140 can be used as a controller, control panel, and / or navigator for the AR environment. For example, the user can interact with the user interface 140 for the placement of virtual content. Existing virtual content can be represented within the user interface 140 by an object or an icon. For example, the virtual content 112 of FIG. 1A is represented within the user interface 140 as an icon 152. The user can select the icon 152 and drag the icon 152 to another location within the user interface 140. In response to moving the icon 152 to another location within the user interface 140, the virtual content 112 can be displayed as an overlay at the corresponding location within the AR environment as seen by a user using the head-mounted AR system 101.

[0031] In some implementations, a surface of an object within the user interface 140 is identified (by the user device 142 or the head-mounted AR system 101) as a proposed surface for content placement. For example, the backrest 154 of the chair of the chair object 146 can be shown in a manner that is emphasized (e.g., in a particular color) to indicate that the backrest of the chair 106 is a recommended location for content. As another example, the top two rows of shelves of the bookshelf object 150 (e.g., shelves 156, 158, 160, and 162) can be emphasized to indicate that the corresponding individual shelves of the bookshelf 110 are suitable for content placement. The lower rows of shelves of the bookshelf 110 may not be identified as proposed surfaces for content because the head-mounted AR system 101 may determine that the lower rows of shelves of the bookshelf 110 are not currently visible to the user (e.g., the table 108 may block the view of the lower shelves). Generally, the proposed surface can be determined based on various factors including the field of view with respect to the current head pose or line of sight, surface contour, surface texture, surface size, the type of content that can be placed on the surface (e.g., the type of existing content or new content), or whether the surface is currently occupied by virtual content.

[0032] In some implementations, the proposed surface is emphasized in response to a notification of new content as a surface suitable for the virtual display of the new content. For example, the proposed surface can be emphasized within the user interface in response to a notification 164 that instructs the user to select a location for a new stock price chart content item. The notification 164 can correspond to the notification 116 and can be automatically displayed within the user interface 140 in conjunction with the display of the notification 116 in the view 102 or in response to a user interaction with the notification 116 (e.g., confirmation of the notification 116).

[0033] In response to notification 164, the user can select the surface of a specific object within user interface 140 as a location for the virtual display of the new stock price chart. The user can select a proposed surface within 3D model 144, such as a location representing a "floating in space" location within room 100, some other surface, or some other location. As indicated by the user's finger 166, the user selects shelf 158 of shelf object 150 to indicate that shelf 111 of bookshelf 110 is the selected location for the virtual display of the new stock price chart.

[0034] An indication of the selection of shelf 158 of shelf object 150 can be transmitted to head-mounted AR system 101. For example, 3D information of shelf 158 (e.g., coordinates, region identifier, etc.) can be transmitted to head-mounted AR system 101.

[0035] In some implementations, in response to the selection of shelf 158 as a location for the virtual display of the new stock price chart, a new icon representing the new stock price chart is displayed within user interface 140, in / above shelf 158. In some implementations, in response to the display of notification 164 (or in conjunction with the display of notification 164), an icon representing the new stock price chart is displayed at an initial / default location within user interface 140. The user can move the new icon to a desired location to finalize the selection of the location for the new stock price chart. The initial / default location can be, for example, the center of user interface 140, a specific proposed surface such as a surface with the highest fitness score, or a surface or location selected based on the configured user preferences. When a new icon representing the new stock price chart is displayed, the new icon can have a different appearance (e.g., different color, flash, other emphasis) from icon 152, at least until the user finalizes the location for the new stock price chart.

[0036] The user interface 140 can be configured such that when an icon is moved to a particular surface, the icon "snaps" to the surface. As another example, the user can move an icon by using a "drag and drop" operation, and in some cases, the user can confirm the final destination / location after the drag and drop operation is complete.

[0037] In response to the selection of the shelf 158, the notification 164 can be removed from the user interface 140. In some implementations, a selection confirmation notification (e.g., "The stock price chart location has been confirmed", "Thank you for selecting the location of the stock price chart") is displayed in response to the selection of the shelf 158. After the selection of the shelf 158, the 3D model 144 can remain displayed within the user interface 140 to enable the user to perform other interactions with the AR environment and receive future notifications of new content.

[0038] FIG. 1C illustrates an updated scene of the room 180 in which the head-mounted AR system 182 virtually displays a new content item. The room 180 corresponds to the room 100, and the view 184 seen through the eyepiece 186 of the head-mounted AR system 182 is updated in response to the selection of a location for virtually displaying the new content item. For example, the head-mounted AR system 182 can receive information indicating the user selection of the shelf 158 within the user interface 140 of FIG. 1B.

[0039] In response to receiving information indicating a user selection of shelf 158 as a location for a new stock price chart content item, the head-mounted AR system 182 can identify shelf 188 of the home shelf 190 as corresponding to shelf 158 of the home shelf object 150. The head-mounted AR system 182 can virtually display a virtual stock price chart content item 192 in / above shelf 188. For example, the virtual stock price chart content item 192 can be rendered and superimposed substantially over the real-world shelf 188 such that it appears to the user as if the virtual stock price chart content item 192 is displayed as an overlay on top of shelf 188.

[0040] The head-mounted AR system 182 can maintain a one-to-one mapping of shelf 188 to the virtual stock price chart content item 192. Using the head-mounted AR system 182, a user can visually perceive the virtual stock price chart content item 192 as if it is appearing on the mapped / matched shelf 188. The virtual stock price chart content item 192 can be displayed such that it appears to be physically attached to shelf 188 as if through projection and perceivable by the user.

[0041] FIG. 2 illustrates an exemplary implementation of an AR system 200 that includes integration of a mobile device. The head-mounted AR system 202 can be configured to present a user with virtual content that can be perceived as an extension to the physical reality. For example, an AR environment generator 204 can provide an image of a virtual object that can be mixed with physical objects within the user's field of view.

[0042] The head-mounted AR system 202 can receive a notification from the new content notification unit 206 that a new content item is available for virtual display by the head-mounted AR system 202. The new content subscriber 208 can, for example, subscribe to and receive the notifications from the new content notification unit 206. As another example, the head-mounted AR system 202 can make an internal determination that new content is available for virtual display to the user.

[0043] The AR environment generator 204 can generate virtual content items to display new content by the head-mounted AR system 202. The virtual content items can be displayed by the head-mounted AR system 202 to virtually appear to the user at a default location, or the virtual content items can be displayed to virtually appear to the user at a location selected by the user. The head-mounted AR system 202 can use the communication interface 212 to send information to the mobile device 210, enabling the user to select a location within the user's environment for virtually displaying the virtual content items using the mobile device 210. The AR environment generator 204 can generate 3D information about the user's environment, and the 3D information can be sent to the mobile device 210 using the communication interface 212. The head-mounted AR system 202 can also send information about the new content items (e.g., a description of the content) and a request regarding the selection of a location for the new content items.

[0044] The mobile device 210 can receive 3D information about the user's environment and information about new content items using the communication interface 214. The 3D model renderer 216 of the AR controller application 218 can render a 3D representation of the user's environment within the AR controller application 218. The 3D representation can include a representation of physical objects within the environment and, where applicable, a representation of existing virtual content that is currently being virtually displayed by the head-mounted AR system 202 within the user's environment.

[0045] The AR controller application 218 can display instructions to the user for selecting a location for the virtual display of a new content item. The content location selector 220 can enable the user to select a location on the 3D representation of the user's environment as corresponding to the location for the virtual display of the new content item. The location can be the surface of a rendered object corresponding to a physical object within the user's environment, or the location can be 3D coordinates that may or may not correspond to a physical object.

[0046] The AR controller application 218 can use the communication interface 214 to transmit location information for the selected location of the new content item to the head-mounted AR system 202. The location information can be 3D coordinates, region information, object information, or other types of information or identifiers. The AR environment generator 204 can present or project the virtual content item so that it appears to the user to be located at a location within the user's physical environment corresponding to the location information received from the mobile device 210.

[0047] Figure 3 depicts a flowchart of a method 300 for displaying content within an AR system. A notification regarding the availability of new content to be displayed within the augmented reality system is received (310). The augmented reality system is configured to present the content on a display system such that the content appears to the user as if it were attached at an assigned location within the user's physical environment. The notification can be received from an external source external to the augmented reality system. As another example, the notification can be received as a result of an internal determination by the augmented reality system that new content is available. The notification can be displayed on the display system.

[0048] A confirmation input indicating acceptance of the new content is received (320). The confirmation input can be an interaction with the notification, a voice command, or some other type of input. The notification can be removed from the display system in response to receiving the confirmation input.

[0049] In response to receiving the confirmation input, three-dimensional information describing the physical environment is provided to an external computing device external to the augmented reality system (330). The three-dimensional information can be provided to enable the external computing device to be used to select an assigned location within the physical environment for the new content. The external computing device can be a mobile device or some other type of computing device.

[0050] The three-dimensional information can include three-dimensional information regarding candidate assigned locations for the new content. The candidate assigned locations can be locations determined to be suitable for overlaying the display of the new content. The candidate assigned locations can correspond to physical surfaces within the physical environment. The three-dimensional information can include the assigned locations of existing content currently being displayed on the display system.

[0051] The 3D representation can correspond to a static representation of the physical environment. As another example, the 3D information can correspond to the user's current viewpoint as seen through the display system. Updated 3D information can be generated in response to detection of a change in the user's viewpoint and provided to an external computing device.

[0052] Location information indicating an assigned location within the physical environment for new content is received from an external computing device (340). The location information can be 3D coordinates, region identifier(s), object identifier, or some other type of location information. The location information can correspond to a location selected by the user on a 3D representation of the physical environment that is displayed on the external computing device.

[0053] A display location is determined based on the location information (350). The display location is a location on the display system for displaying the new content such that the new content appears to the user as if the new content is being overlaid at the assigned location within the physical environment.

[0054] The new content is displayed on the display system at the display location (360).

[0055] In addition, or alternatively, updated assigned locations for a first existing content item can be received from the external computing device. An updated display location on the display system can be determined based on the updated assigned locations. The first existing content item can be displayed on the display system at the updated display location such that the first existing content item appears to the user as if the first existing content item is being displayed as an updated overlay at the updated assigned location within the physical environment.

[0056] The systems, methods, and techniques described may be implemented in digital electronic circuitry, computer hardware, firmware, software, or in combinations of these elements. Apparatuses implementing these techniques may include appropriate input and output devices, a computer processor, and a computer program product tangibly embodied in a machine-readable storage device for execution by a programmable processor. Processes implementing these techniques may be performed by a programmable processor executing a program of instructions to perform the desired functions by operating on input data to produce appropriate output. The techniques may be implemented using one or more computer programs or non-transitory computer-readable storage media comprising instructions executable on a programmable system including at least one programmable processor coupled to receive and transmit data and instructions from and to a data storage system, at least one input device, and at least one output device.

[0057] Each computer program, if desired, may be implemented in a high-level procedure or object-oriented programming language, or in assembly or machine language, and in any case, the language may be a compiler-type language or an interpreter-type language. Suitable processors include, by way of example, both general-purpose microprocessors and special-purpose microprocessors. Generally, a processor receives instructions and data from read-only memory and / or random access memory. Storage devices suitable for tangibly embodying computer program instructions and data include, by way of example, semiconductor memory devices (e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices), magnetic disks (e.g., internal hard disks and removable disks), magneto-optical disks, and compact disk read-only memory (CD-ROM), including any form of non-volatile memory. Any of the foregoing may be supplemented by, or incorporated in, a specially designed application specific integrated circuit (ASIC).

[0058] A computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition that results in a machine-readable propagated signal, or a combination of one or more of them. The term "data processing apparatus" includes, by way of example, all apparatus, devices, and machines for processing data, including programmable processors, computers, or multiple processors or computers. The apparatus may include, in addition to hardware, code for creating an execution environment for the computer program (e.g., code that forms part of processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them). A propagated signal is an artificially generated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver device).

[0059] A computer program, also known as a program, software, software application, script, plugin, or code, may be described in any form of programming language, including a compiler language or an interpreter language, and may be in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use within a computer environment. A computer program does not necessarily correspond to a file in a file system. The program may be stored in a part of a file that holds other programs or data, in a single file dedicated to the program, or in multiple cooperating files. The computer program may be executed on one computer or on multiple computers located on one site or distributed across multiple sites and interconnected by a communication network.

[0060] The processes and logical flows described herein may be implemented by one or more programmable processors executing one or more computer programs to perform actions by operating on input data to produce output. The processes and logical flows may also be implemented by special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and the apparatus may also be implemented as special purpose logic circuitry.

[0061] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read only memory or a random access memory or both.

[0062] The elements of a computer may include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer may also include, or be operatively coupled to, one or more mass storage devices (e.g., magnetic, magneto-optical disks, or optical disks) for storing data for receiving, transmitting, or both receiving and transmitting data. However, a computer may not have such devices. Also, a computer may be embedded within another device (e.g., to name but a few, a tablet computer, a mobile phone, a personal digital assistant (PDA), a mobile audio player, a VAR system). Computer-readable media suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and all forms of non-volatile memory, media, and memory devices including CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated within, special purpose logic circuitry.

[0063] This specification contains many details, but these should be construed as descriptions of features specific to particular embodiments rather than as limitations on the scope of the disclosure or what may be claimed. Features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Also, although a feature may be described above as acting in a certain combination and may thus be claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination. For example, the mapping operation is described as a series of discrete operations, but the various operations may be divided into additional operations, combined into fewer operations, varied in the order of execution, or excluded, depending on the desired implementation.

[0064] Similarly, the separation of the various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together within a single software product or packaged into multiple software products. For example, although some operations are described as being performed by a processing server, one or more of the operations may be performed by a smart meter or other network component.

[0065] As used in this specification, particularly in the appended claims (e.g., the body of the appended claims), terms are generally intended to be "non-limiting" terms (e.g., the term "including" should be construed as "including but not limited to", the term "having" should be construed as "having at least", the term "includes" should be construed as "includes but is not limited to", etc.).

[0066] In addition, if a specific number of claim listings are intended, such intent should be explicitly recited within the claim, and in the absence of such a recitation, no such intent exists. For example, by way of illustration, the following appended claims may contain the use of introductory phrases "at least one" and "one or more" to introduce claim listings. However, the use of such phrases should not be construed to imply that the use of the indefinite article such as "one or more" or "at least one" and "a" or "an" (e.g., "a" and / or "an" should be construed to mean "at least one" or "one or more") in the same claim limits any particular claim containing such an introduced claim listing to embodiments containing only one such listing, and the same applies to the use of the definite article used to introduce claim listings.

[0067] In addition, even when a specific number of recited claims being introduced is explicitly recited, one of ordinary skill in the art should recognize that such recitation should be interpreted as meaning at least the number recited (e.g., a literal recitation of "two recitations" without other qualifying language means at least two recitations or two or more recitations). Further, in those instances where notation similar to "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc." is used, generally such construction is intended to include "A alone," "B alone," "C alone," "A and B together," "A and C together," "B and C together," or "A, B, and C together." The term "and / or" is also intended to be interpreted in this manner.

[0068] The use of terms such as "first," "second," "third," etc. in this specification is not necessarily used to imply a specific order or number of elements. Generally, terms such as "first," "second," "third," etc. are used as general identifiers to distinguish different elements. In the absence of anything indicating that the terms "first," "second," "third," etc. imply a specific order, these terms should not be understood to imply a specific order. Further, in the absence of anything indicating that the terms "first," "second," "third," etc. imply a specific number of elements, these terms should not be understood to imply a specific number of elements. For example, a first widget may be described as having a first side, and a second widget may be described as having a second side. The use of the term "second side" for the second widget is for the purpose of distinguishing such side of the second widget from the "first side" of the first widget, and may not imply that the second widget has two sides.

Claims

**Claim 1**: A computer-implemented method, comprising: providing, on the augmented reality system, a notification indicating that new content is available for display by the augmented reality system; removing the notification from the display on the augmented reality system by the augmented reality system in response to receiving a confirmation input from a user; providing, by the augmented reality system, information describing the physical environment surrounding the user to a mobile device of the user of the augmented reality system; providing, by the augmented reality system, the new content for display at a display location on the display of the augmented reality system based on location information received from a mobile device of the user of the augmented reality system indicating an assigned location within the physical environment selected by the user for the new content as it appears to the user; receiving, from a mobile device of the user of the augmented reality system, an updated assigned location for the new content to be displayed; determining, by the augmented reality system, an updated display location on the display of the augmented reality system for displaying the new content based on the updated assigned location such that the new content appears to the user as an updated overlay at the updated assigned location within the physical environment; A computer-implemented method as described above. **Claim 2**: The computer-implemented method according to claim 1, wherein the mobile device of the user used by the user to indicate the assigned location is the user's mobile phone. **Claim 3**: The computer-implemented method according to claim 1, further comprising receiving, by the augmented reality system, data from an external source external to the augmented reality system and generating a notification based on the augmented reality system indicating that the new content is available. **Claim 4**: The computer-implemented method according to claim 1, further comprising receiving, by the augmented reality system, the location information from a mobile device of the user of the augmented reality system. **Claim 5**: The computer-implemented method according to claim 1, further comprising determining, by the augmented reality system, the display location based on the location information. **Claim 6**: Identifying candidate locations in the physical environment for display of the new content, wherein information provided to the mobile device identifies the candidate locations, the computer-implemented method according to claim 1. **Claim 7**: The information provided to the mobile device includes a static representation of the physical environment around the user, the computer-implemented method according to claim 1. **Claim 8**: A non-transitory computer-readable medium storing one or more instructions, the one or more instructions being executable by a computer system to perform one or more operations, the one or more operations being providing a notification on the augmented reality system indicating that new content is available for display by the augmented reality system; removing the notification from the display on the augmented reality system in response to receiving a confirmation input from the user by the augmented reality system; providing, by the augmented reality system, information describing the physical environment around the user to the mobile device of a user of the augmented reality system; providing, by the augmented reality system, the new content at a display location on a display of the augmented reality system based on location information received from the mobile device of the user of the augmented reality system indicating an assigned location in the physical environment selected by the user for the new content as it appears to the user; receiving, from the mobile device of the user of the augmented reality system, an updated assigned location for the new content to be displayed; determining, by the augmented reality system, an updated display location on the display of the augmented reality system for displaying the new content based on the updated assigned location such that the new content appears to the user as an updated overlay at the updated assigned location in the physical environment; comprising the non-transitory computer-readable medium. **Claim 9**: The mobile device of the user used to indicate the assigned location is the user's mobile phone, the non-transitory computer-readable medium according to claim 8. **Claim 10**: The non-transitory computer-readable medium according to claim 8, comprising receiving, by the augmented reality system, data from an external source external to the augmented reality system and generating a notification based on the augmented reality system indicating that the new content is available. **Claim 11**: The non-transitory computer-readable medium according to claim 8, comprising receiving, by the augmented reality system, the location information from the mobile device of the user of the augmented reality system. **Claim 12**: The non-transitory computer-readable medium according to claim 8, comprising determining, by the augmented reality system, the display location based on the location information. **Claim 13**: Identifying candidate locations within the physical environment for display of the new content, wherein the information provided to the mobile device identifies the candidate locations, the non-transitory computer-readable medium according to claim 8. **Claim 14**: The non-transitory computer-readable medium according to claim 8, wherein the information provided to the mobile device includes a static representation of the physical environment around the user. **Claim 15**: A computer-implemented system, one or more computers, and one or more computer memory devices operatively coupled to the one or more computers comprising, the one or more computer memory devices having a tangible non-transitory machine-readable medium storing one or more instructions, the one or more instructions, when executed by the one or more computers, perform one or more operations, the one or more operations comprising providing a notification on the augmented reality system indicating that new content is available for display by the augmented reality system; removing the notification from the display on the augmented reality system by the augmented reality system in response to receiving a confirmation input from the user; and providing, by the augmented reality system, information describing the physical environment around the user to a mobile device of the user of the augmented reality system. Providing the new content for a display at a display location based on location information received from a mobile device of a user of the augmented reality system, the display location indicating an assigned location within the physical environment selected by the user for the new content that appears to the user as displayed by the augmented reality system; Receiving from the mobile device of the user of the augmented reality system an updated assigned location for the new content to be displayed; Determining, by the augmented reality system, an updated display location on a display of the augmented reality system for displaying the new content based on the updated assigned location, such that the new content appears to the user as an updated overlay at the updated assigned location within the physical environment; A computer-implemented system comprising. The computer-implemented system according to claim 15, wherein the mobile device of the user used by the user to indicate the assigned location is the user's mobile phone. The computer-implemented system according to claim 15, comprising generating, by the augmented reality system, a notification based on the augmented reality system that receives data from an external source external to the augmented reality system and indicates that the new content is available. The computer-implemented system according to claim 15, comprising receiving, by the augmented reality system, the location information from a mobile device of the user of the augmented reality system. The computer-implemented system according to claim 15, comprising determining, by the augmented reality system, the display location based on the location information. The computer-implemented system according to claim 15, comprising identifying candidate locations within the physical environment for display of the new content, wherein information provided to the mobile device identifies the candidate locations.

Images