System and Method for Location Search in Augmented Reality

Abstract

The present disclosure provides a computer-implemented method, system, and device for enabling search in an augmented reality interface. A computing device generates an interface depicting an AR view that includes image data of at least a portion of a physical real-world environment for display by the computing device. The computing device displays one or more filter elements within the interface, each filter element being associated with a type of point of interest. The computing device accesses data from a database of geographical locations that describes a plurality of points of interest within the physical real-world environment. The computing device receives a selection of one of the displayed filter elements. The computing device provides a set of augmented reality elements associated with a set of points of interest for display within the AR view, the set of augmented reality elements representing a filter-based set of points of interest associated with the filter element.

Description

Technical Field

[0001] Related Applications This application claims the benefit of, and priority to, U.S. Provisional Patent Application No. 63 / 390,240, filed Jul. 18, 2022, which is incorporated herein by reference in its entirety. Applicant claims the priority and the benefit of this application and incorporates it herein by reference in its entirety.

[0002] This disclosure generally relates to augmented reality (AR). More specifically, this disclosure relates to location search within an AR view.

Background Art

[0003] Computing devices (e.g., desktop computers, laptop computers, tablet computers, smartphones, wearable computing devices, etc.) are ubiquitous in modern society. They can support communication among their users and provide users with information regarding their environment, current events, the world at large, and the like. A common use of such devices is, for example, to generate and display an augmented reality (AR) view of at least a portion of a physical real-world environment (e.g., the location where one or more of such devices are located). The AR view may be part of an interactive AR experience provided to one or more users of such a device, and such an experience is enhanced by computer-generated information that is perceivable across one or more sensory modalities of the user(s).

Summary of the Invention

[0004] Aspects and advantages of embodiments of the present disclosure are shown in part in the following description, or can be learned from the description, or can be learned through the practice of the embodiments.

[0005] One exemplary aspect of the present disclosure is directed to a computer-implemented method. The method includes generating, by a computing system having one or more processors, an interface depicting an augmented reality (AR) view that includes image data of at least a portion of a physical real-world environment for display by the computing system. The method further includes displaying, by the computing system, one or more filter elements within the interface, wherein each filter element within the one or more filter elements is associated with a type of point of interest. The method further includes accessing, by the computing system from a database of geographical locations, data describing a plurality of points of interest within a portion of the physical real-world environment. The method further includes receiving, by the computing system, a selection of one of the displayed filter elements. The method further includes providing, by the computing system for display within the AR view, a filter-based set of augmented reality elements associated with a set of points of interest, wherein the filter-based set of augmented reality elements represents a filter-based set of points of interest associated with the selected filter element.

[0006] Other exemplary aspects of the present disclosure are directed to a computing device. The computing device includes one or more processors and a computer-readable memory that stores instructions which, when executed by the one or more processors, cause the computing device to perform operations including generating an interface that depicts an augmented reality (AR) view that includes image data of at least a portion of a physical real-world environment for display by a computing system. The operations further include displaying one or more filter elements within the interface, wherein each of the one or more filter elements within the one or more filter elements is associated with a type of point of interest. The operations further include accessing, from a database of geographical locations, data that describes a plurality of points of interest within a portion of the physical real-world environment. The operations further include receiving a selection of one of the displayed filter elements. The operations further include providing a filter-based set of augmented reality elements associated with a set of points of interest for display within the AR view, wherein the filter-based set of augmented reality elements represents a filter-based set of points of interest associated with the selected filter element.

[0007] Other exemplary aspects of the disclosure are directed to non-transitory computer-readable media that store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations including generating an interface that depicts an augmented reality (AR) view that includes image data of at least a portion of a physical real-world environment for display by a computing system. The operations further include displaying one or more filter elements within the interface, wherein each filter element within the one or more filter elements is associated with a type of point of interest. The operations further include accessing, from a database of geographical locations, data that describes a plurality of points of interest within a portion of the physical real-world environment. The operations further include receiving a selection of one of the displayed filter elements. The operations further include providing a filter-based set of augmented reality elements associated with a set of points of interest for display within the AR view, the filter-based set of augmented reality elements representing a filter-based set of points of interest associated with the selected filter element.

[0008] Other aspects of the disclosure are directed to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects, and advantages of the various embodiments of the disclosure will become better understood with reference to the following description of the invention and the accompanying claims. The accompanying drawings, which are incorporated herein and form a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles.

[0010] A detailed description of embodiments directed to those of ordinary skill in the art is set forth in this specification with reference to the following accompanying drawings.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

【Figure An exemplary augmented reality user interface in accordance with an exemplary embodiment of the present disclosure is shown. ​ An exemplary augmented reality user interface in accordance with an exemplary embodiment of the present disclosure is shown. ​ An exemplary augmented reality user interface in accordance with an exemplary embodiment of the present disclosure is shown. ​ An exemplary augmented reality user interface in accordance with an exemplary embodiment of the present disclosure is shown. ​ An exemplary augmented reality user interface in accordance with an exemplary embodiment of the present disclosure is shown. ​ An exemplary augmented reality user interface in accordance with an exemplary embodiment of the present disclosure is shown. ​ An exemplary flowchart for a method of performing filtering and searching of elements within an augmented reality view in accordance with an exemplary embodiment of the present disclosure is shown. **DETAILED DESCRIPTION OF THE INVENTION**

[0012] Reference will now be made in detail to embodiments of the present disclosure, one or more examples of which are illustrated in the drawings. Each example is provided by way of explanation of the disclosure of the invention, not limitation of the disclosure of the invention. Indeed, it will be apparent to those skilled in the art that various modifications and variations can be made in the present disclosure without departing from the scope or spirit of the present disclosure. For example, features illustrated or described as part of one embodiment can be used with other embodiments to yield still further embodiments. Accordingly, it is intended that the present disclosure cover modifications and variations that come within the scope of the appended claims and their equivalents.

[0013] Generally, the present disclosure is directed to a system for providing a search and filtering system for presenting information within an extended reality display. Specifically, a computing system including one or more computing devices can generate an interface depicting an AR view for display by at least one of the computing device(s). Such a view can be of at least a portion of a physical real-world environment. Using this interface, information can be presented to a user on the move (e.g., a user moving through the physical world with a user computing device) from a first-person perspective. For example, a user utilizing one or more of the computing devices can be located near a route of travel within an urban environment, and one or more sensors (e.g., a camera, etc.) of such computing device(s) can generate data representing a view of at least a portion of the urban environment, and the computing device(s) can generate an interface depicting an AR view of at least a portion of the urban environment (e.g., at least partially based on sensor data, map data associated with the geographical location of the computing device(s), etc.). For example, a view including an image of a portion of the urban environment (e.g., an image generated at least partially based on sensor data, etc.), and an image generated at least partially based on map data associated with the geographical location (e.g., overlaid, overlapping, located within, etc. an image of a portion of the urban environment such as the urban environment).

[0014] An AR view can include one or more augmented reality elements (AR elements) overlaid on an image of an urban environment or other real-world location. These augmented reality elements can include one or more indicators of a particular entity within the environment. The entity can include any point of interest within the environment, including but not limited to buildings, landmarks, businesses, etc. Thus, the AR view can include one or more labels that identify particular points of interest within the environment for the user. In this way, the user can have additional information about the environment presented within the AR view. In some examples, such as a dense urban environment, there are more potential points of interest than can be reasonably displayed at one time within the AR view. This is because there is limited space within the AR view to display additional AR elements before the interface becomes difficult to use due to clutter.

[0015] In response to this situation, a computing system that generates an AR view can identify an initial set of AR elements associated with an initial set of points of interest that can be displayed to the user. The initial set can be selected based on a determination of which points of interest are visible to the user within the AR view. This can be accomplished by analyzing the image data displayed within the AR view to detect storefronts or other visible features associated with particular points of interest. In some examples, where a large number of points of interest are visible, the computing system can use a ranking system in which each point of interest is ranked based on its relevance to the user. The most relevant AR elements can be displayed, while other less relevant elements can remain hidden.

[0016] In some examples, the AR display can include a plurality of filter elements. The filter elements can be represented by elements within the interface of the augmented reality display. For example, the filter elements can be selectable buttons or chips within the display. Each filter element can be associated with a particular type of point of interest. For example, the filter elements can be associated with categories such as restaurants, coffee shops, museums, monuments, etc.

[0017] The filter elements can be selected and ordered based on one or more factors. In some examples, the factors can include the number of a particular type of point of interest within the area of the computing device. For example, if there are multiple restaurants in one area, one of the filter elements displayed can be a restaurant. In some examples, the filter elements displayed can be based on the user associated with the computing system. Thus, if the user has previously shown an interest in a particular type of point of interest, the filter elements can be displayed for that type of point of interest.

[0018] In some examples, when the user selects a particular filter element, the interface of the augmented reality view can be updated. For example, the AR view can be changed such that one or more augmented reality elements initially displayed and associated with an initial set of points of interest are replaced with one or more AR elements associated with the selected filter element. In this way, a particular entity, building, or point of interest initially referenced or represented by one or more AR elements within the view can be associated with AR elements associated with entities associated with the selected filter type.

[0019] In some examples, an initial set of augmented reality elements is associated with a point of interest or entity currently visible within the AR view. For example, a computing system can identify one or more buildings, entities, or points of interest currently within the AR view based on image data captured by a camera. For instance, if a particular storefront is currently displayed within the AR view, AR elements associated with that storefront can be displayed above or near the storefront. When a filter element is selected, a filter-based set of augmented reality elements can be displayed. This filter-based set of augmented reality elements can be points of interest related to a type that matches the selected filter. The filter-based set of augmented reality elements can include points of interest that match the selected filter type associated with the points of interest and locations currently displayed within the AR view but are not currently in the AR view. In some examples, the filter-based set of augmented reality elements can include one or more augmented reality elements indicating the direction in which a related entity is located. For example, if the related entity is to the right of the current view, the augmented reality element can be directed near the right side of the screen using an arrow indicating the direction of the entity's location.

[0020] In other examples, a filter-based set of augmented reality elements includes AR elements associated with points of interest that are visible in the AR view but are blocked by a person, object, or building that is between the AR view and the camera of the computing device. In this example, the AR view can be updated to show where the point of interest would be within the AR view if the point of interest were not blocked. In some examples, the AR elements can have different appearances depending on whether the point of interest is currently being displayed or is blocked (including being behind one or more buildings). For example, a first icon can be used for AR elements associated with points of interest currently being displayed in the AR view, and a second icon can be used for AR elements associated with blocked points of interest. The second icon can include information that notifies the user of the direction and distance from the current position of the computing device to the point of interest. In this way, the user can be notified of the direction and distance to a point of interest that matches the selected filter but is blocked and not currently visible in the AR view.

[0021] In some examples, the AR view can have a certain amount of user interface space for displaying augmented reality interface elements. In some examples, if there are too many user interface elements being displayed at once, the augmented reality elements can potentially block other parts of the AR view that are important to the user. For example, if the augmented reality elements obscure one or more display portions of the real world, the usefulness of the augmented reality view can be reduced. As a result, if multiple augmented reality elements are planned to be displayed within the augmented reality interface, the computing system can determine whether the augmented reality elements are within a predetermined threshold distance of each other. If so, one or more of the augmented reality elements can be grouped together to create a cluster of points of interest represented by a single AR element.

[0022] In some examples, the points of interest can be clustered into a single augmented reality interface based on the type or category of the point of interest or entity. For example, multiple hotels can be grouped together into a single interface element on the augmented reality display.

[0023] The augmented reality view can be presented to the user such that the user can select or interact with portions of the AR view. In some examples, the user can select a particular feature or building within the AR view (e.g., via touch input, mouse input, etc.). The computing system can generate information regarding the selected feature or building for presentation to the user in response to that input.

[0024] The systems and methods of the present disclosure provide several technical effects and advantages. As an example, the proposed system can provide for enabling location searches within an augmented reality system. By enabling the user to filter and search from among multiple possible augmented reality elements, the number of augmented reality elements that are displayed at any given time can be reduced without degrading the effectiveness and usefulness of the augmented reality display. By improving the effectiveness of the augmented reality display, the amount of power used and data stored when displaying the augmented reality display can be reduced. By reducing the amount of storage required and the amount of energy used, the cost of the augmented reality system is reduced and the user experience is improved. This represents an improvement in the functionality of the device itself.

[0025] Reference is now made to the drawings, in which exemplary embodiments of the present disclosure are discussed in further detail.

[0026] FIG. 1 shows an exemplary computing device 100 in accordance with an exemplary embodiment of the present disclosure. In some exemplary embodiments, the computing device 100 can be any suitable device including, but not limited to, a smartphone, a tablet, a laptop, a desktop computer, a global positioning system (GPS) device, a computing device integrated into a vehicle, a wearable computing device, or any other computing device configured to enable a person to execute an augmented reality application or access an augmented reality service in a server computing system. The computing device 100 can include one or more processors 102, a memory 104, one or more sensors 110, a location system 112, an augmented reality system 114, and a display system 140.

[0027] The one or more processors 102 can be any suitable processing device, such as a microprocessor, a microcontroller, an integrated circuit, or other suitable processing device. The memory 104 can include any suitable computing system or medium, including, but not limited to, a non-transitory computer-readable medium, RAM, ROM, a hard drive, a flash drive, or other memory devices. The memory 104 can store information accessible by the one or more processors 102, including instructions 108 executable by the one or more processors. The instructions can be any set of instructions that, when executed by the one or more processors 102, cause the one or more processors 102 to provide the desired functionality.

[0028] Specifically, in some devices, the memory 104 can store instructions for implementing the location system 112, the augmented reality system 114, and the display system 140. The computing device 100 can implement the location system 112, the augmented reality system 114, and the display system 140 to present an augmented reality display with one or more filters that enable a user to control the augmented reality elements displayed in the AR view, including executing aspects of the present disclosure.

[0029] It should be understood that the terms "system" or "engine" can refer to specialized hardware, computer logic executed on a more general processor, or some combination thereof. Thus, a system or engine can be implemented in hardware, application-specific circuitry, firmware, and / or software that controls a general-purpose processor. In one embodiment, the system may be implemented as a program code file stored in a storage device, loaded into memory, and executed by a processor, or provided from a computer program product, such as computer-executable instructions stored in a tangible computer-readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0030] The memory 104 may also include data 106 that can be retrieved, manipulated, created, or stored by one or more processors 102, such as map data associated with the location system 112 (e.g., data representing a geographic area including one or more roads and one or more points of interest received from a server system). In some example embodiments, such data can be accessed and displayed to a user of the computing device 100 (e.g., during use of the location system 112 or the augmented reality system 114), or transmitted to a server computing system as needed.

[0031] In some exemplary embodiments, computing device 100 includes a location system 112, an augmented reality system 114, and a display system 140. The location system 112 can determine the location of the computing device 100 based at least in part on data generated by one or more sensors 110. In some examples, the location system 112 can access data within a geographic data store 234. The data stored in the geographic data store 234 can include data used to identify a description of one or more points of interest (e.g., buildings, monuments, merchants, etc.), the type of point of interest, the location of the point of interest, and one or more features associated with the point of interest.

[0032] In some examples, the location system 112 can be associated with a global positioning system (GPS) that provides location system coordinates for determining the location of the computing device 100 using a particular coordinate system. The coordinates provided by the GPS can be used in combination with the information in the geographic data store 234 to identify a particular location of the computing device 100.

[0033] The augmented reality system 114 can include an element presentation system 120, a filtering system 122, a clustering system 124, and an identification system 126. The augmented reality system 114 can be a system that displays an AR view to a user via a display of the computing device 100. The AR view can include a live video of a portion of the physical world captured by a camera associated with the computing device 100, overlaid with additional information provided by the augmented reality system 114, associated with objects and / or locations depicted in the image of the physical real world. In some examples, the displayed image can represent a live video of a portion of the physical world captured by a camera associated with the computing device 100. In this way, the user can use the camera included in the computing device 100 to capture a video of the physical real world, and the augmented reality system 114 can extend the live video by overlaying additional information that can be useful to the user.

[0034] For example, the user can instruct the camera included in the smartphone to capture a video of the row of stores on the road. The smartphone can use its display to present an AR view that includes the captured live video of the row of stores. The augmented reality system 114 can overlay (or otherwise combine) the live video with additional visual images provided by the augmented reality system 114 itself. This additional visual image can include additional information that the user may find useful. In some examples, the augmented reality system 114 can add elements containing supplementary information to the live video display. The supplementary information can include the name of each store, the type of each store, information about the goods and services provided by the store, evaluation information, and the like. This enables the user of the augmented reality system 114 to quickly and easily obtain additional information about the surroundings presented in an easily understandable format (e.g., visually displayed near the corresponding physical location or object).

[0035] The element presentation system 120 can generate one or more augmented reality elements for display within the AR view. The element presentation system 120 can determine which elements need to be shown and where in the AR view the AR elements should be displayed. Each element can include supplementary information about one or more objects, geographical locations, points of interest, or other aspects of the live video displayed in the AR view.

[0036] In some examples, the AR element can be a graphic object that can be inserted into the AR view and can represent information associated with a specific portion of the area near the computing device 100. In some examples, the AR element can be a text label overlaid on a portion of the AR view that provides additional information. For example, a displayed entity (including objects, locations, buildings, roads, streets, routes, etc.) can have an associated AR element that notifies the user of the name of the corresponding entity. The AR element can be an object visually displayed within the AR view, and the user can interact with it to access additional information or accomplish a task within the AR view. For example, a merchant store can have an AR element overlaid on it. The AR element can have the name of the merchant store. Selecting the element (e.g., tapping or clicking on the element) can cause the size of the element to expand and additional information (e.g., the type of merchant or business hours) to be provided.

[0037] In some examples, the element presentation system 120 can present a plurality of filter elements. Each filter element can be associated with a specific type of point of interest. For example, the filter element can include the name "coffee shop" and can thus be associated with merchants that sell coffee. Additionally, the filter element can be provided overlaid on the AR view itself or in the navigation section of the AR view that is located under the live video data.

[0038] When the element hint system 120 displays one or more filter elements, the user can select the filter element by interacting with the AR view. For example, the user can click, tap, or otherwise select a specific filter. In response to determining that the user has selected a specific element, the filtering system 122 can display the AR elements associated with the selected filter. For example, if the selected filter is "park", the AR view can be updated to display the AR elements associated with any park within the view of the live video. In some examples, the AR view can also be updated with AR elements representing parks that are nearby but not currently in the AR view.

[0039] In some examples, before the filter button is selected, the AR view may already display an initial set of AR elements. In some examples, the elements included in the initial set of AR elements are determined based on the most popular AR elements at a specific location. In other examples, the initial set of elements can be selected based on the user's preferences. In some examples, the initial set of AR elements can be determined by a plurality of factors including time, the user's history, a specific location, and past user input (e.g., an AR element with more user interactions than other AR elements may be more likely to be displayed in the initial set of AR elements).

[0040] In some examples, when an initial set of AR elements is already displayed and the user selects a specific filter button, the filtering system 122 can update the AR view to remove one or more AR elements from the initial set of AR elements and AR elements from a filter-based set of one or more AR elements. For example, the filtering system can remove any AR element not associated with the selected filter element. For example, if the user selects the filter element "restaurant", the elements can be updated to remove any AR element not associated with a restaurant.

[0041]

[0041] In other examples, the filtering system 122 can replace an initial set of AR elements with a filter-based set of AR elements. Thus, AR elements not associated with the selected filter can be removed, and new AR elements that are associated with the selected filter but have not been previously displayed can be added to the AR view.

[0042]

[0041] The clustering system 124 can group data from two or more points of interest into a single AR element. The points of interest can be grouped based on their associated point-of-interest types or based on their positions within the AR view. For example, if multiple AR elements associated with the same type of point of interest are in proximity to each other, the clustering system 124 can group them into one group and generate a grouped AR element that references the type of point of interest that the AR elements have in common. The user can then select the grouped AR element, and in response, the AR view can be updated to display information associated with the point of interest included within the group.

[0043] In some examples, the AR elements can be grouped based on their positions within the AR view. For example, the augmented reality system 114 can determine a maximum density of AR elements for a particular AR view. The maximum density can describe a minimum space or range between different AR elements that is required. If two or more elements are closer than what would be allowed by the minimum range, the clustering system 124 can determine whether to combine the AR elements into clusters. If so, clusters can be generated and represented by the AR elements of the entire cluster. In some examples, the clustering system 124 can determine to make one of the AR elements non - visible instead of combining the AR elements into clusters. Determining whether to make an AR element non - visible or to cluster it can be based on the relative relevance of each element, the type of the element, and the user's preferences.

[0044] In some examples, in response to filter elements selected by the user, the clustering system 124 can update the clusters in which the AR elements are grouped. For example, elements associated with the selected filter can be removed from the clusters and displayed separately, while elements not associated with the selected filter can be made non - visible or de - emphasized. In some examples, a group of points of interest associated with the selected filter that are near (e.g., within a particular threshold) the location of the computing device 100 but not currently included in the current AR view can be grouped into a cluster, and the AR elements representing the cluster can be displayed in a portion of the AR view associated with the direction in which the entity or point of interest is located. Thus, if some points of interest associated with a filter are far from the camera and on the right side of the current view, AR elements associated with some of the points of interest can be generated and displayed in the right - hand portion of the current AR view.

[0045] The identification system 126 can receive user input and, in response, identify a specific object or building within the AR view. For example, the user can point the camera of their device at a specific object or location in the real world. The camera can capture an image of that location and provide those images to the augmented reality system 114. The user can indicate via input to the computing device 100 that they want to identify an object, building, or location.

[0046] In response to that input, the identification system 126 can determine information about the selected object, building, or location based on the location information associated with the computing device 100 and the geographic data store 234. In some examples, this information can be displayed to the user and the user interface of the AR view. For example, the name of the location can be displayed on an AR element above the selected location.

[0047] The geographic data store 234 can store various location or geographic data. For example, the geographic data store 234 can include map data. In some examples, the map data can include information describing locations, points of interest, buildings, roads, parks, and other geographic features. The map data can include information that correlates an address to a specific geographic location so that a user can enter an address and the system can display information about that address (e.g., which building is there if that's the location). Similarly, a user can enter the name of a location of interest or a point of interest, and the system can identify the address associated with that location.

[0048] In some examples, the geographic data store 234 can include information associated with a positioning system such as a Global Positioning System (GPS), whereby the location of a particular computing device can be determined with respect to the map data. The geographic data store 234 can include data that enables the augmented reality system 114 to determine one or more points of interest within a predetermined distance from the computing device 100.

[0049] In some examples, the geographic data store 234 can also include geographic data that is specifically generated for display as part of the augmented reality display. For example, the augmented reality system 114 can generate a plurality of annotations for display within the AR view. In some examples, the annotations can be automatically generated for particular points of interest based on information stored in the geographic data store 234. In some examples, the annotations can be used for an initial set of augmented reality elements displayed within the AR view before specific filter elements are selected.

[0050] In some examples, the geographic data store 234 can also include information that enables the augmented reality system 114 to identify one or more objects, buildings, or locations based on their appearance in the live video data presented by the augmented reality system 114. For example, the geographic data store 234 can include reference images, data describing the appearance of points of interest, and user feedback data that enables the augmented reality system 240 to accurately determine which parts of the live video are associated with which geographic features, objects, buildings, locations, etc. within the live video.

[0051] FIG. 2 shows an exemplary client-server environment 200 in accordance with an exemplary embodiment of the present disclosure. The client-server environment 200 includes one or more user computing devices 210 and a server computing system 230. One or more communication networks 220 can interconnect these components. The one or more communication networks 220 can be any of a variety of network types including a local area network (LAN), a wide area network (WAN), a wireless network, a wired network, the Internet, a personal area network (PAN), or a combination of such networks.

[0052] The user computing devices 210 can include, but are not limited to, smartphones, smartwatches, fitness bands, navigation computing devices, laptop computers, and embedded computing devices (e.g., computing devices integrated within other objects such as clothing, vehicles, or other objects). In some examples, the user computing devices 210 can include one or more sensors intended to collect information with the permission of the user associated with the user computing device 210. For example, the user computing device 210 can include a camera 214.

[0053] In some examples, the user computing device 210 can connect to other computing devices, such as a personal computer (PC), laptop, smartphone, tablet, cellular phone, a computing component of a vehicle, or any other computing device capable of communicating with the communication network 220. The user computing device 210 can include one or more applications, such as a search application, a communication application, a navigation application, a productivity application, a game application, a word processing application, an augmented reality application 212, or any other application. The application(s) can include a web browser. The user computing device 210 can send and receive requests to and from the server computing system 230 using the web browser (or other application). The application(s) can include an augmented reality application 212 that generates an augmented reality view for display within the interface of the user computing device 210.

[0054] As shown in FIG. 2, the server computing system 230 can generally be based on a three-layer architecture consisting of a front-end layer, an application logic layer, and a data layer. As will be understood by those skilled in the relevant computer and Internet-related arts, each component shown in FIG. 2 can represent a set of executable software instructions and the corresponding hardware (such as memory and a processor) for executing the instructions. To avoid unnecessary detail, various components and engines not closely related to conveying an understanding of the various examples have been omitted from FIG. 2. However, those skilled in the art will readily recognize that various additional components and engines may be used with the server computing system 230 as shown in FIG. 2 to facilitate additional functionality not specifically described herein. Further, the various components shown in FIG. 2 may be present on a single server computer or may be distributed across several server computers in various arrangements. Further, although FIG. 2 shows the server computing system 230 as having a three-layer architecture, the various exemplary embodiments are in no way limited to this architecture.

[0055] As shown in FIG. 2, the front end can consist of an interface system(s) 222 that receives communications from one or more user computing devices 210 and transmits an appropriate response to the user computing devices 210. For example, the interface system(s) 222 can receive requests in the form of Hypertext Transfer Protocol (HTTP) requests, or other web-based application programming interface (API) requests. The user computing device 210 may execute a conventional web browser application or an application developed for a particular platform that includes any of a wide variety of computing devices and operating systems.

[0056] As shown in FIG. 2, the data layer can include a geographic data store 234. The geographic data store 234 can store various location data. For example, the geographic data store 234 can include map data. In some examples, the map data can include information describing locations, points of interest, buildings, roads, parks, and other geographic features. The map data can include information that correlates addresses with specific geographic locations so that a user can input an address and information about that address (e.g., which building is there if it is at that location). Similarly, a user can input the name of an interesting location or point of interest, and the system can identify the address associated with that location.

[0057] In some examples, the geographic data store 234 can include information associated with a positioning system such as a Global Positioning System (GPS), whereby the location of a particular computing device can be determined with respect to the map data. The geographic data store 234 can include data that enables the augmented reality system 240 to determine one or more points of interest within a predetermined distance from the user computing device 210.

[0058] In some examples, the geographic data store 234 can also include information that enables the augmented reality system 240 to identify one or more points of interest, objects, buildings, or locations based on their appearance in the live video data presented by the augmented reality system 240. For example, the geographic data store 234 can include reference images, data describing the appearance of points of interest, and user feedback data that enables the augmented reality system 240 to accurately determine which parts of the live video are associated with which geographic features, objects, buildings, locations, etc.

[0059] In some examples, the geographic data store 234 can include location characteristic information about a plurality of points of interest, locations, or entities. As described above, the location characteristic information can include the address of the location, the geographical location of the location, the type of the location, the type of the entity, the business hours of the entity, and the like. In some examples, the geographic data store 234 can also include image data, and the image data is associated with one or more geographic regions. The geographic data store 234 may also include satellite image data associated with one or more geographic regions.

[0060] The application logic layer can include application data that can provide a wide range of other applications and services that enable a user to access or receive geographic data for use within the AR system or for other purposes. The application logic layer can include an augmented reality system 240 and a search system 242.

[0061] The user computing device 210 (such as a user computing device like a smartphone) captures live video data of a portion of the world. The user computing device 210 can transmit information regarding the location of the user computing device 210 and the orientation of the camera 214 included in the user computing device 210 to the server computing system 230.

[0062] In response to receiving this information from the user computing device 210, the augmented reality system 240 can generate an augmented reality layer to be displayed on top of the real-world image. In some examples, the augmented reality layer can include one or more augmented reality elements associated with specific points of interest within the real world.

[0063] In some examples, the augmented reality system 240 can generate a list of augmented reality elements to be displayed based on the received position of the user computing device 210 and the orientation of the associated camera 214. In some examples, the list is generated based on the importance or popularity of the points of interest. The augmented reality system 240 can provide one or more filter elements for display within the AR view generated by the augmented reality application 212. Each filter element can be associated with a particular type of point of interest. For example, the filter elements can be based on categories such as restaurants, museums, monuments, historical points of interest, and the like. When the user selects a filter element, the current set of displayed augmented reality elements can be replaced with the set of augmented reality elements associated with the selected filter.

[0064] In some examples, the server computing system 230 prepares a rendered AR layer for presentation on the user computing device 210. In other examples, the augmented reality system 240 provides data (e.g., geographic data, data associated with specific entities, and data describing which AR elements and filter elements to display) to the user computing device 210, and the augmented reality application 212 creates the AR layer.

[0065] In some examples, the augmented reality system 240 can access live image data of the physical world and combine it with the augmented reality layer. In this way, the user computing device 210 can display a live video of the real world that the camera 214 of the user computing device 210 is pointed at. The live video can be overlaid with additional interface elements provided by the augmented reality system 240 based on information within the geographic data store 234. As the user computing device 210 moves and the live image of the physical world changes, the augmented reality layer can also be updated to correctly overlay data over the corresponding portion of the real world within the AR view.

[0066] In some examples, the search system 242 can receive a search input or query from the user computing device 210 and generate data in response to the search input or query. For example, the augmented reality system 240 can generate a plurality of filter elements that can be displayed within the AR view (or just below it within the user interface). Each filter element can be associated with one or more types of points of interest. If the user selects one of these filter elements, this selection can be interpreted as the user performing a search for that topic. The AR view can be updated to display the AR elements associated with the selected filter element. In some examples, the displayed AR elements can be deleted if they do not match the filter and new AR elements can be presented based on the filter.

[0067] In some examples, the user may enter a specific search query into a search field. For example, the search query can be "restaurants nearby". In response, the search system 242 can generate a plurality of query responses based on the information in the geographic data store 234. This information can be sent to the user computing device 210 for display. In some examples, this can be presented as a list of locations within a user interface that is associated with but does not include the AR view. In other examples, each search result can be presented as one or more elements within the AR view. In some examples, the user can swipe left or right on an interface associated with the AR view to switch between a visual representation of the AR elements associated with this search result and a list presentation of the search results.

[0068] FIG. 3 shows an exemplary augmented reality user interface 300 in accordance with an exemplary embodiment of the present disclosure. The exemplary interface includes a live image of the real world within the AR view 302, a plurality of AR elements 304, and a list 306 of one or more filter elements.

[0069] In this example, the live image (e.g., live video) within the AR view 302 includes a video of a street that includes multiple buildings and other objects. As the user computing device (e.g., the user computing device 210 of FIG. 2) moves and changes the viewing angle, the live image within the AR view 302 can be updated.

[0070] In this example, the AR view can include a plurality of AR elements 304. In this example, the AR elements 304 include two elements (a restaurant element and a shopping element), a street name label, and an interface element that provides information regarding the location of the Empire State Building. These AR elements provide additional information to the user.

[0071] In this example, one or more filter elements 306 can include a restaurant filter, a shopping filter, and a coffee filter. The exemplary augmented reality user interface 300 shows filter elements 306 (filter elements based on point-of-interest categories) that can be presented for the user to select (e.g., tap). Filtering elements (or chips) can be generated based on POIs (points of interest) near the user. The filters can be ordered based on various factors including, but not limited to, time (e.g., morning, the user is more interested in "coffee" than "shopping"), the number of POIs (points of interest) of that category nearby, the user's interests, and the like.

[0072] When the user selects a filtering element from one or more filter elements 306, the augmented reality elements shown in the AR view can be updated to include only the AR elements associated with the points of interest associated with the category of the filter button.

[0073] Figure 4 shows an exemplary augmented reality user interface 400 in accordance with an exemplary embodiment of the present disclosure. The exemplary interface includes a live image of the real world within the AR view 402, a plurality of augmented reality interface elements (e.g., 404-1 and 404-2), and a search query field 408. In this example, one of the augmented reality elements is a cluster augmented reality interface element 406. The cluster augmented reality interface element 406 can be associated with a plurality of points of interest (in this case, three nearby points of interest having the type of point of interest "restaurant" are clustered together and represented by a single AR element).

[0074] In this example, the live image (e.g., live video) included within the AR view 402 includes a street video including a plurality of buildings and other objects. As the user computing device (e.g., the user computing device 210 of FIG. 2) moves and changes the viewing angle, the live image within the AR view 402 can be updated.

[0075] In this example, the AR view 402 can include a plurality of AR interface elements 404-1 and 404-2. In this example, the AR elements 404 include two elements associated with a particular restaurant (both of which are represented by restaurant icon elements (404-1 and 404-2)), a street name label, and a cluster augmented reality interface element 406. In some examples, the cluster augmented reality interface element 406 is generated based on the results of a clustering algorithm. The cluster augmented reality interface element 406 can represent three points of interest.

[0076] In some examples, a clustering algorithm can be implemented to avoid overcrowding of results in one section of the AR view 402. Overcrowding within the AR view 402 can make it difficult for the user to read information and still view the live video data. In some examples, points of interest can be clustered based on proximity within the display (e.g., screen space clustering). For example, an augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can determine a minimum allowable distance between two AR elements. If two points of interest within the AR view 402 are close enough such that their associated AR elements would be further apart than the minimum allowable distance, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) clusters them into a group and represents the entire group with a single AR element.

[0077] In some examples, an augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can determine the minimum distance based on the size of the display (e.g., the number of pixels or the physical size of the display) and the size of the AR elements. Thus, two different displays may result in different clustering of the points of interest. Also, the clustering determination can be made based on the size of the AR view 402 associated with the point of interest. For example, if four points of interest of a particular type are visible in the AR view 402, and one of them occupies 50 percent of the display while the other three each occupy only 10 percent of the display, the augmented reality system can cluster the three points of interest into a group while leaving the larger point of interest as a single AR element. In some examples, the AR elements can be resized and their positions shifted to maintain the minimum allowable distance without clustering the points of interest.

[0078] Additionally or alternatively, the points of interest can be clustered based on the type of the point of interest. For example, a group of coffee shops can be combined into a group represented by a single augmented reality element. In some examples, both proximity clustering and semantic clustering can be used to cluster the points of interest. When clustering based on screen space, the system can group all locations within a specific region of the screen. For example, a screen space cluster can include points of interest that are at different distances from the computing device but are all within the same line of sight of the user. When clustering based on type, the system can use the geographical attributes of the location (e.g., the proximity of the point of interest to a street, neighborhood, transportation stop, or popular landmark) to group the results. For example, "5 restaurants near Coit Tower" or "3 clothing stores on Market Street" are examples of clusters defined by proximity to well-known geographical locations.

[0079] In some examples, the clustering methods can be combined (both screen space clustering and semantic clustering), whereby the points of interest within the AR view can be grouped based on screen space proximity, and the points of interest outside the radius can be grouped using type-based clustering.

[0080] FIG. 5 shows an exemplary augmented reality user interface 500 in accordance with an exemplary embodiment of the present disclosure. The exemplary interface includes a live image of the real world in the AR view 502. In this example, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) may enable the user to identify nearby locations. To do this in an efficient and query-free manner, the user computing device (e.g., the user computing device 210 of FIG. 2) may enable the user to select an object or location within the AR view. The augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can generate an identification of the selected object or location.

[0081] In some examples, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can generate an identification of the selected object or location, information regarding the orientation of the camera capturing the live video of the geographical location, an image displayed at the point selected by the user, and stored geographical data, based on location information associated with the user computing device (e.g., location information provided by the user computing device itself).

[0082] For example, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can receive user input indicating a user request to identify an object or location associated with the position at which the user input was received on the touch screen of the user computing device (e.g., the user computing device 210 of FIG. 2). In response to this input, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can determine the current position of the user computing device (e.g., a smartphone). The augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can determine the orientation of the camera capturing an image of the world based on sensors within the user computing device.

[0083] An augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can determine the currently displayed geographical location using the determined position of the device and the orientation of the camera. The augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can access geographical data associated with the displayed geographical location from the geographical data store 234.

[0084] An augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can identify a portion of the displayed image selected or indicated by the user. The augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can analyze the image associated with the selected position and compare it with the geographical data obtained from the geographical data store 234. Based on the analysis of the selected image and the data from the geographical data store, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can identify the specific position or object selected by the user.

[0085] An augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can access information regarding the selected position or object. For example, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can access information such as the name of the position or object, the category or type associated with the position or object, information regarding available services and products, etc. In some examples, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can update the augmented reality view to include information regarding the identified object or position. The information can include AR elements within the AR view. In other examples, the information can include a section of the display specialized to provide additional information regarding the selected object or position.

[0086] In this example, the user is indicating a position on a display within the augmented reality user interface 500. The indicated position can be marked with the augmented reality interface element 506. The augmented reality system can analyze one or more features of the display video (e.g., one or more live images of a portion of the world) to identify specific objects, locations, or points of interest associated with the selected portion of the AR view. In this example, the user selects the front facade of a clothing store. The augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can determine a specific entity (e.g., a merchant) associated with the selected front facade based on one or more of the position of the user computing device and the orientation of the camera. In some examples, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can update the AR view to include an AR element 506 that provides additional information about the selected front facade (e.g., the name of the associated merchant and a link to additional information).

[0087] In some examples, additional information about the selected object, location, or point of interest can be displayed in the information display section 504 of the user interface below the AR view. In this example, the information display section 504 includes the name of the selected merchant, review or rating data associated with the merchant, distance information, type information of the point of interest, and access time information. In some examples, this information can be provided in other parts of the augmented reality user interface 500.

[0088] FIG. 6 shows an exemplary augmented reality user interface 600 according to an exemplary embodiment of the present disclosure. As described above, the augmented reality user interface 600 can include an AR view 602. The AR view 602 can include a live image (e.g., video) of a region of the world. These images can be captured by a camera associated with the user computing device presenting the AR view. These live images can be augmented with additional information provided by the user computing device (e.g., the user computing device 210 of FIG. 2) or a server computing system (e.g., the server computing system 230 of FIG. 2) that provides an augmented reality system to the user computing device via a communication network (e.g., the user computing device 210 of FIG. 2).

[0089] In some examples, the additional information can be used to present one or more augmented reality elements within the AR view 602 proximate to an image of a location, object, or point of interest with which one or more augmented reality elements are associated. For example, if an augmented reality system (e.g., the augmented reality system 114 of FIG. 1) determines that a particular merchant (e.g., a store) is within the AR view 602, the augmented reality system can generate an AR element associated with the particular merchant. The generated AR element can be displayed over or near the image of the merchant within the AR view 602.

[0090] In some examples, the AR view 602 can also include a list of filter elements. The filter elements can be used to indicate the types of augmented reality elements that the user wants to see in the AR view 602. Each filter element can be associated with a particular type of point of interest. When the user selects a particular filter element, the AR view 602 can be updated to include augmented reality elements of the type associated with the selected filter. For example, the user can select a "coffee" filter. In response, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can update the AR view 602 to include augmented reality elements associated with "coffee", which can include, but are not limited to, coffee shops, cafes, fast food restaurants that offer coffee, etc. Additionally, AR elements associated with other topics or types of points of interest can be removed, de-emphasized, or made non-visible.

[0091] In some examples, when the user selects a particular filter element, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can determine a list of points of interest (e.g., objects, locations, buildings, etc.) that are within a particular distance from the user computing device. The points of interest in the list of points of interest are associated with the same type of point of interest as the selected filter element. In some examples, the user interface can display interface elements that can be associated with the list of relevant points of interest. For example, in addition to displaying one or more elements associated with a particular point of interest (e.g., 612, 614, and 616) in the list, the augmented reality user interface 600 can include a "View List" button 620. The "View List" button 620 can be selected by the user. In response, the augmented reality user interface 600 can be updated to display information associated with the list of points of interest.

[0092] In this example, the selected filter element was associated with "shopping". The generated list of points of interest includes one or more stores. When the view list button 620 is selected, the information associated with the one or more stores is displayed by changing the augmented reality user interface 600 to show the list. For example, the list can be displayed on top of the AR view 602 such that the list partially or fully obscures the AR view 602. In other examples, the AR view 602 can be temporarily shifted out of the interface and the list of information can be temporarily shifted into the interface. In some examples, the list view can be interactive, can be ordered by salience, and can allow the user to select a point of interest and directly view the location within the AR view 602. For example, if the user selects a particular point of interest from the list of points of interest, the AR view 602 can be updated to highlight or otherwise indicate the selected point of interest within the AR view 602.

[0093] FIG. 7 shows an exemplary augmented reality user interface 700 according to an exemplary embodiment of the present disclosure. As described above in the description of FIG. 6, a user can select a particular filter element (and thereby select a type of point of interest). In response, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can generate a list of points of interest of that type that are within a threshold distance of the location of the user computing device. In FIG. 6, when the user selects the "view list" button, the list can be presented to the user within a different page of the augmented reality user interface 700. FIG. 7 represents an alternative where the list and the AR view 702 are simultaneously displayed within the same interface. To do this, the list of points of interest can be displayed below (or partially obscuring the bottom portion of) the AR view 702 within a card carousel 720 (e.g., a horizontally scrollable list).

[0094] This layout of the augmented reality user interface 700 can have the advantage of enabling the user to fully interact with the augmented reality user interface 700 with only one hand. For example, if the user is holding a smartphone in one hand and a bag in the other hand, or is otherwise hindered, the user can still scroll the card carousel 720 with a thumb.

[0095] In some examples, the user can swipe through the card carousel 720 to receive additional information about one or more points of interest. As the user swipes through the card carousel 720, the currently displayed card can be made to stand out or be highlighted. Each card within the card carousel 720 can include additional supplementary data regarding the associated point of interest. The supplementary data can include the name of the point of interest, review information, information (if any) associated with products and services available at the point of interest, cost metrics associated with the point of interest (e.g., grouping into specific cost groups), the current distance to the point of interest, business hours, and the like.

[0096] In some examples, the AR view 702 can be controlled to highlight augmented reality elements associated with the card currently displayed within the card carousel 720, enabling the user to identify specific points of interest within the AR view 702. In the illustrated example, the card carousel is currently displaying a card associated with a store called "Brook’s Brothers". The AR element (AR element 712) associated with the Brook’s Brothers store is also visually distinguished.

[0097] Alternatively, the user can select a point of interest by directly tapping an icon within the camera view, in which case the card carousel 720 can be scrolled to display the card associated with the currently selected point of interest. For example, the user can select one of the points of interest by selecting a displayed AR element (e.g., one of 712, 714, or 716), and the card carousel 720 can be updated to reflect the selected point of interest.

[0098] FIG. 8 shows an exemplary augmented reality user interface 800 according to an exemplary embodiment of the present disclosure. To prevent user fatigue during interaction, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can include an alternative user interface for displaying information about the user's surroundings. For example, if the user does not want to raise the user computing device (e.g., the user computing device 210 of FIG. 2) vertically to facilitate capturing video data with a camera integrated into the user computing device, the user can lower and tilt the device to a more horizontal position. When the augmented reality system (e.g., the augmented reality system 114 of FIG. 12) determines, based on sensors included in the user computing device (e.g., the user computing device 210 of FIG. 2), that the tilt of the user computing device (e.g., the user computing device 210 of FIG. 2) has exceeded a threshold angle, the augmented reality system (e.g., the augmented reality system 114 of FIG. 12) can replace the AR view with a map view 804 around the user computing device (e.g., the user computing device 210 of FIG. 2).

[0099] In the map view 804, one or more points of interest (e.g., locations, buildings, objects, etc.) can be indicated using marker elements on the map view 804. In some examples, the specific points of interest marked in the map view are determined based on AR elements that were displayed in the AR view before the map view 804 was displayed. For example, if a particular point of interest has an AR element associated with it within the AR view, the augmented reality system (e.g., the augmented reality system 114 of FIG. 1) can generate a corresponding marker within the map view.

[0100] In this example, a series of markers (806-1 through 806-6) are displayed in the map view 804 at positions associated with corresponding points of interest. In some examples, the displayed markers represent a list of all points of interest having a type that matches the type of selected filter elements within a predetermined distance from the position of the user computing device. In this example, the user computing device (e.g., the user computing device 210 of FIG. 2) is at the position of marker 808, and line 810 represents a predetermined distance from that position. The predetermined distance can be determined based on the distance that can be traveled in a five-minute walk. Other distances or measures of distance can also be used. For example, the predetermined range can be 400 meters. Note that the markers may include points of interest that were not displayed in the AR view when the phone was tilted (because they were outside the camera view) but are included in the map view 804 because they match the type of selected points of interest (e.g., shopping) and are within the predetermined range (indicated by line 810).

[0101] In some examples, a limited list of matching points of interest within a predetermined distance can be displayed. This limited list can rank multiple matching points of interest based on one or more factors (e.g., user preferences, ratings, costs, business hours, degree of matching the filter elements they selected, etc.), and select a specific number of points of interest based on the ranking. The specific number of points selected can be determined based at least in part on the total number of candidate points of interest and the zoom level at which the map view 804 is currently being displayed. Thus, the user can zoom in to display more AR elements associated with the point of interest and zoom out to display fewer AR elements.

[0102] In some examples, the user can select one of the markers 806-2 on the map. When the user selects a particular marker, the user interface 800 of the map view can be updated. For example, instead of displaying one or more filtering tools 812 below the map view 804, the user interface 800 can be updated to display information 820 about the point of interest associated with the selected marker 806-2. This information can include the name, rating, distance, etc. of the point of interest. Enabling the display of the map view when the phone is tilted to a horizontal position allows for comfortable browsing of filter results or search results that match the AR view. When the user finishes browsing the results in a comfortable manner, the user can return the user computing device (e.g., the user computing device 210 of FIG. 2) and tilt it past the threshold angle to cause the extended reality system (e.g., the extended reality system 114 of FIG. 1) to return the interface to the AR view. When the user selects a particular point of interest, the augmented reality elements can be displayed prominently in the AR view when returned to the display.

[0103] FIG. 9 shows an exemplary flowchart for a method of performing filtering and searching of elements within an augmented reality view according to an exemplary embodiment of the present disclosure. One or more portions of this method can be implemented by one or more computing devices, such as, for example, the computing devices described herein. Further, one or more portions of this method can be implemented as algorithms on the hardware components of the device(s) described herein. FIG. 9 shows elements that are executed in a particular order for purposes of illustration and description. Those skilled in the art will understand that any of the elements of the methods discussed herein can be adapted, rearranged, extended, omitted, combined, and / or modified in various ways without departing from the scope of the present disclosure using the disclosure provided herein. This method can be implemented by one or more computing devices, such as one or more of the computing devices shown in FIGS. 1-2.

[0104] A user computing device (e.g., user computing device 210 of FIG. 2) can include one or more processors, memory, and other components, which together, at 902, generate an interface for depicting an augmented reality (AR) view of at least a portion of a physical real-world environment for display by the user computing device (e.g., user computing device 210 of FIG. 2). Accordingly, the AR view can include a user interface on which a live video image is displayed. The live video can be captured by a camera integrated with the user computing device (e.g., user computing device 210 of FIG. 2). The camera can be a camera integrated with a smartphone. In this way, the video displayed in the AR view represents the user's current environment. Next, the AR view can extend the live video to display additional information to the user or present entertainment elements that enhance the user's experience of the current environment.

[0105] In some examples, a user computing device (e.g., the user computing device 210 of FIG. 2) can display one or more filter elements at 904, each filter element being associated with a type of point of interest. For example, a list of filters can be displayed at the bottom of the interface. The filters of this list can be presented under the live video section of the AR view. In other examples, the list of filters can be displayed as transparent buttons or elements on a portion of the live video that is not obstructive.

[0106] At 906, a user computing device (e.g., the user computing device 210 of FIG. 2) can access data that describes a plurality of points of interest within a portion of the physical real-world environment from a geographical location database. In some examples, a user computing device (e.g., the user computing device 210 of FIG. 2) can determine its own location. For example, a computing device (e.g., the user computing device 210 of FIG. 2) can include GPS, which enables the computing device to determine its current location in the world with a certain degree of accuracy. This location can be provided to the augmented reality system and used to determine which points of interest to access.

[0107] In some examples, a plurality of points of interest within a portion of the physical real-world environment can be within a predetermined distance of the location of the computing device. In some examples, the predetermined distance is based on the walking time from the location of the user computing device (e.g., the user computing device 210 in FIG. 2). For example, the predetermined distance can be based on an estimated walking time of 5 minutes from the location of the computing device. Other estimated walking times can also be used.

[0108] In some examples, each point of interest has an associated type (e.g., the type of the point of interest), and the filter element is associated with a particular type of point of interest. One or more of the plurality of points of interest are associated with a merchant. A point of interest can have a plurality of associated types or subtypes. For example, a particular diner can have not only a broad type such as "restaurant" associated with it, but also one or more subtypes such as "breakfast food" and "American food". Additionally, related types such as "coffee" can also be associated with the diner. Types can be associated with existing classification systems (e.g., North American Industry Classification System (NAICS) codes, Geographic Concept Identifier (GCID) codes, etc.). In some examples, one type can be determined to be a primary type and other types can be determined to be secondary types.

[0109] A user computing device (e.g., the user computing device 210 of FIG. 2) can display an initial set of augmented reality elements associated with an initial set of points of interest. The initial set of points of interest can be selected based on overall popularity, past user interaction information, a particular user's preferences, time, etc. The initial set of points of interest displayed for a particular location can be determined such that the most relevant points of interest are automatically highlighted without a particular user request. The initial set of augmented reality elements can be associated with one or more entities currently displayed within the AR view.

[0110] A user computing device (e.g., the user computing device 210 of FIG. 2) can receive a selection of one of the displayed filter elements at 908. For example, if the computing device includes a touch screen, the user can select a particular filter element by touching the screen at a location associated with the desired filter element. The user can select the "coffee" filter element by interacting with the area of the touch screen that includes the "coffee" filter element.

[0111] In response to receiving a selection of one of the displayed filter elements, a user computing device (e.g., the user computing device 210 of FIG. 2) can update the AR view to remove at least one augmented reality element from an initial set of augmented reality elements. Continuing with the above example, if the user selects the "coffee" filter, the user computing device (e.g., the user computing device 210 of FIG. 2) can update the AR by removing one or more augmented reality elements having a type of point of interest other than "coffee".

[0112] A user computing device (e.g., the user computing device 210 of FIG. 2) can provide a filter-based set of augmented reality elements associated with a set of points of interest for display within the AR view at 910, the filter-based set of augmented reality elements representing a filter-based set of points of interest associated with the selected filter element. For example, if the user selects the "coffee" filter, the AR view can be updated to include augmented reality elements (e.g., visual interface elements such as labels, markers, or buttons) that are of the type "coffee" or are associated with points of interest related to coffee.

[0113] As described above, one or more initially displayed augmented reality elements can be deleted and augmented reality elements associated with the "coffee" point of interest can be added. In some examples, one or more of the initial set of points of interest can be included within a filter-based set of points of interest. For example, the initial set of augmented reality elements can include one or more AR elements associated with a coffee shop. Next, when the user selects a "coffee" filter element, those AR elements associated with the coffee shop remain, but AR elements associated with other types are deleted and can be replaced with additional AR elements associated with the type of "coffee" point of interest selected.

[0114] Each AR element within the filter-based set of augmented reality elements can be displayed in proximity to the entity with which it is associated within the AR view. For example, if the AR element provides additional information about a store, the AR element can be positioned to overlap a portion of the image of the building in which the store is located. Thus, the AR view can provide the user with information about where a particular point of interest is, even when its location cannot be clearly determined with available signage.

[0115] A user computing device (e.g., the user computing device 210 of FIG. 2) can provide AR elements in an AR view by accessing image data from a camera associated with the user computing device (e.g., the user computing device 210 of FIG. 2). The user computing device (e.g., the user computing device 210 of FIG. 2) can analyze the image data to identify one or more entities within the image data based on a storefront associated with the entity. For example, the user computing device (e.g., the user computing device 210 in FIG. 2) can determine a user's approximate location using location information from the user computing device (e.g., a smartphone). Image data collected by a camera of the user computing device (e.g., the user computing device 210 in FIG. 2) can be analyzed to identify one or more geographic features (e.g., buildings, roads, markings, signs, etc.). The computing device can use information stored in a geographic data store to determine a specific location represented in the image data. Each geographic feature can be matched to one or more entries within a geographic data store (e.g., the geographic data store 234 of FIG. 1).

[0116] When a user computing device (e.g., the user computing device 210 of FIG. 2) matches features of image data (e.g., live video) to specific geographic entities, the user computing device (e.g., the user computing device 210 of FIG. 2) can display AR elements within the AR view located near features (e.g., storefronts) associated with the corresponding entities.

[0117] In some examples, at least one of the filter-based set of points of interest may not be displayed within the AR view. For example, when a user selects a "coffee" filter element, the user computing device (e.g., user computing device 210 of FIG. 2) can identify a plurality of points of interest near the user (e.g., within the user's threshold distance). Some of these points of interest may be within the range of the threshold distance but are not currently displayed in the image data captured by the camera of the user computing device (e.g., user computing device 210 of FIG. 2). For example, a point of interest may be on the right side of the live video image data. The AR view can be updated with AR elements indicating the direction and distance associated with the points of interest. In this way, the user can receive alerts about nearby points of interest that are not currently within the AR view.

[0118] In some examples, a particular AR element can be associated with a plurality of points of interest. In some examples, each point of interest has a type of point of interest. The user computing device (e.g., user computing device 210 of FIG. 2) can group a plurality of points of interest having the same type of point of interest into a group represented by a single AR element. This grouping can be done when the number of points of interest to be displayed is more than the number that can fit within the AR view. In some examples, all points of interest of a particular type that are currently off-screen can be represented by a single AR element.

[0119] In other examples, a user computing device (e.g., user computing device 210 of FIG. 2) can group points of interest into one group based on each position within the AR view. For example, a user computing device (e.g., user computing device 210 of FIG. 2) can determine that two or more augmented reality elements are displayed within a predetermined distance within the AR view. In response to determining that two or more augmented reality elements are displayed, a user computing device (e.g., user computing device 210 of FIG. 2) can combine the two or more augmented reality elements into a cluster of augmented reality elements within the AR view. In this way, the points of interest can be grouped to ensure that the AR view is not cluttered with too many AR elements.

[0120] In some examples, a user interface of an augmented reality system (e.g., augmented reality system 114 of FIG. 1) can enable a user to select features of image data for identification. For example, a user computing device (e.g., user computing device 210 of FIG. 2) can receive a user input indicating a first entity within the AR view. The user computing device (e.g., user computing device 210 of FIG. 2) can analyze the image data displayed within the AR view to extract one or more characteristics of the first entity. The user computing device (e.g., user computing device 210 of FIG. 2) can access data identifying a particular location within a geographical location database based on one or more characteristics of the first entity from the geographical location database.

[0121] The above description of FIG. 9 describes steps of a method executed by a user computing device (e.g., user computing device 210 of FIG. 2). However, these method steps may also be executed, in whole or in part, by a server computing system (e.g., server computing system 230 of FIG. 2) remote from the user computing system (e.g., user computing device 210 of FIG. 2). In some examples, the steps can be executed by multiple devices.

[0122] The technology discussed herein refers to sensors, servers, databases, software applications, and other computer-based systems, as well as the operations performed and the information sent to and from such systems. The flexibility inherent in computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications can be implemented in a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0123] Although the subject matter has been described in detail with respect to its various specific and illustrative embodiments, each example has been provided for purposes of illustration and is not intended to limit the disclosure. Those of ordinary skill in the art, upon reaching the foregoing understanding, can readily create modifications, variations, and equivalents to such embodiments. Accordingly, the disclosure of the subject matter is not intended to exclude such modifications, variations, and / or additions to the subject matter that would be readily apparent to those of ordinary skill in the art. For example, features illustrated or described as part of one embodiment may be used in other embodiments to create still further embodiments. Accordingly, the disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. A method implemented on a computer that enables search in an augmented reality interface, comprising: generating, by a computing system having one or more processors, an interface for depicting an augmented reality (AR) view including image data of at least a portion of a physical real-world environment for display by the computing system; displaying, by the computing system, one or more filter elements within the interface, wherein each filter element within the one or more filter elements is associated with a type of point of interest; accessing, by the computing system from a geographical location database, data describing a plurality of points of interest within the portion of the physical real-world environment; receiving, by the computing system, a selection of one of the displayed filter elements; providing, by the computing system for display within the AR view, a filter-based set of augmented reality elements associated with a set of points of interest, wherein the filter-based set of augmented reality elements represents a filter-based set of points of interest associated with the selected filter element; A method implemented on a computer, comprising the above steps.

2. The method implemented on a computer according to claim 1, wherein a plurality of the plurality of points of interest are associated with merchants.

3. Before receiving a selection of one of the displayed filter elements, displaying, by the computing system, an initial set of augmented reality elements associated with an initial set of points of interest; in response to receiving the selection of one of the displayed filter elements, updating, by the computing system, the AR view to remove at least one augmented reality element from the initial set of augmented reality elements; The method implemented on a computer according to any one of claims 1 to 2, further comprising the above steps.

4. The computer-implemented method according to claim 3, wherein one or more of the initial set of points of interest are included in the filter-based set of points of interest.

5. The computer-implemented method according to any one of claims 3 to 4, wherein the initial set of augmented reality elements is associated with one or more entities currently displayed in the AR view.

6. The computer-implemented method according to any one of claims 3 to 5, wherein the initial set of augmented reality elements is selected for display based on one or more detected storefronts in the image data used to depict at least a portion of the physical real-world environment within the AR view.

7. The computer-implemented method according to any one of claims 3 to 6, wherein the initial set of augmented reality elements is selected for display based on a visibility determination of one or more points of interest having a predetermined distance from the computing device.

8. The computer-implemented method according to any one of claims 1 to 7, wherein each AR element in the filter-based set of augmented reality elements is displayed proximate to a point of interest associated in the AR view.

9. Providing, by the computing system, a filter-based set of augmented reality elements associated with a set of points of interest for display in the AR view comprises: accessing, by the computing system, image data from a camera associated with the computing system; analyzing, by the computing system, the image data to identify one or more entities in the image data based on storefronts associated with the entities; displaying, by the computing system, AR elements within the AR view disposed near features in the image data associated with corresponding entities; The computer-implemented method according to any one of claims 1 to 8, further comprising.

10. Determining the position of the computing system by the computing system, wherein the plurality of points of interest within the portion of the physical real-world environment are within a predetermined distance of the position of the computing system. The computer-implemented method according to any one of claims 1 to 9, further comprising.

11. The computer-implemented method according to claim 10, wherein the predetermined distance is based on a walking time from the position of the computing system.

12. The computer-implemented method according to claim 11, wherein the predetermined distance is based on an estimated walking time of 5 minutes from the position of the computing system.

13. The computer-implemented method according to any one of claims 1 to 12, wherein at least one point of interest in the filter-based set of points of interest is not displayed within the AR view.

14. The computer-implemented method according to any one of claims 1 to 13, wherein the displayed AR elements are associated with a group comprising a plurality of points of interest.

15. The computer-implemented method according to claim 14, wherein each point of interest has an associated type of point of interest, and the plurality of points of interest included in the group have the same type of point of interest.

16. Determining by the computing system that two or more augmented reality elements are displayed within a predetermined distance within the AR view; In response to determining that two or more augmented reality elements are displayed, combining the two or more augmented reality elements into a signal group augmented reality element within the AR view; The computer-implemented method according to claim 15, further comprising.

17. Receiving, by the computing system, a user input indicating a first entity within the AR view; Analyzing, by the computing system, the image data displayed in the AR view to extract one or more characteristics of the first entity. Accessing, by the computing system from the database of geographical locations, data that identifies a particular location within the database of geographical locations based on the one or more characteristics of the first entity A computer-implemented method according to any one of claims 1 to 16, further comprising **Claim 18** A computing device, comprising One or more processors, and A computer-readable memory storing instructions that, when executed by the one or more processors, cause the computing device to perform the method according to any one of claims 1 to 17 A computing device comprising **Claim 19** A computer-readable medium storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform the method according to any one of claims 1 to 17

Images