System and method for location search in augmented reality

Abstract

The invention provides a computer-implemented method, system and device for implementing search in an augmented reality interface. A computing device generates an interface for display by the computing system that depicts an AR view of image data that includes at least a portion of a physical real-world environment. The computing device displays one or more filter elements within the interface, a respective filter element associated with a point of interest type. The computing system accesses data describing a plurality of points of interest within the physical real-world environment from a database of geographic locations. The computing device receives a selection of one of the displayed filter elements. The computing device provides, for display in the AR view, a set of augmented reality elements associated with a set of points of interest, wherein the set of augmented reality elements represents a set of filter-based points of interest associated with the filter elements.

Description

[0001] Related Applications

[0002] This application is based on and claims the benefit of U.S. Provisional Patent Application No. 63 / 390,240 filed on July 18, 2022, which is incorporated herein by reference. Applicants claim priority and the benefit of this application, and incorporate this application in its entirety by reference. Technical Field

[0003] The present disclosure generally relates to augmented reality (AR). More specifically, the present disclosure relates to place searching within an AR view. Background Art

[0004] Computing devices (e.g., desktop computers, laptops, tablets, smartphones, wearable computing devices, etc.) are ubiquitous in modern society. They can support communication between users, provide their users with information about their environment, current events, the world at large, etc. One common use of such devices is to generate and display, for example, an augmented reality (AR) view of at least a portion of a physical real-world environment (e.g., an environment in which one or more such devices are located, etc.). The AR view can be part of an interactive AR experience provided to one or more users of such devices, where such experience is enhanced by computer-generated information perceptible in one or more sensory modalities of the users, etc. Summary of the invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] An example aspect of the present disclosure relates to a computer-implemented method. The method includes generating an interface for display by a computing system having one or more processors, the interface depicting an augmented reality (AR) view of image data including at least a portion of a physical real-world environment. The method further includes displaying one or more filter elements by the computing system within the interface, wherein a corresponding filter element in the one or more filter elements is associated with a point of interest type. The method further includes accessing data describing a plurality of points of interest within the portion of the physical real-world environment by the computing system from a geographic location database. The method further includes receiving a selection of one of the displayed filter elements by the computing system. The method further includes providing a filter-based augmented reality element set associated with a set of points of interest by the computing system for display in the AR view, wherein the filter-based augmented reality element set represents a filter-based set of points of interest associated with the selected filter element.

[0007] Another example aspect of the present disclosure relates to a computing device. The computing device includes one or more processors and a computer-readable memory, wherein the computer-readable memory stores instructions that, when executed by the one or more processors, cause the computing device to perform operations, the operations including generating an interface for display by the computing system, the interface depicting an augmented reality (AR) view of image data including at least a portion of a physical real-world environment. The operation further includes displaying one or more filter elements within the interface, wherein a corresponding filter element in the one or more filter elements is associated with a point of interest type. The operation further includes accessing data describing a plurality of points of interest within the portion of the physical real-world environment from a geographic location database. The operation further includes receiving a selection of one of the displayed filter elements. The operation further includes providing a filter-based set of augmented reality elements associated with a set of points of interest for display in the AR view, wherein the filter-based set of augmented reality elements represents a filter-based set of points of interest associated with the selected filter element.

[0008] Another example aspect of the present disclosure relates to a non-transitory computer-readable medium storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising generating an interface for display by the computing system, the interface depicting an augmented reality (AR) view of image data including at least a portion of a physical real-world environment. The operation further comprises displaying one or more filter elements within the interface, a corresponding filter element of the one or more filter elements being associated with a point of interest type. The operation further comprises accessing data describing a plurality of points of interest within the portion of the physical real-world environment from a geolocation database. The operation further comprises receiving a selection of one of the displayed filter elements. The operation further comprises providing a filter-based set of augmented reality elements associated with a set of points of interest for display in the AR view, wherein the filter-based set of augmented reality elements represents a filter-based set of points of interest associated with the selected filter element.

[0009] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0012] Figure 1 An example computing device is depicted according to an example embodiment of the present disclosure;

[0013] Figure 2 An example client-server environment is depicted according to an example embodiment of the present disclosure;

[0014] Figure 3 An example augmented reality user interface is shown according to an example embodiment of the present disclosure;

[0015] Figure 4 An example augmented reality user interface is shown according to an example embodiment of the present disclosure;

[0016] Figure 5 An example augmented reality user interface is shown according to an example embodiment of the present disclosure;

[0017] Figure 6 An example augmented reality user interface is shown according to an example embodiment of the present disclosure;

[0018] Figure 7 An example augmented reality user interface is shown according to an example embodiment of the present disclosure;

[0019] Figure 8 shows an example augmented reality user interface according to an example embodiment of the present disclosure; and

[0020] Fig. 9 An example flow chart of a method of performing filtering and searching for elements within an augmented reality view according to an example embodiment of the present disclosure is depicted. DETAILED DESCRIPTION

[0021] Reference will now be made in detail to embodiments of the present disclosure, one or more examples of which are shown in the accompanying drawings. Each example is provided by way of explanation of the present disclosure, rather than by way of limitation. In fact, it will be apparent to those skilled in the art that various modifications and changes may be made to the present disclosure without departing from the scope or spirit of the present disclosure. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to encompass modifications and changes as within the scope of the appended claims and their equivalents.

[0022] In general, the present disclosure relates to a system for providing a search and filtering system for presenting information in an augmented reality display. Specifically, a computing system including one or more computing devices can generate an interface depicting an AR view for display by at least one computing device. Such a view can have at least a portion of a physical real-world environment. The interface can be used to present information to a user on the go (e.g., traveling through the physical world using their user computing device) from a first-person perspective. For example, a user using one or more computing devices may be located near a travel route in an urban environment, and one or more sensors (e.g., cameras, etc.) of such computing devices can generate data representing a view of at least a portion of the urban environment, and the computing device can generate (e.g., at least in part based on sensor data, map data associated with a geographic location of the computing device, etc.) an interface depicting an AR view of at least a portion of the urban environment. For example, the view includes a screen of a portion of the urban environment (e.g., generated at least in part based on sensor data, etc.), and a screen generated at least in part based on map data associated with a geographic location (e.g., superimposed, overlapped, located within a screen of a portion of the urban environment, etc.).

[0023] The AR view may include one or more augmented reality elements (AR elements) superimposed on a screen of an urban environment or other real-world location. These augmented reality elements may include one or more indicators of specific entities in the environment. Entities may include any points of interest within the environment, including but not limited to buildings, landmarks, businesses, etc. Therefore, the AR view may include one or more tags that identify specific points of interest within the user's environment. In this way, the user can present additional information about their environment in the AR view. In some examples, such as dense urban environments, there are more potential points of interest in the AR view than can be reasonably displayed at one time. This is because the AR view has a limited amount of space to display additional AR elements before the interface becomes crowded and difficult to use.

[0024] In response to this situation, the computing system generating the AR view can identify an initial set of AR elements associated with an initial set of points of interest that can be displayed to the user. The initial set can be selected based on determining the points of interest that are visible to the user in the AR view. This can be achieved by analyzing the image data displayed in the AR view to detect storefronts or other visible features associated with particular points of interest. In some examples, if a large number of points of interest are visible, the computing system can use a ranking system in which each point of interest is ranked based on its relevance to the user. The most relevant AR elements can be displayed, while other elements with less relevance can remain hidden.

[0025] In some examples, an AR display may include multiple filter elements. Filter elements may be represented by elements in the interface of the augmented reality display. For example, a filter element may be a selectable button or chip within the display. Each filter element may be associated with a particular type of point of interest. For example, a filter element may be associated with categories such as restaurants, coffee shops, museums, monuments, etc.

[0026] The filter elements may be selected and ordered based on one or more factors. In some examples, the factors may include the number of points of interest of a particular type in an area of ​​the computing device. For example, if there are multiple restaurants in an area, one of the displayed filter elements may be the restaurants. In some examples, the displayed filter elements may be based on a user associated with the computing system. Thus, if a user has previously expressed interest in a particular type of point of interest, a filter element for that type of point of interest may be displayed.

[0027] In some examples, if the user selects a particular filter element, the interface of the augmented reality view can be updated. For example, the AR view can be changed so that one or more augmented reality elements that were originally displayed and associated with the initial set of points of interest can be replaced with one or more AR elements associated with the selected filter element. In this way, a particular entity, building, or point of interest that was originally referenced or represented by one or more AR elements in the view can be replaced with an AR element associated with the entity associated with the selected filter type.

[0028] In some examples, the initial set of augmented reality elements is associated with points of interest or entities currently visible in the AR view. For example, the computing system may identify one or more buildings, entities, or points of interest currently in the AR view based on image data captured by the camera. For example, if a specific storefront is currently displayed in the AR view, an AR element associated with the storefront may be displayed above or near the storefront. Once a filter element is selected, a filter-based set of augmented reality elements may be displayed. The filter-based set of augmented reality elements may be associated points of interest of a type that matches the selected filter. The filter-based set of augmented reality elements may include points of interest currently displayed in the AR view and points of interest that match the selected filter type associated with the location but are not currently in the AR view. In some examples, the filter-based set of augmented reality elements may include one or more augmented reality elements indicating the direction in which the associated entity is located. For example, if the associated entity is on the right side of the current view, the augmented reality element may be toward the right side of the screen, with an arrow indicating the direction of the entity's location.

[0029] In another example, a filter-based set of augmented reality elements includes an AR element associated with a point of interest that would be visible in the AR view but is obscured by a person, object, or building located between it and the camera of the computing device. In this example, the AR view can be updated, indicating where the point of interest would be within the AR view if it were not obscured. In some examples, the AR element may have a different appearance, depending on whether the point of interest is currently displayed or obscured (including behind one or more buildings). For example, a first icon may be used for an AR element associated with a point of interest currently displayed in the AR view, and a second icon may be used for an AR element associated with an obscured point of interest. The second icon may include information that informs the user of the direction of the point of interest and the distance from the current location of the computing device. In this way, the user can be informed of the direction and distance of the point of interest that matches the selected filter but is obscured and therefore not currently visible in the AR view.

[0030] In some examples, an AR view may have a certain amount of user interface space in which augmented reality interface elements are displayed. In some examples, if too many user interface elements are displayed at once, the augmented reality elements may obscure other parts of the AR view that are important to the user. For example, if an augmented reality element obscures one or more displayed parts of the real world, the usefulness of the augmented reality view may be reduced. Therefore, if more than one augmented reality element is planned to be displayed in the augmented reality interface, the computing system can determine whether the augmented reality elements are within a predetermined threshold distance of each other. If so, one or more augmented reality elements can be grouped together to create a cluster of points of interest that are all represented by a single AR element.

[0031] In some examples, points of interest may be clustered into a single augmented reality interface based on the type or category of the point of interest or entity. For example, multiple hotels may be grouped together into a single interface element on an augmented reality display.

[0032] The augmented reality view may be presented to the user so that the user may select or interact with various portions of the AR view. In some examples, the user may select (e.g., via touch input, mouse input, etc.) a particular feature or building within the AR view. The computing system may generate information about the selected feature or building for presentation to the user in response to the input.

[0033] The systems and methods of the present disclosure provide many technical effects and benefits. As an example, the proposed system can implement a place search within an augmented reality system. Enabling a user to filter and search from multiple possible augmented reality elements can reduce the number of augmented reality elements displayed at any given time without reducing the effectiveness and practicality of the augmented reality display. Improving the effectiveness of the augmented reality display can reduce the amount of power used and the amount of data stored when displaying the augmented reality display. Reducing the amount of storage required and the energy used can reduce the cost of the augmented reality system and improve the user experience. This represents an improvement in the functionality of the device itself.

[0034] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0035] Figure 1 An example computing device 100 is depicted in accordance with an example embodiment of the present disclosure. In some example embodiments, computing device 100 may be any suitable device, including, but not limited to, a smartphone, a tablet computer, a laptop computer, a desktop computer, a global positioning system (GPS) device, a computing device integrated into a vehicle, a wearable computing device, or any other computing device configured to allow a person to execute an augmented reality application or access an augmented reality service at a server computing system. Computing device 100 may include one or more processors 102, a memory 104, one or more sensors 110, a positioning system 112, an augmented reality system 114, and a display system 140.

[0036] The one or more processors 102 may be any suitable processing device, such as a microprocessor, a microcontroller, an integrated circuit, or other suitable processing device. The memory 104 may include any suitable computing system or medium, including but not limited to non-transitory computer-readable media, RAM, ROM, a hard drive, a flash drive, or other memory devices. The memory 104 may store information accessible by the one or more processors 102, including instructions 108 executable by the one or more processors. The instructions may be any set of instructions that, when executed by the one or more processors 102, cause the one or more processors 102 to provide desired functionality.

[0037] Specifically, in some devices, the memory 104 may store instructions for implementing the positioning system 112, the augmented reality system 114, and the display system 140. The computing device 100 may implement the positioning system 112, the augmented reality system 114, and the display system 140 to perform aspects of the present disclosure, including presenting an augmented reality display with one or more filters so that a user can control the augmented reality elements displayed in the AR view.

[0038] It should be understood that the term "system" or "engine" can refer to dedicated hardware, computer logic executed on a more general processor, or some combination thereof. Thus, the system or engine can be implemented in hardware, dedicated circuits, firmware, and / or software that controls a general-purpose processor. In one embodiment, the system can be implemented as a program code file stored on a storage device, loaded into a memory, and executed by a processor, or can be provided from a computer program product (e.g., computer executable instructions) stored in a tangible computer-readable storage medium (such as RAM, a hard disk, or an optical or magnetic medium).

[0039] The memory 104 may also include data 106, such as map data associated with the positioning system 112 (e.g., data representing a geographic area including one or more roads and one or more locations of interest received from a server system), which may be retrieved, manipulated, created, or stored by the one or more processors 102. In some example embodiments, such data may be accessed and displayed to a user of the computing device 100 (e.g., during use of the positioning system 112 or the augmented reality system 114) or transmitted to a server computing system as needed.

[0040] In some example embodiments, computing device 100 includes positioning system 112, augmented reality system 114, and display system 140. Positioning system 112 may determine the location of computing device 100 based at least in part on data generated by one or more sensors 110. In some examples, positioning system 112 may access data in geographic data repository 234. The data stored in geographic data repository 234 may include descriptions of one or more points of interest (e.g., buildings, monuments, businesses, etc.), point of interest types, point of interest locations, and data identifying one or more features associated with the point of interest.

[0041] In some examples, positioning system 112 may be associated with a global positioning system (GPS), which provides positioning system coordinates that determine the location of computing device 100 using a particular coordinate system. The coordinates provided by the GPS may be used in conjunction with information from geographic data repository 234 to identify the specific location of computing device 100.

[0042] Augmented reality system 114 may include element rendering system 120, filtering system 122, clustering system 124, and recognition system 126. Augmented reality system 114 may be a system that displays an AR view to a user through a display of computing device 100. The AR view may include an image of the real physical world superimposed with additional information associated with objects and / or locations depicted in the image of the real physical world provided by augmented reality system 114. In some examples, the displayed image may represent a real-time video of a portion of the physical world captured by a camera associated with computing device 100. In this way, a user may use a camera included in computing device 100 to capture a video of the real physical world, and augmented reality system 114 may enhance the real-time video by superimposing additional information that may be helpful to the user.

[0043] For example, a user may direct a camera included in a smartphone to capture a video of a row of stores on a street. The smartphone may use its display to present an AR view that includes a real-time video of a row of stores captured. The augmented reality system 114 may overlay (or otherwise combine) the real-time video with an additional visual screen provided by the augmented reality system 114 itself. This additional visual screen may include additional information that the user may find useful. In some examples, the augmented reality system 114 may add elements including supplementary information to the real-time video display. The supplementary information may include the name of each store, the type of each store, information about the goods and services provided by the store, rating information, and the like. This enables users of the augmented reality system 114 to quickly and easily obtain additional information about their surroundings presented in an easy-to-understand format (e.g., displayed visually near a corresponding physical location or object).

[0044] The element rendering system 120 may generate one or more augmented reality elements for display in the AR view. The element rendering system 120 may determine which elements should be displayed and determine where the AR elements will be displayed in the AR view. Each element may include one or more objects, geographic locations, points of interest, or other aspects of supplemental information to the real-time video displayed in the AR view.

[0045] In some examples, the AR element may be a graphical object that can be inserted into the AR view and may represent information associated with a specific portion of the area near the computing device 100. In some examples, the AR element may be a text label that provides additional information superimposed on a portion of the AR view. For example, a displayed entity (including an object, location, building, road, street, path, etc.) may have an associated AR element to inform the user of the name of the corresponding entity. An AR element may be an object displayed visually in an AR view, with which a user can interact to access additional information or complete a task within the AR view. For example, an AR element may be superimposed on a merchant store. The AR element may have the name of the merchant store. Selecting an element (e.g., tapping or clicking the element) may cause its size to increase and provide additional information (e.g., merchant type or business hours).

[0046] In some examples, the element presentation system 120 may present multiple filter elements. Each filter element may be associated with a particular type of point of interest. For example, a filter element may include the name "coffee shop" and thus be associated with a business that sells coffee. In addition, filter elements may be provided that are superimposed on the AR view itself or in a navigation portion of the AR view that is located below the real-time video data.

[0047] Once the element rendering system 120 displays one or more filter elements, the user can select the filter element by interacting with the AR view. For example, the user can click, tap, or otherwise select a particular filter. In response to determining that the user has selected a particular element, the filtering system 122 can display an AR element associated with the selected filter. For example, if the selected filter is "public parks", the AR view can be updated to display AR elements associated with any public parks in the view of the real-time video. In some examples, the AR view can also be updated with AR elements representing parks that are nearby but not currently in the AR view.

[0048] In some examples, the AR view may already display an initial set of AR elements before the filter button element is selected. In some examples, the elements included in the initial set of AR elements are determined based on the most popular AR elements at a particular location. In another example, the initial set of elements may be selected based on the user's preferences. In some examples, the initial set of AR elements may be determined based on multiple factors, including time of day, user history, specific location, and past user input (e.g., AR elements that have more user interactions than other AR elements may be more likely to be displayed in the initial set of AR elements).

[0049] In some examples, if an initial set of AR elements is already displayed and the user selects a particular filter button, the filtering system 122 can update the AR view to remove one or more AR elements in the initial set of AR elements and one or more AR elements in the filter-based set of AR elements. For example, the filtering system can remove any AR elements that are not associated with the selected filter element. For example, if the user selects the filter element "restaurants," the element can be updated to remove any AR elements that are not associated with restaurants.

[0050] In other examples, the filtering system 122 may replace the initial set of AR elements with the set of AR elements based on the filter. Thus, AR elements not associated with the selected filter may be removed, and new AR elements associated with the selected filter but not previously displayed may be added to the AR view.

[0051] The clustering system 124 can group data from two or more points of interest into a single AR element. Points of interest can be grouped based on an associated point of interest type or based on their location in the AR view. For example, if multiple AR elements associated with the same point of interest type are close to each other, the clustering system 124 can group them into a group and generate a grouped AR element with reference to the point of interest type that the AR elements have in common. The user can then select the grouped AR element, and in response, the AR view can be updated to display information associated with the points of interest included in the group.

[0052] In some examples, AR elements can be grouped based on their location in the AR view. For example, the augmented reality system 114 can determine the maximum density of AR elements for a particular AR view. The maximum density can describe the minimum space or range required between different AR elements. If two or more elements are closer than the distance allowed by the minimum range, the clustering system 124 can determine whether to combine the AR elements into a cluster. If so, a cluster can be generated and can be represented by the AR elements of the entire cluster. In some examples, the clustering system 124 may decide to hide one of the AR elements instead of combining them into a cluster. Whether to hide or cluster AR elements can be decided based on the relative relevance of each element, the type of element, and the user's preferences.

[0053] In some examples, in response to a filter element selected by a user, clustering system 124 may update the clusters into which AR elements are grouped. For example, elements associated with the selected filter may be removed from the cluster and displayed separately, while elements not associated with the selected filter may be hidden or faded. In some examples, a set of points of interest associated with the selected filter that are close to the location of computing device 100 (e.g., within a particular threshold) but not included in the current AR view may be grouped into a cluster, and an AR element representing the cluster may be displayed in a portion of the AR view associated with the direction in which the entity or point of interest is located. Thus, if several points of interest associated with the filter are located outside the camera to the right of the current view, AR elements associated with the several points of interest may be generated and displayed on the right portion of the current AR view.

[0054] The recognition system 126 may receive user input and, in response, recognize a particular object or building within the AR view. For example, a user may direct a camera on their device toward a particular object or location in the real world. The camera may capture images of the location and provide those images to the augmented reality system 114. The user may indicate through input to the computing device 100 that they wish to recognize an object, building, or location.

[0055] In response to the input, recognition system 126 may determine information about the selected object, building, or location based on location information associated with computing device 100 and geographic data repository 234. In some examples, the information may be displayed to the user and the user interface of the AR view. For example, the name of the location may be displayed in an AR element above the selected location.

[0056] Geographic data repository 234 may store various location or geographic data. For example, geographic data repository 234 may include map data. In some examples, map data may include information describing locations, points of interest, buildings, roads, parks, and other geographic features. Map data may include information associating an address with a particular geographic location, so that a user may enter an address and information about the address may be displayed (e.g., which buildings, if any, are at the location). Similarly, a user may enter the name of a location or point of interest, and the system may identify an address associated with the location.

[0057] In some examples, geographic data store 234 may include information associated with a location determination system, such as a global positioning system (GPS), so that the location of a particular computing device can be determined relative to map data. Geographic data store 234 may include data that allows augmented reality system 114 to determine one or more points of interest within a predetermined distance from computing device 100.

[0058] In some examples, geographic data repository 234 may also include geographic data generated specifically for display as part of an augmented reality display. For example, augmented reality system 114 may generate a plurality of annotations for display in an AR view. In some examples, annotations may be automatically generated for particular points of interest based on information stored in geographic data repository 234. In some examples, annotations may be used for an initial set of augmented reality elements displayed in an AR view before a particular filter element is selected.

[0059] In some examples, geographic data repository 234 may also include information that allows augmented reality system 114 to identify one or more objects, buildings, or locations based on their appearance in real-time video data presented by augmented reality system 114. For example, geographic data repository 234 may include reference images, data describing the appearance of points of interest, and user feedback data, enabling augmented reality system 240 to accurately determine which portions of the real-time video are associated with which geographic features, objects, buildings, locations, etc., in the real-time video.

[0060] Figure 2 An example client-server environment 200 is depicted according to an example embodiment of the present disclosure. The client-server environment 200 includes one or more user computing devices 210 and a server computing system 230. One or more communication networks 220 may interconnect these components. The one or more communication networks 220 may be any of a variety of network types, including a local area network (LAN), a wide area network (WAN), a wireless network, a wired network, the Internet, a personal area network (PAN), or a combination of these networks.

[0061] User computing device 210 may include, but is not limited to, smartphones, smart watches, fitness bands, navigation computing devices, laptop computers, and embedded computing devices (e.g., computing devices integrated into other objects such as clothing, vehicles, or other objects). In some examples, user computing device 210 may include one or more sensors designed to collect information associated with user computing device 210 with the user's permission. For example, user computing device 210 may include camera 214.

[0062] In some examples, the user computing device 210 can be connected to another computing device, such as a personal computer (PC), a laptop, a smart phone, a tablet computer, a mobile phone, a computing component of a vehicle, or any other computing device capable of communicating with the communication network 220. The user computing device 210 can include one or more applications, such as a search application, a communication application, a navigation application, a productivity application, a game application, a word processing application, an augmented reality application 212, or any other application. The application can include a web browser. The user computing device 210 can use a web browser (or other application) to send requests to and receive requests from the server computing system 230. The application can include an augmented reality application 212 that generates an augmented reality view to be displayed in an interface of the user computing device 210.

[0063] like Figure 2 As shown, the server computing system 230 may generally be based on a three-tier architecture consisting of a front-end layer, an application logic layer, and a data layer. As those skilled in the relevant computer and Internet related fields understand, Figure 2 Each component shown may represent an executable software instruction set and corresponding hardware (e.g., memory and processor) for executing these instructions. To avoid unnecessary detail, the following description has been omitted. Figure 2 Various components and engines that are not germane to conveying an understanding of the various examples are omitted. However, those skilled in the art will readily recognize that various additional components and engines may be used with the example Figure 2 The server computing system 230 shown is used in conjunction with the server computing system 230 to facilitate additional functionality not specifically described herein. Figure 2 The various components depicted in the FIG2000 may reside on a single server computer or may be distributed across several server computers in various arrangements. Figure 2 , but the various example embodiments are by no means limited to this architecture.

[0064] like Figure 2 As shown, the front end may consist of an interface system 222 that receives communications from one or more user computing devices 210 and transmits appropriate responses to the user computing devices 210. For example, the interface system 222 may receive requests in the form of hypertext transfer protocol (HTTP) requests or other web-based application programming interface (API) requests. The user computing devices 210 may execute conventional web browser applications or applications that have been developed for a specific platform to include any of a variety of computing devices and operating systems.

[0065] like Figure 2As shown, the data layer may include a geographic data repository 234. The geographic data repository 234 may store various location data. For example, the geographic data repository 234 may include map data. In some examples, the map data may include information describing locations, points of interest, buildings, roads, parks, and other geographic features. The map data may include information that associates an address with a particular geographic location, so that a user may enter an address and information about the address (e.g., which buildings, if any, are at the location). Similarly, a user may enter the name of a location or point of interest, and the system may identify an address associated with the location.

[0066] In some examples, geographic data repository 234 may include information associated with a location determination system, such as a global positioning system (GPS), so that the location of a particular computing device can be determined relative to map data. Geographic data repository 234 may include data that allows augmented reality system 240 to determine one or more points of interest within a predetermined distance from user computing device 210.

[0067] In some examples, geographic data repository 234 may also include information that allows augmented reality system 240 to identify one or more points of interest, objects, buildings, or locations based on their appearance in real-time video data presented by augmented reality system 240. For example, geographic data repository 234 may include reference images, data describing the appearance of points of interest, and user feedback data, enabling augmented reality system 240 to accurately determine in real-time video which portions of the real-time video are associated with which geographic features, objects, buildings, locations, etc.

[0068] In some examples, geographic data repository 234 may include location-specific information about a plurality of points of interest, locations, or entities. As described above, the location-specific information may include an address of a location, a geographic location of a location, a type of location, a type of entity, business hours of an entity, and the like. In some examples, geographic data repository 234 may also include image data associated with one or more geographic areas. Geographic data repository 234 may also include satellite image data associated with one or more geographic areas.

[0069] The application logic layer may include application data that may provide a wide range of other applications and services that allow users to access or receive geographic data for use in an AR system or for other purposes. The application logic layer may include an augmented reality system 240 and a search system 242 .

[0070] A user computing device 210 (e.g., a user computing device such as a smartphone) captures real-time video data of a portion of the world. The user computing device 210 may transmit information about the location of the user computing device 210 and the orientation of a camera 214 included in the user computing device 210 to the server computing system 230.

[0071] In response to receiving this information from user computing device 210, augmented reality system 240 may generate an augmented reality layer to display over the display of the real world. In some examples, the augmented reality layer may include one or more augmented reality elements associated with particular points of interest within the real world.

[0072] In some examples, the augmented reality system 240 can generate a list of augmented reality elements that should be displayed based on the received location of the user computing device 210 and the orientation of the associated camera 214. In some examples, the list is generated based on the importance or popularity of the point of interest. The augmented reality system 240 can provide one or more filter elements to be displayed in the AR view generated by the augmented reality application 212. Each filter element can be associated with a specific type of point of interest. For example, the filter element can be based on categories such as restaurants, museums, monuments, historical places, etc. When the user selects a filter element, the currently displayed set of augmented reality elements can be replaced with the set of augmented reality elements associated with the selected filter.

[0073] In some examples, server computing system 230 prepares a rendered AR layer for presentation at user computing device 210. In other examples, augmented reality system 240 provides data (e.g., geographic data, data associated with specific entities, and data describing which AR elements and filter elements to display) to user computing device 210, and augmented reality application 212 creates the AR layer.

[0074] In some examples, augmented reality system 240 can access real-time image data of the physical world and combine it with the augmented reality layer. In this way, user computing device 210 can display a real-time video of the real world that camera 214 of user computing device 210 is pointing at. The real-time video can be overlaid with additional interface elements provided by augmented reality system 240 based on information in geographic data repository 234. As user computing device 210 moves and the real-time image of the physical world changes, the augmented reality layer can also be updated to correctly overlay the data on top of the corresponding part of the real world in the AR view.

[0075] In some examples, the search system 242 can receive a search input or search query from the user computing device 210 and generate data in response to the search input or query. For example, the augmented reality system 240 can generate multiple filter elements that can be displayed in the AR view (or directly below it in the user interface). Each filter element can be associated with one or more point of interest types. If the user selects one of these filter elements, the selection can be interpreted as the user performing a search for the topic. The AR view can be updated to display the AR element associated with the selected filter element. In some examples, if the displayed AR element does not match the filter, the AR element can be removed and a new AR element can be displayed based on the filter.

[0076] In some examples, a user may enter a specific search query in the search bar. For example, the search query may be "restaurants near me". In response, the search system 242 may generate multiple query responses based on the information in the geographic data repository 234. The information may be transmitted to the user computing device 210 for display. In some examples, this may be displayed as a list of locations associated with the AR view but not containing the AR view in the user interface. In other examples, each search result may be displayed as one or more elements in the AR view. In some examples, the user may switch between a visual display of the AR element associated with the search result and a list display of the search results by swiping left or right on the interface associated with the AR view.

[0077] Figure 3 An example augmented reality user interface 300 is shown according to an example embodiment of the present disclosure. The example interface includes a real-time view of the real world in an AR view 302, a plurality of AR elements 304, and a list of one or more filter elements 306.

[0078] In this example, the real-time view (e.g., real-time video) in the AR view 302 includes a video of a street, including multiple buildings and other objects. Figure 2 When the user computing device 210 in the AR view moves and changes the perspective, the real-time picture in the AR view 302 can be updated.

[0079] In this example, the AR view may include multiple AR elements 304. In this example, the AR elements 304 include two elements (restaurant element and shopping element), a street name label, and an interface element that provides information about the location of the Empire State Building. These AR elements provide additional information to the user.

[0080] In this example, one or more filter elements 306 may include a restaurant filter, a shopping filter, and a coffee filter. The example augmented reality user interface 300 shows filter elements 306 (based on point of interest categories) that may be presented for selection (e.g., tapped) by the user. Filter elements (or chips) may be generated based on POIs near the user. The filters may be sorted based on various factors, including, but not limited to: time of day (e.g., in the morning, the user is more interested in "coffee" than "shopping", etc.), the number of nearby POIs in the category, the user's interests, etc.

[0081] Once the user selects a filter element from the one or more filter elements 306 , the augmented reality elements shown in the AR view may be updated to include only AR elements associated with points of interest associated with the filter button category.

[0082] Figure 4 An example augmented reality user interface 400 is shown according to an example embodiment of the present disclosure. The example interface includes a real-time picture of the real world in an AR view 402, multiple augmented reality interface elements (e.g., 404-1 and 404-2), and a search query field 408. In this example, one of the augmented reality elements is a clustered augmented reality interface element 406. The clustered augmented reality interface element 406 can be associated with multiple points of interest (in this case, 3 nearby points of interest of point of interest type "restaurant" are clustered together and represented by a single AR element).

[0083] In this example, the real-time footage (e.g., real-time video) included in the AR view 402 includes a video of a street, including a plurality of buildings and other objects. Figure 2 When the user computing device 210 in the AR view moves and changes the perspective, the real-time picture in the AR view 402 can be updated.

[0084] In this example, AR view 402 may include multiple AR interface elements 404-1 and 404-2. In this example, AR element 404 includes two elements related to a particular restaurant, both represented by restaurant icon elements (404-1 and 404-2), a name label for the street, and cluster augmented reality interface element 406. In some examples, cluster augmented reality interface element 406 has been generated based on the results of a clustering algorithm. Cluster augmented reality interface element 406 may represent three points of interest.

[0085] In some examples, a clustering algorithm can be implemented to avoid crowding of results in one portion of the AR view 402. Crowded AR view 402 can make it difficult for the user to read information and still see real-time video data. In some examples, points of interest can be clustered based on proximity within the display (e.g., screen space clustering). For example, an augmented reality system (e.g., Figure 1 The augmented reality system 114 in the AR view 402 may determine a minimum distance allowed between two AR elements. If two points of interest in the AR view 402 are close enough that their associated AR elements would exceed the minimum distance allowed, the augmented reality system (e.g., Figure 1 The augmented reality system 114 in the example can cluster them into one group and represent the entire group with a single AR element.

[0086] In some examples, an augmented reality system (e.g., Figure 1 The augmented reality system 114 in the AR view 402 may determine the minimum distance based on the size of the display (e.g., the number of pixels or the physical size of the display) and the size of the AR element. Thus, two different displays may result in different clustering of points of interest. Clustering decisions may also be made based on the size of the AR view 402 associated with the points of interest. For example, if four points of interest of a particular type are visible in the AR view 402, but one of the points of interest takes up 50% of the display and the other three points of interest only take up 10% of the display, the augmented reality system may cluster the three points of interest into one group, while retaining the larger point of interest as a single AR element. In some examples, the AR elements may be resized and their positions moved to maintain the minimum allowed distance without clustering the points of interest.

[0087] Additionally or alternatively, points of interest can be clustered based on their type. For example, a group of coffee shops can be combined into a group represented by a single augmented reality element. In some examples, points of interest can be clustered using both proximity clustering and semantic clustering. When clustering based on screen space, the system can group all places within a specific area of ​​the screen. For example, a screen space cluster can contain points of interest that are at different distances from the computing device but are all within the same line of sight of the user. When clustering based on type, the system can group the results using the geographic attributes of a place (e.g., street, block, proximity of the point of interest to a transportation station or popular landmark). For example, "5 restaurants near Coit Tower" or "3 clothing stores on Market Street" are examples of clusters defined based on proximity to well-known geographic locations.

[0088] In some examples, clustering methods (both screen-space clustering and semantic clustering) can be combined so that points of interest within the AR view can be grouped based on screen-space proximity, and points of interest not within the radius can be grouped using type-based clustering.

[0089] Figure 5 An example augmented reality user interface 500 is shown according to an example embodiment of the present disclosure. The example interface includes a real-time picture of the real world in an AR view 502. In this example, an augmented reality system (e.g., Figure 1 The augmented reality system 114 in FIG. 1 may enable a user to identify nearby places. To do this in an efficient and query-free manner, the user computing device (e.g., Figure 2 The user computing device 210 in the AR view may allow the user to select an object or location within the AR view. Figure 1 The augmented reality system 114 in the example may generate an identification of the selected object or location.

[0090] In some examples, an augmented reality system (e.g., Figure 1 The augmented reality system 114 in the embodiment of the present invention can generate an identification of a selected object or location based on location information associated with the user computing device (e.g., provided by the user computing device itself), information about the orientation of a camera capturing real-time video of the geographic location, a display at a user-selected point, and stored geographic data.

[0091] For example, augmented reality systems (e.g. Figure 1 The augmented reality system 114 in the embodiment of the present invention can be used on a user computing device (e.g., Figure 2 In response to the input, the augmented reality system (e.g., Figure 1 The augmented reality system 114 in FIG. 1 may determine the current location of a user computing device (eg, a smartphone). Figure 1 The augmented reality system 114 in the user computing device can determine the orientation of the camera that is capturing the world image based on sensors within the user computing device.

[0092] Augmented reality systems (e.g. Figure 1 The augmented reality system 114 in the device may use the determined device position and the camera orientation to determine the currently displayed geographic location. Figure 1 The augmented reality system 114 in the display can access geographic data related to the displayed geographic location from the geographic data repository 234.

[0093] Augmented reality systems (e.g. Figure 1 The augmented reality system 114 in FIG. 1 may identify the portion of the displayed image that the user has selected or indicated. The augmented reality system (e.g., Figure 1 The augmented reality system 114 in the embodiment of the present invention can analyze the image associated with the selected location and compare it with the geographic data retrieved from the geographic data repository 234. Based on the analysis of the selected image and the data from the geographic data repository, the augmented reality system (e.g., Figure 1 The augmented reality system 114 in the image processing unit can recognize a specific location or object that the user has selected.

[0094] Augmented reality systems (e.g. Figure 1 The augmented reality system 114 in the example may access information about the selected location or object. Figure 1 The augmented reality system 114 in the example may access the name of the location or object, the category or type associated with the location or object, information about available services and products, and the like. Figure 1 The augmented reality system 114 in the AR view may update the augmented reality view to include information about the identified object or location. The information may include AR elements in the AR view. In other examples, the information may include a portion of the display dedicated to providing additional information about the selected object or location.

[0095] In this example, the user has indicated a location on the display within augmented reality user interface 500. The indicated location can be marked with augmented reality interface element 506. The augmented reality system can analyze one or more features of the display video (e.g., one or more real-time images of a portion of the world) to identify a specific object, location, or point of interest associated with the selected portion of the AR view. In this example, the user has selected a clothing store facade. The augmented reality system (e.g., Figure 1 The augmented reality system 114 in the user computing device may determine a particular entity (e.g., a merchant) associated with the selected storefront based on one or more of the location of the user computing device and the orientation of the camera. Figure 1 The augmented reality system 114 in FIG. 5 may update the AR view to include an AR element 506 that provides additional information for the selected storefront (eg, the name of the associated merchant and a link to the additional information).

[0096] In some examples, additional information about the selected object, location, or point of interest can be displayed in the information display portion 504 of the user interface below the AR view. In this example, the information display portion 504 includes the name of the selected business, reviews or ratings data associated with the business, distance information, point of interest type information, and visit time information. In some examples, this information can be provided in another portion of the augmented reality user interface 500.

[0097] Figure 6 An example augmented reality user interface 600 is shown according to an example embodiment of the present disclosure. As described above, the augmented reality user interface 600 may include an AR view 602. The AR view 602 may include real-time images (e.g., video) of an area of ​​the world. These images may be captured by a camera associated with a user computing device that is presenting the AR view. These real-time images may be viewed by a user computing device (e.g., Figure 2 A user computing device 210 in a server computing system (e.g., Figure 2 The server computing system 230 in the embodiment of the present invention is provided to the user computing device (e.g., Figure 2 An augmented reality system is provided to a user computing device 210 in FIG.

[0098] In some examples, the additional information may be used to present one or more augmented reality elements located in the AR view 602 that are proximate to an image of a location, object, or point of interest associated with the one or more augmented reality elements. Figure 1 If the augmented reality system 114 in the AR view 602 determines that a specific merchant (e.g., a store) is located within the AR view 602, the augmented reality system can generate an AR element associated with the specific merchant. The generated AR element can be displayed above the image of the merchant or near the image of the merchant in the AR view 602.

[0099] In some examples, the AR view 602 may also include a list of filter elements. The filter elements may be used to indicate the types of augmented reality elements that the user wishes to see in the AR view 602. Each filter element may be associated with a particular type of point of interest. When the user selects a particular filter element, the AR view 602 may be updated to include augmented reality elements of the type associated with the selected filter. For example, the user may select a "coffee" filter. In response, the augmented reality system (e.g., Figure 1The augmented reality system 114 in the AR view 602 can update the AR view 602 to include augmented reality elements associated with "coffee", including but not limited to coffee shops, cafes, fast food restaurants that serve coffee, etc. In addition, AR elements associated with other themes or types of points of interest can be removed, faded, or hidden.

[0100] In some examples, once a user selects a particular filter element, the augmented reality system (e.g., Figure 1 In some examples, the user interface 600 may display interface elements that may be associated with the list of associated points of interest. For example, in addition to displaying one or more elements (e.g., 612, 614, and 616) associated with a particular point of interest in the list, the augmented reality user interface 600 may also include a “view list” button 620. The user may select the “view list” button 620. In response, the augmented reality user interface 600 may be updated to display information associated with the list of points of interest.

[0101] In this example, the selected filter element is associated with "shopping". The generated list of points of interest includes one or more stores. When the view list button 620 is selected, the list is shown by changing the augmented reality user interface 600, thereby displaying information associated with one or more stores. For example, the list can be displayed on top of the AR view 602 so that the list partially or completely blocks the AR view 602. In another example, the AR view 602 can be temporarily moved out of the interface, and the list of information can be temporarily moved into the interface. In some examples, the list view can be interactive, sorted by prominence, and allow the user to select a point of interest to directly view the location in the AR view 602. For example, if the user selects a particular point of interest from the list of points of interest, the AR view 602 can be updated to highlight or otherwise represent the selected point of interest in the AR view 602.

[0102] Figure 7 An example augmented reality user interface 700 is shown according to an example embodiment of the present disclosure. Figure 6 In the description of , the user can select a particular filter element (and thus a point of interest type). In response, the augmented reality system (e.g., Figure 1 The augmented reality system 114 in the example may generate a list of points of interest of that type within a threshold distance of the user computing device location. Figure 6 In the example embodiment, when the user selects the “View List” button, the list can be displayed to the user in different pages of the augmented reality user interface 700. Figure 7702. An alternative is shown in which the list and the AR view 702 are displayed simultaneously in the same interface. To this end, the list of points of interest can be displayed below the AR view 702 (or partially obstructing the bottom of the AR view 702) in a card carousel 720 (e.g., a horizontally scrollable list).

[0103] This layout of the augmented reality user interface 700 may have the following advantages: it allows the user to fully interact with the augmented reality user interface 700 with only one hand. For example, if the user holds a smartphone in one hand and a bag in the other hand or has other inconveniences, the user can still scroll the card carousel 720 with the thumb.

[0104] In some examples, the user can slide the card carousel 720 to receive additional information about one or more points of interest. When the user slides the card carousel 720, the currently displayed card can be highlighted or highlighted. Each card in the card carousel 720 can contain additional supplemental data about the associated point of interest. The supplemental data may include the name of the point of interest, review information, information about goods and services available at the point of interest (if any), cost indicators associated with the point of interest (e.g., grouped into specific cost groups), the current distance to the point of interest, business hours, and the like.

[0105] In some examples, the AR view 702 can be controlled so that an augmented reality element associated with a card currently displayed in the card carousel 720 can be highlighted, allowing the user to identify a particular point of interest within the AR view 702. In the depicted example, the card carousel is currently displaying a card associated with a store named "Brook's Brot hers." The AR element associated with the Brook's Brot hers store (AR element 712) is also visually distinguished.

[0106] Alternatively, the user may select a point of interest by directly tapping an icon in the camera view, in which case the card carousel 720 may scroll to display cards associated with the currently selected point of interest. For example, the user may select a point of interest by selecting a displayed AR element (e.g., one of 712, 714, or 716), and the card carousel 720 may be updated to reflect the selected point of interest.

[0107] Figure 8 An example augmented reality user interface 800 according to an example embodiment of the present disclosure is shown. To avoid user fatigue due to interaction, an augmented reality system (e.g., Figure 1 The augmented reality system 114 in the embodiment of the present invention may include an alternative user interface for displaying information about the user's surroundings. For example, if the user does not want to hold their user computing device vertically upward (e.g., Figure 212) to capture video data using a camera integrated into the user computing device, the user can then tilt the device down to a more horizontal position. Once the augmented reality system (e.g., augmented reality system 114 in FIG. 12) is based on the user computing device (e.g., Figure 2 The sensor included in the user computing device 210 in the embodiment determines the user computing device (e.g., Figure 2 If the inclination of the user computing device 210 in FIG. 12 exceeds the threshold angle, the augmented reality system (e.g., the augmented reality system 114 in FIG. 12 ) may use the user computing device (e.g., Figure 2 The AR view is replaced by a map view 804 of the surrounding environment of the user computing device 210).

[0108] In the map view 804, one or more points of interest (e.g., locations, buildings, objects, etc.) may be indicated by a marking element on the map view 804. In some examples, the particular point of interest to be marked in the map view is determined based on an AR element displayed in the AR view before the map view 804 is displayed. For example, if the particular point of interest has an AR element associated with it in the AR view, the augmented reality system (e.g., Figure 1 The augmented reality system 114 in the figure can generate corresponding markers in the map view.

[0109] In this example, a series of markers (806-1 through 806-6) are displayed in the map view 804 at locations associated with corresponding points of interest. In some examples, the displayed markers represent a list of all points of interest that have a type that matches the type of the selected filter element within a predetermined distance from the location of the user computing device. In this example, the user computing device (e.g., Figure 2 808 and a predetermined distance from the location. The predetermined distance may be determined based on a distance that can be traveled in a five-minute walk. Other distances or distance metrics may be used. For example, the predetermined range may be 400 meters. Note that the markers may include points of interest that are not displayed in the AR view when the phone is tilted (because they are beyond the camera's field of view), but are included in the map view 804 because they match the selected point of interest type (e.g., shopping) and are within the predetermined range (represented by line 810).

[0110] In some examples, a limited list of matching points of interest within a predetermined distance may be displayed. The limited list may be determined by ranking a plurality of matching points of interest based on one or more factors (e.g., user preferences, ratings, cost, operation time, degree of match with a selected filter element, etc.) and selecting a particular number of points of interest based on the ranking. The specific number of points selected may be determined at least in part based on the total number of candidate points of interest and the zoom level at which the map view 804 is currently being viewed. Thus, the user may zoom in to display more AR elements associated with the points of interest, or zoom out to display fewer AR elements.

[0111] In some examples, the user can select one of the markers 806-2 on the map. If the user selects a particular marker, the user interface 800 of the map view can be updated. For example, instead of displaying one or more filter tools 812 below the map view 804, the user interface 800 can be updated to display information 820 about the point of interest associated with the selected marker 806-2. The information can include the name, ranking, distance, etc. of the point of interest. Displaying the map view when the phone is tilted to a horizontal position can enable comfortable browsing with the AR view. Figure 1 Once the user has finished browsing the results in a comfortable manner, they can then connect the user's computing device (e.g., Figure 2 The user computing device 210 in the embodiment of FIG. 1 is tilted back beyond a threshold angle, causing the augmented reality system (e.g., Figure 1 The augmented reality system 114 in the display returns the interface to the AR view. If the user selects a specific point of interest, the augmented reality element can be highlighted in the AR view when returning to the display.

[0112] Fig. 9 An example flow chart of a method for performing filtering and searching for elements within an augmented reality view according to an example embodiment of the present disclosure is depicted. One or more portions of the method may be implemented by one or more computing devices such as, for example, the computing devices described herein. Furthermore, one or more portions of the method may be implemented as an algorithm on hardware components of the devices described herein. Fig. 9 Elements are depicted as being performed in a particular order for purposes of illustration and discussion. One of ordinary skill in the art, using the disclosure provided herein, will appreciate that elements of any method discussed herein may be adjusted, rearranged, expanded, omitted, combined, and / or modified in various ways without departing from the scope of the present disclosure. The method may be performed by one or more computing devices (such as Figure 1 to Figure 2 The invention may be implemented by one or more of the computing devices depicted in the specification.

[0113] A user computing device (e.g., Figure 2The user computing device 210 in the example may include one or more processors, memory, and other components, which together generate an interface depicting an augmented reality (AR) view of at least a portion of a physical real-world environment at 902 for the user computing device ( Figure 2 Thus, the AR view may include a user interface that displays a real-time video image. The AR view may be integrated into a user computing device (e.g., Figure 2 The camera in the user computing device 210 in the AR view captures real-time video. The camera can be a camera integrated into a smartphone. In this way, the video displayed in the AR view represents the user's current surroundings. The AR view can then enhance the real-time video to show the user additional information or present entertainment elements that enhance the user's experience of the current environment.

[0114] In some examples, a user computing device (e.g., Figure 2 The user computing device 210 in the AR view may display one or more filter elements at 904, each filter element being associated with a point of interest type. For example, a filter list may be displayed at the bottom of the interface. The filters in this list may be displayed below the live video portion of the AR view. In other examples, the filter list may be displayed as a transparent button or element on a portion of the live video without being noticeable.

[0115] A user computing device (e.g., Figure 2 The user computing device 210 in the example may access data describing a plurality of points of interest within a portion of a physical real-world environment from a geolocation database at 906. In some examples, the user computing device (e.g., Figure 2 A user computing device 210 in the example may determine its own location. For example, a computing device (e.g., Figure 2 The user computing device 210 in FIG. 1 may include a GPS that enables the computing device to determine the current location of the computing device within the world with a certain degree of accuracy. The location may be provided to the augmented reality system and used to determine which points of interest to visit.

[0116] In some examples, multiple points of interest within a portion of the physical real world environment may be within a predetermined distance of the location of the computing device. In some examples, the predetermined distance is based on a location from a user computing device (e.g., Figure 2 The predetermined distance may be based on an estimated walking time from the location of the user computing device 210 in FIG. 1 . For example, the predetermined distance may be based on an estimated walking time of five minutes from the location of the computing device. Other estimated walking times may also be used.

[0117] In some examples, each point of interest has an associated type (e.g., point of interest type), and the filter element is associated with a specific point of interest type. One or more of the multiple points of interest are associated with a merchant. A point of interest can have multiple types or subtypes associated with it. For example, a specific restaurant can have a broad type associated with it, such as "restaurant", and one or more subtypes, such as "breakfast food" and "American food". In addition, related types such as "coffee" can also be associated with the restaurant. The types can be associated with existing classification systems (e.g., North American Industry Classification System (NAI CS) codes, Geographic Concept Identifier (GCID) codes, etc.). In some examples, one type can be determined as a primary type, while other types can be determined as secondary types.

[0118] A user computing device (e.g., Figure 2 The user computing device 210 in the AR view may display an initial set of augmented reality elements associated with the initial set of points of interest. The initial set of points of interest may be selected based on overall popularity, past user interaction information, preferences of a particular user, time of day, etc. The initial set of points of interest displayed for a particular location may be determined so that the most relevant points of interest are automatically highlighted without a specific user request. The initial set of augmented reality elements may be associated with one or more entities currently displayed in the AR view.

[0119] A user computing device (e.g., Figure 2 The user computing device 210 in the example embodiment may receive a selection of one of the displayed filter elements at 908. For example, if the computing device includes a touch screen, the user may select a particular filter element by touching the screen at a location associated with the desired filter element. The user may select the "coffee" filter element by interacting with an area of ​​the touch screen that includes the "coffee" filter element.

[0120] In response to receiving a selection of one of the displayed filter elements, the user computing device (e.g., Figure 2 The user computing device 210 in the example above may update the AR view to remove at least one augmented reality element from the initial set of augmented reality elements. Continuing with the above example, if the user selects the “coffee” filter, the user computing device (e.g., Figure 2 The user computing device 210 in FIG. 1 may update the AR by removing one or more augmented reality elements having a point of interest type other than “coffee”.

[0121] A user computing device (e.g., Figure 2The user computing device 210 in the AR view may provide a filter-based set of augmented reality elements associated with the set of points of interest for display in the AR view at 910, wherein the filter-based set of augmented reality elements represents the filter-based set of points of interest associated with the selected filter element. For example, if the user selects a "coffee" filter, the AR view may be updated to include augmented reality elements (e.g., visual interface elements, such as labels, markers, or buttons) associated with points of interest having a type of "coffee" or being associated with coffee.

[0122] As described above, one or more initially displayed augmented reality elements may be removed to add an augmented reality element associated with the “coffee” point of interest. In some examples, one or more points of interest in the initial set of points of interest may be included in the filter-based set of points of interest. For example, the initial set of augmented reality elements may include one or more AR elements associated with coffee shops. Then, if the user selects the “coffee” filter element, those AR elements associated with coffee shops will remain, while AR elements associated with other types may be removed and replaced with additional AR elements associated with the selected “coffee” point of interest type.

[0123] The corresponding AR element in the set of filter-based augmented reality elements can be displayed near the entity associated with it in the AR view. For example, if the AR element provides additional information about a store, the AR element can be positioned to overlay a portion of the image of the building where the store is located. Thus, the AR view can provide the user with information about the location of a specific point of interest even if the available signage does not clearly indicate the location.

[0124] A user computing device (e.g., Figure 2 The user computing device 210 in the example may be accessed by accessing a message from a user computing device (e.g., Figure 2 The user computing device (e.g., Figure 2 The user computing device 210 in FIG. 210 may analyze the image data to identify one or more entities within the image data based on storefronts associated with the entities. For example, the user computing device (e.g., Figure 2 The user computing device 210 in FIG. 1 may use location information from the user computing device (e.g., a smartphone) to determine the approximate location of the user. Figure 2 The computing device may use information stored in a geographic data repository to determine the specific location represented in the image data. Each geographic feature may be associated with a geographic data repository (e.g., Figure 1matches one or more entries in the geographic data repository 234 in .

[0125] Once a user computing device (e.g. Figure 2 The user computing device 210 in the image data (e.g., real-time video) matches features of the image data (e.g., real-time video) to a specific geographic entity, and the user computing device (e.g., Figure 2 A user computing device 210 in the AR view can display AR elements near features associated with the corresponding entity (e.g., a storefront) in the AR view.

[0126] In some examples, at least one point of interest in the set of points of interest based on the filter may not be displayed within the AR view. For example, once the user selects the “coffee” filter element, the user computing device (e.g., Figure 2 A user computing device 210 in the user's computer may identify multiple points of interest near the user (e.g., within a threshold distance of the user). Some of these points of interest may be within the threshold distance but are not currently being viewed by the user computing device (e.g., Figure 2 The AR view may be displayed in image data captured by a camera of a user computing device 210 in the AR view. For example, the point of interest may be to the right of the real-time video image data. The AR view may be updated with AR elements indicating the direction and distance associated with the point of interest. In this way, the user may be alerted to nearby points of interest that are not currently within the AR view.

[0127] In some examples, a particular AR element can be associated with multiple points of interest. In some examples, each point of interest has a point of interest type. The user computing device (e.g., Figure 2 The user computing device 210 in the AR element may group multiple points of interest of the same point of interest type into a group represented by a single AR element. This grouping occurs when the number of points of interest to be displayed is greater than the number that can fit in the AR view. In some examples, all points of interest of a particular type that are currently off-screen may be represented by a single AR element.

[0128] In other examples, a user computing device (e.g., Figure 2 The user computing device 210 in the AR view may group the points of interest into a group based on their corresponding positions in the AR view. Figure 2 The user computing device 210 in the AR view may determine that two or more augmented reality elements are displayed within a predetermined distance in the AR view. In response to determining that two or more augmented reality elements are displayed, the user computing device (e.g., Figure 2 The user computing device 210 in the AR view can combine two or more augmented reality elements into a signal group augmented reality element in the AR view. In this way, points of interest can be grouped to ensure that the AR view is not crowded with too many AR elements.

[0129] In some examples, an augmented reality system (e.g., Figure 1 The user interface of the augmented reality system 114 in the embodiment of the present invention may enable the user to select features of the image data to be recognized. For example, the user computing device (e.g., Figure 2 The user computing device 210 in the AR view may receive user input indicating a first entity in the AR view. The user computing device (e.g., Figure 2 The user computing device 210 in the AR view may analyze the image data displayed in the AR view to extract one or more characteristics of the first entity. The user computing device (e.g., Figure 2 The user computing device 210 in the example may access data from the database of geographic locations that identifies a specific location in the database of geographic locations based on one or more characteristics of the first entity.

[0130] The above Fig. 9 The description describes the method by which a user computing device (e.g., Figure 2 However, these method steps may also be performed in whole or in part by a remote user computing system (e.g., Figure 2 A server computing system (e.g., a user computing device 210 in Figure 2 In some examples, the steps may be performed by more than one device.

[0131] The technology discussed herein relates to sensors, servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0132] Although the present subject matter has been described in detail with respect to various specific example embodiments of the present subject matter, each example is provided by way of explanation rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, modifications, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, variations, and / or additions to the present subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of an embodiment may be used together with another embodiment to produce yet further embodiments. Therefore, the present disclosure is intended to encompass such changes, variations, and equivalents.

Claims

1. A computer-implemented method for performing search in an augmented reality interface, the method comprising: generating, by a computing system having one or more processors, an interface for display by the computing system, the interface depicting an augmented reality (AR) view including image data of at least a portion of a physical real world environment; displaying, by the computing system, within the interface, one or more filter elements, a respective filter element of the one or more filter elements being associated with a point of interest type; accessing, by the computing system, from a database of geographic locations, data describing a plurality of points of interest within the portion of the physical real-world environment; receiving, by the computing system, a selection of one of the displayed filter elements, and A filter-based set of augmented reality elements associated with the set of points of interest is provided by the computing system for display in the AR view, wherein the filter-based set of augmented reality elements represents the filter-based set of points of interest associated with the selected filter element. 2 . The computer-implemented method of claim 1 , wherein a plurality of points of interest in the plurality of points of interest are associated with businesses.

3. The computer-implemented method of any preceding claim, further comprising, prior to receiving a selection of one of the displayed filter elements: displaying, by the computing system, an initial set of augmented reality elements associated with the initial set of points of interest; and In response to receiving the selection of one of the displayed filter elements, updating, by the computing system, the AR view to remove at least one augmented reality element from the initial set of augmented reality elements. 4 . The computer-implemented method of claim 3 , wherein one or more points of interest in the initial set of points of interest are included in the filter-based set of points of interest.

5. The computer-implemented method of any one of claims 3 to 4, wherein the initial set of augmented reality elements is associated with one or more entities currently displayed in the AR view.

6. A computer-implemented method as described in any of claims 3 to 5, wherein the initial set of augmented reality elements is selected for display based on one or more storefronts detected in image data used to depict at least a portion of the physical real-world environment in the AR view.

7. The computer-implemented method of any one of claims 3 to 6, wherein the initial set of augmented reality elements is selected for display based on a visibility determination of one or more points of interest at a predetermined distance from the computing device.

8. The computer-implemented method of any preceding claim, wherein a corresponding AR element in the filter-based set of augmented reality elements is displayed in the AR view proximate to a point of interest associated therewith.

9. The computer-implemented method of any preceding claim, wherein providing, by the computing system, a filter-based set of augmented reality elements associated with a set of points of interest for display in the AR view further comprises: accessing, by the computing system, image data from a camera associated with the computing system; analyzing, by the computing system, the image data to identify one or more entities within the image data based on storefronts associated with the entities; as well as An AR element is displayed, by the computing system, in the AR view proximate a feature in the image data associated with a corresponding entity.

10. The computer-implemented method of any preceding claim, further comprising: A location of the computing system is determined by the computing system; and wherein the plurality of points of interest within the portion of the physical real world environment are within a predetermined distance of the location of the computing system.

11. The computer-implemented method of claim 10, wherein the predetermined distance is based on a walking time from the location of the computing system.

12. The computer-implemented method of claim 11, wherein the predetermined distance is based on an estimated walking time of 5 minutes starting from the location of the computing system.

13. The computer-implemented method of any preceding claim, wherein at least one point of interest in the filter-based set of points of interest is not displayed within the AR view.

14. The computer-implemented method of any preceding claim, wherein the displayed AR element is associated with a group comprising a plurality of points of interest.

15. The computer-implemented method of claim 14, wherein respective points of interest have associated point-of-interest types, and the plurality of points of interest included in the group have the same point-of-interest type.

16. The computer-implemented method of claim 15, further comprising: Determining, by the computing system, to display two or more augmented reality elements within a predetermined distance in the AR view; as well as In response to determining to display two or more augmented reality elements, the two or more augmented reality elements are combined into a signal group augmented reality element in the AR view.

17. The computer-implemented method of any preceding claim, further comprising: receiving, by the computing system, a user input indicating a first entity in the AR view; analyzing, by the computing system, the image data displayed in the AR view to extract one or more characteristics of the first entity; as well as Data identifying a specific location in the database of geographic locations is accessed from the database of geographic locations by the computing system based on the one or more characteristics of the first entity.

18. A computing device, the computing device comprising: one or more processors; as well as A computer readable memory, wherein the computer readable memory stores instructions which, when executed by the one or more processors, cause the computing device to perform the method of any one of claims 1 to 17.

19. A computer readable medium storing instructions which, when executed by one or more computing devices, cause the one or more computing devices to perform the method of any one of claims 1 to 17.

Images