Augmented reality guidance interface

Abstract

Example embodiments described herein relate to an AR guidance system that performs operations, where the operations include detecting a client device at a location within a geo-fenced area, where the geo-fenced area may include a destination of interest therein; determining, within the geo-fenced area, a route from the location of the client device to the destination of interest; causing a presentation of the environment to be displayed within the AR interface at the client device; detecting a display of a real-world signage within the presentation of the environment; in response to detecting the display of the signage within the presentation of the environment, generating a media item, where the media item is based on a route to the destination of interest; and displaying the media item within the AR interface based on the location of the signage within the presentation of the environment.

Description

[0001] This application is a divisional application of the Chinese patent application “Augmented Reality Guidance Interface” with application number 202080023979.8 (filing date March 25, 2020).

[0002] Priority claim

[0003] This application claims the benefit of priority to U.S. patent application serial number 16 / 365,256, filed on March 26, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0004] Embodiments of the present disclosure relate generally to graphical user interfaces (GUIs), and more particularly, to systems for generating and causing display of augmented reality (AR) content within a GUI. Background Art

[0005] Augmented reality (AR) is a vivid direct or indirect view of a physical, real-world environment, elements of which are enhanced by computer-generated sensory input.

[0006] In an unrelated field, turn-by-turn navigation is a feature of some navigation devices in which directions for a selected route are continuously presented to the user either verbally or, in some cases, as visual instructions. Summary of the invention

[0007] According to one aspect of the present disclosure, a method is provided, comprising: determining a destination of interest based on user attributes associated with a user of a client device; determining a route from a location of the client device to the destination of interest; generating a media item based on the route to the destination of interest; and causing a representation of the media item to be displayed at the client device.

[0008] According to another aspect of the present disclosure, a system is provided, comprising: a memory; and at least one hardware processor coupled to the memory and comprising instructions for causing the system to perform operations, the operations comprising: determining a destination of interest based on user attributes associated with a user of a client device; determining a route from a location of the client device to the destination of interest; generating a media item based on the route to the destination of interest; and causing a representation of the media item to be displayed at the client device.

[0009] According to another aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, comprising instructions, which, when executed by one or more processors of a machine, cause the machine to perform operations, the operations comprising: determining a destination of interest based on user attributes associated with a user of a client device; determining a route from a location of the client device to the destination of interest; generating a media item based on the route to the destination of interest; and causing a representation of the media item to be displayed at the client device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0011] Figure 1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network, wherein the messaging system includes an augmented reality (AR) system, in accordance with some embodiments.

[0012] Figure 2 is a block diagram showing further details regarding a messaging system according to an example embodiment.

[0013] Figure 3 is a block diagram illustrating various modules of an AR system according to certain example embodiments.

[0014] Figure 4 is a flow chart illustrating a method for presenting AR content at a client device, according to certain example embodiments.

[0015] Figure 5 is a flow chart illustrating a method for presenting AR content at a client device, according to certain example embodiments.

[0016] Figure 6 is a flow chart illustrating a method for presenting AR content at a client device, according to certain example embodiments.

[0017] Figure 7 is an interface diagram depicting AR content according to some example embodiments.

[0018] Figure 8 is an interface diagram depicting AR content according to some example embodiments.

[0019] Fig. 9 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein and used to implement the various embodiments.

[0020] Fig.10is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0021] As described above, an AR system provides a graphical user interface (GUI) to a user to display a real-time direct or indirect view of a physical, real-world environment, wherein elements of the view are augmented by computer-generated sensor inputs. For example, an AR interface may present media content at locations within the display of the view of the real-world environment such that the media content appears to interact with elements in the real-world environment.

[0022] By using an AR system and implementing it in the context of a turn-by-turn navigation system, an interactive AR guidance system can be configured to use real-world signage to guide a user to a specific location. For example, a user can be navigated through a city by first determining the user's destination and then calculating a route to the destination. The AR guidance system can then identify real-world signage (e.g., sign poles, hanging signs, etc.) and "enhance" the presentation of the signage by adding (i.e., superimposing) additional signage close to the real-world signage within the AR display or modifying existing real-world signage in the AR display. The modified or added signage can be oriented to point the user in the direction the user should go in order to reach their final destination, and in some embodiments can include animation or other media content.

[0023] Thus, example embodiments described herein relate to an AR guidance system that performs operations including: detecting a client device at a location within a geo-fenced area, wherein the geo-fenced area may include a destination of interest therein; determining a route from the location of the client device to the destination of interest within the geo-fenced area; causing a presentation of an environment to be displayed within an AR interface at the client device; detecting a display of real-world signage within the presentation of the environment; generating a media item in response to detecting the display of the signage within the presentation of the environment, wherein the media item is based on the route to the destination of interest; and causing the media item to be displayed within the AR interface based on the location of the signage within the presentation of the environment.

[0024] According to some example embodiments, generating media items based on a route to a destination of interest within a geo-fenced area may include determining a direction of travel based on a location of a client device relative to the destination of interest and the route to the destination of interest, and selecting a direction indicator including a graphical symbol (such as an arrow) to indicate the direction of travel to a user.

[0025] In some embodiments, the AR guidance system can identify attributes or features of a sign detected within the rendering of the environment in order to generate a style template to configure the AR media item. The attributes or features can include the shape of the sign, the color of the sign, the graphics displayed on the sign, physical characteristics (e.g., damage, flashing lights), the font of the text displayed on the sign, the character weight of the text, and the size of the sign. Doing so enables the AR guidance system to generate media items that can be presented within the AR interface in a way that looks like the media items match existing real-world signage.

[0026] In some embodiments, in response to detecting a client device within a threshold distance of a destination of interest, the AR guidance system may retrieve or otherwise access media content associated with the threshold distance (or the threshold distance and the destination of interest). The retrieved media content may then be presented on top of signage detected within the environment within the AR interface. For example, the media content may be presented as an incentive (e.g., “You’re getting close!”) to encourage the user to continue following a path presented by a direction indicator. In some embodiments, the media content may be customized or modified based on one or more user attributes (such as a name or username) associated with a user of the client device and specific demographic information.

[0027] In some example embodiments, in response to detecting a client device within a geo-fenced area, the AR guidance system may access a user profile associated with the client device to retrieve user profile information. The AR guidance system may then use the user profile information to select a destination of interest to the user.

[0028] Consider an illustrative example from the user's perspective. The user may enter a geo-fenced area, such as an amusement park or an urban area. In response to detecting the user within the geo-fenced area, the AR guidance system retrieves user profile information associated with the user and identifies a destination of interest for the user within the geo-fenced area. The AR guidance system then calculates a route to the destination of interest and begins identifying signage presented within the AR interface of the user's client device. The AR guidance system may then enhance the presentation of the user's environment by modifying existing signage or adding signage to existing signage to provide directional indicators that guide the user to the destination of interest. As the user approaches the destination of interest, the AR guidance system may retrieve media content associated with the destination of interest and further use the media content to enhance the signage.

[0029] Figure 11 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes a plurality of client devices 102, each client device hosting a plurality of applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0030] Thus, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and with a messaging server system 108 via the network 106. The data exchanged between messaging client applications 104 and between messaging client applications 104 and messaging server system 108 includes functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0031] The messaging server system 108 provides server-side functionality to certain messaging client applications 104 via the network 106. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client application 104 or by the messaging server system 108, it is understood that the location of certain functions within the messaging client application 104 or within the messaging server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the messaging server system 108, but later migrate the technologies and functions to the messaging client application 104 if the client device 102 has sufficient processing power.

[0032] The messaging server system 108 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to the messaging client applications 104, receiving data from the messaging client applications 104, and processing data generated by the messaging client applications 104. In some embodiments, the data includes, by way of example, message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. In other embodiments, other data is used. The data exchange within the messaging system 100 is invoked and controlled by functions available via the GUI of the messaging client applications 104.

[0033] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0034] Specifically, the API server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the API 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client application 104 in order to call the functions of the application server 112. The API server 110 exposes various functions supported by the application server 112, including account registration, login functions, message sending from a particular messaging client application 104 to another messaging client application 104 via the application server 112, sending of media files (e.g., images or videos) from the messaging client application 104 to the messaging server application 114, and the setting of a collection of media data (e.g., stories) that may be accessed by another messaging client application 104, retrieving a friend list of the user of the client device 102, retrieving such a collection, retrieving messages and content, adding and removing friends from the social graph, the position of friends within the social graph, and open and application events (e.g., related to the messaging client application 104).

[0035] The application server 112 hosts multiple applications and subsystems, including a messaging server application 114, an image processing system 116, a social networking system 122, and an augmented reality (AR) system 124. The messaging server application 114 implements a variety of message processing techniques and functions, particularly those related to aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., referred to as stories, galleries, or collections). These collections are then made available to the messaging client application 104 by the messaging server application 114. In view of the hardware requirements for other processor- and memory-intensive data processing, such processing can also be performed on the server side by the messaging server application 114.

[0036] The application server 112 also includes an image processing system 116 that is dedicated to performing various image processing operations, typically on images or videos received within the payload of a message at the messaging server application 114 .

[0037] The social networking system 122 supports various social networking functional services and makes these functionalities and services available to the messaging server application 114. To do so, the social networking system 122 maintains and accesses an entity graph 304 within the database 120. Examples of functionalities and services supported by the social networking system 122 include identification of other users of the messaging system 100 who have a relationship with a particular user or who the particular user is "following," as well as identification of other entities and interests of the particular user.

[0038] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with messages processed by the messaging server application 114 is stored.

[0039] Figure 2 is a block diagram showing further details about the messaging system 100 according to an example embodiment. Specifically, the messaging system 100 is shown to include a messaging client application 104 and an application server 112, which in turn contain several subsystems, namely a short timer system 202, a collection management system 204, and an annotation system 206.

[0040] The short-term timer system 202 is responsible for enforcing short-term access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the short-term timer system 202 incorporates a plurality of timers that selectively display messages and associated content and enable access to messages and associated content via the messaging client application 104 based on durations and display parameters associated with the messages, message collections, or graphical elements. Further details regarding the operation of the short-term timer system 202 are provided below.

[0041] The collection management system 204 is responsible for managing media collections (e.g., media collections including text collections, image video collections, and audio data collections). In some examples, content collections (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be available for a specified time period (such as the duration of the event to which the content is related). For example, content related to a concert can be available as a "story" during the concert. The collection management system 204 can also be responsible for publishing icons that provide notification to the user interface of the messaging client application 104 that a particular collection exists.

[0042] In addition, the collection management system 204 includes a curation interface 208 that allows collection managers to manage and curate specific content collections. For example, the curation interface 208 enables event organizers to curate content collections related to specific events (e.g., remove inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, compensation can be paid to users to include user-generated content in a collection. In this case, the curation interface 208 operates to automatically pay such users for the use of their content.

[0043] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content (such as user-supported content received by a user to be forwarded or redistributed to one or more recipients). For example, the annotation system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The annotation system 206 provides media overlays operably to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 provides media overlays operably to the messaging client application 104 based on other information such as social network information of the user of the client device 102. Media overlays may include audio-visual content and visual effects and augmented reality overlays. Examples of audio-visual content include pictures, text, logos, animations, and sound effects, as well as animated facial models, image filters, and augmented reality media content. Examples of visual effects include color overlays. Audio-visual content or visual effects can be applied to media content items (e.g., photos or videos or live streams) at the client device 102. For example, media overlays include text that can be superimposed on photos taken by the client device 102. In another example, the media overlay includes an identification of a location overlay (e.g., Venice Beach), the name of a live event, or the name of a business overlay (e.g., Beach Cafe). In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of a business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 120 and accessed through the database server 118.

[0044] In one example embodiment, the annotation system 206 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify situations in which specific media overlays should be provided to other users. The annotation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0045] In another example embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media overlays associated with geographic locations. For example, the annotation system 206 associates the media overlays of the highest-paying merchant with the corresponding geographic location within a predefined amount of time.

[0046] Figure 3 1 is a block diagram illustrating components of an AR system 124 that configure the AR system 124 to detect a client device 102 within a geofenced area, identify a destination of interest based on one or more attributes associated with the client device 102, generate a route from the location of the client device 102 to the destination of interest within the geofenced area, detect signage within a presentation of an environment at the client device 102, select a direction indicator based on the route to the destination of interest, and cause a direction indicator to be displayed at the location based on the signage within the AR interface at the client device 102. The AR system 124 is shown to include a media module 302, a user profile module 304, a global positioning system (GPS) module 306, and a presentation module 308, all of which are configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of these modules may be implemented using one or more processors 310 (e.g., by configuring such one or more processors to perform the functions described for that module), and thus may include one or more processors 310.

[0047] Any one or more of the described modules may be implemented using hardware alone (e.g., one or more of the processors 310 of the machine) or a combination of hardware and software. For example, any module described in the AR system 124 may physically include an arrangement of one or more of the processors 310 (e.g., one or more of the processors 310 of the machine or a subset thereof), which is configured to perform the operations described herein for the module. As another example, any module of the AR system 124 may include software, hardware, or both, which configures an arrangement of one or more processors 310 (e.g., one or more of the processors 310 of the machine) to perform the operations described herein for the module. Therefore, different modules of the AR system 124 may include and configure different arrangements of such processors 310 or a single arrangement of such processors 310 at different points in time. In addition, any two or more modules of the AR system 124 may be combined into a single module, and the functions described herein for a single module may be subdivided among multiple modules. In addition, according to various example embodiments, the modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0048] Figure 44 is a flowchart illustrating a method 400 for displaying an interactive information interface according to some example embodiments. The operations of method 400 may be as described above with respect to Figure 3 The modules described are executed. Figure 4 As shown, method 400 includes one or more operations 402 , 404 , 406 , 408 , 410 , and 412 .

[0049] At operation 402 , the GPS module 306 detects the client device 102 at a location within a geo-fenced area, wherein the geo-fenced area includes one or more destinations of interest.

[0050] In some embodiments, each of the one or more destinations of interest may be associated with a predefined set of user or device attributes, or user context (such as time of day or relative location). In response to detecting the client device 102 within the geo-fenced area, the media module 302 identifies the destination of interest based on the user or device attributes or the user context. The user or device attributes may include user profile information, such as demographic information (e.g., age / age range, location, gender identity) and user behavior (e.g., preferences, interests, etc.). The user context may include the time of day, season, the location of the user within the geo-fenced area (e.g., the location along the border of the geo-fenced area where the user enters the geo-fenced area).

[0051] At operation 404, the GPS module 306 determines a route from the location of the client device 102 to the destination of interest within the geo-fenced area. The route may include a "quickest route" as well as predefined routes based on the destination of interest, user attributes, and user context.

[0052] For example, based on the location of the device and specific user profile information, a destination of interest may be associated with one or more routes so that a user with a particular set of user profile information may be directed to the destination of interest along the quickest route, while a user with another set of user profile information may be directed along a different route.

[0053] At operation 406, the rendering module 308 causes a rendering of the environment to be displayed at the client device 102. The rendering of the environment may be generated by a camera associated with the client device 102 and displayed at the client device 102 within an AR interface.

[0054] At operation 408, the media module 302 detects the display of the signage at the location within the rendering of the environment. According to certain example embodiments, the media module 302 may employ one or more image or pattern recognition techniques.

[0055] In some embodiments, media module 302 can identify signage based on the presence of one or more colors of interest (e.g., shades of red on a stop sign, shades of green on a highway sign, shades of yellow on a traffic sign). For example, media module 302 can use a camera associated with client device 102 to detect the one or more colors of interest.

[0056] In some embodiments, the media module 302 may employ a shape-based detection method or a corner-based detection method. For example, in such embodiments, the media module 302 may be configured to employ a generalized Hough transform for finding shapes in an image, wherein predefined shapes such as polygons may be detected within the image.

[0057] In some embodiments, the media module 302 may detect signs within a presentation of an environment by utilizing machine learning techniques by training the media module 302 using training data that includes images of various types of signs (e.g., street signs, license plates, store signs, etc.).

[0058] At operation 410, based on the location of the sign within the representation of the environment and the route to the destination of interest, the media module 302 generates a media item within the AR interface. For example, the media item may include media content associated with the destination of interest and a directional indicator (e.g., an arrow) that guides the user to the destination of interest.

[0059] In some embodiments, the media module 302 may generate a media item based on a route to a destination of interest and one or more attributes associated with the client device 102, such as user profile information. For example, the media module 302 may retrieve a user identifier associated with a user of the client device 102 and generate a customized media item to guide the user of the client device 102 to the destination of interest (e.g., "Turn left here, User_A!"). In some embodiments, the user profile information may additionally include a language preference.

[0060] At operation 412, presentation module 308 causes display, within the AR interface at client device 102, of the media item generated by media module 302 at the location of the signage within the presentation of the environment. Thus, client device 102 may present an AR interface depicting a directional indicator directing the user to a destination of interest by presenting the directional indicator over the real-world signage presented within the presentation of the environment.

[0061] Figure 5 1 is a flowchart illustrating a method 500 for presenting AR content at a client device 102 according to some example embodiments. The operations of the method 500 may be as described above with respect to Figure 3 The modules described are executed. Figure 5As shown, method 500 includes one or more operations 502, 504, 506, and 508, which can be used as Figure 4 4. The method 400 depicted in the example of FIG. 4 is performed as a portion (eg, a subroutine) of the method 400 depicted in FIG.

[0062] At operation 502, GPS module 306 detects client device 102 within a threshold distance of a destination of interest. For example, the destination of interest itself may contain multiple thresholds, where each threshold is associated with media content to be presented to the device when the threshold is crossed. Thus, in response to determining a route to the destination of interest, as in operation 404 of method 400, GPS module 306 may retrieve the threshold from a memory location associated with the destination of interest at database 120 to trigger an event at client device 102 in response to detecting that client device 102 has crossed the threshold.

[0063] At operation 504, in response to the GPS module 306 detecting the client device 102 within the threshold distance of the destination of interest, the media module 302 accesses media content associated with the threshold of the destination of interest. For example, the media content associated with the threshold may include an indication of the distance to the destination of interest and other incentives to encourage the user of the client device 102 to continue on the route. Such incentives may include images and animations to be presented within the AR interface.

[0064] At operation 506 , the media module 302 detects display of signage within the presentation of the environment at the client device 102 , and at operation 508 , based on the display of the signage, generates and displays a presentation of media content associated with the threshold of the destination of interest at a location within the AR interface.

[0065] For example, the media content may include a graphical element that indicates to the user of client device 102 that the user is a certain distance (eg, fifty feet, twelve parsecs) from a destination of interest.

[0066] Figure 6 6 is a flowchart illustrating a method 600 for presenting AR content at a client device 102 according to some example embodiments. The operations of the method 600 may be as described above with respect to Figure 3 The modules described are executed. Figure 6 As shown, method 600 includes one or more operations 602, 604, and 606, which can be performed as in Figure 4 4. The method 400 depicted in the example of FIG. 4 is performed as a portion (eg, a subroutine) of the method 400 depicted in FIG.

[0067] At operation 602, as in operation 408 of method 400 and operation 506 of method 500, media module 302 detects signage within the rendering of the environment at client device 102. Thus, the signage detected within the rendering of the environment may include one or more graphical characteristics, such as font, character weight, character style, color, shape, size, and faded (i.e., damaged), or any pattern or graphic depicted on the signage.

[0068] At operation 604, the media module 302 generates a style template for the sign based on one or more graphical characteristics of the sign detected by the media module 302. Thus, the style template can define a set of graphical parameters for a media item to be generated and presented within the AR interface. At operation 606, the media module 302 generates a media item based on the one or more graphical characteristics defined by the style template.

[0069] Figure 7 is an interface graphic 700 depicting AR content 715 and 720 presented in an AR interface 705B at a client device 102, as shown in FIG. Figure 4 , 5 and as described in methods 400, 500, and 600 depicted in FIGS.

[0070] As seen in interface graphic 700 , client device 102A may cause presentation of display environment 705A, wherein presentation of environment 705A includes display of signage, such as signage 710A.

[0071] In response to detecting signage 710A, as visible in the rendering of environment 705A, AR guidance system 124 generates and causes display of AR content 715 and 720 , wherein AR content 715 and 720 include media items, such as directional indicators, to guide the user of client device 102 to the destination of interest.

[0072] According to some embodiments and as depicted in interface graphic 700, media items may be overlaid on top of a rendering of environment 705A to generate AR interface 705B. In some embodiments, media items 715 and 720 may be dynamic based on the perspective and position of client device 102 relative to the corresponding signage.

[0073] Figure 8 is an interface graphic 800 depicting AR content 815 presented in an AR interface 805B at a client device 102, as shown in FIG. Figure 4 , 5 and as described in methods 400, 500, and 600 depicted in FIGS.

[0074] As discussed in operation 604 of method 600, the AR guidance system 124 may generate a style template based on one or more graphical characteristics of the sign 810 detected within the rendering of the environment 805A. The one or more graphical characteristics may include, for example, a font, character weight, color, and size of text displayed on the sign 810.

[0075] Based on one or more graphic characteristics, AR guidance system 124 generates and displays AR content 815 at a location within AR interface 805B based on the location of sign 810. As can be seen in interface graphic 800, the AR content includes text that matches the same font and style as the text from sign 810.

[0076] According to some embodiments and as depicted in interface graphic 800, AR content 815 may include overlays on top of existing signage as well as media content including new signage that does not actually exist in real-world signage 810. For example, as visible in AR interface 805B, AR guidance system 124 expands signage 810 to include a directional indicator presented in AR content 815.

[0077] Software Architecture

[0078] Fig. 9 is a block diagram illustrating an example software architecture 906 that may be used in conjunction with the various hardware architectures described herein. Fig. 9 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture 906 may be implemented in hardware (such as Fig.10 1000), which includes, among other things, a processor 1004, a memory 1014, and an I / O component 1018. A representative hardware layer 952 is shown and may represent, for example, Fig. 9 The machine 900 of the embodiment of the present invention. The representative hardware layer 952 includes a processing unit 954 having associated executable instructions 904. The executable instructions 904 represent executable instructions of the software architecture 906, including implementations of the methods, components, etc. described herein. The hardware layer 952 also includes a memory and / or storage module memory / storage device 956 also having executable instructions 904. The hardware layer 952 may also include other hardware 958.

[0079] exist Fig. 9In the example architecture of , software architecture 906 can be conceptualized as a stack of layers, each of which provides specific functionality. For example, software architecture 906 may include layers such as operating system 902, library 920, application 916, and presentation layer 914. In operation, application 916 and / or other components within a layer may call application programming interface (API) API call 908 through a software stack, and receive a response as a response to API call 908. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 918, while other operating systems may provide such layers. Other software architectures may include additional or different layers.

[0080] The operating system 902 can manage hardware resources and provide public services. The operating system 902 may include, for example, a kernel 922, services 924, and drivers 926. The kernel 922 may act as an abstraction layer between the hardware and other software layers. For example, the kernel 922 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 924 may provide other public services to other software layers. Drivers 926 are responsible for controlling the hardware or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 926 include display drivers, camera drivers, drives, flash drives, serial communications drivers (e.g., Universal Serial Bus (USB) drivers), drivers, audio drivers, power management drivers, and more.

[0081] The library 920 provides a common infrastructure used by the application 916 and / or other components and / or layers. The library 920 provides functions that allow other software components to perform tasks in an easier manner than directly connecting to the underlying operating system 902 functions (e.g., kernel 922, service 924, and / or driver 926). The library 920 may include a system library 944 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 920 may include an API library 946, such as a media library (e.g., a library for supporting the presentation and operation of various media formats (such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG)), a graphics library (e.g., an OpenGL framework that can be used to render two-dimensional and three-dimensional graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a network library (e.g., WebKit that can provide network browsing functions), etc. The library 920 may also include various other libraries 948 to provide many other APIs to the application 916 and other software components / modules.

[0082] Framework / middleware 918 (sometimes also referred to as middleware) provides a higher level common infrastructure that can be used by applications 916 and / or other software components / modules. For example, framework / middleware 918 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 918 can provide many other APIs that can be used by applications 916 and / or other software components / modules, some of which can be specific to a particular operating system 902 or platform.

[0083] Applications 916 include built-in applications 938 and / or third-party applications 940. Examples of representative built-in applications 938 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 940 may include applications that are developed by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) to develop applications and can be used on mobile operating systems (such as IOS TM ANDROID TM , The third party application 940 may call an API call 908 provided by a mobile operating system (such as the operating system 902) to facilitate the functions described herein.

[0084] Applications 916 may use built-in operating system functionality (e.g., kernel 922, services 924, and / or drivers 926), libraries 920, and framework / middleware 918 to create a user interface to interact with a user of the system. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 914. In these systems, the application / component "logic" may be separated from the aspects of the application / component that interact with the user.

[0085] Fig.10 1 is a block diagram illustrating components of a machine 1000 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Fig.10A graphical representation of a machine 1000 in the example form of a computer system is shown within which instructions 1010 (e.g., software, programs, applications, applet, application software, or other executable code) for causing the machine 1000 to perform any one or more of the methods discussed herein may be executed. Thus, the instructions 1010 may be used to implement the modules or components described herein. The instructions 1010 convert a general purpose, unprogrammed machine 1000 into a specific machine 1000 that is programmed to perform the functions described and illustrated in the manner described. In alternative embodiments, the machine 1000 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1000 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1010, sequentially or otherwise, specifying actions to be taken by the machine 1000. In addition, while only a single machine 1000 is shown, the term "machine" should also be construed to include a collection of machines that individually or jointly execute instructions 1010 to implement any one or more of the methodologies discussed herein.

[0086] The machine 1000 may include a processor 1004, a memory / storage device 1006, and an I / O component 1018, which may be configured to communicate with each other, such as via a bus 1002. The memory / storage device 1006 may include a memory 1014 (such as a main memory or other memory storage device) and a storage unit 1016, both of which are accessible by the processor 1004, such as via the bus 1002. The storage unit 1016 and the memory 1014 store instructions 1010 that embody any one or more of the methods or functions described herein. The instructions 1010 may also reside, in whole or in part, within the memory 1014, within the storage unit 1016, within at least one processor 1004 (e.g., within a cache memory of a processor), or any suitable combination thereof during execution of the instructions 1010 by the machine 1000. Thus, the memory 1014, the storage unit 1016, and the memory of the processor 1004 are examples of machine-readable media.

[0087] I / O components 1018 may include a variety of components to receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 1018 included in a particular machine 1000 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a terminalless server machine will likely not include such a touch input device. It should be understood that I / O components 1018 may be included in Fig.10 Many other components not shown in the figure. The I / O components 1018 are grouped according to function only to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1018 may include output components 1026 and input components 1028. The output components 1026 may include visual components (e.g., displays (such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs)), projectors, or cathode ray tubes (CRTs)), sound components (e.g., speakers), tactile components (e.g., vibration motors, resistor mechanisms), other signal generators, etc. The input components 1028 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide the location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0088] In other example embodiments, the I / O component 1018 may include a biometric component 1030, a motion component 1034, an environmental component 1036, or a positioning component 1038, among a wide range of other components. For example, the biometric component 1030 may include a component that detects expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measures biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifies a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1034 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 1036 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects hazardous gas concentrations for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 938 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or a barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0089] Various technologies may be used to implement communications. I / O components 1018 may include a communication component 1040 operable to couple machine 1000 to network 1032 or device 1020 via coupling 1022 and coupling 1024, respectively. For example, communication component 1040 may include a network interface component or other suitable device for interfacing with network 1032. In further examples, communication component 1040 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low power consumption), Device 1020 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via a universal serial bus (USB)).

[0090] In addition, the communication component 1040 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1040 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode (such as a universal product code (UPC) barcode), a multi-dimensional barcode (such as a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Supercode, UCCRSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal with a tag). In addition, various information can be derived via the communication component 1040, such as a location via an Internet Protocol (IP) geographic location, a location via a location, a location via a location, and a location. The location of signal triangulation, the location of NFC beacon signals that can indicate a specific location via detection, etc.

[0091] Glossary

[0092] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media that facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0093] In this context, a "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communication device that a user may use to access a network.

[0094] In this context, a "communication network" refers to one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless or cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0095] In this context, a "short message" refers to a message that is accessible for a limited duration of time. A short message can be text, image, video, etc. The access time of a short message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0096] In this context, "machine-readable medium" refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be considered to include a single medium or multiple media capable of storing instructions (e.g., centralized or distributed databases, or associated caches and servers). The term "machine-readable medium" should also be considered to include any medium or combination of multiple media capable of storing instructions (e.g., code) executed by a machine so that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or device as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" excludes the signal itself.

[0097] In this context, "component" refers to a device, physical entity, or logic with boundaries defined by function or subroutine calls, branch points, application program interfaces (APIs), or other technologies that provide partitioning or modularization of specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processes. A component can be a packaged functional hardware unit that is designed to be used with other components and a part of a program that generally performs a specific function of related functions. A component can constitute a software component (e.g., a code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., an independent computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or an application portion) to operate as a hardware component that performs a specific operation as described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform a specific operation. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component may also include a programmable logic or circuit that is temporarily configured by software to perform a specific operation. For example, a hardware component may include software executed by a general purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function and is no longer a general purpose processor. It should be understood that the decision to mechanically implement a hardware component in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a specific manner or perform a specific operation described herein. Considering an embodiment in which a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, where a hardware component includes a general-purpose processor that is configured by software as a special-purpose processor, the general-purpose processor may be configured as respectively different special-purpose processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to, for example, constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components existing at the same time, communication can be achieved by signal transmission (e.g., through appropriate circuits and buses) between two hardware components or more hardware components. In embodiments where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing and retrieving information in a memory structure accessible to multiple hardware components. For example, a hardware component can perform an operation and store the output of the operation in a storage device coupled to its communication. Then, another hardware component can access the storage device later to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device and can operate on a resource (e.g., an information set). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether temporarily configured or permanently configured, such a processor can constitute a processor-implemented component for performing one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein can be at least partially processor-implemented, where one or more specific processors are examples of hardware. For example, at least some operations in the method can be performed by one or more processors or components implemented by the processor. In addition, one or more processors can also operate to support the performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some operations can be performed by a group of computers (as an example of a machine including a processor), which can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interface (API)). The performance of certain operations can be distributed between processors, not only residing in a single machine, but also deployed on multiple machines. In some example embodiments, a processor or a component implemented by a processor can be located in a single geographical location (e.g., in a home environment, an office environment, or a server cluster). In other example embodiments, a processor or a component implemented by a processor can be distributed in multiple geographical locations.

[0098] In this context, a "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine code," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may further be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0099] In this context, a "timestamp" refers to a sequence of characters or encoded information that identifies when a particular event occurred, such as giving a date and time of day, sometimes with accuracy to fractions of a second.

Claims

1. A method comprising: determining a destination of interest based on a user attribute associated with a user of the client device; determining a route from the location of the client device to the destination of interest; generating a media item based on the route to the destination of interest; as well as Display of a representation of the media item is caused at the client device.

2. The method according to claim 1, wherein: Generating the media item based on the route to the destination of interest includes: determining a direction of travel based on the location of the client device and the destination of interest; and Based on the direction of travel, the media item is generated.

3. The method according to claim 2, wherein: Based on the traveling direction, generating the media item comprises: Based on the direction of travel, a graphical symbol is selected from a graphical symbol database.

4. The method according to claim 1, further comprising: detecting, at the client device, display of signage within image data displayed at the client device; as well as The media item is generated based on the route to the destination of interest and the display of the signage.

5. The method according to claim 4, wherein: The display of the signage includes one or more attributes, and wherein generating the media item includes: The media item is generated based on the route to the destination of interest and the one or more attributes of the display of the sign.

6. The method according to claim 1, further comprising: detecting that the client device is within a threshold distance of the destination of interest; accessing media content associated with the destination of interest; as well as The media content is presented at the client device.

7. The method according to claim 1, wherein: The destination of interest is a first destination of interest within a geo-fenced area, the geo-fenced area includes a plurality of destinations of interest, and the method further includes: accessing a user profile of a user associated with the client device, the user profile comprising user profile information, the user profile information comprising the user attributes; and The first destination of interest is selected from the plurality of destinations of interest based on the user profile information.

8. A system comprising: Memory; as well as at least one hardware processor coupled to the memory and comprising instructions for causing the system to perform operations comprising: determining a destination of interest based on a user attribute associated with a user of the client device; determining a route from the location of the client device to the destination of interest; generating a media item based on the route to the destination of interest; and Display of a representation of the media item is caused at the client device.

9. The system according to claim 8, wherein: Generating the media item based on the route to the destination of interest includes: determining a direction of travel based on the location of the client device and the destination of interest; and Based on the direction of travel, the media item is generated.

10. The system according to claim 9, wherein: Based on the traveling direction, generating the media item comprises: Based on the direction of travel, a graphical symbol is selected from a graphical symbol database.

11. The system according to claim 8, wherein: The operations also include: detecting, at the client device, display of signage within image data displayed at the client device; and The media item is generated based on the route to the destination of interest and the display of the signage.

12. The system according to claim 11, wherein: The display of the signage includes one or more attributes, and wherein generating the media item includes: The media item is generated based on the route to the destination of interest and the one or more attributes of the display of the sign.

13. The system according to claim 8, wherein: The operations also include: detecting that the client device is within a threshold distance of the destination of interest; accessing media content associated with the destination of interest; and The media content is presented at the client device.

14. The system according to claim 8, wherein: The destination of interest is a first destination of interest within a geo-fenced area, the geo-fenced area includes a plurality of destinations of interest, and the operations further include: accessing a user profile of a user associated with the client device, the user profile comprising user profile information, the user profile information comprising the user attributes; and The first destination of interest is selected from the plurality of destinations of interest based on the user profile information.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: determining a destination of interest based on a user attribute associated with a user of the client device; determining a route from the location of the client device to the destination of interest; generating a media item based on the route to the destination of interest; as well as Display of a representation of the media item is caused at the client device.

16. The non-transitory machine-readable storage medium of claim 15, wherein: Generating the media item based on the route to the destination of interest includes: determining a direction of travel based on the location of the client device and the destination of interest; and Based on the direction of travel, the media item is generated.

17. The non-transitory machine-readable storage medium of claim 16, wherein: Based on the traveling direction, generating the media item comprises: Based on the direction of travel, a graphical symbol is selected from a graphical symbol database.

18. The non-transitory machine-readable storage medium of claim 15, wherein: The operations also include: detecting, at the client device, display of signage within image data displayed at the client device; and The media item is generated based on the route to the destination of interest and the display of the signage.

19. The non-transitory machine-readable storage medium of claim 18, wherein: The display of the signage includes one or more attributes, and wherein generating the media item includes: The media item is generated based on the route to the destination of interest and the one or more attributes of the display of the sign.

20. The non-transitory machine-readable storage medium of claim 15, wherein: The operations also include: detecting that the client device is within a threshold distance of the destination of interest; accessing media content associated with the destination of interest; and The media content is presented at the client device.

Images