Location-based augmented reality systems

Abstract

A location-based augmented reality system generates and causes display of augmented reality content including three-dimensional layouts based on the viewing angle and location of a client device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. patent application serial number 16 / 818,540, filed on March 13, 2020, and U.S. provisional application serial number 62 / 829,571, filed on April 4, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments of the present disclosure relate generally to mobile computing technology and, more particularly, but not by way of limitation, to the presentation of augmented reality and virtual reality displays. Background Art

[0004] Augmented reality (AR) is a real-time direct or indirect image of a physical, real-world environment, whose elements are supplemented or "augmented" by computer-generated sensory input (e.g., sound, video, graphics, etc.). Thus, the technology is used to enhance the user's perception of reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] To easily delineate the discussion of any particular element or action, the highest significant digit or digits in a reference number refer to the drawing number that first introduces the element.

[0006] Figure 1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network, wherein the messaging system includes a location-based augmented reality (AR) system, in accordance with some embodiments.

[0007] Figure 2 is a block diagram showing more details regarding a messaging system according to an example embodiment.

[0008] Figure 3 is a block diagram illustrating various modules of a location-based AR system, according to certain example embodiments.

[0009] Figure 4 is a flow chart illustrating a method of generating and causing display of location-based AR content, according to certain example embodiments.

[0010] Figure 5 is a flow chart illustrating a method of generating and causing display of location-based AR content, according to certain example embodiments.

[0011] Figure 6 is a flow chart illustrating a method for generating and causing display of location-based AR content, according to certain example embodiments.

[0012] Figure 7is a flow chart illustrating a method for generating and causing display of location-based AR content, according to certain example embodiments.

[0013] Figure 8 is an interface flow diagram illustrating presentation of location-based AR content according to certain example embodiments.

[0014] Figure 9 is a block diagram illustrating a representative software architecture that can be used in conjunction with the various hardware architectures described herein and to implement the various embodiments.

[0015] Figure 10 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0016] As discussed above, augmented reality, or “AR,” is a real-time, direct or indirect representation of a physical, real-world environment, elements of which are supplemented or “augmented” by computer-generated sensory input (e.g., sound, video, graphics, etc.). Thus, “AR content” includes media content that includes AR features to be presented within a representation of the real-world environment, such as within a display of a client device. According to certain embodiments, a location-based AR system is configured to associate AR content with a location and, in response to detecting a client device at or within the location, access and present the AR content at the client device.

[0017] In some embodiments, AR content includes a media overlay or "lens," where the lens modifies or transforms an image or video presented at a client device in some manner. For example, lens data (i.e., AR content) can be used to perform complex additions or transformations on an image or video, such as adding bunny ears to a person's head in a video clip, adding a floating heart with the background color to a video clip, changing the scale of a person's features in a video clip, or many other such transformations. Location-based AR content can be associated with a set of geolocation coordinates such that, in response to detecting a client device at that location, a lens associated with that location can be accessed and presented on the client device's display.

[0018] Reference will now be made in detail to specific example embodiments for carrying out the inventive subject matter of the present disclosure. In the following description, specific details are set forth in order to provide a thorough understanding of the subject matter. It should be understood that these embodiments can be practiced without some or all of these specific details. Thus, disclosed is a location-based AR system that performs operations comprising: accessing a surface model of an object or environment, wherein the surface model comprises a plurality of three-dimensional (3D) features; determining a relative positioning of the plurality of 3D features based on a location of a client device; determining a perspective of the client device based on the location of the client device and the relative positioning of the plurality of 3D features; accessing media content associated with the location of the client device; and applying the media content to a positioning within a presentation of an image presented at the client device, the positioning being based on the perspective of the client device.

[0019] In some example embodiments, the media content includes 3D topography. For example, a location-based AR system may assign a set of geographic location coordinates to the 3D topography and present the 3D topography as AR content in a presentation of an image at a client device. The text string of the 3D topography may be generated based on user input, based on user profile data associated with the client device, or in some embodiments may be predefined.

[0020] In some example embodiments, presenting media content based on the perspective of the client device may include applying one or more transformations to the media content based on the perspective by a process of distorting, stretching, expanding, splitting, projecting, or otherwise changing the perspective so as to give an impression of height, width, depth, and positioning relative to the viewing point (i.e., perspective) of the client device.

[0021] A location-based AR system may include a video and image acquisition system that records and / or presents images of an environment, and a graphical interface configured to display a presentation of the environment (i.e., a display of a client device). In some example embodiments, to apply media content to the presentation of the environment, the location-based AR system accesses a surface model associated with a location corresponding to the environment, wherein the surface model is a topographical representation of the environment, or one or more objects in the environment, that includes 3D features such as surfaces, contours, and shapes. For example, the surface model may include a wire mesh form that includes a 3D representation of an object within the environment or a form of the environment.

[0022] In some embodiments, the location-based AR system can generate a surface model based on image data received from a client device. In such embodiments, to generate the surface model, the location-based AR system applies various computer vision techniques to the image data.

[0023] In a further embodiment, the location-based AR system accesses a surface model database that includes surface models of the environment organized based on the geolocation coordinates of corresponding locations depicted by the surface models. For example, the location-based AR system determines the location of the client device and retrieves or otherwise accesses the corresponding surface model from the surface model database based on the location.

[0024] The location-based AR system identifies a set of features represented by a surface pattern to determine a perspective of a client device. The set of features may include distinguishing points or features, such as outlines, markers, or other features in space that can be used as graphical markers. For example, the distinguishing points or features may include landmarks and identifiable objects such as windows and / or doors. Having identified a set of features of the surface model, the location-based AR system determines the relative positioning of each distinguishing point or feature with respect to each other. For example, the relative positioning may indicate a distance between the distinguishing points or features. The location-based AR system determines a perspective of the client device based on the relative positioning of the distinguishing points or features, wherein the perspective indicates a representation of the space relative to the mobile device.

[0025] Consider an illustrative example from a user perspective. A user of a client device may cause a display of image data generated by a camera associated with the client device. The location-based AR system may receive a request from the client device, wherein the request includes location data identifying a location of the client device. The location-based AR system may then access a media repository to access AR content including 3D typography, wherein the AR content is associated with the location.

[0026] To present AR content at a client device, a location-based AR system accesses a surface model representing one or more surfaces of an environment and determines the perspective of the client device based on the 3D features of the surface model and the location of the client device. The AR content can then be applied to the image based on the perspective of the client device. From the user's perspective, the presentation of image data can include display of image data depicting an area, such as a street or urban area including one or more buildings, where each building can be associated with a corresponding surface model generated by the location-based AR system. The 3D typesetting can then be presented to the user so that it appears as if the 3D typesetting is floating around the surfaces of the buildings. The user can then provide input to edit or change the text string of the 3D typesetting, or in some embodiments, the location-based AR system can modify or edit one or more characteristics of the 3D typesetting based on user profile data associated with the user of the client device.

[0027] Figure 11 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. Messaging system 100 includes multiple client devices 102, each hosting multiple applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of messaging client applications 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0028] Thus, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and the messaging server system 108 via the network 106. The data exchanged between messaging client devices 104 and between messaging client applications 104 and the messaging server system 108 includes functionality (e.g., commands to call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0029] The messaging server system 108 provides server-side functionality to specific messaging client applications 104 via the network 106. Although certain functions of the messaging system 100 are described herein as being performed by either the messaging client application 104 or the messaging server system 108, it should be understood that the location of certain functions within the messaging client application 104 or the messaging server system 108 is a design choice. For example, it may be technically preferable to first deploy certain technologies and functions within the messaging server system 108 and then migrate the technologies and functions to the messaging client application 104 where the client device 102 has sufficient processing power.

[0030] The messaging server system 108 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to the messaging client applications 104, receiving data from the messaging client applications 104, and processing data generated by the messaging client applications 104. In some embodiments, the data includes, by way of example, message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. In other embodiments, other data is used. Data exchange in the messaging system 100 is invoked and controlled by functions available through the user interface (UI) of the messaging client applications 104.

[0031] Turning now specifically to messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to application server 112. Application server 112 is communicatively coupled to database server(s) 118, which facilitate access to database(s) 120 storing data associated with messages processed by application server 112.

[0032] In particular, an application program interface (API) server 110 is handled, which receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client application 104 in order to invoke functionality of the application server 112. The application program interface (API) server 110 exposes various functions supported by the application server 112, including: account registration, login functionality, sending messages from a particular messaging client application 104 to another messaging client application 104 via the application server 112, sending media files (e.g., images or videos) from a messaging client application 104 to the messaging server application 114 and for possible access by another messaging client application 104, setting up collections of media data (e.g., stories), retrieval of such collections, retrieval of a friend list of a user of a client device 102, retrieval of messages and content, adding and removing friends from a social graph, position of friends within a social graph, open and application events (e.g., related to the messaging client application 104).

[0033] The application server 112 hosts multiple applications and subsystems, including a messaging server application 114, an image processing system 116, a social networking system 122, and a location-based AR system 124. The messaging server application 114 implements multiple message processing techniques and functions, which particularly relate to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). The messaging server application 114 then makes these collections available to the messaging client applications 104. Given the hardware requirements of such processing, other processor- and memory-intensive processing of the data can also be performed on the server side by the messaging server application 114.

[0034] The application server 112 also includes an image processing system 116 that is dedicated to performing various image processing operations, typically with respect to images or videos received within the payload of messages at the messaging server application 114 .

[0035] The social networking system 122 supports various social networking functional services and makes these functionalities and services available to the messaging server application 114. To this end, the social networking system 122 maintains and accesses an entity graph 304 within the database(s) 120. Examples of functionalities and services supported by the social networking system 122 include identifying other users of the messaging system 100 with whom a particular user has a relationship or with whom the particular user is "following," as well as identifying other entities and interests of the particular user. The location-based AR system 124 provides functionality for generating and causing the display of morphing media within a representation of space.

[0036] The application server 112 is communicatively coupled to one or more database servers 118 , which facilitate access to database(s) 120 where data associated with messages processed by the messaging server application 114 is stored.

[0037] Figure 2 1 is a block diagram illustrating further details regarding messaging system 100 according to an example embodiment. Specifically, messaging system 100 is shown as including messaging client application 104 and application server 112, which in turn embody a number of subsystems, namely, short-term timer system 202, collection management system 204, and annotation system 206.

[0038] The short-term timer system 202 is responsible for enforcing temporary access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the short-term timer system 202 incorporates a plurality of timers that selectively display and enable access to messages and associated content (such as transformed media) via the messaging client application 104 based on duration and display parameters associated with the message, or collection of messages (e.g., a SNAPCHAT story). More details regarding the operation of the short-term timer system 202 are provided below.

[0039] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, images, video, and audio data). In some examples, collections of content (e.g., messages including distorted media, images, video, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be available for a specified period of time (e.g., the duration of the event to which the content is related). For example, content related to a concert (e.g., distorted media displayed at a specific location) can be available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing an icon that provides notification to the user interface of the messaging client application 104 of the existence of a particular collection.

[0040] The collection management system 204 also includes a curation interface 208 that allows collection managers to manage and curate specific content collections. For example, the curation interface 208 enables event organizers to curate a collection of content related to a specific event (e.g., removing inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, users can be compensated for including user-generated content in a collection. In this case, the curation interface 208 operates to automatically pay such users for the use of their content.

[0041] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The annotation system 206 can operatively provide media overlays (e.g., SNAPCHAT filters) to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 can operatively provide media overlays to the messaging client application 104 based on other information (such as social network information of the user of the client device 102). Media overlays can include audio and visual content and visual effects. Examples of audio and video content include distorted media, pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays or projection of distorted media items onto a presentation that depicts space. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, a media overlay includes text that can be superimposed on a photo or video stream generated by the client device 102. In another example, the media overlay includes an overlay of a location (e.g., Venice Beach), the name of a live event, or an overlay of a business name (e.g., Beach Cafe). In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of the business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database(s) 120 and accessible via the database server(s) 118.

[0042] In one example embodiment, the annotation system 206 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify under what circumstances specific media overlays should be provided to other users. The annotation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0043] In another example embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media overlays associated with geographic locations through a bidding process. For example, the annotation system 206 associates the media overlay of the merchant with the highest bid with the corresponding geographic location within a predetermined amount of time.

[0044] Figure 33 is a block diagram 300 illustrating components of a location-based AR system 124, according to various example embodiments, that configures the location-based AR system 124 to present AR content including 3D layouts based on the location and perspective of a client device. The location-based AR system 124 is shown to include a location module 302, a rendering module 304, an identification module 306, and an AR module 308, all of which are configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of these modules may be implemented using one or more processors 310 (e.g., by configuring such one or more processors to perform the functionality described for that module) and thus may include one or more of the processors 310.

[0045] Any one or more of the described modules may be implemented using hardware alone (e.g., one or more of the processors 310 of the machine) or a combination of hardware and software. For example, any module of the described location-based AR system 124 may physically include an arrangement of one or more of the processors 310 (e.g., a subset of the one or more processors of the machine or a subset thereof) configured to perform the operations described herein for that module. As another example, any module of the engagement tracking system 310 may include software, hardware, or both that configures an arrangement of one or more processors 310 (e.g., in one or more processors of the machine) to perform the operations described herein for that module. Thus, different modules of the location-based AR system 124 may include and configure different arrangements of the processors 310 or a single arrangement of the processors 310 at different points in time. Furthermore, any two or more modules of the location-based AR system 124 may be combined into a single module, and the functionality described herein for a single module may be subdivided into multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0046] Figure 4 is a flow chart illustrating various operations performed by the location-based AR system 124 in a method 400 for generating and causing display of AR content at a client device according to certain example embodiments. The operations of the method 400 may be as described above with respect to Figure 3 The modules described are executed. Figure 4 As shown, method 700 includes one or more operations 402 , 404 , 406 , 408 , and 410 .

[0047] At operation 402, the location module 302 accesses a surface model of an environment, wherein the surface model includes a plurality of 3D features. In some embodiments, to access the surface model of the environment, the location module 302 may access a repository including a collection of surface models associated with locations or landmarks, wherein the collection of surface models includes the surface model associated with the environment referenced above.

[0048] For example, the location module 302 can determine the location of the client device 102 based on location data generated by the client device 102 or by identifying one or more landmarks in image data generated and displayed at the client device 102. In response to determining the location of the client device 102, the location module 302 can access a repository (i.e., database 120) to retrieve a surface model of the environment corresponding to the location. Thus, the surface model can include a topographical representation of the environment or one or more objects in the environment, including 3D features such as surfaces, contours, and shapes. For example, the surface model can include a wire mesh form that includes a 3D representation of an object within the environment or the form of the environment.

[0049] At operation 404, the identification module 306 determines the relative positioning of the plurality of 3D features. In some embodiments, the identification module 306 may identify a set of features depicted by the surface model of the environment, which may include distinctive points or features, such as contours, markers, or other features that may serve as graphical markers in space. In this embodiment, a set of features of the surface model is identified, and the identification module 306 determines the relative positioning of each distinctive point or feature with respect to each other, wherein the relative positioning indicates the distance between the distinctive points or features.

[0050] At operation 406 , the AR module 308 determines a viewing angle of the client device 102 based on the position of the client device 102 and the relative positioning of the plurality of 3D features, wherein the viewing angle represents a viewing position of the client device 102 .

[0051] At operation 408, the AR module 308 accesses media content associated with the location at the client device 102. In some embodiments, the media repository (i.e., database 120) may contain a collection of media content, wherein the collection of media content is indexed within the repository based on location, landmark, or location attributes. Thus, by referencing a particular location identifier, landmark, or location attribute, the AR module 308 can identify the corresponding media content.

[0052] A "location" may include an actual geographic location that can be identified based on coordinates, as well as the type of location, such as a business (i.e., restaurant, coffee shop), a specific business (i.e., SNAP headquarters), or a place (i.e., park, beach). Attributes of a location may include weather conditions (i.e., cloudy, sunny), how crowded the environment is, temperature, and time of day.

[0053] At operation 410, the rendering module 304 applies the media content to a position within the image rendered at the client device 102, wherein the positioning is based on the viewing angle of the client device 102. In some embodiments, rendering the media content based on the viewing angle of the client device 102 may include applying one or more transformations to the media content based on the viewing angle through a process of distorting, stretching, expanding, splitting, projecting, or otherwise changing the viewing angle to give the impression of height, width, depth, and positioning relative to the viewing point (i.e., viewing angle) of the client device 102.

[0054] Figure 5 is a flow chart illustrating various operations of the location-based AR system 124 in performing a method 500 for generating and causing display of AR content at a client device according to certain example embodiments. The operations of the method 500 may be as described above with respect to Figure 3 The modules described are executed. Figure 5 As shown, according to some example embodiments, method 500 includes one or more operations 502 and 504 , which may be performed as part of (eg, a predecessor task, a subroutine, or a portion of) method 400 .

[0055] At operation 502, the presentation module 304 accesses a user profile associated with the client device 102, wherein the user profile includes user profile data. In some embodiments, in response to detecting a client device associated with a user at a location (i.e., the client device 102), the presentation module 304 can access the user profile of the user to obtain the user profile data.

[0056] According to certain example embodiments, user profile data may include user selections defining the user's affinities and interests, as well as user demographic information, language preferences, a list of user connections (ie, a buddy list), and a log of user actions performed by the user.

[0057] At operation 504, the AR module 308 configures the media content based on user profile data from a user profile associated with the client device 102. For example, Figure 8 As shown in the depicted interface diagram 804, media content 806 may include media attributes including text features such as qualifiers or phrases. In some embodiments, the AR module 308 may configure the media attributes of the media content 806 based on user profile data associated with the client device 102. By doing so, the media content 806 may be presented differently for different users on their corresponding devices.

[0058] As illustrative examples, the AR module 308 may translate textual features of the media content 806 based on a language preference associated with the user, or may select and display graphical features (i.e., color, size, shape, etc.) of the media content 806 based on user profile data associated with the client device 102.

[0059] In some embodiments, the location at which to display the media content 806 may also be determined based on user profile data corresponding to the client device 102. For example, the user of the client device 102 may have corresponding user profile data that identifies one or more locations associated with the user, such as addresses (workplace, home, gym, etc.), based on explicit user input or implicit user activity (i.e., frequency of visits to a location over a period of time, duration of each visit to a location, etc.). In this embodiment, the AR module 308 may select and present media content (i.e., media content 806) at the location based on the location from the user profile data. As an illustrative example, in response to identifying a home address associated with the user of the client device 102, the AR module 308 may present the media content at a location based on the user's home address.

[0060] Figure 6 is a flow chart illustrating various operations of the location-based AR system 124 in performing a method 600 for generating and causing display of AR content at a client device according to certain example embodiments. The operations of the method 600 may be as described above with respect to Figure 3 The modules described are executed. Figure 6 As shown, according to some example embodiments, method 600 includes one or more operations 602 and 604 , which may be performed as part of (eg, a predecessor task, a subroutine, or a portion of) method 400 .

[0061] At operation 602, the AR module 308 receives a request to edit a feature of media content from the client device 102, wherein the feature includes a first text string and the request includes a second text string. For example, a user can configure a feature of the media content by providing input selecting the media content to be displayed at the client device 102. In response to the input, the presentation module 304 can cause the display of an interface that receives input for editing the feature of the selected media content.

[0062] At operation 604, the AR module 308 edits the presentation of the media content at the client device based on the second request included in the request. As an illustrative example, the user of the client device 102 may provide a selection of media content (such as Figure 8The user of the client device 102 may then provide input including the second text string. The AR module 308 may then change the presentation of the media content 806 based on the second text string in the request.

[0063] Figure 7 is a flow chart illustrating various operations of the location-based AR system 124 in performing a method 700 for generating and causing display of AR content at a client device according to certain example embodiments. The operations of the method 700 may be as described above with respect to Figure 3 The modules described are executed. Figure 7 As shown, according to some example embodiments, method 700 includes one or more operations 702 , 704 , 706 , and 708 , which may be performed as part of (eg, a predecessor task, a subroutine, or a portion of) method 400 .

[0064] At operation 702, the AR module 308 accesses a surface model. As discussed above, a surface model is a topographical representation of an environment or one or more objects within the environment that includes 3D features such as surfaces, contours, and shapes. For example, a surface model may include a wire mesh that includes a 3D representation of an object within the environment or the form of the environment.

[0065] In some embodiments, accessing the surface model may include generating the surface model. For example, the AR module 308 may generate the surface model based on image data received from the client device. In this embodiment, to generate the surface model, the AR module 308 applies various computer vision techniques to the image data to identify surface features and generates the surface model based on the surface features.

[0066] At operation 704, the location module 302 assigns a surface model generated by the AR module 308 to the location based on one or more of the surface features corresponding to the location and the geo-location coordinates associated with the location. In some embodiments, the location module 302 can index the surface model within a surface model database (i.e., database 120), wherein the surface model database includes a plurality of surface models assigned to the location.

[0067] In this embodiment, by referencing position or geographic location coordinates based on a set of surface features, a corresponding surface model may be identified.

[0068] At operation 706, as in Figure 4 In operation 402 of the depicted method 400 , the AR module 308 receives a request from the client device 102 , wherein the request includes location information (ie, a set of surface features, geographic location coordinates). For example, the location information may identify the current location of the client device 102 .

[0069] In response to receiving the request including the location information, at operation 708, the AR module 308 accesses a surface model associated with the location information within the database 120. Accordingly, the AR module 308 may then render and display media content within the representation of the image data based on the surface model corresponding to the location.

[0070] Figure 8 1 is an interface flow diagram illustrating the presentation of location-based AR content presented by the location-based AR system 124 according to some example embodiments. Figure 8 As can be seen in the figure, the interface flow chart includes interface diagram 802 and interface diagram 804.

[0071] According to some example embodiments, the client device 102 may cause the display of a presentation of the interface diagram 802. For example, the client device 102 may capture image data and generate the interface depicted by the interface diagram 802. Figure 4 The method 400 and Figure 7 As discussed in the depicted method 700 , modules of the location-based AR system 124 may access a surface model associated with the location depicted by the interface diagram 802 based on surface features from an image or based on location data accessed at the client device 102 .

[0072] As can be seen in interface diagram 804, the location-based AR system 124 can access media content within a repository (i.e., database 120) based on the location of the client device 102. The media content (i.e., media content 806) can be associated with a location within the media repository such that a reference to the location within the repository can identify the media content 806.

[0073] The location-based AR system 124 may then cause a rendering of the media content 806 to be displayed at the location within the GUI, as seen in interface diagram 804 .

[0074] Software Architecture

[0075] Figure 9 is a block diagram illustrating an example software architecture 906 that may be used in conjunction with the various hardware architectures described herein. Figure 9 is a non-limiting example of a software architecture, and it will be understood that numerous other architectures may be implemented to facilitate the functionality described herein. The software architecture 906 may be implemented in a system such as Figure 10 1000, including, among other things, a processor 1004, a memory 1014, and I / O components 1018. A representative hardware layer 952 is shown and may represent, for example, Figure 10The machine 1000 is a computer system. A representative hardware layer 952 includes a processing unit 954 having associated executable instructions 904. Executable instructions 904 represent executable instructions of a software architecture 906, including implementations of the methods, components, and the like described herein. The hardware layer 952 also includes memory and / or storage modules, memory / storage devices 956, also having executable instructions 904. The hardware layer 952 may also include other hardware 958.

[0076] exist Figure 9 In the example architecture of , software architecture 906 can be conceptualized as a stack of layers, wherein each layer provides specific functionality. For example, software architecture 906 may include layers such as operating system 902, library 920, application 916, and presentation layer 914. In operation, applications 916 and / or other components within a layer can call application program interface (API) API call 908 through the software stack and receive responses in response to API call 908. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 918, while other operating systems may provide such layers. Other software architectures may include additional or different layers.

[0077] The operating system 902 can manage hardware resources and provide common services. The operating system 902 may include, for example, a kernel 922, services 924, and drivers 926. The kernel 922 may act as an abstraction layer between the hardware and other software layers. For example, the kernel 922 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 924 may provide other common services to other software layers. Drivers 926 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 926 include display drivers, camera drivers, drives, flash drives, serial communication drivers (such as Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0078] The libraries 920 provide a common infrastructure used by applications 916 and / or other components and / or layers. The libraries 920 provide functionality that allows other software components to perform tasks more easily than by directly interfacing with underlying operating system 902 functionality (e.g., kernel 922, services 924, and / or drivers 926). The libraries 920 may include system libraries 944 (e.g., the C standard library), which may provide functionality such as memory allocation, string manipulation, and mathematical functions. Furthermore, the libraries 920 may include API libraries 946, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats (e.g., MPREG4, H.264, MP3, AAC, AMR, JPG, and PNG)), graphics libraries (e.g., the OpenGL framework for rendering 2D and 3D graphics content on a display), database libraries (e.g., SQLite, which may provide various relational database functions), networking libraries (e.g., WebKit, which may provide web browsing functionality), and the like. The library 920 may also include a variety of other libraries 948 to provide a variety of other APIs to the applications 916 and other software components / modules.

[0079] The framework / middleware 918 (sometimes also referred to as middleware) provides a high-level, general-purpose infrastructure that can be used by applications 916 and / or other software components / modules. For example, the framework / middleware 918 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 918 can provide a wide range of other APIs that can be used by applications 916 and / or other software components / modules, some of which may be specific to a particular operating system 902 or platform.

[0080] Applications 916 include built-in applications 938 and / or third-party applications 940. Examples of representative built-in applications 938 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 940 may include applications that are created by entities other than the vendor of a particular platform using Android TM or IOS TM Software Development Kit (SDK) for developing applications and can be used on mobile operating systems such as IOS TM ANDROID TM 、 Third-party applications 940 may invoke API calls 908 provided by a mobile operating system (such as operating system 902) to facilitate the functionality described herein.

[0081] Applications 916 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernel 922, services 924, and / or drivers 926), libraries 920, and framework / middleware 918. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 914. In these systems, the application / component "logic" may be separated from aspects of the application / component that interact with the user.

[0082] Figure 10 is a block diagram illustrating components of a machine 1000 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. Specifically, Figure 10 A diagrammatic representation of a machine 1000 in the example form of a computer system is shown within which instructions 1010 (e.g., software, programs, applications, applet, application program, or other executable code) may be executed to cause the machine 1000 to perform any one or more of the methodologies discussed herein. Thus, the instructions 1010 may be used to implement the modules or components described herein. The instructions 1010 convert a general-purpose, unprogrammed machine 1000 into a specific machine 1000 that is programmed to perform the functions described and illustrated in the manner described. In alternative embodiments, the machine 1000 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a network deployment, the machine 1000 may operate as a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1000 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 1010 that specify actions to be taken by the machine 1000. Furthermore, while only a single machine 1000 is shown, the term "machine" shall also be taken to include a collection of machines that individually or collectively execute instructions 1010 to perform any one or more of the methodologies discussed herein.

[0083] The machine 1000 may include a processor 1004, memory storage / storage devices 1006, and I / O components 1018, which may be configured to communicate with each other, such as via a bus 1002. The memory / storage devices 1006 may include a memory 1014 (such as a main memory or other memory storage device) and a storage unit 1016, which may be accessed by the processor 1004, such as via the bus 1002. The storage unit 1016 and the memory 1014 store instructions 1010 that embody any one or more of the methodologies or functions described herein. During execution by the machine 1000, the instructions 1010 may also reside, in whole or in part, within the memory 1014, within the storage unit 1016, within at least one of the processors 1004 (e.g., within a cache memory of the processor), or any suitable combination thereof. Thus, the memory 1014, the storage unit 1016, and the memory of the processor 1004 are examples of machine-readable media.

[0084] The I / O components 1018 may include a variety of components to receive input, provide output, generate output, send information, exchange information, collect measurements, etc. The specific I / O components 1018 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that the I / O components 1018 may be included in Figure 10 The I / O components 1018 are grouped according to their functionality for the purpose of simplifying the following discussion only and are by no means limiting. In various exemplary embodiments, the I / O components 1018 may include an output component 1026 and an input component 1028. The output component 1026 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an auditory component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistive mechanism), other signal generators, and the like. The input component 1028 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides touch location and / or force or touch gestures, or other tactile input component), an audio input component (e.g., a microphone), and the like.

[0085] In further example embodiments, the I / O component 1018 may include, among other components, a biometric component 1030, a motion component 1034, an environmental component 1036, or a position component 1038. For example, the biometric component 1030 may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 1034 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental component 1036 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The position component 1038 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects the altitude from which the air pressure can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0086] A variety of technologies can be used to implement communications. The I / O components 1018 may include a communications component 1040 operable to couple the machine 1000 to the network 1032 or the device 1020 via coupling 1022 and coupling 1024, respectively. For example, the communications component 1040 may include a network interface component or other suitable device that interfaces with the network 1032. In further examples, the communications component 1040 may include a wired communications component, a wireless communications component, a cellular communications component, a near field communications (NFC) component, a Components (e.g. Low energy consumption), Device 1020 may be another machine or any of a variety of peripheral devices, such as peripheral devices coupled via a universal serial bus (USB).

[0087] In addition, the communication component 1040 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1040 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as universal product codes (UPC) barcodes, multi-dimensional barcodes (e.g., Quick Response (QR) codes, Aztec codes, Data Matrix, Digital Graphics, Maximal codes, PDF417, Supercodes, UCC RSS-2D barcodes), and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various information can be obtained via the communication component 1040, such as location via Internet Protocol (IP) geolocation, location information via Signal triangulation to obtain location, obtaining location via detection of NFC beacon signals that can indicate a specific location, etc.

[0088] Glossary

[0089] In this context, "carrier signal" means any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium through a network interface device and using any of a number of well-known transmission protocols.

[0090] In this context, a "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistant (PDA), smartphone, tablet computer, ultrabook, netbook, laptop computer, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communication device that a user can use to access a network.

[0091] In this context, a "communication network" refers to one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless or cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling may implement any of a variety of types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards development organizations, other long range protocols, or other data transmission technologies.

[0092] In this context, a "temporary message" is a message that is accessible for a limited duration. A temporary message can be text, images, videos, and more. The access time for a temporary message can be set by the sender of the message. Alternatively, the access time can be set by default or specified by the recipient. Regardless of the setting technique, the message is temporary.

[0093] In this context, a "machine-readable medium" refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (such as erasable programmable read-only memory (EPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (such as a centralized or distributed database, or associated caches and servers) that can store instructions. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media that can store instructions (such as code) executed by a machine, such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Thus, a "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" itself does not include signals.

[0094] In this context, a "component" refers to a device, physical entity, or logic that has boundaries defined by function or subroutine calls, branch points, application program interfaces (APIs), or other techniques for partitioning or modularizing specific processing or control functions. Components can be combined through their interfaces with other components to execute a machine process. A component can be a packaged functional hardware unit designed to be used with other components and a portion of a program that typically performs a specific function related to the function. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that is capable of performing certain operations and can be configured or arranged in some physical manner. In various example embodiments, one or more computer systems (e.g., stand-alone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) to operate as a hardware component to perform certain operations as described herein. A hardware component can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component can be a dedicated processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Hardware components may also include programmable logic or circuits that are temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or specific component of a machine) specifically customized to perform the configured function and is no longer a general-purpose processor. It should be understood that the decision to implement the hardware component mechanically in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be made for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein. Considering embodiments in which hardware components are temporarily configured (e.g., programmed), each hardware component does not need to be configured or instantiated at any instance of time. For example, in the case where the hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as different special-purpose processors (e.g., including different hardware components) at different times. Thus, software configures one or more specific processors accordingly, for example, to constitute a specific hardware component at one instance in time and to constitute a different hardware component at a different instance in time. Hardware components can provide information to other hardware components, or receive information from other hardware components. Thus, the described hardware components can be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between two or more hardware components (e.g., on appropriate circuits and buses). In embodiments in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing and obtaining information in a memory structure accessible to multiple hardware components. For example, a hardware component can perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, another hardware component can access the memory device at a later time to obtain and process the stored output. The hardware component can also initiate communication with an input or output device and can operate on a resource (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily (e.g., by software) configured or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations of the method may be performed by one or more processors or processor-implemented components. In addition, one or more processors may also be operable to support the performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some operations may be performed by a group of computers (as an example of a machine including a processor), which can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)). The performance of certain operations may be distributed among the processors, not only residing in a single computer, but also deployed on multiple computers. In some example embodiments, the processor or processor-implemented component may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or processor-implemented component may be distributed over multiple geographic locations.

[0095] In this context, a "processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executed on an actual processor) that manipulates data values according to control signals (e.g., "commands," "opcodes," "machine code," etc.) and generates corresponding output signals for operating a machine. A processor may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may further be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0096] In this context, a "timestamp" refers to a sequence of characters or encoded information that identifies when a specific event occurred, such as giving a date and time, sometimes accurate to a fraction of a second.

Claims

1. A system comprising: Memory; as well as at least one hardware processor coupled to the memory and comprising instructions that cause the system to perform operations comprising: accessing a surface model of an environment, the surface model comprising a wire mesh form comprising a topographic representation of a plurality of three-dimensional (3D) features of the environment; determining a relative position of the plurality of 3D features with respect to each other; determining a viewing angle of the client device based on a position of the client device and the relative positioning of the plurality of 3D features within the environment, wherein the viewing angle indicates a representation of space relative to the client device; accessing media content associated with the location of the client device; and The media content is applied to a position within an image rendered at the client device, the positioning being based on a viewing angle of the client device.

2. The system according to claim 1, wherein: The media content includes 3D typography including a text string.

3. The system according to claim 2, wherein: The instructions also include: accessing a user profile associated with the client device, the user profile including user profile data; and The text string is generated based on the user profile data from the user profile associated with the client device.

4. The system according to claim 2, wherein: The text string is a first text string, and the instructions further include: receiving a request from the client device to edit the media content, the request including a second text string; and The media content is edited based on the second text string from the request.

5. The system according to claim 1, wherein: Accessing the surface model involves: generating the surface model; assigning the surface model to a set of geographic location coordinates within a surface model database; receiving a request from the client device, the request including the geographic location coordinates assigned to the surface model within the surface model database; and In response to the request from the client device, the surface model associated with the geographic location coordinates is accessed.

6. The system according to claim 1, wherein: The media content includes media characteristics, and wherein the instructions further include: accessing a user profile associated with the client device, the user profile including user profile data; and The media characteristics of the media content are determined based on the user profile data.

7. The system according to claim 6, wherein: The media characteristic includes a language, the user profile data includes a language preference, and the instructions further include: The media content is configured based on the language preference identified by the user profile data.

8. A method comprising: accessing a surface model of an environment, the surface model comprising a wire mesh form comprising a topographic representation of a plurality of three-dimensional (3D) features of the environment; determining a relative position of the plurality of 3D features with respect to each other; determining a viewing angle of the client device based on a position of the client device and the relative positioning of the plurality of 3D features within the environment, wherein the viewing angle indicates a representation of space relative to the client device; accessing media content associated with the location of the client device; as well as The media content is applied to a position within an image rendered at the client device, the positioning being based on a viewing angle of the client device.

9. The method according to claim 8, wherein The media content includes 3D typography including a text string.

10. The method according to claim 9, wherein: The method further comprises: accessing a user profile associated with the client device, the user profile including user profile data; and The text string is generated based on the user profile data from the user profile associated with the client device.

11. The method according to claim 9, wherein The text string is a first text string, and the method further comprises: receiving a request from the client device to edit the media content, the request including a second text string; and The media content is edited based on the second text string from the request.

12. The method according to claim 8, wherein Accessing the surface model involves: generating the surface model; assigning the surface model to a set of geographic location coordinates within a surface model database; receiving a request from the client device, the request including the geographic location coordinates assigned to the surface model within the surface model database; and In response to the request from the client device, the surface model associated with the geographic location coordinates is accessed.

13. The method according to claim 8, wherein The media content includes media characteristics, and wherein the method further comprises: accessing a user profile associated with the client device, the user profile including user profile data; and The media characteristics of the media content are determined based on the user profile data.

14. The method according to claim 13, wherein The media characteristic includes language, the user profile data includes language preference, and the method further includes: The media content is configured based on the language preference identified by the user profile data.

15. A machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: accessing a surface model of an environment, the surface model comprising a wire mesh form comprising a topographic representation of a plurality of three-dimensional (3D) features of the environment; determining a relative position of the plurality of 3D features with respect to each other; determining a viewing angle of the client device based on a position of the client device and the relative positioning of the plurality of 3D features within the environment, wherein the viewing angle indicates a representation of space relative to the client device; accessing media content associated with the location of the client device; as well as The media content is applied to a position within an image rendered at the client device, the positioning being based on a viewing angle of the client device.

16. The machine-readable storage medium of claim 15, wherein: The media content includes 3D typography including a text string.

17. The machine-readable storage medium of claim 16, wherein: The operations further include: accessing a user profile associated with the client device, the user profile including user profile data; and The text string is generated based on the user profile data from the user profile associated with the client device.

18. The machine-readable storage medium of claim 16, wherein: The text string is a first text string, and the operations further include: receiving a request from the client device to edit the media content, the request including a second text string; and The media content is edited based on the second text string from the request.

19. The machine-readable storage medium of claim 15, wherein: The accessing the surface model comprises: generating the surface model; assigning the surface model to a set of geographic location coordinates within a surface model database; receiving a request from the client device, the request including the geographic location coordinates assigned to the surface model within the surface model database; and In response to the request from the client device, the surface model associated with the geographic location coordinates is accessed.

20. The machine-readable storage medium of claim 15, wherein: The media content includes media characteristics, and wherein the operations further comprise: accessing a user profile associated with the client device, the user profile including user profile data; and The media characteristics of the media content are determined based on the user profile data.

Images