Virtual vision system

Abstract

Context-based augmented reality systems can be used to display augmented reality elements on live video feeds on client devices. Augmented reality elements can be selected based on multiple contextual inputs generated by the client device. Contextual inputs can include the client device's location data and location data of nearby physical locations with pre-configured augmented elements. The pre-configured augmented elements can be pre-configured to display a design scheme corresponding to the physical location.

Description

[0001] This application is a divisional application of patent application No. 201880016074.0, filed on March 6, 2018, entitled "Virtual Vision System".

[0002] Related Applications

[0003] This application is in priority to U.S. Patent Application Serial No. 15 / 654,429 entitled “Virtual Vision System”, filed July 19, 2017, which claims priority to U.S. Provisional Patent Application Serial No. 62 / 467,693 entitled “Context-Based Augmented Reality System”, filed March 6, 2017, both of which are incorporated herein by reference in their entirety. Technical Field

[0004] Embodiments of this disclosure generally relate to generating interactive content, and more specifically, but not limited to, context-based computer vision systems. Background Technology

[0005] Digital devices (such as smartphones, tablets, and laptops) can be used as navigation devices to display the location of physical places (such as restaurants and shops). However, such displays often lack interactivity and customization, leading to a poor user experience. Attached Figure Description

[0006] To facilitate identification of any discussion of a particular element or action, the highest valid number or number in the reference numerals refers to the figure number (“Figure”) that first introduced the element or action.

[0007] Figure 1 This is a block diagram illustrating an example messaging system for exchanging data (e.g., messages and associated content) over a network;

[0008] Figure 2 This is a block diagram illustrating further details of a messaging system with an integrated virtual object machine learning system according to an example embodiment;

[0009] Figure 3 This is a schematic diagram illustrating data that can be stored in a database of a messaging server system according to certain example embodiments;

[0010] Figure 4 This is a schematic diagram illustrating the structure of a message generated by a messaging client application according to some embodiments;

[0011] Figure 5This is a schematic diagram illustrating an example access restriction process according to an example embodiment, in which access to content (e.g., short-lived messages and associated multimedia payloads of data) or a collection of content (e.g., short-lived message stories) may be time-limited (e.g., made short-lived).

[0012] Figure 6 This illustrates some example embodiments in Figure 1 A block diagram of the example components provided within the system;

[0013] Figure 7 A flowchart of a method for implementing a virtual vision system according to some example embodiments is shown;

[0014] Figure 8 A flowchart of a method for selecting a lens object according to some example embodiments is shown;

[0015] Figure 9 A flowchart of a method for determining geographic data according to some example embodiments is shown;

[0016] Figure 10 A flowchart of a method for determining context data according to some example embodiments is shown;

[0017] Figure 11 A flowchart of a method for determining a lens object according to some example embodiments is shown;

[0018] Figure 12A-12B An example user interface for implementing orientation toward a relevant lens object, according to some example embodiments, is shown;

[0019] Figure 13 An example user interface for implementing an outdoor-related lens object is shown according to some example embodiments;

[0020] Figure 14 An example user interface for implementing variations of a physically location-dependent lens object, according to some example embodiments, is shown;

[0021] Figure 15 An example user interface for implementing variations of a physically location-dependent lens object, according to some example embodiments, is shown;

[0022] Figure 16 An example user interface for implementing a variant of the depicted project-related shot object, according to some example embodiments, is shown;

[0023] Figure 17 This is a block diagram illustrating a representative software architecture, which can be used in conjunction with the various hardware architectures described in this document; and

[0024] Figure 18 This is a block diagram illustrating components of a machine, according to some example embodiments, capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more methods discussed herein. Detailed Implementation

[0025] The following description includes systems, methods, techniques, instruction sequences, and computer program products embodying illustrative embodiments of the present disclosure. In the following description, numerous specific details are set forth for purposes of explanation in order to provide an understanding of various embodiments of the subject matter of the invention. However, it will be apparent to those skilled in the art that embodiments of the subject matter of the invention can be practiced without specific details. Generally, it is not necessary to show in detail well-known examples of instructions, protocols, structures, and techniques.

[0026] Digital devices (e.g., smartphones, tablets, laptops) can be used as navigation devices to display the location of physical places (e.g., restaurants, stadiums); however, such displays often lack interactivity and customization, resulting in a poor user experience. To improve the user experience, context-based augmented reality (AR) systems can be used to display customized display elements on a live feed (e.g., a video stream) based on the context of a given user's situation and geographic location. While live video feeds are discussed below, it should be understood that shot data can be selected in a similar manner to cover a single image.

[0027] According to some example embodiments, a user can use an application on his / her phone client device to watch live video captured by the phone's rear camera. The video is dynamically captured and displayed on the phone screen. The application then determines the user's context by analyzing the current time, the user's geographic location, identified items displayed in the live feed, audio data and / or other data of the user's environment (e.g., Wi-Fi access, Bluetooth® beacon data) to select the shot to be displayed on the live feed. A shot is a video filter that overlays the displayed elements on the live feed. The displayed elements can be customized to the user's general area (e.g., a city, a shopping mall with many shops), the user's sub-regional location (e.g., a shop within a mall), or a nearby location. A nearby location is a physical location where a shot has been created and associated, as discussed in further detail below. In this way, a user can use his / her phone to view the world via a live feed and load location-specific or physically-location-specific shots based on the user's context, with little or no user input.

[0028] The term "lens object" (e.g., a lens virtual object) refers to a data packet or virtual object (e.g., an instantiation of a class) that includes or references data specific to a given lens object. When data is stored with a lens object, the data is included and referenced by providing an address where the data can be retrieved (e.g., over a network, locally on a client device). The included or referenced data may include pre-configured elements (e.g., text, avatars, cartoons, arrows) and associations with specific context parameters (e.g., time, identified item, approximate area, its sub-areas). Associations with specific context parameters can be used to select a given lens object. For example, a lens object may be specified as being in a given city and can be selected when client device 102 is in that city (e.g., GPS data indicates that client device 102 is in a given city). See the user interface below for further examples of pre-configured display elements.

[0029] Figure 1 This is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and related content) over a network. The messaging system 100 includes multiple client devices 102, each of which hosts multiple applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a message server system 108 via a network 106 (e.g., the Internet).

[0030] Therefore, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and the messaging server system 108 via network 106. The data exchanged between messaging client applications 104 and between messaging client applications 104 and messaging server system 108 includes functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0031] The messaging server system 108 provides server-side functionality to a specific messaging client application 104 via network 106. While some functions of the messaging system 100 are described herein as being performed by either the messaging client application 104 or the messaging server system 108, it should be understood that the location of certain functions within the messaging client application 104 or the messaging server system 108 is a design choice. For example, it may be technically preferred to initially deploy certain technologies and functions within the messaging server system 108, but later migrate these technologies and functions to the messaging client application 104 when the client device 102 has sufficient processing power.

[0032] The messaging server system 108 supports various services and operations provided to the messaging client application 104. These operations include sending data to the messaging client application 104, receiving data from the client application 104, and processing data generated by the client application 104. As an example, this data may include message content, client device information, geolocation information, media comments and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system 100 is invoked and controlled through functions available in the user interface (UI) of the messaging client application 104.

[0033] Now, specifically to message server system 108, application programming interface (API) server 110 is coupled to application server 112 and provides it with a programming interface. Application server 112 is communicatively coupled to database server 118, which provides easy access to database 120, in which data associated with messages processed by application server 112 is stored.

[0034] API server 110 receives and sends message data (e.g., commands and message payloads) between client device 102 and application server 112. Specifically, API server 110 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by messaging client application 104 to invoke functions of application server 112. API server 110 exposes various functions supported by application server 112, including: account registration, login functionality, sending from one messaging client application 104 to another messaging client application 104 via application server 112, sending media files (e.g., images or videos) from messaging client application 104 to messaging server application 114 for possible access by another messaging client application 104, setting media data collections (e.g., stories), retrieving such collections, retrieving the friend list of the user of client device 102, retrieving messages and content, adding and deleting friends in the social graph, the location of friends in the social graph, and opening application events (e.g., related to messaging client application 104).

[0035] Application server 112 hosts multiple applications and subsystems, including messaging server application 114, image processing system 116, and social networking system 122. Messaging server application 114 implements various messaging techniques and functions, specifically involving the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., referred to as stories or galleries). Messaging server application 114 then makes these collections available to messaging client application 104. Given the hardware requirements of other processor- and memory-intensive data processing, this processing can also be performed on the server side by messaging server application 114.

[0036] Application server 112 also includes an image processing system 116, which is dedicated to performing various image processing operations, typically relating to images or videos received within the payload of messages at message server application 114.

[0037] Application server 112 also includes an image processing system 116, which is dedicated to performing various image processing operations, typically relating to images or videos received within the payload of messages at message delivery server application 114.

[0038] Social networking system 122 supports various social networking functions and services, and makes these functions and services available to messaging server application 114. To this end, social networking system 122 maintains and accesses an entity graph within database 120 (e.g., Figure 11 (Entity Figure 304 in the diagram). Examples of functions and services supported by the social networking system 122 include identifiers of a specific user or other users of the messaging system 100 who have a relationship with or are “followed” by the specific user, as well as identifiers of other entities and interests of the specific user.

[0039] Application server 112 is communicatively coupled to database server 118, which provides easy access to database 120, where data associated with messages processed by message server application 114 is stored.

[0040] Figure 2 This is a block diagram illustrating further details of a messaging system 100 according to an example embodiment. Specifically, the messaging system 100 is shown as including a messaging client application 104 and an application server 112, which in turn embodies multiple subsystems, namely a short-lived timer system 202, a collection management system 204, an annotation system 206, and a virtual vision system 210.

[0041] The transient timer system 202 is responsible for implementing temporary access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the transient timer system 202 includes multiple timers that selectively display and enable access to messages and associated content via the messaging client application 104 based on duration and display parameters associated with a message or set of messages. Further details regarding the operation of the transient timer system 202 are provided below.

[0042] The collection management system 204 is responsible for managing media collections (e.g., collections of text, images, video, and audio data). In some examples, collections of content (e.g., messages that include images, videos, text, and audio) may be organized into “event libraries” or “event stories.” Such collections may be available for a specified period of time (such as the duration of the event the content relates to). For example, content related to a concert may be available as a “story” for the duration of the concert. The collection management system 204 may also be responsible for publishing icons that provide notifications of the existence of a specific collection to the user interface of the messaging client application 104.

[0043] The collection management system 204 also includes a curation interface 208, which allows the collection manager to manage and curate specific collections of content. For example, the curation interface 208 enables event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, users may be paid compensation to include user-generated content in the collection. In this case, the curation interface 208 operates to automatically pay these users for using their content.

[0044] Annotation system 206 provides various functionalities that enable users to annotate or otherwise modify or edit media content associated with messages. For example, annotation system 206 provides functionality related to the generation and publication of media overlays for messages processed by messaging system 100. Annotation system 206 operatively provides media overlays (e.g., filters) to messaging client application 104 based on the geolocation of client device 102. In another example, annotation system 206 operatively provides media overlays to messaging client application 104 based on other information, such as the social network information of the user of client device 102. Media overlays may include audio and visual content as well as visual effects. Examples of audio and visual content include images, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos) at client device 102. For example, media overlays may include text that can be overlaid on top of photos generated by client device 102. In another example, media overlays may include identifiers for location overlays (e.g., Venice Beach), names of live events, or names of business overlays (e.g., beach cafes). In another example, annotation system 206 uses the geolocation of client device 102 to identify media overlays that include the name of a business at the geolocation of client device 102. The media overlays may include other tags associated with the business. The media overlays may be stored in database 120 and accessed through database server 118.

[0045] In one example embodiment, annotation system 206 provides a user-based publishing platform that allows users to select geographic locations on a map and upload content associated with those locations. Users can also specify an environment in which specific content should be offered to other users. Annotation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0046] In another example embodiment, annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media coverage associated with geographic location through a bidding process. For example, annotation system 206 associates the media coverage of the highest bidder with a predefined amount of time corresponding to the geographic location.

[0047] According to some example embodiments, the virtual vision system 210 manages the tracking of objects in different images. References below... Figure 6-16 Further details of the virtual vision system 210 are discussed. Although the virtual vision system 210... Figure 2As shown in the example embodiment, which is integrated into the messaging client application 104, it should be understood that in some example embodiments, the virtual vision system 210 is integrated into other systems, such as application server 112. Furthermore, in some example embodiments, some engines of the virtual vision system 210 may be integrated into application server 112 (e.g., to provide server-side support for client requests), and some engines may be integrated into client device 102 (e.g., to generate client requests).

[0048] Figure 3 This is a schematic diagram illustrating data 300 that can be stored in a database 120 of a messaging server system 108 according to some example embodiments. Although the contents of the database 120 are shown as including multiple tables, it should be understood that the data can be stored in other types of data structures (e.g., as an object-oriented database).

[0049] Database 120 includes message data stored in message table 314. Entity table 302 stores entity data, including entity diagram 304. Entities whose records are maintained in entity table 302 may include individuals, company entities, organizations, objects, locations, etc. Regardless of type, any entity with respect to the data stored in message server system 108 can be an identifiable entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0050] Entity graph 304 also stores information about the relationships and associations between entities. For example, such relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based.

[0051] Database 120 also stores annotation data in annotation table 312 in the form of filters. The filters for which data is stored in annotation table 312 are associated with and applied to videos (whose data is stored in video table 310) and / or images (whose data is stored in image table 308). In one example, a filter is an overlay displayed as an overlay on an image or video during display to the recipient user. Filters can be of various types, including user-selected filters from a filter library presented to the sending user by messaging client application 104 while the sending user is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the sending user based on geographic location. For example, based on geographic location information determined by the Global Positioning System (GPS) unit of client device 102, messaging client application 104 can present neighborhood-specific or location-specific geolocation filters within the user interface. Another type of filter is a data filter, which can be selectively presented to the sending user by messaging client application 104 based on other input or information collected by client device 102 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed at which the sending user is traveling, the battery life of client device 102, or the current time.

[0052] Other annotation data that can be stored in image table 308 is so-called "shot" data. A "shot" can be a real-time special effect and sound that can be added to an image or video.

[0053] As described above, video table 310 stores video data, which in one embodiment is associated with messages for which records are maintained in message table 314. Similarly, image table 308 stores image data associated with messages for which message data is stored in entity table 302. Entity table 302 can associate various annotations from annotation table 312 with various images and videos stored in image table 308 and video table 310.

[0054] Story table 306 stores data about collections of messages and associated image, video, or audio data, compiled into collections. The creation of a specific collection can be initiated by a specific user (e.g., each user for whom records are maintained in entity table 302). A user can create a "personal story" in the form of a collection of content that has already been created and sent / broadcast by that user. For this purpose, the user interface of messaging client application 104 may include user-selectable icons that allow the sending user to add specific content to his or her personal story.

[0055] Collections can also constitute "live stories," which are collections of content from multiple users, created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated flow of user-submitted content from various locations and events. For instance, the option to contribute content to a specific live story can be presented to a user via the user interface of messaging client application 104, whose client device has location services enabled and is at a public location event at a specific time. Live stories can be identified to the user by messaging client application 104 based on their location. The end result is a "live story" told from a community perspective.

[0056] Another type of content collection is called a "location story," which allows users whose client devices 102 are located in a specific geographic location (e.g., within a college or university campus) to contribute to a specific collection. In some embodiments, contributions to a location story may require a second level of authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0057] Figure 4 This is a schematic diagram illustrating the structure of a message 400 according to some embodiments. The message 400 is generated by a messaging client application 104 for communication with another messaging client application 104 or a messaging server application 114. Specifically, the content of the message 400 is used to populate a message table 314 stored in a database 120, which is accessible by the messaging server application 114. Similarly, the content of the message 400 is stored in memory as "in transit" or "in flight" data of the client device 102 or application server 112. The message 400 is shown to include the following components:

[0058] • Message Identifier 402: A unique identifier that identifies message 400.

[0059] • Message text payload 404: Text generated by the user via the user interface of client device 102 and included in message 400.

[0060] • Message image payload 406: Image data captured by the camera component of the client device 102 or obtained from the memory of the client device 102, and included in message 400.

[0061] • Message video payload 408: Video data captured by the camera component or obtained from the memory component of the client device 102 and included in message 400.

[0062] • Message audio payload 410: Audio data, which is captured by the microphone or obtained from the memory component of the client device 102 and included in message 400.

[0063] • Message annotation 412: Annotation data (e.g., filters, stickers, or other enhancements) that represents annotations to be applied to message image payload 406, message video payload 408, or message audio payload 410 of message 400.

[0064] • Message Duration Parameter 414: A parameter value, in seconds, indicating the amount of time that the message content (e.g., message image payload 406, message video payload 408, message audio payload 410) will be presented to or accessible to the user via the messaging client application 104.

[0065] • Message geolocation parameter 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of message 400. Multiple message geolocation parameter 416 values ​​may be included in the payload, each of which is associated with a content item included in the content (e.g., a specific image within message image payload 406 or a specific video within message video payload 408).

[0066] • Message Story Identifier 418: An identifier value that identifies one or more sets of content (e.g., "story") associated with a specific content item in the message image payload 406 of message 400. For example, multiple images within the message image payload 406 may each be associated with multiple sets of content using an identifier value.

[0067] • Message Tag 420: One or more tags, each of which indicates the subject of the content included in the message payload. For example, in the case where a specific image depicts an animal (e.g., a lion) is included in the message image payload 406, a tag value may be included within message tag 420 indicating the relevant animal. Tag values ​​may be manually generated based on user input, or may be automatically generated using, for example, image recognition.

[0068] • Message sender identifier 422: An identifier (e.g., a messaging system identifier, email address, or device identifier) ​​that indicates a user of such a client device 102 who generates and sends message 400 on and from the client device 102.

[0069] • Message recipient identifier 424: An identifier (e.g., a messaging system identifier, email address, or device identifier) ​​that indicates the user of the client device 102 to which message 400 is addressed.

[0070] The content (e.g., values) of various components of message 400 can be pointers to locations in tables where the content data values ​​are stored. For example, image values ​​in message image payload 406 can be pointers to locations (or their addresses) within image table 308. Similarly, values ​​in message video payload 408 can point to data stored in video table 310, values ​​in message annotation 412 can point to data stored in annotation table 312, values ​​in message story identifier 418 can point to data stored in story table 306, and values ​​in message sender identifier 422 and message receiver identifier 424 can point to user records stored in entity table 302.

[0071] Figure 5 This is a schematic diagram illustrating an access restriction process 500 in which access to content (e.g., a short message 502 and associated multimedia payloads of data) or a collection of content (e.g., a short message story 504) may be time-restricted (e.g., made short-lived).

[0072] A short message 502 is shown in association with a message duration parameter 506, the value of which determines the amount of time the short message client application 104 will display the short message 502 to the receiving user. In one embodiment, when the messaging client application 104 is an application client, the short message 502 may be viewed by the receiving user for up to 10 seconds, depending on the amount of time specified by the sending user using the message duration parameter 506.

[0073] Message duration parameter 506 and message receiver identifier 424 are shown as inputs to message timer 512, which is responsible for determining the amount of time for which brief message 502 is displayed to a specific receiving user identified by message receiver identifier 424. Specifically, brief message 502 will only be displayed to the relevant receiving user for a period of time determined by the value of message duration parameter 506. Message timer 512 is shown as providing output to a more general brief timer system 202, which is responsible for timing the overall content displayed to the receiving user (e.g., brief message 502).

[0074] Brief message 502 Figure 5This is shown as being included within a short message story 504 (e.g., a personal SNAPCHAT story or an event story). The short message story 504 has an associated story duration parameter 508, the value of which determines the duration for which the short message story 504 is presented to and accessible to users of the messaging system 100. For example, the story duration parameter 508 could be the duration of a concert, where the short message story 504 is a collection of content related to that concert. Alternatively, when setting up and creating the short message story 504, the user (owner user or curator user) can specify the value of the story duration parameter 508.

[0075] Additionally, each short message 502 within a short message story 504 has an associated story participation parameter 510, the value of which determines the duration for which the short message 502 will be accessible within the context of the short message story 504. Therefore, a particular short message story 504 can "expire" and become inaccessible within its context before the short message story 504 itself expires in terms of the story duration parameter 508. The story duration parameter 508, the story participation parameter 510, and the message receiver identifier 424 each provide input to a story timer 514, which first operationally determines whether a particular short message 502 of the short message story 504 will be displayed to a specific receiving user, and if so, for how long. It should be noted that, as a result of the message receiver identifier 424, the short message story 504 also recognizes the identity of the specific receiving user.

[0076] Therefore, the story timer 514 operatively controls the overall lifetime of the associated transient message story 504, as well as the individual transient messages 502 included within the transient message story 504. In one embodiment, each transient message 502 within the transient message story 504 remains visible and accessible for a period of time specified by the story duration parameter 508. In another embodiment, a particular transient message 502 may expire within the context of the transient message story 504 based on the story participation parameter 510. It should be noted that even within the context of the transient message story 504, the message duration parameter 506 can still determine the duration for which a particular transient message 502 is displayed to the receiving user. Therefore, the message duration parameter 506 determines the duration for which a particular transient message 502 is displayed to the receiving user, regardless of whether the receiving user is viewing a transient message 502 inside or outside the context of the transient message story 504.

[0077] The ephemeral timer system 202 can also operationally remove a specific ephemeral message 502 from the ephemeral message story 504 based on the determination that a specific ephemeral message 502 has exceeded its associated story engagement parameter 510. For example, when the sending user has established a story engagement parameter 510 for 24 hours from the date of publication, the ephemeral timer system 202 will remove the relevant ephemeral message 502 from the ephemeral message story 504 after the specified 24 hours. The ephemeral timer system 202 is also operational to remove the ephemeral message story 504 when the story engagement parameter 510 of each ephemeral message 502 within the ephemeral message story 504 has expired or when the ephemeral message story 504 itself has expired in terms of the story duration parameter 508.

[0078] In certain use cases, the creator of a specific ephemeral message story 504 may specify an indefinite story duration parameter 508. In this case, the expiration of the story participation parameter 510 for the last remaining ephemeral message 502 within the ephemeral message story 504 will determine when the ephemeral message story 504 itself expires. In this case, adding a new ephemeral message 502 with a new story participation parameter 510 to the ephemeral message story 504 effectively extends the lifespan of the ephemeral message story 504 to a value equal to the story participation parameter 510.

[0079] In response to the ephemeral timer system 202 determining that the ephemeral message story 504 has expired (e.g., is no longer accessible), the ephemeral timer system 202 communicates with the messaging system 100 (and, for example, the messaging client application 104) to cause the markers (e.g., icons) associated with the relevant ephemeral message story 504 to no longer be displayed in the user interface of the messaging client application 104. Similarly, when the ephemeral timer system 202 determines that the message duration parameter 506 of a particular ephemeral message 502 has expired, the ephemeral timer system 202 causes the messaging client application 104 to no longer display the markers (e.g., icons or text identifiers) associated with the ephemeral message 502.

[0080] Figure 6 A block diagram illustrating components provided within a virtual vision system 210 according to some embodiments is shown. The components are communicatively coupled (e.g., via suitable interfaces) to each other and various data sources to allow information to be transferred between applications, or to allow applications to share and access public data. Furthermore, the components access database 126 via database server 124. Figure 6As shown, the virtual vision system 210 includes an interface engine 605, a lens context engine 610, a lens display engine 615, and a sensor engine 620. In short, according to some example embodiments, the interface engine 605 manages interactions with a server to retrieve lens object data for new locations. Furthermore, according to some example embodiments, the interface engine 605 is configured to interact with a server to receive lens objects selected by the lens context engine 610, which executes on the server.

[0081] The lens context engine 610 receives input parameters and selects a lens object based on those parameters. The parameters include location data and contextual data, such as the time of day or an item identified in the image, as described in further detail below. According to some example embodiments, the lens display engine 615 is configured to manage the execution of the selected lens object by displaying display elements on a live video feed being displayed on the client device 102.

[0082] Sensor engine 620 is configured to manage access to one or more sensors of client device 102. For example, according to some example embodiments, sensor engine 620 may access controls to activate a rear camera of client device 102, access GPS data via a GPS sensor integrated into client device 102, access compass data via accelerometer and gyroscope sensors of client device 102, create audio data via microphone sensor of client device 102, and access user interface controls for visual data of objects displayed on the screen of client device 102.

[0083] According to some embodiments, one or more engines in the virtual vision system 210 may be integrated into the server-side context-based augmented reality system 150, rather than into the virtual vision system 210 on the client side. For example, if the camera context engine 610 runs from the server side, then the client device 102 can generate context data (in... Figure 10 (Discussed in the middle), the context data is sent to the camera context engine 610 on the server side, which then... Figure 11 As discussed further, the lens object is selected based on the received context data.

[0084] Figure 7A flowchart of a method 700 for implementing a virtual vision system 210 according to some example embodiments is shown. In operation 705, the lens context engine 610 identifies lens objects using at least location data provided from the client device 102. For example, the lens context engine 610 requests current location data (e.g., latitude / longitude) by invoking a GPS service integrated into the operating system of the client device 102. The lens context engine 610 then determines whether any lens is associated with the retrieved current location data. In some embodiments, the lens data is stored locally on the client device 102. In those embodiments, the lens context engine 610 then matches the current location with the corresponding locally stored lens object. In some embodiments, the lens context engine 610 contacts the server-side virtual vision system 210 to identify a lens. For example, if the user is in a new geographic area (e.g., a new city), the client device 102 may not have any locally stored lens objects. In this case, the interface engine 605 can submit the current location to the server 140 and, in response, receive a lens object from the server-side virtual vision system 210.

[0085] In operation 710, the lens display engine 615 activates the lens object. In some embodiments, object activation includes identifying geographic coordinates stored in the object for a nearby physical location or destination. In operation 715, the sensor engine 620 generates orientation data for the lens object. According to some example embodiments, the orientation data includes the direction the client device 102 is facing, along with latitude / longitude data in degrees from north. The geographic coordinates of the physical location are compared with the orientation data of the client device 102 to determine how to guide the user to that physical location. For example, if the physical location is 0.3 miles away at 30 degrees north, the orientation data can be used to select one or more pre-configured display elements of the lens object in the following operation.

[0086] In operation 720, the lens display engine 615 uses orientation data to display elements of the selected lens object. Continuing the example above, the lens display engine 615 selects a left arrow and displays it on the live feed of the client device 102. In response to the display of the left arrow, the user can rotate their client device 102 so that its orientation matches the orientation of the physical location (e.g., from 30 degrees north). When the physical location is displayed in the live feed, additional display elements can be overlaid on the live feed. In some example embodiments, the lens display engine 615 determines when to depict the physical location in the live feed by determining when the orientation of the client device 102 matches the orientation of the physical location. Alternatively, the lens display engine 615 can use computer vision algorithms to actively scan the live feed for image feature data that matches the physical location image feature data. The image feature data of the physical location is included in the lens object data, and when a lens object is selected in operation 705, the image feature data of the physical location is loaded into the computer vision algorithm of the lens display engine 615.

[0087] Figure 12A-12B An example user interface 1200 for implementing method 700 is shown, according to some example embodiments. For example... Figure 12A As shown, client device 102 is smartphone 1205, which is running an application (e.g., messaging client application 104) that is displaying a live feed 1210 of video captured by the rear camera (not shown) of smartphone 1205. As described above, the application determines the current location of smartphone 1205, identifies a nearby physical location 1215 for which a lens object with display elements has been pre-configured. The application compares the orientation of physical location 1215 with the orientation of smartphone 1205 and selects a pre-configured display element 1220 that minimizes the difference between the two orientations. The pre-configured display element 1220 is pre-configured because it presents a design scheme (e.g., logo, color scheme, descriptive text) associated with physical location 1215.

[0088] In response to the user rotating his / her smartphone 1205 in the direction of display element 1220, the final physical location 1215 will appear in the live feed 1210. While the physical location 1215 is in the live feed 1210, an additional pre-configured display element 1225, such as a cartoon hamburger trail, can show the user directions to the physical location 1215. The user interface 1210 also includes a capture image button displayed as a white circle at the bottom center of the user interface 1210. Selecting the capture image button generates an image that can be published to a social media platform managed by application server 112 using a share button (not shown) for the captured image. The image can depict the physical location 1215, with pre-configured display elements (e.g., display element 1220) overlaid in front of a real-world background (e.g., a hamburger restaurant, a beach). In this way, the user can share the image with his / her social media friends to indicate that the user is heading to the physical location 1215, thus allowing for an interactive and shareable navigation method using the pre-configured display elements for the physical location 1215. Other user interfaces (e.g., Figure 13-11 Those depicted in the image can also be imaged and shared using the same methods.

[0089] In some example embodiments, even if physical location 1215 is covered or otherwise obscured by other objects, pre-configured display elements 1220 are still overlaid on the live feed 1210. For example, if a building is located between the user of client device 102 and physical location 1215, display elements 1220 and 1225 can still be displayed to indicate to the user that physical location 1215 is behind a building when the user holds client device 102 upward so that live feed 1210 displays a view of the building.

[0090] Furthermore, as shown in the figure, when the orientation of the smartphone 1205 matches the orientation of the physical location 1215, the pre-configured display element 1220 can be updated to indicate the distance to the physical location 1215. In this way, by seamlessly selecting and displaying the pre-configured display elements 1220 and 1225, the user experience becomes more interactive and customizable, requiring almost no user operation.

[0091] Figure 8A flowchart of a method 800 for selecting a lens object according to some example embodiments is shown. Method 800 includes a more detailed method for selecting a lens object. In operation 805, sensor engine 620 determines the orientation of client device 102. The orientation of client device 102 may include the direction in which client device 102 faces relative to north (e.g., 270 degrees off north). Furthermore, in some example embodiments, the orientation of client device 102 may include geographic location data, such as the current latitude and longitude of client device 102. In operation 810, lens context engine 610 generates context parameters detailing the context of client device 102. The context may include sensor inputs and time inputs from client device 102. In operation 815, lens context engine 610 uses location data and context parameters to identify lens objects. Each lens object may be associated with one or more context parameters of client device 102 and a specified set of context parameters for a given current location. After operation 815, the method may return to... Figure 7 Operation 720, as described above, involves displaying the display elements of the identified lens object.

[0092] Figure 9 A flowchart of a method 900 for determining geographic data according to some example embodiments is shown. Method 900 is an example embodiment of operation 805. For example, method 900 can be called as a subroutine to complete operation 805. The start bar refers to the method call. The end or return bar outputs or otherwise stores data generated from the preceding operation in memory. Method 900 can be referenced using a wrapper function, which can be stored on the client device 102 or within the sensor engine 620. Calling the wrapper function performs multiple operations, such as those shown in method 900. In operation 905, sensor engine 620 determines the orientation data of client device 102, including the geographic location of client device 102 and the orientation of client device 102. In some example embodiments, a GPS sensor of client device 102 is used to generate longitude and latitude data of client device 102. In addition, a compass sensor of client device 102 is used to determine the direction that client device 102 is facing relative to north.

[0093] In operation 910, sensor engine 620 determines the approximate location or current area of ​​client device 102. For example, given the latitude and longitude of the current location, client device 102 can determine that it is located in the approximate area of ​​Venice, California. In some example embodiments, the latitude and longitude are sent to a server, and the server determines the approximate area of ​​client device 102 and returns this information as a response to client device 102. Furthermore, according to some example embodiments, the approximate area is not necessarily a city, but rather an approximate area that includes more than one physical location 1215. For example, the approximate area could refer to a shopping mall with multiple stores.

[0094] In operation 915, sensor engine 620 determines the sub-region location of client device 102. A sub-region is an area within a general area. For example, if the general area is a city (e.g., San Francisco), the sub-region could be a neighboring area within the city (e.g., Mount Portrero). As another example, if the area is a shopping mall, the sub-region could be a specific store within the shopping mall. In some example embodiments, the same mechanism used to determine the general area is used to determine the sub-region (e.g., looking up data locally on client device 102, or sending location data to a server and receiving the sub-region as a response). Furthermore, according to some example embodiments, sensor engine 620 can use location data from an enhanced location service (e.g., Foursquare™) to determine the sub-region to pinpoint the sub-region to a given store within a shopping mall. In short, Foursquare™ is a service that can pinpoint the location of a device by analyzing information available to client device 102, such as current location, connected Wi-Fi networks, nearby / available cell towers, nearby / available Wi-Fi networks, and other information. Enhanced location services can be provided from third-party servers. The sensor engine 620 sends a request to the server, which then retrieves sub-region data from a third-party server via API server 110. The sub-region data is then sent back to the sensor engine 620 for further processing.

[0095] Figure 10A flowchart of a method 1000 for determining context data according to some example embodiments is shown. Method 1000 is an example embodiment of operation 810. For example, method 1000 can be called as a subroutine to complete operation 810. The start bar refers to the method call. The end or return bar outputs or otherwise stores data generated from the previous operation in memory. Method 1000 can be referenced using a wrapper function, which may be stored on the client device 102 or within the sensor engine 620. Calling the wrapper function performs multiple operations, such as those shown in method 1000. In operation 1005, the sensor engine 620 determines whether the client device 102 is indoors or outdoors based on computer vision analysis of a live feed 1210 displayed on the screen of the client device 102. For example, the sensor engine 620 can determine the ambient light level by analyzing the pixel values ​​of frames in the live feed 1210. If the pixel level indicates a high level of light, the sensor engine 620 determines that the client device 102 is outdoors. However, if the pixel level indicates a medium to low level of light, sensor engine 620 determines that client device 102 is indoors. In operation 1010, sensor engine 620 uses audio data generated from the microphone of client device 102 to determine the environment (e.g., restaurant, beach, restroom). For example, sensor engine 620 can use computer vision algorithms to identify items being displayed in live feed 1210. The identified items (e.g., pool table, bar stool, neon lights) indicate the type of environment (e.g., dive bar) and can later be used to select camera objects based on the identified items. Furthermore, in operation 1010, sensor engine 620 can access the microphone of client device 102 to analyze audio data from the surrounding environment. For example, if sensor engine 620 determines that the clinking sound is the sound of plates colliding and there is a high ambient sound level, sensor engine 620 can determine that the user is in a public place, such as a restaurant. In operation 1015, sensor engine 620 can further use item recognition to provide context without knowing the surrounding environment, as discussed in further detail below. In operation 1020, sensor engine 620 can request the local time of client device 102. The local time can be used to provide additional contextual information for selecting time-based shot objects (e.g., morning-oriented shot objects, happy hour shot objects, late-night shot objects).

[0096] Figure 11A flowchart of method 1100 for determining a lens object according to some example embodiments is shown. Method 1100 is an example embodiment of operation 815. For example, method 1100 can be called as a subroutine to complete operation 815. The start bar refers to the method call. The end or return bar outputs or otherwise stores data generated from the previous operation in memory. Method 1100 can be referenced using a wrapper function, which can be stored on the client device 102 or within the sensor engine 620. Calling the wrapper function performs multiple operations, such as those shown in method 1100. In operation 1105, the lens context engine 610 determines whether the client device 102 is indoors or outdoors. For example, the lens context engine 610 can analyze image or pixel data from the live feed 1210 and determine that the client device 102 is outdoors if the pixel is blue or the lighting is bright. If the camera context engine 610 determines that the client device 102 is outdoors, then in operation 1110, the camera context engine 610 selects an outdoor camera object for the approximate area of ​​the current location of the client device 102.

[0097] In some example embodiments, if the camera context engine 610 determines that the client device 102 is outdoors, but there are no camera objects for the general area, then in operation 1115, the camera context engine 610 performs (via server 150) a search for camera objects associated with locations near the current location of the client device 102. Once a camera object for the nearby location is identified, it can be displayed using pre-configured display elements 1220, 1225, as referenced above. Figure 12A and 12B The subject of discussion.

[0098] Continuing with decision operation 1105, if the shot context engine 610 determines that client device 102 is inside, then the shot context engine 610 proceeds to decision operation 1120, where the shot context engine 610 analyzes whether client device 102 is located within a sub-region of the approximate region as described above (e.g., programmatically accessing enhanced location services via an API). Assuming client device 102 is within a sub-region of the approximate region, the method continues to operation 1125, where the shot context engine 610 determines the sub-region of client device 102. In operation 1130, the shot context engine 610 determines the item depicted in the live feed 1210 or image. In operation 1135, the shot context engine 610 determines the local time of client device 102. In operation 1140, the shot context engine 610 selects a shot object using the sub-region, item, and / or time.

[0099] Continuing with decision operation 1120, assuming client device 102 is not in the sub-region, in operation 1145, the shot context engine 610 determines the item depicted in the live feed 1210 or image. In operation 1150, the shot context engine 610 determines the local time of client device 102. In operation 1155, the shot context engine 610 selects a shot object using the depicted item and / or time.

[0100] Figure 13 An example user interface 1300 for implementing an outdoor-related lens object is shown according to some example embodiments. In operation 1110, select... Figure 13 The camera object, wherein the camera context engine 610 determines that the client device 120 is outside the general area of ​​the Venice beach. The pre-configured display elements for the selected camera object include city-related display elements 1305 describing “Venice” and an avatar 1310 for the user of the client device 102, which has been pre-configured to display the avatar 1310 of a shopping spree on the main passageway of the general area (e.g., the Venice pontoon bridge).

[0101] Figure 14 An example user interface 1400 for implementing a variant of a lens object related to physical location 1215, according to some example embodiments, is shown. Selecting in operation 1140... Figure 14 The camera context engine 610 determines that the client device 102 is in the sub-region (Coffee Shop) and also indicates the time, indicating that it is early morning. Pre-configured display elements 1220, 1225 include a sub-region descriptor 1405, which describes the name of the sub-region (“Mennoti’s Coffee Shop”) and displays a coffee shop avatar. Furthermore, the pre-configured display elements include an avatar 1410 pre-configured to display as a sleepy avatar, and also include time-related phrases such as “Good Morning”.

[0102] Figure 15 An example user interface 1500 for implementing changes to a lens object related to physical location 1215, according to some example embodiments, is shown. In operation 1140, selection... Figure 15The camera context engine 610 determines that the client device 102 is in a sub-region (“Townhouse”) and further determines that the time indicates it is happy hour (e.g., 5 p.m. to 8 p.m.). Pre-configured display elements include a sub-region descriptor 1505, which describes the name of the sub-region (“Townhouse” in a stylized font). Additionally, pre-configured display elements include display element 1510, which is pre-configured to display a well-dressed avatar holding a tropical cocktail, and also includes a time-based phrase such as “Happy Hour”.

[0103] It should be understood that the above are merely example phrases, avatars, fonts, etc., and are only examples of pre-configured elements selected using context, and other variations can be implemented. For example, instead of the time-related "Happy Hour," the phrase might be related to other features of the sub-area. For instance, if the townhouse is known for its high-quality margaritas, then the phrase displayed in element 1510 could describe "Worlds Best Margarita." Furthermore, according to some example embodiments, if the townhouse is currently promoting an item, the phrase could describe "Margaritas only $1 during happy hour!" Other variations are also possible.

[0104] Figure 16 The illustration shows a variation of the depicted object-related lens object for example user interface 1600, according to some example embodiments. Figure 11 Operation 1155 selection Figure 16 The camera object, the camera context engine 610, uses the depicted object 1605 and time to select the camera object. Figure 16 In the live feed 1210 displayed on the screen of client device 102, a pancake is depicted. In operation 1015, sensor engine 620 can use a locally executed computer vision algorithm to identify the depicted item 1605 as an egg and a pancake, or optionally send one or more images of the item to server 112, where server 112 can then run a more rigorous computer vision analysis algorithm to identify the depicted item 1605 as an egg and a pancake. After identifying the item as an egg and a pancake, the time of day, retrieved from operation 1150, can be used to confirm that it is morning. Figure 16The pre-configured display element 1610 includes: a user's avatar, appearing to be waking up and stretching, and the phrase "Morning". Other avatars and phrases can also be implemented. In this way, the pre-configured display element 1610 can be used to select objects when no general area or sub-area data is available.

[0105] Figure 17 This is a block diagram illustrating example software architecture 1706, which can be used in conjunction with various hardware architectures described herein. Figure 17 This is a non-limiting example of software architecture, and it should be understood that many other architectures can be implemented to enhance the functionality described herein. Software Architecture 1706 can be applied to, for example... Figure 18 It executes on hardware such as machine 1800, which includes a processor, memory, and I / O components. A representative hardware layer 1752 is shown, and it can represent, for example... Figure 18 The machine 1800. A representative hardware layer 1752 includes a processing unit 1754 with associated executable instructions 1704. The executable instructions 1704 represent executable instructions of the software architecture 1706, including implementations of the methods, components, etc., described herein. Hardware layer 1752 also includes memory / memory 1756, which also has the executable instructions 1704. Hardware layer 1752 may also include other hardware 1758.

[0106] exist Figure 17 In the example architecture. For example... Figure 17 As shown, software architecture 1706 can be conceptualized as a stack of layers, where each layer provides specific functionality. For example, software architecture 1706 may include layers such as operating system 1702, libraries 1720, framework / middleware 1718, applications 1716, and an image layer 1714. Operationally, application 1716 and / or other components within a layer can invoke application programming interface (API) calls 1708 and receive responses in the form of message 1712. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 1718, while others may provide such a layer. Other software architectures may include additional or different layers.

[0107] Operating system 1702 can manage hardware resources and provide public services. Operating system 1702 may include, for example, kernel 1722, services 1724, and drivers 1726. Kernel 1722 can act as an abstraction layer between hardware and other software layers. For example, kernel 1722 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 1724 can provide other public services to other software layers. Drivers 1726 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1726 may include display drivers, camera drivers, Bluetooth drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi drivers, audio drivers, power management drivers, etc., depending on the hardware configuration.

[0108] Library 1720 provides a common infrastructure used by application 1716 and / or other components and / or layers. Library 1720 provides functionality that allows other software components to perform tasks more easily than by directly interfaced with the underlying operating system 1702 functions (e.g., kernel 1722, services 1724, and / or drivers 1726). Library 1720 may include system libraries 1744 (e.g., the C standard library), which provide functions such as memory allocation, string manipulation, mathematical functions, etc. Furthermore, Library 1720 may include API libraries 1746, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, or PNG), graphics libraries (e.g., the OpenGL framework for rendering 2D and 3D graphics content on a display), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functionality), and so on. Library 1720 may also include various other libraries 1748 to provide many other APIs to application 1716 and other software components / modules.

[0109] Framework / Middleware 1718 provides a higher level of common infrastructure that can be used by application 1716 and / or other software components / modules. For example, framework / middleware 1718 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 1718 can provide a wide range of other APIs that can be used by application 1716 and / or other software components / modules, some of which may be specific to a particular operating system 1702 or platform.

[0110] Application 1716 includes built-in applications 1738 and / or third-party applications 1740. Examples of representative built-in applications 1738 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1740 may include applications developed by entities other than the vendor of a specific platform using the Android™ or iOS™ Software Development Kit (SDK), and may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® phones, or other mobile operating systems. Third-party applications 1740 may invoke API calls 1708 provided by the mobile operating system (such as operating system 1702) to facilitate the functions described herein.

[0111] Application 1716 can use built-in operating system functions (e.g., kernel 1722, services 1724, and / or drivers 1726), libraries 1720, and frameworks / middleware 1718 to create user interfaces to interact with the system's users. Alternatively or additionally, in some systems, interaction with the user can occur through an image layer (e.g., image layer 1714). In these systems, the application / component "logic" can be decoupled from the aspects of the application / component that interact with the user.

[0112] Figure 18 This is a block diagram illustrating components of a machine 1800 according to some example embodiments, the machine 1800 being capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any or more methods discussed herein. Specifically, Figure 18An illustration shows a machine 1800 in an example form of a computer system, wherein instructions 1816 (e.g., software, programs, applications, applets, or other executable code) for causing machine 1800 to execute can perform any or more of the methods discussed herein. Thus, instructions 1816 can be used to implement the modules or components described herein. Instructions 1816 transform a generic, unprogrammed machine 1800 into a specific machine 1800 programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, machine 1800 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1800 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1800 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones 1205, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart devices), other smart devices, network devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1816, which specify the actions to be taken by machine 1800. Furthermore, although only a single machine 1800 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 1816 to perform any one or more methods discussed herein.

[0113] Machine 1800 may include processor 1810, memory / memory 1830, and I / O components 1850, which may be configured to be accessed, for example, via bus 1802. Memory / memory 1830 may include memory 1832, such as main memory, or other storage devices, and memory cell 1836, both of which may be accessed by processor 1810, for example, via bus 1802. Memory cell 1836 and memory 1832 store instructions 1816 embodying any one or more methods or functions described herein. During execution of machine 1800, instructions 1816 may also reside wholly or partially within memory 1832, memory cell 1836, at least one of processor 1810 (e.g., within the processor's cache memory), or any suitable combination thereof. Thus, the memories of memory 1832, memory cell 1836, and processor 1810 are examples of machine-readable media.

[0114] I / O component 1850 may include a wide variety of components to receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O component 1850 included in a particular machine 1800 will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 1850 may include... Figure 18 Many other components are not shown. The functional division of I / O component 1850 is merely for the purpose of simplifying the following discussion and is by no means limiting. In various example embodiments, I / O component 1850 may include output component 1852 and input component 1854. Output component 1852 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistive mechanisms), other signal generators, and so on. Input component 1854 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical photographic keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads), trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing position and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), and so on.

[0115] In another example embodiment, I / O component 1850 may include a wide range of other components, such as biometric component 1856, motion component 1858, environmental component 1860, or position component 1862. For example, biometric component 1856 may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1858 may include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental component 1860 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects the concentration of hazardous gases to ensure safety or to measure pollutants in the atmosphere), or other components that provide indications, measurements, or signals corresponding to the surrounding physical environment. Position component 1862 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0116] Various technologies can be used to implement communication. I / O component 1850 may include communication component 1864, operable to couple machine 1800 to network 1880 or device 1870 via coupling 1882 and coupling 1872, respectively. For example, communication component 1864 may include network interface component or other suitable device interface with network 1880. In further examples, communication component 1864 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, Bluetooth® component (e.g., Bluetooth Low Energy®), Wi-Fi® component, and other means of providing communication. Device 1870 may be another machine or any of various peripheral devices (e.g., peripheral devices coupled via Universal Serial Bus (USB)).

[0117] Furthermore, the communication component 1864 may detect identifiers or include components operable to detect identifiers. For example, the communication component 1864 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF418, Ultra Code, UCCRSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals of the tag). Additionally, various information can be derived via the communication component 1864, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location by detecting NFC beacon signals that can indicate a specific location, etc.

[0118] Glossary:

[0119] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions 1816 for execution by machine 1800, and includes digital or analog communication signals or other intangible media to facilitate the communication of these instructions 1816. Instructions 1816 can be sent or received on network 1880 using a transmission medium via a network interface device and using any of many well-known transmission protocols.

[0120] In this context, "client device" refers to any machine 1800 that interfaces with communication network 1880 to obtain resources from one or more server systems or other client devices 102. Client device 102 may be, but is not limited to, mobile phones, desktop computers, laptops, PDAs, smartphones 1205, tablets, ultrabooks, netbooks, multiprocessor systems, microprocessor-based or programmable consumer electronics systems, game consoles, set-top boxes, or any other communication device that a user can use to access network 1880.

[0121] In this context, "communication network" refers to one or more parts of network 1880, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network or part of network 1880 may include a wireless or cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular network or wireless coupling. In this example, coupling can be implemented using any of various types of data transmission technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, GSM evolution enhanced data rate (EDGE) technology, the 3rd generation partnership program (3GPP) including 3G, fourth-generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high-speed packet access (HSPA), global microwave access interoperability (WiMAX), long-term evolution (LTE) standards, and other standards defined by various standards setting organizations, other remote protocols, or other data transmission technologies.

[0122] In this context, a "brief message" refers to a message 400 that is accessible for a limited time. A brief message 502 can be text, an image, video, etc. The access time for a brief message 502 can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, this message 400 is temporary.

[0123] In this context, "machine-readable medium" means a component, device, or other tangible medium capable of temporarily or permanently storing instructions 1816 and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., erasable programmable read-only memory (EPROM)), and / or any suitable components thereof. The term "machine-readable medium" should be considered to include a single medium or multiple media capable of storing instructions 1816 (e.g., a centralized or distributed database, or associated caches and servers). The term "machine-readable medium" should also be considered to include any medium or combination of media capable of storing instructions (e.g., code) for execution by machine 1800, such that instructions 1816, when executed by machine 1800, cause the machine to perform any or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or apparatuses. The term "machine-readable medium" does not include the signal itself.

[0124] In this context, a “component” refers to a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies, that provides specific processing or control functions, or a partition or modularity thereof. Components can be combined with other components through their interfaces to perform machine processes. A component can be an encapsulated functional hardware unit designed for use with other components, or part of a program that typically performs a specific function related to that function. Components can constitute software components (e.g., code embodied on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various example embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., processor 1812 or a set of processors 1810) can be configured by software (e.g., an application program or an application portion) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of machine 1800), uniquely tailored to perform the configured functions, and is no longer the general-purpose processor 1810. It should be understood that cost and time considerations can drive the decision to implement hardware components in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., software-configured). Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed), that operate or perform certain operations described herein in a certain way.

[0125] Consider embodiments where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any given time. For example, in the case where the hardware components include a general-purpose processor 1812 configured by software as a dedicated processor, the general-purpose processor 1812 can be configured at different times as different dedicated processors (e.g., including different hardware components). The software thus configures a particular processor 1812 or processor 1810, for example, constituting a particular hardware component at one time and a different hardware component at different times.

[0126] Hardware components can provide and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled. In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more of the hardware components (e.g., via appropriate circuitry and buses). In embodiments where multiple hardware components are configured or instantiated at different times, communication between these hardware components can be achieved, for example, by storing and retrieving information in a memory structure accessible to the multiple hardware components. For example, one hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then later access the memory device to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., information sets).

[0127] Various operations of the example methods described herein can be performed, at least in part, by one or more processors 1810 configured, either temporarily (e.g., by software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors 1810 can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, “processor-implemented component” refers to a hardware component implemented using one or more processors 1810. Similarly, the methods described herein can be implemented, at least in part, by processors, with a particular processor 1812 or processor 1810 being an example of such hardware. For example, at least some operations of the methods can be performed by one or more processors 1810 or processor-implemented components. Furthermore, one or more processors 1810 can also operate to support the performance of the relevant operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some operations can be performed by a group of computers (e.g., a machine 1800 including processor 1810), which can be accessed via a network 1880 (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The performance of certain operations can be distributed among processors 1810, residing not only within a single machine 1800 but also deployed across multiple machines 1800. In some example embodiments, processor 1810 or processor-implemented components may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other example embodiments, processor 1810 or processor-implemented components may be distributed across multiple geographic locations.

[0128] In this context, "processor" refers to any circuit or virtual circuit (physical circuitry simulated by logic executed on the actual processor 1812) that operates data values ​​according to control signals (e.g., "commands", "opcodes", "machine codes", etc.) and produces corresponding output signals applied to the operating machine 1800. For example, a processor can be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), or any combination thereof. Processor 1810 can also be a multi-core processor having two or more independent processors 1812, 1814 (sometimes referred to as "cores") capable of simultaneously executing instructions 1816.

[0129] In this context, a "timestamp" refers to a sequence of characters or encoded information that identifies when a particular event occurred, such as a date and time, sometimes accurate to a fraction of a second.

[0130] This patent document contains a portion of copyrighted material. The copyright holder does not object to any fax copy of the patent document or patent disclosure as it appears in the patent documents or records of the Patent and Trademark Office, but otherwise reserves all copyright.

[0131] The following numbered examples are implementation examples:

[0132] 1. A system comprising: One or more processors of the machine; and A memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations, including: On the client device, a camera virtual object is determined at least in part based on the current geographic location of the client device, the camera virtual object being designated as one or more display elements pre-configured for physical locations near the current geographic location of the client device; Orientation data is determined using at least the current geographic location of the client device and the destination geographic location of a physical location specified in an identified lens virtual object, the current geographic location being generated at least in part by invoking one or more location services of the client device, the orientation data specifying the direction of the destination geographic location relative to the current orientation of the client device; and This causes a directional display element from one or more pre-configured display elements to be presented on one or more images displayed on the client device, the directional display element indicating the direction of the destination geographic location relative to the current orientation of the client device.

[0133] 2. The system as described in Example 1, further comprising: Identify updates to the orientation data that indicate the physical location is within the field of view of one or more images, and that these updates are triggered in response to physical movement of the client device; and In response to the physical location being within the field of view of the one or more images, one or more physical location display elements from the pre-configured display elements are displayed, the one or more physical location display elements indicating the location of the physical location in the one or more images.

[0134] 3. The system as described in Example 1, the operation further includes: receiving pre-configured display elements for the destination geographic location of the physical location from an attached client device.

[0135] 4. The system as described in Example 1, wherein the pre-configured display elements are pre-configured to display a business design associated with the physical location.

[0136] 5. The system as described in Example 4, wherein the design scheme includes one or more trademark design schemes of the business, including at least one of the following: color scheme, one or more logos, and one or more words.

[0137] 6. The system as described in Example 2, wherein the physical location is obscured by one or more physical objects in the field of view, and wherein the physical location display element overlays the one or more physical objects.

[0138] 7. The system as described in Example 1, wherein each lens virtual object is configured to invoke one or more location services of the client device using a wrapper function that specifies multiple location services of the operating system executed on the client device.

[0139] 8. The system as described in Example 1, wherein the current geographic location includes latitude and longitude data generated by the GPS sensor of the client device, and wherein the destination geographic location includes latitude and longitude data of the physical location.

[0140] 9. The system as described in Example 1, wherein the direction display element uses a pointer display element to indicate direction.

[0141] 10. The system as described in Example 1, wherein the operation of determining the orientation data uses operations including the following: The absolute location of the physical location is identified using one or more of the location services; and When the client device physically moves, the orientation data is dynamically updated using the orientation sensor on the client device.

[0142] 11. The system as described in Example 2, wherein the operation of indicating the physical location in one or more image feed operations uses the operation of determining that the current orientation of the client device is consistent with the orientation of the physical location.

[0143] 12. The system as described in Example 2, wherein the operation of instructing the physical location in the one or more image feeds uses operations including: Determine the difference between the current orientation of the client device and the orientation of the physical location.

[0144] 13. The system according to Example 1, wherein the one or more images are video feeds, the video feeds are displayed on the display device when frames of the video feeds are generated, and wherein physical location display elements are dynamically displayed on frames of the video feeds.

[0145] 14. A method comprising: At least one processor is used on the client device to determine a camera virtual object based at least in part on the current geographic location of the client device, the camera virtual object being designated as one or more display elements pre-configured for physical locations near the current geographic location of the client device; Orientation data is determined using at least the current geographic location of the client device and the destination geographic location of a physical location specified in an identified lens virtual object, the current geographic location being generated at least in part by invoking one or more location services of the client device, the orientation data specifying the direction of the destination geographic location relative to the current orientation of the client device; and This causes a directional display element from one or more pre-configured display elements to be presented on one or more images displayed on the client device, the directional display element indicating the direction of the destination geographic location relative to the current orientation of the client device.

[0146] 15. The method described in Example 14, further comprising: Identify updates to the orientation data that indicate the physical location is within the field of view of one or more images, and that these updates are triggered in response to physical movement of the client device; and In response to the physical location being within the field of view of the one or more images, one or more physical location display elements from the pre-configured display elements are displayed, the one or more physical location display elements indicating the location of the physical location in the one or more image feeds.

[0147] 16. The method as described in Example 14, further comprising: Receive pre-configured display elements for the destination geolocation of the physical location from the attached client device.

[0148] 17. The method as described in Example 14, wherein the pre-configured display element is pre-configured to display a commercial design associated with the physical location.

[0149] 18. The method as described in Example 17, wherein the design scheme includes one or more trademark design schemes of the business, including at least one of the following: color scheme, one or more logos, and one or more words.

[0150] 19. A machine-readable storage device containing instructions that, when executed by a machine, cause the machine to perform the operations of the methods of any of the examples 14-18.

Claims

1. A system comprising: One or more processors of the machine; and A memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations, including: On the client device, a lens virtual object is selected based at least in part on the client device’s current geographic location and time parameters, wherein the lens virtual object refers to a data packet that includes or references data specific to a given lens object, wherein the lens is a video filter that overlays display elements in a live feed from a camera displayed on the client device, and the lens virtual object is specified as one or more display elements pre-configured for a physical location near the client device’s current geographic location; Orientation data is determined using at least the current geographic location of the client device and the destination geographic location of the physical location specified in the identified lens virtual object, the current geographic location being generated at least in part by invoking one or more location services of the client device, and the orientation data specifying the direction of the destination geographic location relative to the current orientation of the client device; This causes one or more pre-configured display elements from the lens virtual object to be presented on one or more images displayed on the client device, the direction display elements indicating the direction of the destination geographic location relative to the current orientation of the client device; Determining that the physical location is within the live feed; and Based on the physical location being in the live feed, one or more additional pre-configured display elements of the lens virtual object are presented on the one or more images.

2. The system of claim 1, further comprising: Identify updates to the orientation data, the updates indicating that the physical location is within the field of view of one or more images, and the updates to the orientation data are caused in response to physical movement of the client device; and In response to the physical location being within the field of view of the one or more images, one or more physical location display elements from the pre-configured display elements are displayed, the one or more physical location display elements indicating the location of the physical location in the one or more images.

3. The system of claim 1, wherein the operation further includes: Receive pre-configured display elements for the destination geolocation of the physical location from the attached client device.

4. The system of claim 1, wherein the pre-configured display element is pre-configured to display a commercial design associated with a physical location.

5. The system of claim 4, wherein the design scheme includes one or more trademark design schemes of the business, including at least one of the following: color scheme, one or more logos, and one or more words.

6. The system as claimed in claim 2, wherein, The physical location is obscured by one or more physical objects in the field of view, and the physical location display element is overlaid on the one or more physical objects.

7. The system as claimed in claim 1, wherein, Each camera virtual object is configured to invoke one or more location services of the client device using a wrapper function, the wrapper function specifying multiple location services of the operating system executed on the client device.

8. The system of claim 1, wherein, The current geographic location includes latitude and longitude data generated by the GPS sensor of the client device, and the destination geographic location includes the latitude and longitude data of the physical location.

9. The system of claim 1, wherein the direction display element uses a pointer display element to indicate direction.

10. The system of claim 1, wherein the operation of determining the orientation data includes the following operations: The absolute location of the physical location is identified using one or more of the location services; and When the client device physically moves, the orientation data is dynamically updated using the orientation sensor on the client device.

11. The system of claim 2, wherein instructing the physical location in the one or more image feeds uses operations including: Determine that the current orientation of the client device is consistent with the orientation of the physical location.

12. The system of claim 2, wherein instructing the physical location in the one or more image feeds uses operations including: Determine the difference between the current orientation of the client device and the orientation of the physical location.

13. The system of claim 1, wherein the one or more images are video feeds, the video feeds are displayed on the display device when frames of the video feeds are generated, and wherein the physical location display elements are dynamically displayed on frames of the video feeds.

14. A method comprising: Using at least one processor on a client device, a lens virtual object is selected based at least in part on the current geographic location and time parameters of the client device, wherein the lens virtual object refers to a data packet that includes or references data specific to a given lens object, wherein the lens is a video filter that overlays display elements in a live feed from a camera displayed on the client device, and the lens virtual object is specified as one or more display elements pre-configured for a physical location near the current geographic location of the client device; Orientation data is determined using at least the current geographic location of the client device and the destination geographic location of the physical location specified in the identified lens virtual object, wherein the current geographic location is generated at least in part by invoking one or more location services of the client device, and the orientation data specifies the direction of the destination geographic location relative to the current orientation of the client device; This causes one or more pre-configured display elements from the lens virtual object to be presented on one or more images displayed on the client device, the direction display elements indicating the direction of the destination geographic location relative to the current orientation of the client device; The physical location is determined to be within the live feed; as well as Based on the physical location being in the live feed, one or more additional pre-configured display elements of the lens virtual object are presented on the one or more images.

15. The method of claim 14, further comprising: Identify updates to the orientation data, the updates indicating that the physical location is within the field of view of one or more images, the updates to the orientation data being caused in response to physical movement of the client device; as well as In response to the physical location being within the field of view of the one or more images, one or more physical location display elements from the pre-configured display elements are displayed, the one or more physical location display elements indicating the location of the physical location in the one or more image feeds.

16. The method of claim 14, further comprising: Receive pre-configured display elements for the destination geolocation of the physical location from the attached client device.

17. The method of claim 14, wherein, The pre-configured display elements are pre-configured to display commercial designs associated with the physical location.

18. The method of claim 17, wherein, The design scheme includes one or more trademark design schemes for the business, including at least one of the following: color scheme, one or more logos, and one or more words.

19. A machine-readable storage device comprising instructions that, when executed by a machine, cause the machine to perform the operations of the methods of any of the examples 14-18.

Images