Context-based selection of augmented reality experiences

Abstract

The experience of a user participating in an AR technology that allows the user to interact with its environment and with each other is enhanced by automatically selecting an augmented reality (AR) experience suitable for use given the user's physical environment. The physical environment of the user is a physical environment of a computing device of the user. The physical environment may include objects and / or conditions, such as others, animals, and smart devices, present in proximity to a user's computing device.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. application Serial No. 17 / 661,231, filed on April 28, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to facilitating interaction between devices hosting messaging applications. Background Art

[0004] As the popularity of computer-implemented tools that allow users to access content online and interact with content and other users continues to grow, various computer-implemented tools are being developed to allow users to interact with other users and share content through messaging applications. For example, a messaging system can host a backend service for an associated messaging client that is configured to allow users to interact asynchronously via messages and also synchronously via audio and video interactions. A messaging system can also allow users to participate in shared experiences, such as multiplayer games. Users in a messaging system are represented by corresponding profiles that store information about the associated users. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the accompanying drawings, like reference numerals may describe similar components in different views, and the drawings are not necessarily drawn to scale. To easily identify the discussion of any particular element or action, one or more of the highest digits in the reference numerals refer to the figure number in which the element is first introduced. Some examples are shown by way of example and not limitation in the figures of the accompanying drawings, in which:

[0006] Figure 1 is a diagrammatic representation of a selected networked environment in which a context-based augmented reality experience may be deployed, according to some examples.

[0007] Figure 2 is a diagrammatic representation of a messaging system having both client-side and server-side functionality according to some examples.

[0008] Figure 3 is a diagrammatic representation of the data structures maintained in a database according to some examples.

[0009] Figure 4 is a graphical representation of messages according to some examples.

[0010] Figure 5 is a flow chart of an access restriction process according to some examples.

[0011] Figure 6is a flow diagram of a method for providing selection of a context-based augmented reality experience according to some examples.

[0012] Figure 7 is a diagram illustrating a map user interface (UI) displayed by a messaging client executing on a user device according to some examples.

[0013] Figure 8 is a diagram illustrating a participant UI for obtaining information about the physical environment of a user device according to some examples.

[0014] Fig. 9 is a diagram illustrating an IoT device UI for obtaining information about the physical environment of a user device according to some examples.

[0015] Fig.10 is a diagram illustrating a selection UI that includes a reference to an identified suitable AR experience, according to some examples.

[0016] Fig.11 is a diagram showing a user profile UI that displays references to content generated during a user's participation in an real-life (IRL) AR experience, according to some examples.

[0017] Fig.12 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure improve the functionality of electronic messaging software and systems by enhancing the experience of users who wish to interact with their environment and / or with each other using augmented reality (AR) techniques through their respective user devices.

[0019] AR is an environment in which the real world, such as that viewed by a digital camera, is enhanced by superimposing virtual content on a view of the real world presented on a display of a user device. Computing applications that use AR technology and can be accessed through a messaging client are referred to as AR experiences. AR experiences can be configured to utilize certain aspects of the physical environment of a user device that executes a messaging client. For example, an AR experience may require at least two human participants or at least two user devices that execute a messaging client to be physically near each other. In another example, an AR experience configured to mimic a ghost requires a smart light. In yet another example, a dog or cat may be a necessary aspect of the physical environment of a user device that executes a messaging client in order to participate in a specific AR experience. A user may not know every AR experience available to him through a messaging client, and for a user, such a situation can be frustrating: exploring interesting AR experiences that can be accessed through a messaging client, only to learn that the AR experience cannot be used because the physical environment lacks one or more aspects required for the AR experience. On the other hand, a user may not know an AR experience that is very suitable for the user's current physical environment.

[0020] An example of an AR experience requiring one user device and multiple participants is an AR experience that tracks the heads of participants in the output of a digital image sensor of a camera and displays instructions to pass a user device from one participant to another. An example of an AR experience that requires multiple user participants and multiple user devices is an AR experience that provides a virtual object that participants can each move around on their respective screens in an attempt to "hit" the virtual object with the other participants as they are seen in the output of the digital image sensor of the camera. An example of an AR experience that incorporates the movement of an animal is an AR experience that tracks a representation of an animal (e.g., a dog or cat) in a display of the output of a digital image sensor of a camera of a user device and generates UI elements and / or instructions for the user based on the movement or sounds of the animal detected by the digital image sensor and / or microphone.

[0021] By automatically selecting AR experiences available given the user's physical environment, the user's experience of engaging with AR technologies that allow the user to interact with their environment and each other is enhanced. The user's physical environment, i.e., the physical environment of a user device controlled by the user, may include objects and / or conditions present near the user device, such as other people, animals, smart devices, and conditions such as a cloudy sky. For the purposes of this specification, the physical environment of a user device executing a messaging client is referred to as a physical context. In some examples, the automatic selection of available AR experiences given the user's physical environment is performed by an AR experience selection system provided by the messaging system.

[0022] Automatic selection of an AR experience available given the user's physical environment can be triggered by a predetermined event detected by the AR experience selection system. For the purposes of this specification, a predetermined event that triggers the selection of an AR experience available given the physical environment is referred to as a triggering event or entry point. An example triggering event detected by the AR experience selection system is the co-location of a user device and another user device executing a messaging client. Figure 2 Describes a mapping system to detect co-location of two devices. Figure 7 An example of a map UI is shown in , which includes an indication of co-location and an invitation to participate in an AR experience that exploits the physical presence of another participant. Another entry point for participating in the AR experience selection system is associated with a camera scan, where a trigger event is the detection of a specific type of physical object, such as a person or a cat, in the output of a digital sensor of a camera of a user device. In some examples, a trigger event can be detected when a user is taking a selfie using a camera UI of a messaging client. The camera UI is configured to display the output of a digital image sensor of a camera with an associated user device, and to display user-selectable elements that are executable to capture an image through the camera or to start and stop video recording.

[0023] In some examples, the AR experience selection system is configured to generate, for example, Figure 8 and Fig. 9 FIG. 1 is a UI for obtaining information about the physical environment (physical context) of a user device to identify a suitable AR experience based on the physical context and generate an AR experience as shown in FIG. Fig.10 , including a reference to an identified suitable AR experience. The reference to the identified suitable AR experience can be executed to launch the AR experience at the user device.

[0024] The AR experience selection system can be configured to allow content generated during participation in AR experiences that utilize aspects of the physical environment of the user device executing the messaging client to be saved and shared. Such content can be tagged with real life (IRL) tags and saved in the user's profile as IRL content. In some examples, such as in an example where the entry point for selecting an AR experience is a physical co-location identified by a mapping system, the IRL content is saved in a shared profile representing an IRL experience shared by two or more users of the messaging system. The messaging system can be configured to allow users to store IRL content and view saved IRL content, and share the IRL content with other users of the messaging system. An AR experience using IRL features can be described as a digital space in which various aspects of a user's interaction with a physical environment are tracked and recorded. Examples of IRL features include tracking representations of people and / or animals displayed in a camera view UI, or interacting with smart devices such as fitness trackers or smart lights.

[0025] Details of a messaging system configured to include a location-based shared AR experience system are described below.

[0026] Networked computing environment

[0027] Figure 1 1 is a block diagram 100 illustrating an example messaging system for exchanging data (e.g., messages and associated content) over a network. The messaging system includes a messaging client 104 and multiple instances of a messaging server system 108 executed at respective client devices, such as a client device 102. Each messaging client 104 is communicatively coupled to other instances of the messaging client 104 and the messaging server system 108 via a network 106 (e.g., the Internet). The client device 102 is a smart phone, AR glasses, or another type of device capable of displaying AR content, having access to a global positioning provider (e.g., GPS), and having a network connection.

[0028] The messaging client 104 is able to communicate and exchange data with another messaging client 104 and a messaging server system 108 via the network 106. The data exchanged between the messaging clients 104 and between the messaging clients 104 and the messaging server system 108 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0029] The messaging server system 108 provides server-side functionality to certain messaging clients 104 via the network 106. Although certain functions of the messaging system are described herein as being performed by the messaging client 104 or by the messaging server system 108, the location of certain functions in the messaging client 104 or within the messaging server system 108 may be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the messaging server system 108, but to later migrate the technologies and functions to the messaging client 104 where the client device 102 has sufficient processing power.

[0030] The messaging server system 108 supports various services and operations provided to the messaging clients 104. Such operations include sending data to the messaging clients 104, receiving data from the messaging clients 104, and processing data generated by the messaging clients 104. As examples, the data may include message content, client device information, geographic location information, media enhancements and overlays, message content persistence conditions, social network information, live event information, and images and videos captured using the front-facing camera of the associated client device using customized image reprocessing. Data exchange within the messaging system is activated and controlled by functions available via the user interface (UI) of the messaging client 104.

[0031] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 that stores data associated with messages processed by the application server 112. Similarly, a web server 124 is coupled to the application server 112 and provides a web-based interface to the application server 112. To this end, the web server 124 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0032] The application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that the messaging client 104 can call or query to activate the functions of the application server 112. The application program interface (API) server 110 exposes various functions supported by the application server 112, including: account registration; login functionality; sending messages from a particular messaging client 104 to another messaging client 104 via the application server 112; sending media files (e.g., images or videos) from a messaging client 104 to the messaging server 114 and for possible access by another messaging client 104; setting of media data collections (e.g., stories); retrieving a friend list of a user of the client device 102; retrieving such collections; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to a messaging client 104).

[0033] The application server 112 hosts several server applications and subsystems, including, for example, a messaging server 114, an image processing server 116, and a social network server 122. The messaging server 114 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 104. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 104. In view of the hardware requirements for other processor and memory intensive processing of data, such processing can also be performed on the server side by the messaging server 114.

[0034] Application server 112 also includes image processing server 116, which is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at messaging server 114. Some of the various image processing operations may be performed by various AR components, collectively referred to as AR engines, which may be hosted or supported by image processing server 116. In some examples, the AR engine is used to facilitate the functionality provided by the AR experience selection system described herein.

[0035] The social network server 122 supports various social networking functions and services and makes them available to the messaging server 114. To this end, the social network server 122 maintains and accesses the entity graph 306 (e.g., Figure 3 ). Examples of functions and services supported by social network server 122 include identifying other users of the messaging system that have a "friend" relationship with a particular user or that the particular user is "following," and also identifying interests and other entities of a particular user.

[0036] In some examples, game server 126 is configured as the authoritative source of user actions and user action effects (also referred to as events) in a multiplayer game. Game server 126 is the authoritative source of events in a multiplayer game in the sense that each user device participating in the same multiplayer game maintains a version of the game state that is consistent with the corresponding versions of the game state available to other users via their respective user devices. In some examples, the multiplayer game is a shared AR experience.

[0037] System Architecture

[0038] Figure 2 2 is a block diagram 200 showing additional details about a messaging system according to some examples. Specifically, the messaging system is shown as including a messaging client 104 and an application server 112. The messaging system contains several subsystems that are supported on the client side by the messaging client 104 and on the server side by the application server 112. These subsystems include, for example, an AR experience selection system 202, an augmentation system 206, a map system 210, and a game system 212.

[0039] The AR experience selection system 202 is configured to automatically select one or more AR experiences available given the nearby physical environment of a user device. The AR experience selection system 202 determines the physical context of a messaging client executed at a user device in response to detecting a triggering event at the user device. Example triggering events include detecting the co-location of a user device and another user device executing a messaging client, and detecting a particular type of physical object, such as a human or a cat, in the output of a digital sensor of a camera of the user device. The physical context includes features of the physical environment of the user device controlled by the user, and may include objects and / or conditions present in the vicinity of the user device, such as other people, animals, and smart devices.

[0040] Enhancement system 206 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content that may be associated with a message. For example, enhancement system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system. Media overlays may be stored in database 120 and accessed through database server 118.

[0041] In some examples, the augmentation system 206 is configured to provide access to an AR component that can be implemented using a programming language suitable for application development, such as JavaScript or Java, and identified in the messaging server system by a corresponding AR component identifier. The AR component may include or reference various image processing operations corresponding to image modification, filtering, media overlay, transformation, etc. These image processing operations can provide an interactive experience of a real-world environment, in which objects, surfaces, backgrounds, lighting, etc. captured by a digital image sensor or camera are enhanced by computer-generated perceptual information. In this case, the AR component includes a collection of data, parameters, and other assets required to apply a selected augmented reality experience to an image or video feed.

[0042] In some embodiments, the AR component includes a module configured to modify or transform image data presented within a graphical user interface (GUI) of a client device in some manner. For example, AR component data can be used to perform complex additions or transformations to content images, such as adding bunny ears to a person's head in a video clip, adding a floating heart with a background color to a video clip, changing the scale of a person's features within a video clip, or many other such transformations. This includes both real-time modifications that modify the image using the AR component modification when the image is captured using a camera associated with the client device and then displayed on the screen of the client device, as well as modifications to stored content, such as video clips in a gallery that can be modified using the AR component.

[0043] Various augmented reality functions that can be provided by the AR component include: detecting objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), tracking these objects as they leave, enter, and move around the field of view in a video frame, and modifying or transforming these objects as they are tracked. In various embodiments, different methods for implementing such transformations can be used. For example, some embodiments may involve generating a 3D mesh model of one or more objects, and using animated textures and transformations of the model within a video to implement the transformation. In other embodiments, tracking of points on an object can be used to place an image or texture at the tracked location, which image or texture can be two-dimensional or three-dimensional. In other embodiments, neural network analysis of video frames can be used to place images, models, or textures in content (e.g., frames of an image or video). Therefore, AR component data refers to both images, models, and textures used to create transformations in content, and additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement. In some embodiments, the augmentation system 206 is used as an AR engine by the AR experience selection system 202 to perform tasks such as generating and tracking one or more virtual objects that can be provided through a shared AR experience.

[0044] The mapping system 210 provides various geolocation functions and supports the presentation of map-based media content and messages through the messaging client 104. For example, the mapping system 210 enables the display of user icons or visualizations (e.g., stored in the profile data 308) on a map to indicate the current or past locations of the user's "friends", as well as the display of media content (e.g., a collection of messages including photos and videos) generated by such friends within the context of the map. For example, a message posted to the messaging system 100 by a user from a particular geographic location may be displayed within the context of the map to the particular user's "friends" on the map interface of the messaging client 104 at that particular location. In addition, a user may share his or her location and status information (e.g., using an appropriate status visualization) with other users of the messaging system 100 via the messaging client 104, where the location and status information is similarly displayed to the selected user within the context of the map interface of the messaging client 104. In some implementations, AR experience selection system 202 uses mapping system 210 to determine co-location of a user device and another user device executing a messaging client, and generates a map user interface that includes an indication of the co-location and an invitation to participate in an AR experience that utilizes the physical presence of another participant. An example of a map UI that includes an indication of the co-location and an invitation to participate in an AR experience that utilizes the physical presence of another participant is shown in FIG. Figure 7 is shown in and discussed below.

[0045] The gaming system 212 provides various gaming functions within the context of the messaging client 104. The messaging client 104 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the messaging client 104 and played with other users of the messaging system 100, including one or more AR experiences.

[0046] Data Architecture

[0047] Figure 3 is a diagram illustrating a data structure 300 that may be stored in a database 120 of a messaging server system 108 according to some examples. Although the contents of the database 120 are illustrated as including several tables, it should be understood that data may be stored in other types of data structures (e.g., as an object-oriented database).

[0048] The database 120 includes message data stored in a message table 302. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Figure 4 Additional details regarding information that may be included in a message and included within the message data stored in message table 302 are described.

[0049] The entity table 304 stores entity data and is linked (e.g., referenced) to the entity graph 306 and profile data 308. The entities for which records are maintained within the entity table 304 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity about which the messaging server system 108 stores data may be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0050] The entity graph 306 stores information about relationships and associations between entities. Such relationships may be social, professional (e.g., working in a common company or organization), interest-based, or activity-based, by way of example only.

[0051] Profile data 308 stores multiple types of profile data about a particular entity. As explained above, users in the messaging system are represented by corresponding profiles that store information about the associated user. Based on the privacy settings specified by the particular entity, profile data 308 can be selectively used and presented to other users of the messaging system. In the case where the entity is a person, profile data 308 includes, for example, a user name, phone number, address, settings (e.g., notification and privacy settings), and a user-selected visualization (or a collection of such visualizations). Profile data 308 may include saved IRL content that a user of the messaging system can access by viewing the profile of another user of the messaging system, which is a way to view details about the user's friend relationship with another user in the messaging system. The saved IRL content is associated with an IRL AR experience that the two users experience together.

[0052] The database 120 also stores enhancement data in the enhancement table 310. The enhancement data is associated with and applied to videos (whose data is stored in the video table 314) and images (whose data is stored in the image table 316). In some examples, the enhancement data is used by various AR components including the AR component. As described above, an example of enhancement data is a target media content object, which can be associated with an AR component and used to generate an AR experience for a user.

[0053] Another example of augmented data is an augmented reality (AR) tool that can be used in an AR component to implement image transformation. Image transformation includes real-time modification of an image (e.g., a video frame) when it is captured using a digital image sensor of the client device 102. The modified image is displayed on the screen of the client device 102 together with the modification. AR tools can also be used to modify stored content, such as modifying video clips or still images stored in a gallery. In a client device 102 that has access to multiple AR tools, a user can apply different AR tools (e.g., by engaging different AR components configured to utilize different AR tools) to a single video clip to see how different AR tools will modify the same video clip. For example, by selecting different AR tools for the same captured content, multiple AR tools that apply different pseudo-random movement models can be applied to the same captured content. Similarly, real-time video capture can be used with the modifications shown to show how the video image currently captured by the digital image sensor of the camera provided by the client device 102 will modify the captured data. Such data may be displayed only on the screen and not stored in memory, or the content captured by the digital image sensor may be recorded and stored in memory with or without modification (or in both cases). The messaging client 104 may be configured to include a preview feature that can simultaneously display what modifications produced by different AR tools will look like in different windows of the display. For example, this can allow a user to simultaneously view multiple windows with different pseudo-random animations presented on the display.

[0054] In some examples, when a specific modification is selected together with the content to be transformed, the element to be transformed is identified by the computing device, and then if the element to be transformed is present in the frame of the video, the element to be transformed is detected and tracked. The elements of the object are modified according to the request for modification, thereby transforming the frame of the video stream. For different types of transformations, the frame transformation of the video stream can be performed by different methods. For example, for frame transformations that mainly involve changing the form of the elements of the object, the characteristic points of each element of the object are calculated (for example, using an active shape model (ASM) or other known methods). Then, a grid based on the characteristic points is generated for each element of at least one element of the object. The grid is used for the subsequent stage of tracking each element of the object in the video stream. In the tracking process, the grid mentioned for each element is aligned with the position of each element. Then, additional points are generated on the grid. A first set of first points is generated for each element based on the modification request, and a set of second points is generated for each element based on the set of first points and the modification request. Then, the frame of the video stream can be transformed by modifying the elements of the object based on the set of first points and the set of second points and the grid. In such a method, the background of the modified object can also be changed or distorted by tracking and modifying the background.

[0055] Stories table 312 stores data about a collection of messages and associated image, video, or audio data that is compiled into a collection (e.g., a story or gallery). The creation of a particular collection may be initiated by a particular user (e.g., each user whose record is maintained in entity table 304). A user may create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by that user. To this end, the user interface of messaging client 104 may include a user-selectable icon to enable the sending user to add specific content to his or her personal story. In some examples, stories table 312 may store content generated by the AR experience selection system, e.g., IRL content.

[0056] A collection may also constitute a "Live Story," which is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "Live Story" may constitute a curated stream of user-submitted content from different locations and events. For example, a user whose client device has location services enabled and who is at a common location event at a particular time may be presented with the option to contribute content to a particular Live Story via the user interface of the messaging client 104. The messaging client 104 may identify a Live Story to the user based on his or her location. The end result is a "Live Story" told from the perspective of the community.

[0057] Another type of content collection is called a "location story," which enables users whose client devices 102 are located within a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributions to location stories may require secondary authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0058] As mentioned above, the video table 314 stores video data, which in some examples is associated with a message whose record is maintained within the message table 302. Similarly, the image table 316 stores image data associated with a message whose message data is stored in the entity table 304. The entity table 304 may associate various enhancements from the enhancement table 310 with various images and videos stored in the image table 316 and the video table 314.

[0059] Data communication architecture

[0060] Figure 4 is a schematic diagram illustrating the structure of a message 400 according to some examples, the message 400 being generated by a messaging client 104 for transmission to another messaging client 104 or a messaging server 114. The content of a particular message 400 is used to populate a message table 302 stored in a database 120, which is accessible by a messaging server 114. Similarly, the content of the message 400 is stored in memory as "in-flight" or "in-flight" data of a client device 102 or an application server 112. In some examples, the content of the message 400 includes an image or video created using an AR component. The message 400 is shown as including the following example components:

[0061] Message identifier 402 : a unique identifier that identifies the message 400 .

[0062] Message text payload 404 : Text to be generated by a user via the user interface of the client device 102 and included in the message 400 .

[0063] Message image payload 406: Image data captured by a camera component of client device 102 or retrieved from a memory component of client device 102 and included in message 400. Image data for a sent or received message 400 may be stored in image table 316. Image data may include content generated by the AR experience selection system.

[0064] Message video payload 408: Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400. Video data for a sent or received message 400 may be stored in the video table 314. The video data may include content generated by the AR experience selection system.

[0065] Message audio payload 410 : audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400 .

[0066] Message enhancement data 412: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing enhancements to be applied to the message image payload 406, message video payload 408, message audio payload 410 of the message 400. The enhancement data for a sent or received message 400 may be stored in the enhancement table 310.

[0067] Message duration parameter 414: A parameter value indicating the amount of time in seconds that the content of a message (eg, message image payload 406, message video payload 408, message audio payload 410) will be presented to or made accessible to a user via messaging client 104.

[0068] Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 416 values ​​may be included in the payload, each of which is associated with a content item included in the content (e.g., a specific image within a message image payload 406, or a specific video within a message video payload 408).

[0069] Message story identifier 418: An identifier value that identifies one or more content collections (e.g., a "story" identified in stories table 312) associated with a particular content item in message image payload 406 of message 400. For example, an identifier value may be used to associate multiple images within message image payload 406 with each of multiple content collections.

[0070] Message tags 420: Each message 400 may be tagged with a plurality of tags, each of which indicates the subject of the content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value indicating the relevant animal may be included within the message tags 420. The tag values ​​may be manually generated based on user input, or may be automatically generated using, for example, image recognition.

[0071] • Message sender identifier 422: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 on which the message 400 was generated and from which the message 400 was sent.

[0072] • Message recipient identifier 424: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 to which the message 400 is addressed.

[0073] The content (e.g., values) of the various components of message 400 may be pointers to locations in a table where content data values ​​are stored. For example, an image value in message image payload 406 may be a pointer to a location (or address of a location) within image table 316. Similarly, a value within message video payload 408 may point to data stored within video table 314, a value stored within message enhancement 412 may point to data stored in enhancement table 310, a value stored within message story identifier 418 may point to data stored in story table 312, and values ​​stored within message sender identifier 422 and message recipient identifier 424 may point to user records stored in entity table 304.

[0074] Time-based access restriction architecture

[0075] Figure 5 5 is a schematic diagram illustrating access restriction processing 500, according to which access to content (e.g., ephemeral message 502 and associated multimedia payload of data) or a collection of content (e.g., ephemeral message group 504) can be time-limited (e.g., ephemeral). In some examples, the content of ephemeral message 502 includes an image or video created by the AR experience selection system.

[0076] The ephemeral message 502 is shown to be associated with a message duration parameter 506, the value of which determines the amount of time that the ephemeral message 502 will be displayed by the messaging client 104 to the receiving user of the ephemeral message 502. In some examples, the receiving user may view the ephemeral message 502 for up to 10 seconds, depending on the amount of time specified by the sending user using the message duration parameter 506.

[0077] The message duration parameter 506 and the message recipient identifier 424 are shown as inputs to a message timer 512, which is responsible for determining the amount of time that the transient message 502 is shown to the particular receiving user identified by the message recipient identifier 424. In particular, the transient message 502 is only shown to the relevant receiving user for the period of time determined by the value of the message duration parameter 506. The message timer 512 is shown as providing an output to a more general transient timer system 202, which is responsible for the overall timing of displaying content (e.g., the transient message 502) to the receiving user.

[0078] exist Figure 5 1. Ephemeral message 502 is shown as being included within an ephemeral message group 504 (e.g., a collection of messages in a personal story or event story). Ephemeral message group 504 has an associated group duration parameter 508, the value of which determines the duration for which ephemeral message group 504 is presented and accessible to users of messaging system 100. For example, group duration parameter 508 may be the duration of a concert, wherein ephemeral message group 504 is a collection of content related to the concert. Alternatively, a user (an owning user or a curator user) may specify the value of group duration parameter 508 when performing setup and creation of ephemeral message group 504.

[0079] In addition, each ephemeral message 502 within the ephemeral message group 504 has an associated group participation parameter 510, the value of which determines the duration for which the ephemeral message 502 will be accessible within the context of the ephemeral message group 504. Thus, a particular ephemeral message group 504 may "expire" and become inaccessible within the context of the ephemeral message group 504 before the ephemeral message group 504 itself expires according to the group duration parameter 508. The group duration parameter 508, the group participation parameter 510, and the message recipient identifier 424 each provide inputs to a group timer 514, which first operatively determines whether a particular ephemeral message 502 of the ephemeral message group 504 will be displayed to a particular receiving user, and if so, how long to display it. Note that the identity of the particular receiving user is also known to the ephemeral message group 504 due to the message recipient identifier 424.

[0080] Thus, the group timer 514 operatively controls the total lifespan of the associated ephemeral message group 504 and the individual ephemeral messages 502 included in the ephemeral message group 504. In some examples, each ephemeral message 502 within the ephemeral message group 504 remains viewable and accessible for a period of time specified by the group duration parameter 508. In other examples, within the context of the ephemeral message group 504, a particular ephemeral message 502 may expire based on the group participation parameter 510. Note that even within the context of the ephemeral message group 504, the message duration parameter 506 may still determine the duration for which a particular ephemeral message 502 is displayed to a receiving user. Thus, the message duration parameter 506 determines the duration for which a particular ephemeral message 502 is displayed to a receiving user, regardless of whether the receiving user is viewing the ephemeral message 502 within or outside the context of the ephemeral message group 504.

[0081] In addition, the transient timer system 202 may be operative to remove a particular transient message 502 from the transient message group 504 based on a determination that the particular transient message 502 has exceeded the associated group participation parameter 510. For example, when the sending user has established a group participation parameter 510 of 24 hours from posting, the transient timer system 202 will remove the associated transient message 502 from the transient message group 504 after the specified 24 hours. The transient timer system 202 is also operative to remove a transient message group 504 when the group participation parameter 510 for each transient message 502 within the transient message group 504 has expired, or when the transient message group 504 itself has expired according to the group duration parameter 508.

[0082] In some use cases, the creator of a particular ephemeral message group 504 may specify an indefinite group duration parameter 508. In this case, the expiration of the group participation parameter 510 of the last remaining ephemeral message 502 within the ephemeral message group 504 will determine when the ephemeral message group 504 itself expires. In this case, a new ephemeral message 502 with a new group participation parameter 510 added to the ephemeral message group 504 effectively extends the life of the ephemeral message group 504 to a value equal to the group participation parameter 510.

[0083] In response to the transient timer system 202 determining that the transient message group 504 has expired (e.g., is no longer accessible), the transient timer system 202 communicates with the messaging system 100 (and, for example, in particular, the messaging client 104) to cause the indicia (e.g., icon) associated with the relevant transient message group 504 to no longer be displayed within the user interface of the messaging client 104. Similarly, when the transient timer system 202 determines that the message duration parameter 506 of a particular transient message 502 has expired, the transient timer system 202 causes the messaging client 104 to no longer display the indicia (e.g., icon or text identification) associated with the transient message 502.

[0084] Processing Flow

[0085] Figure 6 is a flow diagram of a method 600 for selection of a context-based AR experience, according to some examples.

[0086] Although the described flowcharts may show the operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. When the operations of a process are completed, the process is terminated. A process may correspond to a method, procedure, algorithm, etc. The operations of a method may be performed in whole or in part, may be performed in combination with some or all of the operations in other methods, and may be performed by any number of different systems, such as the systems described herein or any portion thereof, such as a processor included in any system. In one example embodiment, some or all of the processing logic resides in Figure 1 The client device 102 and / or Figure 1 The client device 102 may be a smart phone or AR glasses.

[0087] Method 600 begins at operation 610, when Figure 2 The AR experience selection system 202 determines a physical context of a messaging client executed at a user device. The physical context includes features of the physical environment of the user device controlled by the user. For example, the physical context may include objects and / or conditions present near the user device, such as other people, animals, Internet of Things (IoT) devices, and conditions such as a cloudy sky. An IoT device is a device that is capable of exchanging data with other devices over a network and may include sensors and data processing capabilities. Examples of IoT devices (also referred to as smart devices) include fitness trackers, voice-activated speakers, smart lights, and the like.

[0088] At operation 620, the AR experience selection system 202 selects an AR experience suitable for the physical environment represented by the physical context from a plurality of AR experiences provided by the messaging system. For example, the physical context may include a second user device that is executing a messaging client and is within a predetermined distance of the user device. The AR experience suitable for the physical environment represented by the physical context may be a shared AR experience that allows a user to manipulate virtual objects from the user device and from another user device.

[0089] In another example, the physical context may be an indication of a person and / or animal in the camera view UI of the messaging client. An AR experience appropriate for the physical environment represented by the physical context may be an AR experience configured to track the indication of the animal or person in the camera view UI. In the example where the physical context is a smart light, a suitable AR experience is an AR experience configured to communicate with the smart light.

[0090] At operation 630, AR experience selection system 202 generates an experience selection UI including a reference to the AR experience. The reference is executable to launch the AR experience at the user device, and at operation 640 causes the selection UI to be displayed on the display of the user device.

[0091] Method 600 may be triggered in response to determining that two devices are co-located, which may be accomplished using Figure 2 Map system 210 or use Method 600 may also be triggered in response to AR experience selection system 202 receiving a request to participate in an AR experience appropriate for the physical environment of the user device. Method 600 may also be triggered by AR experience selection system 202 detecting that the output of a digital image sensor of a camera of the user device is integrated into a camera scanning function.

[0092] user interface

[0093] Figure 7 700 is a diagram illustrating a map UI displayed by a messaging client executed on a first user device controlled by a user represented in the map UI by reference numeral 710, according to some examples. In addition to user 710, the map UI also illustrates another user represented by reference numeral 720. User 720 controls a second user device that is executing the messaging client and is within a predetermined physical proximity (e.g., six feet or less) of the first user device. In response to determining that the first user device and the second user device are within a predetermined physical proximity of each other, Figure 2 The AR experience selection system 202 causes an invitation 730 to be presented, inviting the user to participate in the IRL AR experience. Corresponding instances of the map UI are displayed on both the first user device and the second user device.

[0094] Figure 8 is a diagram 800 illustrating a participant UI for obtaining information about people and animals that may be present in a nearby physical environment of a user device, according to some examples. Figure 8 Region 810 in is configured to accept input from a user of the number of people in physical proximity to the user. Region 820 is configured to accept input from a user of the number of dogs in physical proximity to the user. Region 830 is configured to accept input from a user of the number of cats in physical proximity to the user.

[0095] Fig. 9 900 is a diagram illustrating an IoT device UI for obtaining information about IoT devices present in a nearby physical environment of a user device according to some examples. Fig. 9 As shown in , area 910 is configured to accept user input of the number of computing devices (e.g., smart phones) executing a messaging client that are physically close to the user, and area 920 is configured to accept user input of the number of smart lights that are activated and physically close to the user's device. An example of an AR experience using IoT devices such as smart lights is an AR experience that generates UI elements and / or instructions for a player based on the behavior of the smart lights, such as changes in the color and / or brightness of the smart lights.

[0096] Fig.10 1000 is a diagram illustrating a selection UI including a reference to an identified suitable AR experience, identified by reference numeral 1010. The AR experience selection system 202 identifies a suitable AR experience based on a physical context indicating that two users are physically close to each other and the availability of an IoT device (smart light).

[0097] Fig.11 is a diagram 1100 illustrating a user profile UI that displays references to content generated during a user's participation in an IRL AR experience, according to some examples. Fig.11 , the user profile UI shows an avatar and name (JONATHAN) of the associated user, and also includes respective references 1110, 1120, and 1130, each executable to access content generated during the user's participation in the respective associated IRL AR experience. Diagram 1100 shows references to content generated during participation in the viewer's IRL AR experience with a particular user (JONATHAN). In some examples, the user identified as JONATHAN is also allowed to view content generated during his participation in the IRL AR experience.

[0098] Machine Architecture

[0099] Fig.121200 within which instructions 1208 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 1200 to perform any one or more of the methods discussed herein. For example, the instructions 1208 may cause the machine 1200 to perform any one or more of the methods described herein. The instructions 1208 convert a general-purpose, non-programmed machine 1200 into a specific machine 1200 that is programmed to perform the functions described and illustrated in the manner described. The machine 1200 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1200 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1200 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 1208 specifying actions to be taken by the machine 1200. In addition, although only a single machine 1200 is shown, the term "machine" should also be considered to include a collection of machines that individually or jointly execute instructions 1208 to perform any one or more of the methods discussed herein. For example, the machine 1200 may include any of the client device 102 or a plurality of server devices that form part of the messaging server system 108. In some examples, the machine 1200 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.

[0100] The machine 1200 may include a processor 1202, a memory 1204, and an input / output I / O component 1238, which may be configured to communicate with each other via a bus 1240. In an example, the processor 1202 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1206 that executes instructions 1208 and a processor 1210. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Fig.12 Multiple processors 1202 are shown, but the machine 1200 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0101] The memory 1204 includes a main memory 1212, a static memory 1214, and a storage unit 1216, which are all accessible by the processor 1202 via the bus 1240. The main memory 1204, the static memory 1214, and the storage unit 1216 store instructions 1208 that embody any one or more of the methodologies or functions described herein. The instructions 1208 may also reside, completely or partially, within the main memory 1212, within the static memory 1214, within the machine-readable medium 1218 within the storage unit 1216, within at least one of the processors 1202 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 1200.

[0102] I / O components 1238 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 1238 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O components 1238 may include Fig.1212 and 13. In various examples, the I / O components 1238 may include a user output component 1224 and a user input component 1226. The user output component 1224 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an auditory component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 1226 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a pointing-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or another pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides location and force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0103] In another example, the I / O component 1238 may include a biometric component 1228, a motion component 1230, an environmental component 1232, or a position component 1234, as well as a variety of other components. For example, the biometric component 1228 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1230 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0104] Environmental components 1232 include, for example, one or more cameras (with still imaging / video and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect concentrations of hazardous gases for safety or measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0105] With respect to cameras, client device 102 can have a camera system including, for example, a front-facing camera on a front surface of client device 102 and a rear-facing camera on a rear surface of client device 102. The front-facing camera can, for example, be used to capture still images and videos (e.g., “selfies”) of a user of client device 102, which can then be enhanced using the enhancement data (e.g., filters) described above. The rear-facing camera can, for example, be used to capture still images and videos in a more traditional camera mode, where the images are similarly enhanced using the enhancement data. In addition to the front-facing camera and the rear-facing camera, client device 102 can also include a 360° camera for capturing 360° photos and videos.

[0106] Additionally, the camera system of the client device 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even three, four, or five rear camera configurations on the front and back sides of the client device 102. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0107] The location component 1234 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0108] Various technologies may be used to implement communications. The I / O components 1238 also include a communication component 1236 that is operable to couple the machine 1200 to the network 1220 or device 1222 via a corresponding coupling or connection. For example, the communication component 1236 may include a network interface component or another suitable device for interfacing with the network 1220. In other examples, the communication component 1236 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low power consumption), Device 1222 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0109] In addition, the communication component 636 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 636 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes and other optical codes) or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various information can be obtained via the communication component 1236, such as obtaining positioning via Internet Protocol (IP) geolocation, ... Positioning is obtained through signal triangulation, positioning is obtained via detection of NFC beacon signals that can indicate a specific position, etc.

[0110] Various memories (e.g., main memory 1212, static memory 1214, and memory of processor 1202) and storage unit 1216 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1208) when executed by processor 1202 cause various operations to implement the disclosed examples.

[0111] The instructions 1208 may be sent or received over the network 1220 using a transmission medium via a network interface device (e.g., a network interface component included in the communication component 1236) and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1208 may be sent or received via a coupling (e.g., a peer-to-peer coupling) with the device 1222 using a transmission medium.

[0112] Glossary

[0113] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by the machine and includes digital or analog communications signals or other intangible media to facilitate communication of these instructions. Instructions may be sent or received over a network via a network interface device using a transmission medium.

[0114] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0115] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, other type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0116] "Component" refers to a device, a physical entity, or logic with boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization for specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components and can be part of a program that generally performs a specific function in a related function. A component can constitute a software component (e.g., a code implemented on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., an independent computer system, a client computer system, or a server computer system) or one or more hardware components (e.g., a processor or a processor group) of a computer system can be configured by software (e.g., an application or an application part) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and is no longer a general-purpose processor. It should be understood that the decision to implement the hardware component mechanically, in a dedicated and permanently configured circuit or in a temporarily configured (e.g., configured by software) circuit can be driven by cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an embodiment in which the hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Thus, software configures one or more specific processors to, for example, constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between or among two or more hardware components in the hardware components. In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device to which it is communicatively coupled. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some of the operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some of the operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., API). The execution of certain operations can be distributed between processors, not only resident in a single machine, but deployed across multiple machines. In some example embodiments, a processor or a part implemented by a processor can be located in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other example embodiments, a processor or a part implemented by a processor can be distributed across multiple geographic locations.

[0117] "Computer-readable storage media" refers to both machine storage media and transmission media. Thus, the term includes both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0118] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and may be used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0119] “Non-transitory computer-readable storage medium” refers to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.

[0120] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the transfer of software or data. The term "signal medium" should be deemed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode the information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

Claims

1. A method comprising: In a messaging system that hosts a backend service of a messaging client and provides a plurality of augmented reality (AR) experiences, the messaging client providing a camera view user interface (UI) that includes an output of a digital image sensor of a camera, determining a physical context of the messaging client executing at a user device; selecting an AR experience from the plurality of AR experiences based on the physical context; generating an experience selection UI including a reference to the AR experience, the reference being executable to launch the AR experience at the user device; as well as The selection UI is caused to be displayed on a display of the user equipment.

2. The method according to claim 1, wherein: The physical context is other user equipment executing the messaging client.

3. The method according to claim 2, wherein: Determining the physical context includes determining that the user equipment is within a predetermined distance of the other user equipment.

4. The method according to claim 2, wherein: The AR experience is a shared AR experience that allows a user to manipulate virtual objects from the user device and from the other user devices.

5. The method according to claim 1, wherein: Determining the physical context includes using an output of a digital sensor of the camera.

6. The method according to claim 5, wherein: The physical context is an indication of an animal in the camera view UI.

7. The method according to claim 6, wherein: The AR experience is configured to track an indication of the animal in the camera view UI.

8. The method according to claim 5, wherein: The physical context is an indication of a person, wherein the AR experience is configured to track the indication of the person in the camera view UI.

9. The method according to claim 1, wherein: The physical context is a smart light, wherein the AR experience is configured to communicate with the smart light.

10. The method according to claim 1, wherein: The user device is a smart phone or AR glasses.

11. A system comprising: one or more processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: In a messaging system that hosts a backend service of a messaging client and provides a plurality of augmented reality (AR) experiences, the messaging client providing a camera view user interface (UI) that includes an output of a digital image sensor of a camera, determining a physical context of the messaging client executing at a user device; selecting an AR experience from the plurality of AR experiences based on the physical context; generating an experience selection UI including a reference to the AR experience, the reference executable to launch the AR experience at the user device; and The selection UI is caused to be displayed on a display of the user equipment.

12. The system according to claim 11, wherein: The physical context is other user equipment executing the messaging client.

13. The system according to claim 12, wherein: Determining the physical context includes determining that the user equipment is within a predetermined distance of the other user equipment.

14. The system according to claim 12, wherein: The AR experience is a shared AR experience that allows a user to manipulate virtual objects from the user device and from the other user devices.

15. The system according to claim 11, wherein: Determining the physical context includes using an output of a digital sensor of the camera.

16. The system of claim 15, wherein: The physical context is an indication of an animal in the camera view UI.

17. The system of claim 16, wherein: The AR experience is configured to track an indication of the animal in the camera view UI.

18. The system of claim 15, wherein: The physical context is an indication of a person, wherein the AR experience is configured to track the indication of the person in the camera view UI.

19. The system of claim 11, wherein: The physical context is a smart light, wherein the AR experience is configured to communicate with the smart light.

20. A machine-readable non-transitory storage medium having instruction data executable by a machine to cause the machine to perform operations comprising: In a messaging system that hosts a backend service of a messaging client and provides a plurality of augmented reality (AR) experiences, the messaging client providing a camera view user interface (UI) that includes an output of a digital image sensor of a camera, determining a physical context of the messaging client executing at a user device; selecting an AR experience from the plurality of AR experiences based on the physical context; generating an experience selection UI including a reference to the AR experience, the reference being executable to launch the AR experience at the user device; as well as The selection UI is caused to be displayed on a display of the user equipment.

Images