Customized virtual store

Abstract

Aspects of the present disclosure comprise a system comprising a computer-readable storage medium storing a method and a program for performing operations, wherein the operations include: receiving a request from a first user from a client device of a first user to participate in an AR shopping experience curated by a store; identifying a first real-world product available for purchase from the store; receiving an image of the first user’s real-world environment; creating a first AR item representing the first real-world product; comparing the visual attributes of the first AR item with the physical layouts of a plurality of real-world objects depicted in the image of the real-world environment; and overlaying the first AR item on a first real-world object among a plurality of real-world objects in the image in response to comparing the visual attributes of the first AR item with the physical layouts of the plurality of real-world objects.

Description

Technology Field

[0001] Claim of priority

[0002] This application claims the benefit of priority to U.S. Patent Application No. 17 / 324,350 filed May 19, 2021, the entirety of which is incorporated herein by reference.

[0003] Technology field

[0004] The present disclosure generally relates to providing augmented reality (AR) / virtual reality (VR) experiences using messaging applications. Background Technology

[0005] Augmented reality is a modification of a virtual environment. For example, in virtual reality, the user is completely immersed in a virtual world, whereas in AR, the user is immersed in a world where virtual objects are combined with or overlaid on the real world. AR systems aim to create and present virtual objects that interact realistically with the real-world environment. Examples of AR applications may include single- or multi-player video games, instant messaging systems, and the like. Brief explanation of the drawing

[0006] In drawings that are not necessarily drawn to scale, the same number may describe similar components in different drawings. To easily identify the discussion of any specific element or operation, the top digit or numbers in the reference numbers refer to the drawing number where the element is first introduced. Some non-limiting examples are illustrated in the drawings of the attached drawings. FIG. 1 is a schematic representation of a networked environment in which the present disclosure may be arranged, according to some examples. Figure 2 is a schematic representation of a messaging client application according to some examples. Figure 3 is a schematic representation of a data structure maintained in a database according to some examples. Figure 4 is a schematic representation of a message according to some examples. FIGS. 5, FIGS. 6, FIGS. 7a, FIGS. 7b, FIGS. 8, and FIGS. 9 are schematic representations of the outputs of a shared shopping experience system according to some examples. FIG. 10 is a flowchart illustrating exemplary operations of a messaging application server according to examples. FIG. 11 is a schematic representation of a machine in the form of a computer system in which a set of instructions can be executed to enable the machine to perform one or more of the methodologies discussed in this specification, according to some examples. FIG. 12 is a block diagram illustrating a software architecture in which examples can be implemented internally. Specific details for implementing the invention

[0007] The following description includes systems, methods, techniques, instruction sequences, and computing machine program products that implement exemplary examples of the present disclosure. In the following description, for the purposes of explanation, many specific details are provided to facilitate an understanding of various examples. However, it will be apparent to those skilled in the art that these examples may be practiced without such specific details. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily presented in detail.

[0008] Typically, virtual reality (VR) and augmented reality (AR) shopping systems allow users to browse virtual products from the comfort of their own homes. These systems enable users to preview how physical items look on the screen, simplifying the purchasing process. However, these systems are individual user-based and do not provide a way for users to virtually share their shopping experiences. To obtain feedback from friends regarding products of interest, users must navigate the virtual store themselves, select an item of interest, and then send that item to their friends for feedback. Receiving this feedback can take time, and the lack of real-time communication regarding browsing items typically leads to missed purchasing opportunities. While these systems generally function well, the passive and individual nature of the shopping experiences they provide detracts from their overall appeal. Specifically, users of these systems must expend significant effort searching and navigating multiple user interfaces and pages of information to identify items of interest, and then share those items with their friends, which takes significantly more time. These tasks can be difficult and time-consuming, which diminishes overall interest in using these systems and wastes resources.

[0009] Furthermore, some AR shopping systems allow users to select items of interest to place in their environment. However, item placement does not take into account how the item appears in AR relative to other real-world objects. These systems may experience presentation issues due to environmental conditions, user actions, and unexpected visual interruptions between the camera and the rendered object. This can cause virtual objects to disappear or behave erratically, destroying the illusion of virtual objects existing in the real world.

[0010] The disclosed technologies improve the efficiency of using electronic devices that implement or otherwise access AR / VR systems by providing a shared AR / VR shopping experience in which multiple users can interact with each other in a virtual world to shop for products, such as purchasing corresponding physically or electronically consumable items (e.g., clothing, accessories, makeup, toys, video items, music items, or video game items). Because users interact with each other in the shared AR / VR shopping experience, they can receive and provide real-time feedback, thereby improving the overall shopping experience. This prevents missing shopping and purchasing opportunities and increases the overall appeal of online shopping. Furthermore, the disclosed technologies consider the physical layout and geometry of the real-world environment when placing virtual products or items, thereby maintaining the illusion of virtual objects existing in the real world and avoiding presentation problems.

[0011] Specifically, the disclosed technologies receive a request from a first user to participate in an AR shopping experience curated by a store from a first user's client device. In some examples, a first real-world product available for purchase from the store is identified, and an image of the first user's real-world environment is received from the camera of the client device. The disclosed technologies generate a first AR item representing the first real-world product and compare one or more visual properties of the first AR item with the physical layouts of a plurality of real-world objects depicted in the image of the real-world environment. The disclosed technologies overlay the first AR item on the first real-world object among the plurality of real-world objects in the image (in response thereto) as a result of comparing one or more visual properties of the first AR item with the physical layouts of the plurality of real-world objects. This improves the user's overall experience in using the electronic device and reduces the total amount of system resources required to accomplish the task.

[0012] Networked computing environment

[0013] FIG. 1 is a block diagram illustrating an exemplary messaging system (100) for exchanging data (e.g., messages and associated content) over a network. The messaging system (100) includes multiple instances of client devices (102), each of which hosts multiple applications including a messaging client (104) and other external applications (109) (e.g., third-party applications). Each messaging client (104) is communically coupled to a messaging server system (108) and external app(s) servers (110) via a network (112) (e.g., the Internet) and other instances of the messaging client (104) (e.g., hosted on each other client device (102)). The messaging client (104) may also communicate with locally hosted third-party applications (109) using APIs (Applications Program Interfaces).

[0014] A messaging client (104) can communicate with other messaging clients (104) and with a messaging server system (108) via a network (112) and exchange data. The data exchanged between messaging clients (104) and between a messaging client (104) and a messaging server system (108) includes functions (e.g., commands to activate functions) as well as payload data (e.g., text, audio, video, or other multimedia data).

[0015] The messaging server system (108) provides server-side functionality to a specific messaging client (104) via the network (112). Although specific functions of the messaging system (100) are described herein as being performed by the messaging client (104) or by the messaging server system (108), the location of specific functions within the messaging client (104) or the messaging server system (108) may be a design choice. For example, it may be technically desirable to initially place specific technologies and functions within the messaging server system (108), but later move these technologies and functions to the messaging client (104) where the client device (102) has sufficient processing capacity.

[0016] The messaging server system (108) supports various services and operations provided to the messaging client (104). These operations include sending data to the messaging client (104), receiving data from it, and processing data generated by it. This data may include, for example, message content, client device information, geolocation information, media augmentations and overlays, message content persistence conditions, social network information, and live event information. Data exchanges within the messaging system (100) are initiated and controlled through functions available via the user interfaces of the messaging client (104).

[0017] Now, referring specifically to the messaging server system (108), an API (Application Program Interface) server (116) is coupled to the application servers (114) to provide a programmatic interface. The application servers (114) are communicably coupled to a database server (120), which facilitates access to a database (126) that stores data associated with messages processed by the application servers (114). Similarly, a web server (128) is coupled to the application servers (114) and provides web-based interfaces to the application servers (114). To this end, the web server (128) processes incoming network requests via HTTP (Hypertext Transfer Protocol) and some other related protocols.

[0018] The API server (116) receives and transmits message data (e.g., commands and message payloads) between the client device (102) and the application servers (114). Specifically, the API server (116) provides a set of interfaces (e.g., routines and protocols) that can be initiated or queried by the messaging client (104) to call the functionality of the application servers (114). The API server (116) provides account registration; login functionality; transmission of messages through the application servers (114) from a specific messaging client (104) to another messaging client (104); transmission of media files (e.g., images or videos) from the messaging client (104) to the messaging server (118) and for possible access by the other messaging client (104); settings of a collection of media data (e.g., stories); retrieval of a list of friends of the user of the client device (102); retrieval of such collections; retrieval of messages and content; It exposes various functions supported by application servers (114), including adding and deleting entities (e.g., friends) to an entity graph (e.g., social graph); checking the location of friends within the social graph; and opening application events (e.g., related to a messaging client (104)).

[0019] Application servers (114) host a number of server applications and subsystems, including, for example, a messaging server (118), an image processing server (122), and a social network server (124). The messaging server (118) implements a number of message processing techniques and functionalities particularly related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from a number of instances of the messaging client (104). As further described in detail, text and media content from a number of sources may be aggregated into collections of content (e.g., referred to as stories or galleries). Next, these collections are made available to the messaging client (104). Other processor- and memory-intensive processing of data may also be performed on the server side by the messaging server (118), taking into account the hardware requirements for such processing.

[0020] The application servers (114) also include an image processing server (122) dedicated to performing various image processing operations with respect to images or videos in the payload of a message typically transmitted from or received from a messaging server (118).

[0021] An image processing server (122) is used to implement the scanning functionality of the augmentation system (208). The scanning functionality includes activating and providing one or more augmented reality experiences on the client device (102) when an image is captured by the client device (102). Specifically, a messaging client (104) on the client device (102) may be used to activate a camera. The camera displays one or more real-time images or videos to the user along with one or more icons or identifiers of one or more augmented reality experiences. The user may select one of the given identifiers to launch a corresponding augmented reality experience or perform a desired image modification (e.g., launching a shared shopping experience with one or more friends / users or a single-user AR shopping experience, as discussed in relation to FIGS. 5 through 10 below).

[0022] In some examples, the first user's client device (102) may generate and display a barcode associated with the shared shopping experience. One or more other users or friends of the first user may join and participate in the shared shopping experience with the first user by using their respective client devices (102) to capture or scan an image of the barcode displayed on the first user's client device (102). In some examples, the barcode may be physically placed on the surface of a real-world object (e.g., store shelves, store entrances, hidden locations, walls, ceilings, floors, or other physical locations), such as in a store or other real-world environment. Different users or friends may join and participate in the shared shopping experience by using their respective client devices (102) to capture or scan an image of the barcode physically placed on the surface of the real-world object. In some examples, the barcode may be personalized or associated with a password, login credentials, or a designated user group. That is, the client device (102) of the first user can generate a barcode and associate some restrictions with the barcode, such as a pre-selected group of friends or users who are allowed to participate in a shared shopping experience with the first user. After generating the barcode, the client device (102) can store the barcode along with the restrictions locally or on a remote server.

[0023] When a barcode is scanned by another user's client device (102), the link associated with the barcode is retrieved and accessed by the other user's client device (102). The other user's client device (102) also provides a user identifier to the server associated with the link. The server determines that the user identifier matches or satisfies the restrictions associated with the barcode (e.g., that the user identifier matches the user identifier associated with the barcode specified by the first user). In response, the server authorizes the other user's client device (102) to join the same AR session of the first user's shopping experience. In this way, the first user and the other user can interact with each other virtually and in AR to browse a virtual store, such as wearing different AR items (e.g., AR clothing or makeup). In response to the determination that the user identifier does not match or satisfy the restrictions associated with the barcode, the server prevents the other user's client device (102) from accessing or joining the same AR session of the first user's shopping experience.

[0024] In some embodiments, in response to launching an AR shopping experience (e.g., when a barcode is scanned by the client device (102)), a single image or video clip of the real-world environment is captured. Specifically, the client device (102) may present a set of commands to the user for capturing an image or video clip of the real-world environment. For example, the client device (102) may command the user to start activating the rear (world) camera of the client device (102) to start capturing a video clip of a first real-world object (e.g., a sofa). The client device (102) may command the user to start rotating the client device (102) in a specific direction until the user rotates 360 degrees. In response, a video clip capturing 360 degrees of the user's environment, including multiple real-world objects within the real-world environment, is captured. A video clip is transmitted by a client device (102) to a remote server to process and customize a virtual store within a real-world environment (e.g., to select AR objects corresponding to the physical layouts of real-world objects). After the server customizes the virtual store, the virtual store is presented by overlaying selected AR objects within the real-world environment depicted on images captured by one or more users or on a camera feed.

[0025] The social network server (124) supports various social networking functions and services and makes these functions and services available to the messaging server (118). To this end, the social network server (124) maintains and accesses an entity graph (308) (shown in FIG. 3) within a database (126). Examples of functions and services supported by the social network server (124) include the identification of other users of the messaging system (100) with whom a specific user has relationships or is "following," and also the identification of other entities and the interests of the specific user.

[0026] Returning to the messaging client (104), features and functions of an external resource (e.g., a third-party application (109) or an applet) are made available to the user through the interface of the messaging client (104). The messaging client (104) receives a user selection of an option to launch or access features of an external resource (e.g., a third-party resource), such as external apps (109). The external resource may be a third-party application (external apps (109)) installed on the client device (102) (e.g., a "native app"), or a small version of a third-party application (e.g., an "applet") hosted on the client device (102) or remotely from the client device (102) (e.g., on third-party servers (110)). A small version of a third-party application includes a subset of the features and functions of the third-party application (e.g., a full-scale, native version of the third-party independent application) and is implemented using markup-language documents. In one example, a small version of the third-party application (e.g., an "applet") is a web-based, markup-language version of the third-party application and is embedded in a messaging client (104). In addition to using markup-language documents (e.g., .*ml files), the applet may include a scripting language (e.g., .*js files or .json files) and a style sheet (e.g., .*ss files).

[0027] In response to receiving a user selection of an option to launch an external resource (external app (109)) or access its features, the messaging client (104) determines whether the selected external resource is a web-based external resource or a locally installed external application. In some cases, external applications (109) installed locally on the client device (102) may be launched independently and separately from the messaging client (104), such as by selecting an icon corresponding to the external application (109) on the home screen of the client device (102). Smaller versions of these external applications may be launched or accessed through the messaging client (104), and in some examples, any part of the small external application may not be accessible outside the messaging client (104), or only limited parts may be accessible. A small external application can be launched by a messaging client (104) that receives a markup-language document associated with the small external application from an external app(s) server (110) and processes such document.

[0028] In response to determining that the external resource is a locally installed external application (109), the messaging client (104) commands the client device (102) to launch or access the external application (109) by executing locally stored code corresponding to the external application (109). In response to determining that the external resource is a web-based resource, the messaging client (104) communicates with the external app(s) servers (110) to obtain a markup-language document corresponding to the selected resource. Next, the messaging client (104) processes the obtained markup-language document and presents the web-based external resource within the user interface of the messaging client (104).

[0029] A messaging client (104) may notify a user of a client device (102) or other users associated with such user (e.g., “friends”) of activity occurring in one or more external resources. For example, the messaging client (104) may provide participants of a conversation (e.g., a chat session) in the messaging client (104) with notifications regarding current or recent use of an external resource by one or more members of a group of users. One or more users may be invited to join an active external resource or to launch an external resource that was recently used but is currently inactive (in the group of friends). The external resource may provide participants of the conversation, each using a messaging client (104), with the ability to share items, situations, states, or locations in the external resource with one or more members of a group of users entering the chat session. The shared items may be interactive chat cards, which may be used to allow members of the chat to interact, for example, to launch a corresponding external resource, view specific information within the external resource, or take a member of the chat to a specific location or state within the external resource. Within a given external resource, response messages can be sent to users on a messaging client (104). The external resource may optionally include different media items in the responses based on the current context of the external resource.

[0030] The messaging client (104) may present to the user a list of available external resources (e.g., third-party or external applications (109) or applets) to launch or access a given external resource. This list may be presented in a context-sensing menu. For example, icons representing different external applications (109) (or applets) may change based on how the menu is launched by the user (e.g., from a conversational interface or from a non-conversational interface).

[0031] System Architecture

[0032] FIG. 2 is a block diagram illustrating additional details regarding a messaging system (100) according to some examples. Specifically, the messaging system (100) is illustrated as including a messaging client (104) and application servers (114). The messaging system (100) implements a number of subsystems supported on the client side by the messaging client (104) and on the server side by the application servers (114). These subsystems include, for example, a short-term timer system (202), a collection management system (204), an augmentation system (208), a map system (210), a game system (212), and an external resource system (220).

[0033] The short-term timer system (202) is responsible for enforcing temporary or time-limited access to content by the messaging client (104) and the messaging server (118). The short-term timer system (202) includes a plurality of timers that selectively enable access to messages and associated content (e.g., for presentation and display) through the messaging client (104) based on duration and display parameters associated with a message or a collection of messages (e.g., a story). Further details regarding the operation of the short-term timer system (202) are provided below.

[0034] The collection management system (204) is responsible for managing sets or collections of media (e.g., collections of text, image, video, and audio data). Collections of content (e.g., messages including images, video, text, and audio) may be organized into "event galleries" or "event stories." These collections may be available for a specified period, such as the duration of the event to which the content is related. For example, content related to a music concert may be available as a "story" for the duration of the music concert. The collection management system (204) may also be responsible for posting an icon on the user interface of the messaging client (104) that provides notification of the existence of a specific collection.

[0035] The collection management system (204) further includes a curation interface (206) that allows a collection manager to manage and curate specific collections of content. For example, the curation interface (206) enables an event organizer to curate a collection of content related to a specific event (e.g., removing inappropriate content or duplicate messages). Additionally, the collection management system (204) automatically curates content collections using machine vision (or image recognition technology) and content rules. In certain examples, a reward may be paid to users for including user-generated content in a collection. In these cases, the collection management system (204) operates to automatically pay these users for their use of the content.

[0036] The augmentation system (208) provides various functions that enable a user to augment media content associated with a message (e.g., annotate or otherwise modify or edit). For example, the augmentation system (208) provides functions related to the creation and publication of media overlays for messages processed by the messaging system (100). The augmentation system (208) operatively supplies a media overlay or augmentation (e.g., an image filter) to the messaging client (104) based on the geolocation of the client device (102). In another example, the augmentation system (208) operatively supplies a media overlay to the messaging client (104) based on other information, such as the social network information of the user of the client device (102). The media overlay may include auditory and visual content and visual effects. Examples of auditory and visual content include images, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Auditory and visual content or visual effects may be applied to media content items (e.g., photos) on the client device (102). For example, a media overlay may include text, graphic elements, or images that can be overlaid on a photo taken by the client device (102). In another example, the media overlay may include a location identification overlay (e.g., Venice beach), the name of a live event, or a merchant name overlay (e.g., Beach Coffee House). In another example, the augmentation system (208) uses the geolocation of the client device (102) to identify a media overlay containing the name of a merchant located at the geolocation of the client device (102). The media overlay may include other indications associated with the merchant.Media overlays are stored in a database (126) and can be accessed through a database server (120).

[0037] In some examples, the augmentation system (208) provides a user-based publishing platform that enables users to select a geolocation on a map and upload content associated with the selected geolocation. Users may also specify situations in which a particular media overlay should be provided to other users. The augmentation system (208) generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geolocation.

[0038] In other examples, the augmentation system (208) provides a merchant-based publishing platform that enables merchants to select a specific media overlay associated with a geolocation through a bidding process. For example, the augmentation system (208) associates the media overlay of the highest bidder with a corresponding geolocation for a predefined amount of time. The augmentation system (208) communicates with an image processing server (122) to acquire augmented reality experiences and presents identifiers of these experiences in one or more user interfaces (e.g., as icons on a real-time image or video, or as thumbnails or icons in interfaces dedicated to the presented identifiers of the augmented reality experiences). Once an augmented reality experience is selected, one or more images, videos, or augmented reality graphic elements are retrieved and presented as an overlay on an image or video captured by the client device (102). In some cases, the camera is switched to a forward view (e.g., the front camera of the client device (102) is activated in response to the activation of a specific augmented reality experience), and images from the front camera of the client device (102) begin to be displayed on the client device (102) instead of the rear camera of the client device (102). One or more images, videos, or augmented reality graphic elements are retrieved and presented as an overlay on top of the images captured and displayed by the front camera of the client device (102).

[0039] In other examples, the augmentation system (208) provides a merchant-based publishing platform that enables merchants to select specific AR items to display within a real-world environment to represent one or more virtual stores through a bidding process. For example, the augmentation system (208) associates the AR items of the highest bidder with presentations of specific real-world objects associated with high rankings. That is, the augmentation system (208) can detect multiple real-world objects in a real-world environment. The augmentation system (208) can rank real-world objects based on their physical layouts by comparing the physical layouts or attributes of the detected real-world objects with one another. In some examples, the augmentation system (208) can rank real-world objects based on size. In some examples, the augmentation system (208) can rank real-world objects based on available free physical space on top or inside (e.g., shelves) parts. In some examples, the augmentation system (208) may rank real-world objects based on their real-world locations within the real-world environment (e.g., objects closer to the kitchen are ranked higher than objects closer to the bathroom or door). After the real-world objects are ranked, the augmentation system (208) places or overlays the highest bid or highest-ranked AR items on or inside the free physical space of the highest-ranked real-world object. The remaining AR objects are similarly positioned and overlaid based on their respective ranks relative to the rank of the real-world object.

[0040] In other examples, the augmentation system (208) may communicate with another augmentation system (208) on another client device (102) and with a server via a network (106) and exchange data. The data exchanged may include a session identifier identifying a shared AR session, a common origin, a common coordinate frame, functions (e.g., commands for initiating functions), as well as a conversion between the first client device (102) and the second client devices (102) used to align the shared AR session to other payload data (e.g., text, audio, video, or other multimedia data) (e.g., the plurality of client devices (102) includes the first and second devices).

[0041] The augmentation system (208) transmits a transformation to the second client device (102) so that the second client device (102) can adjust the AR coordinate system based on the transformation. In this way, the first and second client devices (102) synchronize their coordinate systems and frames to display content in an AR session, for example, in a shared AR shopping experience. Specifically, the augmentation system (208) computes the origin of the second client device (102) in the coordinate system of the first client device (102). Next, the augmentation system (208) can determine an offset in the coordinate system of the second client device (102) based on the position of the origin from the perspective of the second client device (102) in the coordinate system of the second client device (102). This offset is used to generate a transformation so that the second client device (102) generates AR content according to a common coordinate system or frame as the first client device (102).

[0042] The augmentation system (208) may communicate with a client device (102) and / or a shared shopping experience system (224) to establish individual (single user) or shared AR / VR sessions. The augmentation system (208) may also be coupled to a messaging server (118) to establish an electronic group communication session (e.g., group chat, instant messaging) for the client devices (102) in a shared AR session. An electronic group communication session may be associated with a session identifier provided by the client devices (102) to obtain access to the electronic group communication session and to the shared AR / VR session. In one example, the client devices (102) first obtain access to the electronic group communication session and then obtain a session identifier from the electronic group communication session that allows the client devices (102) to access the shared AR / VR session. In some examples, the client devices (102) may access the shared AR / VR session at the application servers (114) without assistance or communication with the augmentation system (208).

[0043] The map system (210) provides various geographic location functions and supports the presentation of map-based media content and messages by the messaging client (104). For example, the map system (210) enables the display of user icons or avatars (e.g., stored in profile data (316)) on the map to display the current or past locations of the user's "friends" as well as media content created by these friends (e.g., collections of messages including photos and videos) within the context of the map. For example, a message posted by a user to the messaging system (100) from a specific geographic location can be displayed to the specific user's "friends" on the map interface of the messaging client (104) within the context of the map at that specific location. The user can also share their location and status information with other users of the messaging system (100) through the messaging client (104) (e.g., using an appropriate status avatar), and this location and status information is similarly displayed to selected users within the context of the map interface of the messaging client (104).

[0044] The game system (212) provides various game functions within the context of the messaging client (104). The messaging client (104) provides a game interface that provides a list of available games (e.g., web-based games or web-based applications) that can be launched by a user within the context of the messaging client (104) and played with other users of the messaging system (100). The messaging system (100) further enables a specific user to invite other users to participate in the play of a specific game by issuing invitations to these other users from the messaging client (104). The messaging client (104) also supports both voice and text messaging (e.g., chats) within the context of gameplay, provides a leaderboard for games, and also supports the provision of in-game rewards (e.g., coins and items).

[0045] The external resource system (220) provides an interface for the messaging client (104) to communicate with external app(s) servers (110) to launch or access external resources. Each external resource (app) server (110) hosts, for example, a markup language (e.g., HTML5) based application or a small version of an external application (e.g., a game, utility, payment, or ride-sharing application outside the messaging client (104). The messaging client (104) can launch a web-based resource (e.g., an application) by accessing an HTML5 file from the external resource (app) servers (110) associated with the web-based resource. In certain examples, applications hosted by the external resource servers (110) are programmed in JavaScript utilizing a Software Development Kit (SDK) provided by the messaging server (118). The SDK includes APIs (Application Programming Interfaces) having functions that can be called or launched by web-based applications. In certain examples, the messaging server (118) includes a JavaScript library that provides access to a given third-party resource for specific user data of the messaging client (104). Although HTML5 is used as an exemplary technology for programming games, applications and resources programmed based on other technologies may be used.

[0046] To integrate the functions of the SDK into a web-based resource, the SDK is downloaded from the messaging server (118) by the external resource (apps) server (110) or otherwise received by the external resource (apps) server (110). Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate specific functions of the SDK to integrate the features of the messaging client (104) into the web-based resource.

[0047] The SDK stored on the messaging server (118) effectively provides a bridge between external resources (e.g., third-party or external applications (109) or applets) and the messaging client (104). This provides the user with a seamless experience communicating with other users on the messaging client (104), while also preserving the look and feel of the messaging client (104). To bridge communication between the external resources and the messaging client (104), in certain examples, the SDK facilitates communication between the external resource servers (110) and the messaging client (104). In certain examples, the WebViewJavaScriptBridge running on the client device (102) establishes two unidirectional communication channels between the external resources and the messaging client (104). Through these communication channels, messages are transmitted asynchronously between the external resources and the messaging client (104). Each SDK function activation is transmitted as a message and a callback. Each SDK function is implemented by configuring a unique callback identifier and sending a message having that callback identifier.

[0048] Using the SDK, not all information from the messaging client (104) is shared with the external resource servers (110). The SDK limits which information is shared based on the needs of the external resources. In certain examples, each external resource server (110) provides an HTML5 file corresponding to a web-based external resource to the messaging server (118). The messaging server (118) can add a visual representation of the web-based external resource (such as box art or other graphics) to the messaging client (104). Once the user selects a visual representation or commands the messaging client (104) to access the features of the web-based external resource through the messaging client's (104) GUI, the messaging client (104) obtains the HTML5 file and instantiates the resources necessary to access the features of the web-based external resource.

[0049] The messaging client (104) presents a graphical user interface (e.g., a landing page or a title screen) for an external resource. While presenting the landing page or title screen, or before or after, the messaging client (104) determines whether the launched external resource has previously been authorized to access the messaging client (104)'s user data. In response to the determination that the launched external resource has previously been authorized to access the messaging client (104)'s user data, the messaging client (104) presents another graphical user interface of the external resource, including the functions and features of the external resource. In response to determining that the launched external resource has not previously been authorized to access the messaging client (104) user data, after a threshold period (e.g., 3 seconds) during which the external resource's landing page or title screen is displayed, the messaging client (104) slides up a menu to authorize the external resource to access user data (e.g., animated by raising the menu from the bottom of the screen to the middle or other part of the screen). This menu identifies the type of user data that the external resource is authorized to use. In response to receiving a user selection of an accept option, the messaging client (104) adds the external resource to a list of authorized external resources and allows the external resource to access user data from the messaging client (104). In some examples, the external resource is authorized by the messaging client (104) to access user data according to the OAuth 2 framework.

[0050] The messaging client (104) controls the type of user data shared with external resources based on the type of external resources being authorized. For example, external resources including full-scale external applications (e.g., third-party or external applications (109)) are provided with access to a first type of user data (e.g., just two-dimensional avatars of users with or without different avatar features). As another example, external resources including small versions of external applications (e.g., web-based versions of third-party applications) are provided with access to a second type of user data (e.g., payment information, two-dimensional avatars of users, three-dimensional avatars of users, and avatars with various avatar features). Avatar features include different ways of customizing the look and feel of the avatar, such as different poses, facial features, clothing, etc.

[0051] The shared shopping experience system (224) provides a virtual AR store for a single user or multiple users to browse and shop for products virtually in AR together or individually. In particular, the shared shopping experience system (224) may provide a shared shopping experience associated with a specific merchant or store. The shared shopping experience system (224) creates options for the user to join or participate in the shared shopping experience. In response to receiving the user's selection of options from the client device (102), the shared shopping experience system (224) presents a screen that allows the user to invite one or more friends to join the shared shopping experience with the user. Input identifying one or more other users or friends of the user on the messaging client (104) may be received from the user's client device (102). In response to this, the shared shopping experience system (224) sends a communication to the identified friends that allows the friends to launch the shared shopping experience and join with the user.

[0052] In some examples, in response to receiving a user selection of options from a client device (102) to participate in a shared shopping experience, the shared shopping experience system (224) presents a screen that allows the user to select types of AR items to include in the shared shopping experience. For example, the shared shopping experience system (224) may identify one or more real-world objects in the camera feed of the user's client device (102). The shared shopping experience system (224) identifies a list of different types of real-world products that are compatible and can be displayed on one or more real-world objects. The shared shopping experience system (224) generates a list of different types of real-world products (e.g., makeup, pants, shirts, jewelry, etc.). The shared shopping experience system (224) receives a user selection of one or more of the different types of real-world products and generates multiple AR items representing one or more of the selected real-world products. Multiple generated AR items are overlaid on one or more different real-world objects depicted in the camera feeds of the user's client device (102) and the client devices (102) of other users invited to join the user in the shared shopping experience.

[0053] In some examples, the shared shopping experience system (224) receives images or video clips depicting various real-world objects in the user's real-world environment. The shared shopping experience system (224) detects the configuration or physical layout of the real-world objects and selects one or more AR items having visual attributes that match or correspond to the physical layout of the real-world objects. For example, the shared shopping experience system (224) may receive a user request to browse or shop for shirts and pants. The shared shopping experience system (224) may generate a first set of AR items representing shirts available for purchase and a second set of AR items representing pants available for purchase. The shared shopping experience system (224) may identify a table having some free space on top in the real-world environment. The shared shopping experience system (224) may also identify a wardrobe having some available free hanging space in the real-world environment.

[0054] The shared shopping experience system (224) may determine that a first set of AR items includes a vertical stack of shirts. That is, the AR shirts can be folded so that one is stacked on top of another. The shared shopping experience system (224) may determine that such a vertical stack does not correspond to a wardrobe because the vertical stack cannot be hung inside the wardrobe and cannot be placed in the wardrobe in any realistic way. The shared shopping experience system (224) may determine that the visual properties of the vertical stack of shirts correspond to free space on a tabletop or a set of shelves. In response to this, the shared shopping experience system (224) constructs a first set of AR items to be displayed on a real-world table depicted in a real-world environment. In this way, whenever any user captures an image depicting a real-world table while browsing the virtual store, the first set of AR items is presented on the table.

[0055] The shared shopping experience system (224) may determine that a second set of AR items each comprises a horizontal stack of pants on a hanger. The shared shopping experience system (224) may determine that such a horizontal stack does not correspond to a table because the horizontal stack cannot be placed on top of a table in any realistic manner. The shared shopping experience system (224) may determine that the visual properties of the horizontal stack of pants correspond to free space within a closet or other suitable clothing rack. In response to this, the shared shopping experience system (224) constructs a second set of AR items to be displayed inside a real-world closet depicted in a real-world environment. In this way, whenever any user captures an image depicting a real-world closet while browsing a virtual store, the second set of AR items is presented as hanging on a rack within the closet.

[0056] In some examples, the shared shopping experience system (224) may identify multiple different types of AR items that include visual attributes corresponding to the physical layout of a given real-world object. For example, the shared shopping experience system (224) may identify a first set of AR items (including folded shirts in a vertical stack) and a third set of AR items (including shorts stacked on top of each other in a vertical stack). Both the first and third sets of AR items may be placed or overlaid on top of the free space on a real-world table. The shared shopping experience system (224) may determine that the available free space on the real-world table is insufficient (or barely sufficient) to overlay one of the two sets of AR items. In response, the shared shopping experience system (224) determines the likelihood of interest or that the user will purchase the first and third sets of AR items. For example, the shared shopping experience system (224) may determine that the user generally enjoys purchasing shorts rather than shirts. In these cases, the shared shopping experience system (224) chooses to display a third set of AR items on the free space of a real-world table instead of a first set of AR items.

[0057] After users join or launch a shared shopping experience, each user is presented on their own client device (102) with an instance of a virtual shared shopping experience of a specific merchant or store configured based on a previously captured real-world environment. While the shared shopping experience system (224) provides AR displays or VR stores to users, other users can join and leave the shared shopping experience in real time. The shared shopping experience system (224) presents virtual AR items representing products offered for sale by the merchant or store on the displays of the users' client devices (102). The virtual shopping experience provides a virtual store that looks like the merchant's physical store. That is, the users' client devices (102) display virtual AR items overlaid on real-world objects depicted in the camera feed of each client device (102). For example, the first user's first client device (102) can capture an image of a first part of the real-world environment and display one or more AR items on the first part of the real-world environment. A second client device (102) of a first user can capture an image of a second part of a real-world environment and display one or more AR items on the second part of the real-world environment. The first client device (102) can be physically moved in 3D space and can be oriented toward the second part of the real-world environment such that at least a portion of the image displayed by the first client device (102) includes the same real-world object depicted in the image displayed by the second client device (102). In these cases, both the first and second client devices (102) display or overlay the same set of AR items on the same real-world object depicted in the images displayed by each client device (102).In this way, the shopping experience is synchronized across multiple client devices (102) that are within a threshold proximity of each other. The threshold proximity can be predetermined (e.g., a 50-foot proximity threshold), dynamic, or depend on the physical environment that can be captured by different client devices (102). For example, client devices (102) may be more than 50 feet apart from each other, but both can capture images of the same physical real-world object. In these cases, AR items are displayed at different scales on the same physical real-world object depicted on each client device (102).

[0058] Users can explore and browse virtual AR items within a virtual AR store individually and independently. For example, a first user can explore a virtual store on the first user's client device (102) by moving around a shared real-world environment (e.g., a shared room in a house, a shared real-world store, or an outdoor public space). That is, as the first user moves along a specific path in three dimensions (3D), the shared shopping experience system (224) can determine the user's trajectory and update the display of the virtual AR store presented to the first user to move the user along the determined trajectory. As the first user turns the client device (102) to the right or in a given direction in 3D, the shared shopping experience system (224) changes the appearance of the virtual AR store to show the user's AR content of the virtual store appearing from the right or in the given direction in 3D.

[0059] The second user may similarly browse through the virtual store independently of the first user. That is, the second user may be browsing virtual items in the first room of the virtual store, and the display of the first room may be provided to the second user on the second user's client device (102). While the display of the first room is provided to the second user's client device (102), the first user may be browsing AR items in the second room of the virtual AR store, and accordingly, the display of the second room may be presented to the first user's client device (102). The shared shopping experience system (224) maintains the virtual 3D locations or coordinates of each user currently participating in the shared shopping experience system (224). Specifically, the shared shopping experience system (224) can assign first coordinates representing the current 3D location of the first user in the virtual store (e.g., the first 3D location in the second room) and assign second coordinates representing the current 3D location of the second user in the virtual store (e.g., the second 3D location in the first room).

[0060] In some cases, the shared shopping experience system (224) may present indicators on each of the users currently participating in the shared shopping experience that identify which users are participating in the shared shopping experience and their current 3D locations. The indicators may be displayed conditionally based on whether a given user's client device (102) is oriented toward another user's 3D direction. For example, the first user's client device (102) may be capturing a video feed containing images corresponding to the first part of the shared real-world environment while the second user is physically located in the second part of the shared real-world environment, which is 180 degrees away from the first part. As a result, the second user is not initially within the field of view corresponding to the first part (for example, the second user is not within the camera feed captured by the first user's client device (102). In these cases, the indicator of the second user is presented or overlaid on the camera feed captured and displayed by the first user's client device (102) to indicate the 3D location of the second user. For example, the indicator may identify the second user (e.g., may be the second user's avatar) and may include an arrow that identifies the second user or is oriented at a position physically relative to the first user. As input is received from the first user's client device (102) indicating that the first user's client device (102) has moved (e.g., rotated 180 degrees) so that the camera feed now includes a depiction of the second part of the real-world environment, the field of view of the first user's client device (102) is updated to depict the second part.As a result, the first user's client device (102) is now oriented toward the direction of the second user's physical location, and the second user appears in the video feed captured by the first user's client device (102). In response, the shared shopping experience system (224) removes the display of an indicator uniquely identifying the second user within the display of the virtual store on the first user's client device (102). In some cases, the indicator may be displayed only when the second user is depicted within the camera feed of the first user's client device (102). In these situations, the indicators may indicate the current activity being performed by the second user associated with the indicator.

[0061] In some cases, indicators of the physical locations of other users within the virtual store include pins. In some cases, indicators of the physical locations of other users include the avatars of other users. The pins or avatars may include the name of each user represented by the indicator. The pins or avatars may also include information regarding current activities being performed by other users, such as whether the other user associated with the indicator is accessing detailed information about a given product, virtually trying on a product, viewing and purchasing a product, picking up a product, or performing some other action.

[0062] In some embodiments, the shared shopping experience system (224) presents a list of AR items corresponding to real-world products within the field of view of the first user's client device (102), such as on one or more real-world objects depicted in the camera feed of the first user's client device (102). The list of AR items may represent different types of real-world products, such as shirts, shoes, and pants. The shared shopping experience system (224) may update the display attributes of the list of virtual reality items presented on the first user's client device (102) based on the types of actions performed by the second user operating the other client device (102). For example, the shared shopping experience system (224) may detect that the second user has selected a given AR item to wear or to observe more details. In response to this, the shared shopping experience system (224) may modify display attributes in a first manner to present a visual indicator (e.g., a highlight area) on one of the AR items in a list of AR items to indicate that a given AR item is being acted upon or interacted with by another user. For example, if the shared shopping experience system (224) detects that a second user has selected to wear virtual reality pants, the shared shopping experience system (224) may modify display attributes in a first manner, such as displaying a blue border around the first virtual reality item among the AR items corresponding to the physical product of pants.

[0063] For example, the shared shopping experience system (224) may present a first stack of AR items corresponding to pants and a second stack of AR items corresponding to shirts. When the second user chooses to wear pants, the shared shopping experience system (224) may modify the display properties in a first manner to display a blue border around the first stack of AR items presented on the first user's client device (102) to indicate that the second user has performed an action type to wear pants. When the second user chooses to purchase pants, the shared shopping experience system (224) may modify the display properties in a second manner to display a red border around the first stack of AR items presented on the first user's client device (102) to indicate that the second user has performed an action type to purchase pants. If the second user chooses to wear the shirts, the shared shopping experience system (224) may modify the display properties in a third way to display a blue border around the second stack of AR items presented on the first user's client device (102) to indicate that the second user has performed an action type for wearing the shirts. If the second user chooses to purchase the shirts, the shared shopping experience system (224) may modify the display properties in a fourth way to display a red border around the first stack of AR items presented on the first user's client device (102) to indicate that the second user has performed an action type for purchasing the shirts.

[0064] In some cases, the shared shopping experience system (224) may reduce the quantity of AR items in a list presented on the first user's client device (102) in response to detecting that the second user has chosen to wear an AR item or to view more details about it. For example, the shared shopping experience system (224) may present a first stack of AR items corresponding to pants that appears to include five pairs of pants and a second stack of AR items corresponding to shirts that appears to include seven shirts. If the second user chooses to try on pants, the shared shopping experience system (224) may reduce the quantity of pants included in the first stack of AR items presented on the client device (102) from five to four. If the second user chooses to wear shirts, the shared shopping experience system (224) may reduce the quantity of shirts included in the second stack of AR items presented on the first user's client device (102) from seven to six. The shared shopping experience system (224) can detect that the second user has chosen not to purchase the shirts worn. In response, the shared shopping experience system (224) can increase the number of shirts included in the second stack of AR items presented on the first user's client device (102) from 6 to 7.

[0065] In some embodiments, the shared shopping experience system (224) may access the store inventory associated with the virtual store. The shared shopping experience system (224) may choose how many AR items to present based on the store inventory. For example, the shared shopping experience system (224) may determine that there are 30 pairs of pants available in the store inventory. In response, the shared shopping experience system (224) may display up to 7 AR items corresponding to the pants in a first stack of AR items. As another example, the shared shopping experience system (224) may determine that there are 6 shirts available in the store inventory. In response, the shared shopping experience system (224) may display 6 or some other quantity fewer than the total number of available shirts in a second stack of AR items corresponding to the shirts. As another example, the shared shopping experience system (224) may determine that there is one dress available in the store inventory. In response to this, the shared shopping experience system (224) may display one AR item corresponding to the dress. The shared shopping experience system (224) may detect input from a client device (102) indicating the selection of an AR item corresponding to the dress. The shared shopping experience system (224) may determine that the selected AR item corresponds to the last available dress. In response to this, the shared shopping experience system (224) removes the AR item from being presented to any other user participating in the virtual store. That is, the shared shopping experience system (224) may associate a non-fungible token (NFT) with each real-world physical item available in the store's inventory. The shared shopping experience system (224) may create a unique AR item corresponding to each real-world physical item using the NFT associated with each real-world physical item.In this way, each AR item presented to users of the shared shopping experience in the virtual store uniquely represents a real-world product available in the store inventory.

[0066] In some examples, in response to determining that a selected AR item corresponds to the last available dress, the shared shopping experience system (224) may identify the physical layout of the real-world object where the last dress was displayed. The shared shopping experience system (224) may search to identify a different type of AR item corresponding to the physical layout of the real-world object. The shared shopping experience system (224) may search for a set of AR items of a different type (e.g., long-sleeved shirts or pants on hangers) and present this set of AR items of a different type instead of where the selected AR item corresponding to the last dress was displayed. In this way, the free physical space of different real-world objects is maximized in use for presenting AR items corresponding to the store. As the last AR item of a given type is selected from a given real-world object, a different type of AR item is selected to be presented in its place. If the shared shopping experience system (224) fails to identify other types of AR items corresponding to products available for purchase to be displayed in the free available space of a real-world object, the shared shopping experience system (224) may search for one or more decorative AR items corresponding to a virtual store. For example, the shared shopping experience system (224) may search for an AR flowerpot or logo representing a virtual store to be displayed in the free available space of a real-world object.

[0067] In some embodiments, the shared shopping experience system (224) presents a virtual or real-world mannequin in a virtual store. Two or more users participating in the shared shopping experience system (224) may have access to a field of view containing the virtual or real-world mannequin. For example, two or more users may each be capturing images on their respective client devices (102) depicting the real-world mannequin. As another example, two or more users may each be capturing images on their respective client devices (102) depicting a real-world object on which the virtual AR mannequin is displayed. The shared shopping experience system (224) may receive input from the first user's client device (102) to add clothing items, such as an AR hat, to the virtual or real-world mannequin. In response, the shared shopping experience system (224) modifies the virtual or real-world mannequins depicted in images displayed on each of the client devices (102) of two or more users to include an AR hat selected by the first user. The shared shopping experience system (224) may receive input from the client device (102) of the second user to add other clothing items to the virtual or real-world mannequins, such as an AR shirt. In response, the shared shopping experience system (224) modifies the virtual or real-world mannequins depicted in images displayed on each of the client devices (102) to include an AR hat selected by the first user and an AR shirt selected by the second user. In this way, each user can uniquely modify the components of the virtual store in a manner that is presented to other users participating in the virtual store.

[0068] The shared shopping experience system (224) may also allow users participating in the virtual shared shopping experience to exchange messages. For example, the shared shopping experience system (224) may allow a first user to type a message. The shared shopping experience system (224) may then present the message as an overlay of the virtual store presented to each user participating in the shared shopping experience. That is, the content of the messages may be overlaid on AR items of the virtual store presented to each user browsing the virtual store. As another example, the content of the messages may be overlaid on a real-world environment depicted in images captured and displayed on each client device (102) of the users browsing the virtual store. The shared shopping experience system (224) may receive responses from users through each of their client devices (102) and may update a list of messages to include the responses.

[0069] In some examples, the shared shopping experience system (224) may receive input from a client device (102) of a second user participating in the shared shopping experience requesting to virtually wear a given AR item. For example, the client device (102) of the second user may receive a tapping on the screen of the client device at a location where the given AR item is overlaid on the camera feed. In response, a set of options including an option to virtually wear the given AR item is displayed. In response to receiving the selection of an option to virtually wear the given AR item, the shared shopping experience system (224) activates the front camera of the client device (102) and captures one or more images of the second user. The shared shopping experience system (224) overlays the given AR item onto the second user depicted in one or more images. The shared shopping experience system (224) may detect that the client device (102) of the first user is oriented toward the second user. For example, the shared shopping experience system (224) may determine that the first user's client device (102) is capturing a video feed depicting the second user. In response, the shared shopping experience system (224) also displays a given AR item overlaid on the second user in the camera feed displayed by the first user's client device (102).

[0070] The shared shopping experience system (224) may receive input from the first user to add another AR item to the second user or to replace a given AR item with a different AR item. In response, the shared shopping experience system (224) adds another AR item to the depiction of the second user in the camera feeds of the client devices of the first and second users. In this way, the second user appears to be wearing the given AR item selected by the second user along with the other AR item selected by the first user. If the input from the first user chooses to replace the given AR item, the shared shopping experience system (224) replaces the display of the given AR item on the second user with the other AR item in the depiction of the second user in the camera feeds of the client devices of the first and second users. This allows two users to interact in AR, virtually wear different AR items, and provide feedback to each other by exchanging one or more messages.

[0071] In some examples, the shared shopping experience system (224) may detect an outside person within the camera feed of one or more of the client devices (102) of the users participating in the shared shopping experience. The outside person may be a person or object that has not been invited to join the users' shared shopping experience. For example, the shared shopping experience system (224) may process images received from a given client device (102) of the users participating in the shared shopping experience. The shared shopping experience system (224) may perform object recognition and face detection on the images to extract facial features (e.g., identity) of the person depicted in the images. The shared shopping experience system (224) may also include a set of facial features or other identifying attributes (or identities) of each user participating in the shared shopping experience. The shared shopping experience system (224) may determine that the facial features (identities) of a person depicted in the images do not match the facial features or identifying attributes (identities) of users participating in the shared shopping experience, and may determine that such a person is an outsider. In response, the shared shopping experience system (224) may perform a blocking process to remove or block the outsider from the camera feed of the client device (102) in which the person is depicted. For example, the shared shopping experience system (224) may perform body tracking using one or more machine learning techniques (e.g., one or more trained neural networks) to segment the entire body of the outsider in the images. The shared shopping experience system (224) may then blend each pixel within the segmented entire body of the outsider with the background of the shared shopping experience.In another example, the shared shopping experience system (224) can replace a part of an image containing a segmented whole body of an external person with another virtual object (e.g., a cartoon or other AR item available for purchase or on a mannequin).

[0072] The shared shopping experience system (224) is a component that can be accessed by an AR / VR application implemented on a client device (102). The AR / VR application uses an RGB camera to capture images of a room in a house or other real-world environment. The AR / VR application applies various trained machine learning techniques to the captured images of the room to create a virtual or augmented reality store that includes various virtual or AR items representing products associated with the store and decorative AR items corresponding to the store.

[0073] Data Architecture

[0074] FIG. 3 is a schematic diagram illustrating data structures (300) that may be stored in a database (126) of a messaging server system (108) according to specific examples. Although the contents of the database (126) are depicted as including a number of tables, it will be recognized that data may be stored in data structures of other types (e.g., as an object-oriented database).

[0075] The database (126) contains message data stored in the message table (302). For any specific message, this message data includes at least message sender data, message receiver (or receiver) data, and a payload. Further details regarding information that may be included in the message and in the message data stored in the message table (302) are described below with reference to FIG. 4.

[0076] The entity table (306) stores entity data and is linked to the entity graph (308) and profile data (316) (e.g., for reference). Entities for which records are maintained within the entity table (306) may include individuals, legal entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity for which the messaging server system (108) stores data may be a recognized entity. Each entity is provided with an entity type identifier (not shown) as well as a unique identifier.

[0077] The entity graph (308) stores information about relationships and associations between entities. These relationships may be, for example, social, professional (for example, work in a common corporation or organization), interest-based, or activity-based.

[0078] Profile data (316) stores multiple types of profile data regarding a specific entity. Profile data (316) may be optionally used and presented to other users of the messaging system (100) based on privacy settings specified by the specific entity. If the entity is an individual, profile data (316) includes, for example, a username, phone number, address, settings (e.g., notification and privacy settings), and user-selected avatar representations (or a collection of such avatar representations). Next, a specific user may optionally include one or more of these avatar representations within the content of messages communicated through the messaging system (100) and on map interfaces displayed to other users by messaging clients (104). A collection of avatar representations may include "status avatars" that present a graphic representation of a state or activity that a user may choose to communicate at a specific time. Profile data (316) may include facial features or other identifying attributes of users participating in a shared shopping experience.

[0079] If the entity is a group, the profile data (316) for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings (e.g., notifications) for the related group.

[0080] The database (126) also stores augmentation data, such as overlays or filters, in the augmentation table (310). The augmentation data is associated with and applied to videos (for which data is stored in the video table (304)) and images (for which data is stored in the image table (312)).

[0081] The database (126) may also store data regarding individual and shared AR sessions. This data may include data communicated between an AR session client controller of the first client device (102) and another AR session client controller of the second client device (102), and data communicated between an AR session client controller and an augmentation system (208). The data may include data used to establish a common coordinate frame of a shared AR scene, transformations between devices, a session identifier, images depicting a body, skeletal joint positions, wrist joint positions, feet, etc.

[0082] In one example, the filters are overlays that are displayed overlaid on an image or video during presentation to a recipient user. The filters may be of various types, including user-selected filters from a set of filters presented to the sender by the messaging client (104) when the sender is composing a message. Other types of filters include geolocation filters (also known as geo-filters) that may be presented to the sender based on geolocation. For example, specific geolocation filters for neighborhoods or specific locations may be presented within the user interface by the messaging client (104) based on geolocation information determined by the Global Positioning System (GPS) unit of the client device (102).

[0083] Another type of filter is a data filter that may be optionally presented to the sending user by the messaging client (104) based on information or other inputs collected by the client device (102) during the message generation process. Examples of data filters include the current temperature at a specific location, the current speed at which the sending user is moving, the battery life of the client device (102), or the current time.

[0084] Other augmented data that may be stored in the image table (312) includes augmented reality content items (e.g., corresponding to applying augmented reality experiences). Augmented reality content items or augmented reality items may be real-time special effects and sounds that can be added to an image or video.

[0085] As described above, augmented data includes similar terms referring to augmented reality content items, overlays, image transformations, AR images, and modifications that can be applied to image data (e.g., videos or images). This includes real-time modifications that modify an image as it is captured using device sensors (e.g., one or more cameras) of the client device (102) and then displayed on the screen of the client device (102) along with the modifications. This also includes modifications to stored content, such as video clips in a gallery that can be modified. For example, on a client device (102) having access to multiple augmented reality content items, a user can see how different augmented reality content items modify a stored clip using a single video clip containing multiple augmented reality content items. For example, multiple augmented reality content items applying different pseudo-random movement models can be applied to the same content by selecting different augmented reality content items for the content. Similarly, real-time video capture may be used with the illustrated modifications to show how the video images currently being captured by the sensors of the client device (102) will modify the captured data. This data may simply be displayed on the screen and not stored in memory, or the content captured by the device sensors may be written and stored in memory with or without the modifications (or both). In some systems, a preview feature may show how different augmented reality content items will appear in different windows on the display simultaneously. This may, for example, enable multiple windows with different pseudo-random animations to be displayed simultaneously on the display.

[0086] Accordingly, data and various systems using augmented reality content items or other such transformation systems to modify content using this data may involve the detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), tracking of these objects as they leave, enter, and move around the field of view in video frames, and modification or transformation of these objects as they are tracked. In various examples, different methods may be used to achieve these transformations. Some examples may involve generating a 3D mesh model of an object or objects, and using transformations of this model and animated textures within the video to achieve the transformation. In other examples, tracking of points on an object may be used to place an image or texture (which may be 2D or 3D) at a tracked location. In yet other examples, neural network analysis of video frames may be used to place images, models, or textures within the content (e.g., images or frames of the video). Therefore, augmented reality content items refer to both the images, models, and textures used to generate transformations in the content, as well as the additional modeling and analysis information required to achieve these transformations through object detection, tracking, and placement.

[0087] Real-time video processing can be performed with any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and store them in the device's memory, or generate video streams using the device's sensors. Additionally, any objects can be processed using computer animation models, such as human faces and parts of the human body, animals, or inanimate objects like chairs, cars, or other objects.

[0088] In some examples, when a specific modification is selected along with the content to be transformed, the elements to be transformed are identified by the computing device and subsequently detected and tracked if they exist in the frames of the video. The elements of the object are modified according to the request for modification, thereby transforming the frames of the video stream. The transformation of the frames of the video stream can be performed by different methods for different types of transformations. For example, for frame transformations that primarily refer to changing the shapes of the elements of an object, unique points for each element of the object are calculated (e.g., using the Active Shape Model (ASM) or other known methods). Next, a mesh is generated based on the unique points for each of at least one element of the object. This mesh is used in the next step of tracking the elements of the object in the video stream. In the tracking process, the mesh mentioned for each element is aligned with the location of each element. Next, additional points are generated on the mesh. A first set of first points is generated for each element based on a request for modification, and a second set of points is generated for each element based on the first set of points and the request for modification. Next, frames of a video stream can be transformed by modifying the elements of an object based on the first and second sets of points and a mesh. In this method, the background of the modified object can likewise be changed or distorted by tracking and modifying the background.

[0089] In some examples, transformations that modify parts of an object's regions using the object's elements can be performed by calculating unique points for each element of the object and generating a mesh based on the calculated unique points. Points are generated on the mesh, and then various regions based on these points are generated. The object's elements are then tracked by aligning the regions for each element with the positions for at least one element, and the characteristics of these regions can be modified based on requests for modification, thereby transforming the frames of the video stream. Depending on specific requests for modification, the characteristics of the mentioned regions may be transformed in different ways. Such modifications may involve changing the color of the regions; removing at least some parts of the regions from the frames of the video stream; including one or more new objects in the regions based on requests for modification; and modifying or distorting the regions or the elements of the objects. In various examples, any combination of these modifications or other similar modifications may be used. For specific models to be animated, some distinctive points may be selected as control points to be used in determining the entire state-space of options for model animation.

[0090] In some examples of computer animation models that transform image data using face detection, faces are detected on the image using a specific face detection algorithm (e.g., Viola-Jones). Next, the Active Shape Model (ASM) algorithm is applied to the face region of the image to detect face feature reference points.

[0091] Other methods and algorithms suitable for face detection may be used. For example, in some examples, features are located using landmarks that represent distinguishable points present in most of the images under consideration. For face landmarks, for example, the position of the left pupil may be used. If the initial landmark is not identifiable (e.g., if a person is wearing an eye patch), auxiliary landmarks may be used. These landmark identification procedures can be applied to any such objects. In some examples, a set of landmarks forms a feature. Features can be represented as vectors using the coordinates of points within the feature. One feature is aligned to another feature using a similarity transformation (allowing for translation, scaling, and rotation) that minimizes the average Euclidean distance between feature points. The average feature is the average of the aligned training features.

[0092] In some examples, a search for landmarks from average features aligned with the position and size of the face determined by a global face detector is initiated. Next, this search repeats the steps of proposing a tentative feature by adjusting the positions of feature points through template matching of image textures around each point, and fitting the tentative feature to the global feature model until convergence occurs. In some systems, individual template matching is unreliable, and the feature model pools the results of weak template matchings to form a stronger global classifier. The global search is repeated at each level of the image pyramid, from coarse resolution to fine resolution.

[0093] The transformation system can capture an image or video stream on a client device (e.g., client device (102)) and perform complex image manipulations locally on the client device (102) while maintaining a suitable user experience, computation time, and power consumption. Complex image manipulations may include size and shape changes, emotion transitions (e.g., changing a face from a frown to a smile), state transitions (e.g., aging a subject, reducing apparent age, changing gender), style transitions, graphic element applications, and any other suitable image or video manipulations implemented by a convolutional neural network configured to run efficiently on the client device (102).

[0094] In some examples, a computer animation model for transforming image data may be used by a system capable of capturing a user's image or video stream (e.g., a selfie) using a client device (102) having a neural network that operates as part of a messaging client (104) operating on the client device (102). A transformation system operating within the messaging client (104) determines the presence of a face in the image or video stream and provides modification icons associated with the computer animation model to transform the image data, or the computer animation model may exist as being associated with an interface described herein. The modification icons include changes that may be the basis for modifying the user's face in the image or video stream as part of a modification operation. Once a modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., creating a smiling face on the user). The modified image or video stream may be presented in a graphical user interface displayed on the client device (102) as soon as the image or video stream is captured and the specified modification is selected. The transformation system can implement complex convolutional neural networks on a portion of an image or video stream to generate and apply selected modifications. That is, a user can capture an image or video stream and, once the modification icon is selected, be presented with the modified result in real-time or near real-time. Additionally, modifications can be persistent as long as the video stream is being captured and the selected modification icon remains toggled. Machine-trained neural networks can be used to enable these modifications.

[0095] A graphical user interface presenting modifications performed by the conversion system may provide the user with additional interaction options. These options may be based on the interface used to initiate content capture and the selection of specific computer animation models (e.g., initiated from the content creator user interface). In various examples, modifications may be persistent after the initial selection of a modification icon. The user may toggle modifications on or off by tapping or otherwise selecting the face being modified by the conversion system, and save this for later viewing or browsing to other areas of the capture application. If multiple faces are modified by the conversion system, the user may toggle modifications on or off globally by tapping or selecting a single face being modified and displayed within the graphical user interface. In some examples, among a group of multiple faces, individual faces may be modified individually, or these modifications may be toggled individually by tapping or selecting an individual face or a series of individual faces displayed within the graphical user interface.

[0096] The story table (314) stores data regarding collections of messages and associated image, video, or audio data that are compiled into collections (e.g., stories or galleries). The creation of a specific collection may be undertaken by a specific user (e.g., each user for whom a record is maintained in the entity table (306)). A user may create a “personal story” in the form of a collection of content created and transmitted / broadcasted by that user. To this end, the user interface of the messaging client (104) may include a user-selectable icon to enable the transmitting user to add specific content to their personal story.

[0097] The collection may also constitute a “live story,” which is a collection of content from multiple users generated manually, automatically, or using a combination of manual and automatic techniques. For example, a “live story” may constitute a curated stream of user-submitted content from various locations and events. The client devices enable location services, and for users who are at a common location event at a specific time, an option to contribute content to a specific live story may be presented, for example, through the user interface of the messaging client (104). A live story may be identified to the user by the messaging client (104) based on their location. The final result is a “live story” in terms of community.

[0098] A type of addition to a content collection is known as a "location story," which enables a user whose client device (102) is located within a specific geographic location (e.g., a college or university campus) to contribute to a specific collection. In some examples, contribution to a location story may require a second degree of authentication to verify whether the end user belongs to a specific organization or other entity (e.g., a student on a university campus).

[0099] As mentioned above, the video table (304) stores video data associated with messages for which records are maintained in the message table (302), in one example. Similarly, the image table (312) stores image data associated with messages for which message data is stored in the entity table (306). The entity table (306) can associate various augmentations from the augmentation table (310) with various images and videos stored in the image table (312) and the video table (304).

[0100] Data communication architecture

[0101] FIG. 4 is a schematic diagram illustrating the structure of a message (400) according to some examples, generated by a messaging client (104) for communication with an additional messaging client (104) or a messaging server (118). The content of a particular message (400) is used to populate a message table (302) stored in a database (126) accessible by the messaging server (118). Similarly, the content of the message (400) is stored in memory as "in-transit" or "in-flight" data of the client device (102) or application servers (114). The message (400) is illustrated as comprising the following exemplary components:

[0102] · Message identifier (402): A unique identifier that identifies the message (400).

[0103] · Message text payload (404): Text to be generated by the user through the user interface of the client device (102) and included in the message (400).

[0104] · Message image payload (406): Image data captured by the camera component of the client device (102) or retrieved from the memory component of the client device (102) and included in the message (400). Image data for the transmitted or received message (400) may be stored in the image table (312).

[0105] · Message video payload (408): Video data included in the message (400) that is captured by the camera component or retrieved from the memory component of the client device (102). Video data for the transmitted or received message (400) may be stored in the video table (304).

[0106] · Message audio payload (410): Audio data captured by a microphone or retrieved from a memory component of a client device (102) and included in a message (400).

[0107] · Message augmentation data (412): Augmentation data (e.g., filters, stickers, or other annotations or enhancements) representing augmentations to be applied to the message image payload (406), message video payload (408), or message audio payload (410) of the message (400). Augmentation data for a transmitted or received message (400) may be stored in an augmentation table (310).

[0108] · Message duration parameter (414): A parameter value indicating the amount of time, in seconds, during which the content of the message (e.g., message image payload (406), message video payload (408), message audio payload (410)) is presented to or made accessible to the user through the messaging client (104).

[0109] · Message geolocation parameter (416): Geolocation data associated with the content payload of the message (e.g., latitude and longitude coordinates). Multiple message geolocation parameter (416) values ​​may be included in the payload, and each of these parameter values ​​is associated with content items included in the content (e.g., a specific image in the message image payload (406), or a specific video in the message video payload (408)).

[0110] · Message story identifier (418): Identifier values ​​that identify one or more content collections (e.g., "stories" identified in the story table (314)) to which a specific content item in the message image payload (406) of the message (400) is associated. For example, multiple images within the message image payload (406) may each be associated with multiple content collections using identifier values.

[0111] · Message Tag (420): Each message (400) may be tagged with multiple tags, each of which indicates the subject of the content included in the message payload. For example, if a specific image included in the message image payload (406) depicts an animal (e.g., a lion), a tag value indicating the relevant animal may be included in the message tag (420). The tag values ​​may be generated manually based on user input, or, for example, automatically using image recognition.

[0112] · Message sender identifier (422): An identifier indicating the user of the client device (102) on which the message (400) was generated and from which the message (400) was transmitted (e.g., a messaging system identifier, an email address, or a device identifier).

[0113] · Message recipient identifier (424): An identifier indicating the user of the client device (102) to which the message (400) is addressed (e.g., a messaging system identifier, an email address, or a device identifier).

[0114] The contents (e.g., values) of the various components of the message (400) may be pointers to locations in tables where content data values ​​are stored internally. For example, an image value in the message image payload (406) may be a pointer (or its address) to a location in the image table (312). Similarly, values ​​in the message video payload (408) may point to data stored in the video table (304), values ​​in the message augmentation data (412) may point to data stored in the augmentation table (310), values ​​in the message story identifier (418) may point to data stored in the story table (314), and values ​​in the message sender identifier (422) and message receiver identifier (424) may point to user records stored in the entity table (306).

[0115] FIGS. 5, FIGS. 6, FIGS. 7a, FIGS. 7b, FIGS. 8, and FIGS. 9 are schematic representations of the outputs of a shared shopping experience system according to some examples. For example, as illustrated in FIG. 5, a user interface (500) is provided on a client device (102) of a first user. The user interface (500) includes a portal page that identifies a merchant or store by name (510). The user interface (500) includes a first option (520) for browsing the AR virtual store of the identified merchant or store alone. The user interface (500) includes a second option (530) for browsing the AR virtual store of the identified merchant or store with one or more other users.

[0116] In response to receiving input from the client device (102) indicating the selection of the first option (520), the shared shopping experience system (224) launches an AR version of the store. The AR version of the store includes a plurality of virtual items corresponding to real-world items (e.g., physical items) sold by the merchant or store. The shared shopping experience system (224) allows the user to individually browse the contents of the store, wear the AR items, access additional information about the AR items, and purchase real-world products corresponding to the AR items being browsed. As discussed above, in response to receiving input indicating the selection of the first option (520), the shared shopping experience system (224) presents a screen that allows the user to select the types of real-world products that the user is interested in browsing. The types of real-world products included in the screen may be determined based on one or more attributes or physical layouts of real-world objects depicted in the camera feed of the user's client device (102). After the user selects one or more types of real-world products, the shared shopping experience system (224) generates AR items corresponding to the selected one or more types of real-world products and overlays the AR items on real-world objects depicted in the camera feed.

[0117] In some examples, in response to receiving an input indicating the selection of the first option (520), the shared shopping experience system (224) accesses the user's profile and automatically selects different types of real-world products that match the user's profile. The types of real-world products included on the screen may be determined based on one or more attributes or physical layouts of real-world objects depicted in the camera feed of the user's client device (102). After one or more types of real-world products are automatically selected, the shared shopping experience system (224) generates AR items corresponding to the selected one or more types of real-world products and overlays the AR items on the real-world objects depicted in the camera feed based on the physical layouts of the real-world objects (e.g., types of real-world objects and / or amount of free space available on the real-world objects).

[0118] In response to receiving input from the client device (102) indicating the selection of the second option (530), the shared shopping experience system (224) presents a user interface to the first user's client device (102) for inviting one or more friends to a virtual store. The shared shopping experience system (224) can search for the user's friends on the messaging client (104) that are determined to be within a threshold proximity to the user. For example, the shared shopping experience system (224) can access location information (if shared) by the user's friends' client devices (102). The shared shopping experience system (224) can compare the accessed location information with the location of the user's client device (102) to calculate the distance between the locations of the friends' client devices (102) and the user's client device (102). If it is determined that the distance between a friend's given client device (102) and the user's client device (102) is less than a threshold proximity (e.g., less than 50 feet), the shared shopping experience system (224) includes an identifier of the corresponding friend within the screen that allows the user to invite the friend to join the user in the shared shopping experience. If it is determined that the distance between a friend's given client device (102) and the user's client device (102) is greater than or equal to a threshold proximity (e.g., 50 feet or more), the shared shopping experience system (224) excludes the identifier of the corresponding friend within the screen to prevent the user from inviting the friend to join the user in the shared shopping experience.

[0119] The shared shopping experience system (224) may receive input from the user selecting one or more friends from the user interface. As discussed above, the user interface presented in response to receiving input indicating the selection of the second option (530) may also include options that allow the user to select the types of real-world products that the user is interested in browsing. The types of real-world products included on the screen may be determined based on one or more attributes or physical layouts of real-world objects depicted in the camera feed of the user's client device (102). After the user selects one or more types of real-world products, the shared shopping experience system (224) generates AR items corresponding to the selected one or more types of real-world products and overlays the AR items on the real-world objects depicted in the camera feed.

[0120] The shared shopping experience system (224) can send messages to one or more selected friends on their respective client devices (102). The messages can identify the types of real-world products selected by the user that are represented in the shared shopping experience. The messages may include an option to join the shared shopping experience and notify one or more friends that the first user wishes to join the shared shopping experience with one or more friends. In some cases, a second option (530) may be provided within a communication session participating in a group chat interface. In response to receiving the user's selection of the second option (530) within the group chat interface, the shared shopping experience system (224) invites all members within the group chat who are within a threshold proximity to each other to join the shared shopping experience together.

[0121] After users and friends choose to join a shared shopping experience, an AR store is presented to the users on their respective client devices (102). The AR store includes one or more AR items that are overlaid on real-world objects depicted in the camera feed of the first user's client device (102). The AR items include visual attributes corresponding to the physical layout of the real-world objects depicted in the real-world environment in the camera feed. For example, as illustrated in FIG. 6, a user interface (600) is presented on the first user's client device (102), where one or more AR items are overlaid on real-world objects depicted in the camera feed of the first user's client device (102). The second user may interface with the second user's client device (102) and request the first user to join the shared shopping experience. In response, the shared shopping experience system (224) displays one or more AR items of the same AR store on real-world objects depicted in the camera feed of the second user's client device (102).

[0122] The shared shopping experience system (224) may present a message or prompt (610) on the display of the first user's client device (102) indicating that a second user has joined and / or identifying all users currently participating in the shared shopping experience. The message or prompt (610) may include avatars or identifiers (614) of all users currently in the AR store and participating in the virtual shopping experience. The message or prompt (610) may also include a unique identifier (612) of a merchant or store associated with the virtual shopping experience, such as the source of real-world items represented by AR items within the virtual store shown to users participating in the shared shopping experience. The message or prompt (610) may provide status updates of actions performed by each user in the virtual shopping experience continuously or periodically. For example, if it is detected that a second user is virtually wearing an AR item, a message or prompt (610) may be provided to all other users participating in the shared shopping experience who identify the second user and the action being performed (e.g., virtually wearing an item). The shared shopping experience system (224) may receive input from a given user who selects the message or prompt (610), and in response, the shared shopping experience system (224) may notify the user of how to navigate in the real-world environment to reach the second user's real-world location in order to begin capturing a camera feed depicting the second user performing the action (e.g., virtually wearing an AR item).

[0123] The shared shopping experience system (224) may present presence indicators (620). The presence indicators (620) identify all users currently participating in the shared shopping experience, such as by using avatars. A given presence indicator (620) may be selected to send a message to a user associated with the given presence indicator (620) or to communicate with him in another way.

[0124] A virtual or AR mannequin (630) may be presented to two or more users who are capturing a camera feed of the real-world portion associated with the display of the virtual or AR mannequin (630). A shared shopping experience system (224) may receive input from a first user's client device (102) to add clothing items to the virtual or AR mannequin (630), such as an AR hat. In response, the shared shopping experience system (224) modifies the virtual or AR mannequin (630) to include the AR hat selected by the first user. The shared shopping experience system (224) may receive input from a second user's client device (102) to add other clothing items to the virtual or AR mannequin (630), such as an AR shirt. In response, the shared shopping experience system (224) modifies the virtual or AR mannequin (630) to include the AR hat selected by the first user and the AR shirt selected by the second user. In this way, each user can uniquely modify the components of the AR (virtual) store in a manner that is displayed to other users participating in the virtual store.

[0125] In some embodiments, the shared shopping experience system (224) may present indicators on each of the users currently participating in the shared shopping experience that identify which users are participating in the shared shopping experience and their current real-world 3D locations or coordinates. The indicators may be displayed conditionally based on whether a given user's client device (102) is oriented toward another user's 3D direction. For example, as illustrated in FIG. 6, the first user's client device (102) may be oriented to capture a camera feed corresponding to a first part of the virtual store where a first collection of virtual items is displayed, whereas the second user is physically located in a second part of the virtual store where a second collection of virtual items is displayed, 180 degrees away from the first part. The second user's physical location is not initially within the camera feed corresponding to the first part. As input is received from the first user's client device (102) that the first user's client device (102) has moved (e.g., rotated 180 degrees), the camera feed of the first user's client device (102) is updated to depict the second part as illustrated in FIG. 7a. As a result, the user interface (700) (Fig. 7a) is now presented on the first user's client device (102) which is capturing a camera feed that depicts the second user (720) and excludes the display of the indicator.

[0126] A user interface (700) presented on a first user's client device (102) includes a first real-world object (790). The first real-world object (790) may include real-world shelves having available free space. A shared shopping experience system (224) selects one or more AR items (722) that match the physical layout of the real-world shelves and displays one or more AR items (722) on a depiction of the first real-world object (790) included in a camera feed captured by the first user's client device (102). For example, the shared shopping experience system (224) searches for and selects AR items that can be realistically placed on top of the real-world object. That is, the shared shopping experience system (224) can search for AR items that are vertically stacked on top of each other, such as a folded shirt, folded shorts, a folded skirt, folded pants, a book, etc. The shared shopping experience system (224) may select a given AR item among the identified AR items based, for example, on the amount of available free space and / or user preferences. The shared shopping experience system (224) may then display a given one of the identified AR items as an AR item (722) on a first real-world object (790).

[0127] A user interface (700) presented on a first user's client device (102) includes a second real-world object (794). The physical layout of the second real-world object (794) may include a real-world ceiling. A shared shopping experience system (224) selects one or more AR items (792) having visual attributes that match the physical layout of the real-world ceiling and displays one or more AR items (792) on a depiction of the second real-world object (794) included in a camera feed captured by the first user's client device (102). One or more AR items (792) may correspond to decorative aspects of a virtual store that are not related to real-world products sold by the store. For example, one or more AR items (792) may include one or more AR spotlights. One or more AR items (792) can generate virtual light that is projected onto one or more other real-world objects depicted in the camera feed and / or onto one or more other AR items that are overlaid on the real-world objects depicted in the camera feed.

[0128] A user interface (700) presented on the first user's client device (102) includes a third real-world object (796). The physical layout of the third real-world object (790) may correspond to a rack, such as the interior of a closet, having available free hanging space. The shared shopping experience system (224) selects one or more AR items (799) that match the physical layout of the real-world racks and displays one or more AR items (799) on a depiction of the third real-world object (796) included in a camera feed captured by the first user's client device (102). For example, the shared shopping experience system (224) searches for and selects AR items that can be realistically placed on the racks of the real-world object. That is, the shared shopping experience system (224) may search for AR items in a horizontal stack (or array), such as those hanging on hangers. AR items in the horizontal stack (or array) may include hanging jewelry, hanging shirts, hanging skirts, hanging pants, etc. The shared shopping experience system (224) may select a given AR item among the identified AR items based, for example, on the amount of available free space and / or user preferences. The shared shopping experience system (224) may then display a given one among the identified AR items as an AR item (799) in relation to or inside a third real-world object (796).

[0129] The shared shopping experience system (224) may determine that the second user is currently wearing an AR item (722) on the second user's client device (102). In response, the shared shopping experience system (224) presents the AR item (722) on the depiction of the second user (720) presented on the first user's client device (102). The shared shopping experience system (224) may receive input from the first user's client device (102) to add more AR items to the depiction of the second user (720) and / or replace the AR item (722) with a different AR item. Any modifications made by the first user are reflected and presented to the second user on the second user's client device (102). That is, the second user's client device (102) may update the AR item overlaid on the camera feed depicting the second user with new AR items selected by the first user.

[0130] The user interface (700) includes an indicator (710) that uniquely identifies a third user within a display of a virtual store on the first user's client device (102). That is, the shared shopping experience system (224) may determine that the third user is not within the camera feed being captured by the first user's client device (102) and, in response, may display the indicator (710) of the third user. The indicator (710) may identify the real-world location or coordinates of the third user. The indicator (710) may be used to provide the first user with commands regarding how to navigate the real-world environment to reach the third user and begin capturing a camera feed depicting the third user.

[0131] In some embodiments, the shared shopping experience system (224) presents a user interface (701) (Fig. 7b) that enables users to exchange one or more messages (740) with one another. The messages (740) can identify the user who sent the message by the user's name. The messages (740) may be overlaid or displayed on each user interface of the shared shopping experience (virtual stores) on each of their respective client devices (102). As discussed above, the virtual mannequin may be presented to two or more users on each of their respective client devices (102) in response to the client devices (102) capturing a camera feed of real-world objects on which the virtual or AR mannequin is displayed. The shared shopping experience system (224) may receive input from the first user's client device (102) to add clothing items to the virtual or AR mannequin, such as an AR shirt (750). In response, the shared shopping experience system (224) modifies the virtual or AR mannequin to include the AR shirt (750) selected by the first user. The shared shopping experience system (224) may receive input from the second user's client device (102) to add other clothing items to the virtual or AR mannequin, such as AR shorts (752). In response, the shared shopping experience system (224) modifies the virtual or AR mannequin to include the AR shirt (750) selected by the first user and the AR shorts (752) selected by the second user. In this way, each user can uniquely modify the components of the virtual store in AR in a manner that is displayed to other users participating in the virtual store.

[0132] In some embodiments, the shared shopping experience system (224) may receive input from a second user’s client device (102) indicating a selection of a given AR item (722), such as one corresponding to real-world shirts. The selection may correspond to an action to view additional information regarding the AR item (722). In response, the shared shopping experience system (224) presents a full-screen view of the selected AR item (722) on the second user’s client device (102). For example, as illustrated in FIG. 8, the shared shopping experience system (224) displays a full-screen user interface (800) depicting the selected AR item (722). The full-screen user interface (800) displays the selected AR item (722) as a 3D image of a first real-world product corresponding to the AR item (722) in a second size larger than the first size in which the selected AR item (722) is displayed to other users. For example, an AR shirt corresponding to a virtual reality item (722) is displayed in a stack of AR items at a first size, as shown in FIG. 7a. After receiving an input requesting additional information about the AR item (722), the shared shopping experience system (224) displays a full-screen view of the AR item (820) (e.g., AR shirt) at a second size larger than the first size.

[0133] The full-screen user interface (800) is presented to a second user while other users, such as the first user, continue to present the appearance of the AR store, such as AR items, on their respective client devices (102). The full-screen user interface (800) includes a navigation area (810). The shared shopping experience system (224) may receive input from the second user navigating the navigation area (810). The navigation area (810) may include options for modifying the aspects or visual attributes of the selected AR item (722). For example, the navigation area (810) allows the user to select alternative sizes, colors, or styles for the selected AR item (722). In response, the shared shopping experience system (224) updates the appearance of the AR item (820) according to the selection from the navigation area (810). The modifications to the selected AR item (722) are presented to any client device (102) capturing a camera feed depicting the second user. The shared shopping experience system (224) can display wear options (822).

[0134] In response to receiving the selection of the wear option (822), the shared shopping experience system (224) may allow the second user to virtually wear the AR item (820). That is, the shared shopping experience system (224) may activate the front camera of the second user's client device (102). The shared shopping experience system (224) may capture a video of the second user's entire body and present the video of the entire body on the display of the second user's client device (102). The shared shopping experience system (224) may overlay or augment the entire body within the video with the selected AR item (820) to allow the user to see how the AR item looks on the user. In some cases, the AR item (820) corresponds to furniture. In these cases, the shared shopping experience system (224) may activate the rear camera of the second user's client device (102). The shared shopping experience system (224) can capture a video of the second user's real-world environment and present the video of the real-world environment on the display of the second user's client device (102). The shared shopping experience system (224) can overlay or augment the real-world environment within the video with the selected AR item (820) to allow the user to see how the AR item looks in the real-world environment. Any modifications to the selected AR item (722) performed by the second user are presented to the second user or any client device (102) that is capturing a camera feed depicting a part of the real-world environment where the second user requested to display the AR item (e.g., AR furniture item).

[0135] In response to receiving the selection of the wear option (822), the shopping experience system (224) may update the prompt (610) presented on other client devices (102) to indicate that the second user is wearing the AR item. The shopping experience system (224) may also update the display properties of the AR item (722) on the display of the AR item (722) provided on the client devices (102) of the first user and other users to indicate the type of action performed by the second user in association with the AR item (722). For example, as illustrated in FIG. 9, the user interface (900) includes a depiction of the AR item (722) having a blue border (910) to indicate that another user, such as the second user, is wearing the AR item (722). That is, the shared shopping experience system (224) modifies the display properties of the AR item (722) presented to the first user in a first manner to indicate that the second user is performing a first type of action, such as wearing the AR item (722). The shared shopping experience system (224) may also update the quantity of AR items included in a stack of AR items shown to users in response to detecting that a given user has selected an AR item and / or based on the store's inventory. For example, referring again to FIG. 7a, multiple stacks (or other arrangements) of AR items are depicted. Each stack may contain different quantities of items. The quantity of items presented to users in the virtual store is increased or decreased based on actions of other users, such as whether other users have selected items, are browsing items, are purchasing items, have added items to their shopping carts, or are virtually wearing items.

[0136] FIG. 10 is a flowchart of a process (1000) according to some examples. While such a flowchart may describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. Additionally, the order of these operations may be rearranged. The process terminates when the operations are completed. The process may correspond to a method, procedure, etc. The steps of these methods may be performed wholly or partially, and may be performed together with some or all of the steps in other methods, and may be performed by any number of different systems, or any part thereof, such as a processor included in any of these systems.

[0137] In operation 1001, the shared shopping experience system (224) receives a request from the first user from the first user’s client device to participate in an augmented reality (AR) shopping experience curated by the store as discussed above.

[0138] In operation 1002, the shared shopping experience system (224) identifies a first real-world product that can be purchased from a store, as discussed above.

[0139] In operation 1003, the shared shopping experience system (224) receives an image of the first user's real-world environment from the camera of the client device, as discussed above.

[0140] In operation 1004, the shared shopping experience system (224) generates a first AR item representing a first real-world product, as discussed above.

[0141] In operation 1005, the shared shopping experience system (224) compares one or more visual attributes of the first AR item with the physical layouts of a plurality of real-world objects depicted in an image of a real-world environment, as discussed above.

[0142] In operation 1006, the shared shopping experience system (224) overlays the first AR item on the first real-world object among the multiple real-world objects in the image as a result of comparing one or more visual attributes of the first AR item with the physical layouts of the multiple real-world objects as discussed above.

[0143] Machine Architecture

[0144] FIG. 11 is a schematic representation of a machine (1100) on which instructions (1108) (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine (1100) to perform one or more of the methodologies discussed herein. For example, instructions (1108) can cause the machine (1100) to perform one or more of the methods described herein. Instructions (1108) convert a general-purpose unprogrammed machine (1100) into a specific machine (1100) programmed to perform the described and illustrated functions in the described manner. The machine (1100) may operate as a standalone device or may be coupled to other machines (e.g., networked). In a networked deployment, the machine (1100) may operate at the capacity of a server machine or a client machine in a server-client network environment, or may operate as a peer machine in a peer-to-peer (or distributed) network environment. The machine (1100) may include, but is not limited to, a server computer, a client computer, a PC (personal computer), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart device), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing commands (1108) specifying actions to be taken by the machine (1100), sequentially or otherwise.Additionally, although only a single machine (1100) is exemplified, the term “machine” may also be taken to include a collection of machines that execute instructions (1108) individually or jointly to perform one or more of the methodologies discussed herein. The machine (1100) may include, for example, a client device (102) or any one of a plurality of server devices forming part of a messaging server system (108). In some examples, the machine (1100) may also include both client and server systems, and specific operations of a specific method or algorithm are performed on the server side and specific operations of a specific method or algorithm are performed on the client side.

[0145] The machine (1100) may include processors (1102), memory (1104), and I / O (input / output) components (1138) that can be configured to communicate with each other via a bus (1140). In the example, the processors (1102) (e.g., a CPU (Central Processing Unit), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an RFIC (Radio-Frequency Integrated Circuit), other processors, or any suitable combination thereof) may include, for example, a processor (1106) and a processor (1110) that execute instructions (1108). The term “processor” is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as “cores”) capable of executing instructions simultaneously. FIG. 11 illustrates multiple processors (1102), but the machine (1100) may include a single processor having a single core, a single processor having multiple cores (e.g., a multi-core processor), multiple processors having a single core, multiple processors having multiple cores, or any combination thereof.

[0146] The memory (1104) includes a main memory (1112), a static memory (1114), and a storage unit (1116), both of which are accessible to processors (1102) via a bus (1140). The main memory (1104), the static memory (1114), and the storage unit (1116) store instructions (1108) that implement one or more of the methodologies or functions described herein. The instructions (1108) may also exist, wholly or partially, during execution by the machine (1100), in the main memory (1112), in the static memory (1114), in the machine-readable medium (1118) within the storage unit (1116), in at least one of the processors (1102) (e.g., in the processor's cache memory), or any suitable combination thereof.

[0147] The I / O components (1138) may include a wide variety of components for receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components (1138) included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones may include touch input devices or other such input mechanisms, whereas a headless server machine may not include such touch input devices. It will be recognized that the I / O components (1138) may include many other components not shown in FIG. 11. In various examples, the I / O components (1138) may include user output components (1124) and user input components (1126). User output components (1124) may include visual components (e.g., displays such as a PDP (plasma display panel), an LED (light-emitting diode) display, an LCD (liquid crystal display), a projector, or a CRT (cathode ray tube), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc.User input components (1126) may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing mechanism), haptic input components (e.g., a physical button, a touch screen providing the position and force of touches or touch gestures, or other haptic input components), audio input components (e.g., a microphone), etc.

[0148] In additional examples, I / O components (1138) may include biometric components (1128), motion components (1130), environment components (1132), or location components (1134), among a wide variety of other components. For example, biometric components (1128) may include components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye-tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identify a person (e.g., voice identification, retinal identification, face identification, fingerprint identification, or electroencephalogram-based identification), etc. Motion components (1130) include acceleration sensor components (e.g., accelerometers), gravity sensor components, and rotation sensor components (e.g., gyroscopes).

[0149] Environmental components (1132) include, for example, one or more cameras (having still image / photo and video capabilities), a light sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases for safety or measuring contaminants in the atmosphere), or other components capable of providing indications, measurements, or signals corresponding to the surrounding physical environment.

[0150] Regarding cameras, the client device (102) may have a camera system including, for example, front cameras on the front of the client device (102) and rear cameras on the rear of the client device (102). The front cameras may be used, for example, to capture still images and videos (e.g., "selfies") of the user of the client device (102), which may then be augmented with the augmentation data (e.g., filters) described above. The rear cameras may be used, for example, to capture still images and videos in a more traditional camera mode, and these images may similarly be augmented with augmentation data. In addition to the front and rear cameras, the client device (102) may also include a 360° camera for capturing 360° photos and videos.

[0151] Additionally, the camera system of the client device (102) may include dual rear cameras (e.g., a depth-sensing camera as well as a main camera) or even triple, quadruple, or quintuple rear camera configurations on the front and rear sides of the client device (102). These multiple camera systems may include, for example, a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0152] The position components (1134) include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect atmospheric pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), etc.

[0153] Communication can be implemented using a wide variety of technologies. The I / O components (1138) further include communication components (1136) operable to connect the machine (1100) to a network (1120) or devices (1122) through respective connections or accesses. For example, the communication components (1136) may include a network interface component, or other devices suitable for interfacing with the network (1120). In additional examples, the communication components (1136) may include wired communication components, wireless communication components, cellular communication components, NFC (Near Field Communication) components, Bluetooth ® Components (e.g., Bluetooth) ® Low Energy), Wi-Fi ® It may include other communication components that provide communication through components and other aspects. The devices (1122) may be any of other machines or a wide variety of peripheral devices (e.g., peripheral devices connected via USB).

[0154] Furthermore, the communication components (1136) may include components capable of detecting identifiers or operable to detect identifiers. For example, the communication components (1136) may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., optical sensors for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or acoustic detection components (e.g., microphones for identifying tagged audio signals). Additionally, various information, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via NFC beacon signal detection capable of indicating a specific location, etc., may be derived through the communication components (1136).

[0155] Various memories (e.g., main memory (1112), static memory (1114), and memory of the processors (1102)) and storage unit (1116) may store one or more sets of instructions and data structures (e.g., software) that implement or are used by any one or more of the methodologies or functions described herein. When these instructions (e.g., instructions (1108)) are executed by the processors (1102), various operations enable the implementation of the disclosed examples.

[0156] Commands (1108) may be transmitted or received through a network (1120) using a transmission medium, through a network interface device (e.g., a network interface component included in the communication components (1136)) and using any one of some well-known transmission protocols (e.g., HTTP (hypertext transfer protocol)). Similarly, commands (1108) may be transmitted or received using a transmission medium through a connection to the devices (1122) (e.g., a peer-to-peer connection).

[0157] Software Architecture

[0158] FIG. 12 is a block diagram 1200 illustrating a software architecture (1204) that may be installed in one or more of the devices described herein. This software architecture (1204) is supported by hardware, such as a machine (1202) that includes processors (1220), memory (1226), and I / O components (1238). In this example, the software architecture (1204) may be conceptualized as a stack of layers, each layer providing specific functionality. The software architecture (1204) includes layers such as an operating system (1212), libraries (1210), frameworks (1208), and applications (1206). Operationally, applications (1206) initiate API calls (1250) through the software stack and receive messages (1252) in response to the API calls (1250).

[0159] The operating system (1212) manages hardware resources and provides common services. The operating system (1212) includes, for example, a kernel (1214), services (1216), and drivers (1222). The kernel (1214) acts as an abstraction layer between the hardware and other software layers. For example, the kernel (1214) provides, among other functionalities, memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services (1216) may provide other common services for other software layers. Drivers (1222) are responsible for controlling the underlying hardware or interfacing with it. For example, drivers (1222) may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), WI-FI® drivers, audio drivers, power management drivers, etc.

[0160] Libraries (1210) provide a common low-level infrastructure used by applications (1206). Libraries (1210) may include system libraries (1218) (e.g., the C standard library) that can provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the libraries (1210) may include API libraries (1224), such as media libraries (e.g., libraries that support the presentation and manipulation of various media formats such as MPEG4 (Moving Picture Experts Group-4), H.264 or AVC (Advanced Video Coding), MP3 (Moving Picture Experts Group Layer-3), AAC (Advanced Audio Coding), AMR (Adaptive Multi-Rate) audio codecs, JPG or JPEG (Joint Photographic Experts Group), or PNG (Portable Network Graphics), graphics libraries (e.g., OpenGL frameworks used to render graphic content on a display in two dimensions (2D) and three dimensions (3D)), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functionality). Libraries (1210) may also include a wide variety of other libraries (1228) to provide many different APIs to applications (1206).

[0161] Frameworks (1208) provide high-level common infrastructure used by applications (1206). For example, frameworks (1208) provide various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Frameworks (1208) may provide a wide spectrum of other APIs that can be used by applications (1206), some of which may be specific to a particular operating system or platform.

[0162] In the example, the applications (1206) may include a wide range of other applications such as a home application (1236), a contacts application (1230), a browser application (1232), a book reader application (1234), a location application (1242), a media application (1244), a messaging application (1246), a game application (1248), and an external application (1240). The applications (1206) are programs that execute functions defined in the programs. Various programming languages ​​structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language), may be used to create one or more of the applications (1206). In a specific example, the external application (1240) (e.g., ANDROID by an entity other than the vendor of a specific platform) TM or iOS TM Applications developed using the SDK (software development kit) are iOS TM , ANDROID TMIt may be mobile software running on a mobile operating system, such as a WINDOWS® Phone or other mobile operating system. In this example, an external application (1240) may initiate API calls (1250) provided by the operating system (1212) to facilitate the functionality described herein.

[0163] Glossary

[0164] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium through a network interface device.

[0165] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop, PDAs (portable digital assistants), a smartphone, a tablet, an ultrabook, a netbook, a laptop, a multi-processor system, a microprocessor-based or programmable consumer electronic device, a game console, a set-top box, or any other communication device that a user can use to access the network.

[0166] The term “communication network” refers to one or more parts of a network, which may be an ad-hoc network, intranet, extranet, VPN (virtual private network), LAN (local area network), wireless LAN (WLAN), WAN (wide area network), wireless WAN (WWAN), MAN (metropolitan area network), Internet, part of the Internet, part of the PSTN (Public Switched Telephone Network), POTS (plain old telephone service) network, cellular telephone network, wireless network, Wi-Fi® network, other types of networks, or a combination of two or more of these networks. For example, a network or part of a network may include a wireless or cellular network, and a connection may be a CDMA (Code Division Multiple Access) connection, a GSM (Global System for Mobile communications) connection, or other types of cellular or wireless connections.In this example, the connection may implement any of various types of data transmission technologies, such as 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3GPP (third Generation Partnership Project) including 3G, 4G (fourth generation wireless) networks, UMTS (Universal Mobile Telecommunications System), HSPA (High Speed ​​Packet Access), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution) standards, other things defined by various standards-setting organizations, other long-range protocols, or other data transmission technologies.

[0167] A "component" refers to a device, physical entity, or logic having boundaries defined by functions or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularization of specific processing or control functions. Components may be combined with other components through their interfaces to perform machine processes. A component may be a packaged functional hardware unit designed to be used with other components and a part of a program that generally performs a specific function among the associated functions.

[0168] Components may comprise software components (e.g., code implemented on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing specific operations and may be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or part of an application) as hardware components that operate to perform specific operations as described herein.

[0169] Hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuits or logic that are permanently configured to perform specific operations. A hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuits that are temporarily configured by software to perform specific operations. For example, a hardware component may include software executed by a general-purpose processor or another programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of machines) uniquely tailored to perform the configured functions and are no longer general-purpose processors. It will be recognized that the decision to implement a hardware component mechanically, in dedicated, permanently configured circuits, or in temporarily configured circuits (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the phrase “hardware component” (or “hardware-implemented component”) should be understood to encompass tangible entities, which are physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular way or to perform the specific operations described herein.

[0170] When considering examples where hardware components are configured (e.g., programmed) temporarily, each hardware component does not need to be configured or instantiated at any single point in time. For example, in the case of a general-purpose processor configured by software to have a hardware component become a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors at different times (e.g., including different hardware components). Thus, the software configures a specific processor or processors to configure a specific hardware component at one time and a different hardware component at another time.

[0171] Hardware components can provide information to other hardware components and receive information from them. Thus, the described hardware components can be considered as being communicably coupled. When multiple hardware components exist simultaneously, communication can be achieved through the transmission of signals between or between two or more of the hardware components (e.g., via appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between these hardware components can be achieved, for example, through the storage and retrieval of information in memory structures accessed by multiple hardware components. For example, one hardware component may perform an operation and store the output of that operation in a communicably coupled memory device. Additional hardware components may subsequently access the memory device to retrieve and process the stored output. Hardware components may also engage in communication with input or output devices and operate on resources (e.g., collections of information).

[0172] Various operations of the exemplary methods described herein may be performed at least partially by one or more processors that are permanently configured to perform the relevant operations or temporarily configured (e.g., by software). Whether temporarily or permanently configured, these processors may comprise processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be at least partially processor-implemented, and specific processors or processors are examples of hardware. For example, at least some of the operations of the methods may be performed by one or more processors (1102) or processor-implemented components. Furthermore, one or more processors may also operate to support the performance of the relevant operations in a "cloud computing" environment or as "software as a service (SaaS)." For example, at least some of the operations may be performed by a group of computers (as an example of a machine containing processors), and these operations may be accessible via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., APIs). The performance of specific operations may be distributed among processors, not only residing within a single machine but also deployed across multiple machines. In some examples, processors or processor-implementation components may be located in a single geographic location (e.g., a home environment, an office environment, or within a server farm). In other examples, processors or processor-implementation components may be distributed across multiple geographic locations.

[0173] "Computer-readable storage medium" refers to a machine storage medium. Accordingly, these terms include storage devices / mediums. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0174] An "ephemeral message" refers to a message accessible for a limited duration. Ephemeral messages can be text, images, videos, etc. The access time for an ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is transitory.

[0175] "Machine storage medium" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Accordingly, this term may be taken to include, but is not limited to, solid-state memories including memory inside or outside processors, and optical and magnetic media. Specific examples of machine storage media, computer storage media, and device storage media include, by example, non-volatile memory including semiconductor memory devices, e.g., EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), FPGAs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine-storage media," "computer-storage media," and "device-storage media" specifically exclude carriers, modulated data signals, and other such media, at least some of which are covered under the term "signal medium."

[0176] "Non-transitory computer-readable storage medium" refers to a type of medium capable of storing or encoding instructions for execution by a machine.

[0177] "Signal medium" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be taken to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal in which one or more of its characteristics are set or changed in such material to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0178] Changes and modifications to the disclosed examples may be made without departing from the scope of the present disclosure. Such other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the following claims.

Claims

Claim 1 A method comprising: establishing a virtual experience with a second user's second computing device by one or more processors of a first user's first computing device; determining that the first computing device is oriented toward a three-dimensional (3D) location excluding the second user's current location; and presenting an indicator of the second user identifying the second user's current location on the first computing device in response to the determination that the first computing device is oriented toward a 3D location excluding the second user's current location. Claim 2 A method according to claim 1, further comprising: a step of identifying a first real-world product; a step of receiving an image of the first user's real-world environment from a camera of the second computing device; and a step of generating a first AR item representing the first real-world product. Claim 3 A method according to claim 2, further comprising: a step of comparing one or more visual attributes of the first AR item with physical layouts of a plurality of real-world objects depicted in an image of the real-world environment; and a step of overlaying the first AR item on a first real-world object among the plurality of real-world objects in the image as a result of comparing one or more visual attributes of the first AR item with physical layouts of the plurality of real-world objects. Claim 4 A method according to claim 1, further comprising: generating a plurality of real-world products of purchasable types for display on the first computing device; and receiving an input from the first computing device for selecting a type of real-world product from the plurality of types, wherein the selected type is associated with the real-world product of purchasable types. Claim 5 A method according to claim 1, further comprising the step of accessing a profile associated with the first user; and the step of selecting a type of real-world product from a plurality of types based on the profile associated with the first user, wherein the selected type is associated with a real-world product available for purchase. Claim 6 A method according to claim 1, further comprising: a step of generating a first AR item representing a first real-world product; and a step of overlaying the first AR item on a second real-world object in response to determining that one or more visual attributes of the first AR item correspond to the physical layout of a second real-world object among a plurality of real-world objects. Claim 7 A method according to claim 1, further comprising: a step of generating a first AR item representing a decorative component of a store; and a step of overlaying the first AR item on a second real-world object in response to determining that one or more visual attributes of the second AR item correspond to the physical layout of the second real-world object among a plurality of real-world objects. Claim 8 A method according to claim 1, further comprising: receiving a video of the real-world environment that depicts the geometry of the real-world environment including physical layouts of real-world objects; and customizing the AR configuration of AR items corresponding to real-world products available for purchase from a store based on the geometry of the real-world environment. Claim 9 A method according to claim 8, further comprising the step of generating the customized AR configuration of the AR items within the video of the real-world environment for display. Claim 10 A method according to claim 1, further comprising: a step of determining by one or more processors that the second user is interacting with the first AR item; and a step of overlaying the first AR item on the second user in the image in response to detecting that the second user is depicted in an image captured by the first user's computing device. Claim 11 A method according to claim 10, further comprising the step of generating a list of messages exchanged between the first user and the second user within a virtual experience on the computing devices of the first user and the second user for display. Claim 12 A method according to claim 11, further comprising: a step of determining that the computing devices of the first user and the second user are within a threshold proximity to each other; and a step of generating the virtual experience to display on each of the computing devices of the first user and the second user in response to determining that the computing devices of the first user and the second user are within a threshold proximity to each other. Claim 13 A method according to claim 1, further comprising: a step of creating a virtual mannequin for display within the virtual experience on the computing device of the first user; and a step of applying a first clothing item to the virtual mannequin based on a first input received from the first computing device of the first user selecting a second AR item. Claim 14 A method according to claim 1, further comprising the steps of: accessing an inventory of a store represented by the virtual experience; and modifying the quantity of a plurality of AR items represented on a display of the virtual experience based on the inventory of the store. Claim 15 A method according to claim 14, further comprising the step of associating a non-fungible token (NFT) with each of the plurality of AR items to track the availability of real-world products represented by the plurality of AR items. Claim 16 A system comprising a processor configured to perform operations, wherein the operations include: an operation of establishing a virtual experience with a second user’s second computing device by a first user’s first computing device; an operation of determining that the first computing device is oriented toward a three-dimensional (3D) location excluding the second user’s current location; and an operation of presenting an indicator of the second user identifying the second user’s current location on the first computing device in response to the determination that the first computing device is oriented toward the 3D location excluding the second user’s current location. Claim 17 A system according to claim 16, wherein the above operations include: an operation of identifying a first real-world product available for purchase from a store; an operation of receiving an image of the first user's real-world environment from a camera of the second computing device; and an operation of generating a first AR item representing the first real-world product. Claim 18 A system according to claim 17, wherein the operations include: an operation of comparing one or more visual attributes of the first AR item with physical layouts of a plurality of real-world objects depicted in an image of the real-world environment; and an operation of overlaying the first AR item on the first real-world object among the plurality of real-world objects in the image as a result of comparing one or more visual attributes of the first AR item with physical layouts of the plurality of real-world objects. Claim 19 In claim 16, the above operations include: an operation of generating a plurality of real-world products of purchasable types for display on the first computing device; and an operation of receiving from the first computing device an input for selecting a type of real-world product from the plurality of types, wherein the selected type is associated with the purchasable real-world product. Claim 20 A non-transient machine-readable storage medium comprising instructions, wherein the instructions, when executed by one or more processors of a machine, cause the machine to perform operations including: establishing a virtual experience with a second user's second computing device by a first user's first computing device; determining that the first computing device is oriented toward a three-dimensional (3D) location excluding the second user's current location; and, in response to the determination that the first computing device is oriented toward a 3D location excluding the second user's current location, presenting an indicator of the second user identifying the second user's current location on the first computing device.

Images